Abstract
BACKGROUND:
Recruiting and enrolling participants into randomized controlled trials is difficult. Reviews of the extent to which trials achieve targets on time and within budget commonly report failure or delay. PROMISE—Promoting Readiness of Minors in Supplemental Security Income (SSI)—a six program cooperative study of randomized trials testing effectiveness of service programs for transition-aged youth with disabilities receiving SSI benefits on employment and educational outcomes, provides an unusual opportunity to describe successful recruitment and enrollment into large-scale trials.
OBJECTIVE:
The purpose is to profile recruitment strategies used within and across the six PROMISE programs for meeting enrollment targets of SSI youth and families.
METHODS:
From descriptive data extracted from process analysis reports of each of the six PROMISE programs, we constructed cross-program profiles of recruitment strategies.
RESULTS:
All six programs met their enrollment targets on time. Although they contacted most potential participants through initial mailings and telephone calls, the programs reported that multiple contacts using multiple strategies, especially resource-intensive in person meetings and assignment of staff full-time to recruitment activities, were needed to meet enrollment targets.
CONCLUSIONS:
Because all PROMISE programs met their required enrollment targets, researchers designing large-scale, field-based randomized controlled trials may benefit from using a mix of recruitment strategies deployed by full-time recruiters.
Keywords
Introduction
In transitioning from school to postsecondary education and employment, youth with disabilities receiving Supplemental Security Income (SSI) benefits encounter formidable obstacles to economic self-sufficiency. Compared to their peers without disabilities, they exhibit lower employment rates, and experience poorer health, social isolation, greater service needs, and lower earned income. Unfortunately, efforts specifically directed toward improving these outcomes in education, employment, and income have met with relatively mixed success (Enayati & Karpur, 2019).
In 2013, the U.S. Department of Education awarded cooperative agreements to five states and one consortium of six states1 under PROMISE—Promoting Readiness of Minors in Supplemental Security Income (SSI)–a joint federal venture of the U.S. Department of Education (ED), the Social Security Administration (SSA), the U.S. Department of Health & Human Services, and the U.S. Department of Labor.2 In randomized controlled trials, each of the six PROMISE programs tested the effectiveness of innovative services and supports for transition-aged SSI recipients and their families designed to: (a) increase youth educational attainment; (b) improve youth and parent/guardian employment outcomes; (c) reduce household reliance on SSI; and (d) increase total household income and economic well-being. SSA contracted with Mathematica Policy Research (Mathematica) to evaluate the outcomes and service implementation for each of the six PROMISE programs.
ED required that the PROMISE programs provide a set of core services to youth and families: (a) case management for optimizing transition planning and service coordination; (b) benefits counseling and financial literacy training about SSA work incentives, eligibility for public assistance programs, and economic self-sufficiency; (c) career and work-based learning experiences, including at least one paid work experience in an integrated setting during high school; and (d) parent training for supporting youth’s attainment of education and employment goals (Honeycutt, Gionfriddo & Livermore, 2018). To coordinate multiple agency services to meet each youth’s needs, the PROMISE programs formed partnerships among state agencies responsible for vocational rehabilitation, education, workforce development, Medicaid, Temporary Assistance for Needy Families, and developmental disabilities and mental health.
All youth who enrolled into the PROMISE study were SSI beneficiaries. SSA administers the SSI program, a means-tested income support program, helping low-income families offset costs of managing their child’s disability. SSI eligibility for children requires they must (a) be under the age of 18 (or 22 for full-time students), (b) be unmarried, and (c) meet criteria for disability or blindness, citizenship/residency, and income and resources. Disability is defined as a medically determinable physical or mental impairment that (a) results in severe functional limitations, and (b) has lasted or can be expected to last for a continuous period of one year, or can be expected to result in death. Low-income is defined as household income at or below the federal poverty level. Children who receive SSI face redetermination of their SSI eligibility at age 18 to assess whether they qualify under adult criteria. Approximately 1.15 million children receive SSI, and the average monthly benefit is $674.3
SSA stipulated that within a two-year window, each program must recruit and enroll at least 2000 SSI youth4 between the ages of 14 and 16, with half randomized to the PROMISE services condition and half to the control (i.e. usual) services condition. Each PROMISE program developed recruitment and enrollment plans for youth and their families. At program start-up, and at subsequent six-month intervals, SSA provided each program with a list of all eligible youth residing within their geographic jurisdictions,5 including their contact information, from which to recruit and enroll. SSA encouraged programs to begin recruitment by mailing an information packet to each youth and family. Programs deployed a mix of recruitment strategies, such as follow-up mailings, telephone calls, emails, text messages, and in-person meetings.
Challenges to recruiting participants to field-based randomized clinical trials
For field-based randomized controlled trials to yield valid inferences of intervention effectiveness and generalizability, they must meet enrollment targets determined by appropriate power analyses on time and within budget. However, recruiting participants into trials is difficult. Although the research literature on recruitment into experimental and quasi-experimental studies is broad, when considered across areas of clinical problems, settings, and service types, it lacks depth with very few strategies determined with high certainty as effective (e.g., McDonald, 2006; Sully, 2013). For example, a recent Cochrane Collaboration meta-analysis of methods to increase recruitment into randomized and quasi-experimental trials (Treweek et al., 2018) quantified effectiveness of various recruitment strategies reported in 68 studies conducted in a variety of health care settings involving 72 comparisons with more than 74,000 participants. Unfortunately, for only two strategies did they find high-certainty evidence of effectiveness: (a) clear description of all trial procedures in initial mailings to potential participants; and (b) telephoning (or text messaging) potential participants who did not respond to initial mailings. Surprisingly, high-certainty evidence of ineffectiveness was found for the content, format, and appearance of user-tested information leaflets. Disappointingly, a small proportion of other strategies showed only moderate-certainty evidence of effectiveness (e.g., how potential participants process information, financial incentives), and none of these strategies could be recommended with confidence for use in future trials.
Consequently, faced with uncertainty about which recruitment strategies to use in field-based randomized controlled trials, researchers may be tempted to select strategies on a “trial and error” basis, which risks wasting time, money, and resources. Worse yet, some researchers may not fully appreciate the difficulty of trial recruitment, and therefore allocate insufficient resources to meet enrollment targets (e.g. Lemons, Fuchs, Gilbert & Fuchs, 2014).
In this paper, using descriptive data extracted from Mathematica’s process analysis reports for each of the six PROMISE programs, we profile these programs’ recruitment strategies. We hope to present researchers in transition, secondary education, and vocational rehabilitation with some suggestions of feasible strategies for meeting enrollment targets in future trials of field-based interventions for transition-aged youth with disabilities.
Methods
Recruitment and enrollment data sources
Mathematica, the independent evaluator of the six PROMISE programs, produced interim process analysis reports for each of the six PROMISE programs, summarizing program service implementation; demographic and clinical profiles of enrolled youth; and recruitment and enrollment outcomes. Mathematica’s findings rested upon data derived from quantifiable recruitment strategies recorded by designated program staff into electronic data bases, and from telephone interviews and visits with program staff: Arkansas (Honeycutt et al., 2018), ASPIRE Consortium (Anderson et al., 2018), California (Matulewicz et al., 2018), Maryland (Kauff et al., 2018), New York State (McCutcheon et al., 2018), and Wisconsin (Selekman et al., 2018). The reports present (a) the type, frequency, and percentage of each program’s quantifiable recruitment strategies, (b) the quarterly rates of youth enrollment across the two-year recruitment period, (c) the number of contacts needed to enroll one youth calculated in two ways—by overall average and by four rank-ordered categories; and (d) narrative descriptions about recruitment strategies unique to each program.
Characteristics of the target population
A total of 13,172 youth and families enrolled across the six PROMISE programs. Because demographic and clinical characteristics of enrollees were similar across the six programs, we summarize the following characteristics in the aggregate. Age: About 40 percent of youth were age 14, except the Maryland program (25%). Gender: Approximately two-thirds of enrolled youth were male, and one-third female. Race/ethnicity: SSA discourages researchers from analyzing the race variable in its SSI and other administrative data systems, because after 2002, it discontinued publication of data by race after altering its method for assigning new SSNs. In Mathematica’s PROMISE program process evaluation reports, about two-thirds of enrolled participants are identified as “other/unknown,” which may reflect individuals identifying as biracial or multiracial. Poverty: The study’s selection criteria of current SSI beneficiary indicates that youth and families had household incomes at or below the federal poverty level. Urbanicity: Complete residential profiles of the six programs’ enrolled participants were not available to us.
Types of disabilities: Mathematica recorded primary disabling conditions according to the SSA’s Listing of Impairments, and combined them into five categories listed in order of decreasing frequency: (a) intellectual or developmental disabilities, (b) other mental impairments, together accounting for 75 percent of enrolled youth. Remaining categories included (c) physical disabilities, (d) speech, hearing, or visual impairments, and (e) other. Primary Spoken Language: More than 85 percent of enrolled youth identified English as their preferred language, except for the California program (65%), and Spanish as the second most commonly preferred language.
Recruiter competencies with reasonable accommodations and cultural sensitivity
All programs trained and ensured staff involved in recruiting and obtaining consent and assent from youth and parents/guardians, were skilled with accommodating those with: (a) language barriers (e.g., low levels of literacy; languages other than English); (b) mental impairments (e.g., conduct disorders); (c) physical disabilities (e.g., spinal cord injuries); (d) sensory impairments (e.g., vision and hearing loss); and (e) cognitive impairments (e.g., borderline intellectual functioning). Similarly, all programs trained and ensured staff manifested cultural sensitivity as indicated by understanding one’s own culture and biases; accurately appraising the cultures of people living in one’s recruitment area; and learning how to respond to cultural beliefs that might impede recruiting and collecting data from youth and parents/guardians.
Strategies for recruiting, enrolling, and obtaining consent/assent
Youth and parents/guardians entered PROMISE programs in one of two ways: (a) they enrolled in response to invitations from direct mailings, telephone calls, or other program solicitations, or (b) in hearing about the study program in other ways, they initiated contact with program staff. Initial program contacts may have involved youth and their parents/guardians separately or together, depending on which specific recruitment method they encountered first. Each of the program’s research teams developed its own recruitment and enrollment plans, which included mixes of initial and follow-up mailings, telephone calls, emails, text messages, and in-person meetings.
Initial mailings, telephone calls, and emails
To aid programs’ recruitment and enrollment of youth and families, Mathematica developed a comprehensive procedures manual; supplied sample templates for initial mailings, consent forms for parents/guardians and assent forms for youth; and scripts for telephone calls, emails, and in-person meetings that could be customized by each program according to its specific population needs and staffing assignments. Mathematica provided technical assistance on recruitment and enrollment to the PROMISE programs on an as-needed basis.
Templates, scripts, and consent/assent forms covered purposes of studies, random assignment, services to be provided by research conditions (PROMISE versus usual), and how data collected from study participation would be used and kept confidential by the researchers. Each program’s research team and institutional review boards approved each program’s customized templates, scripts, and consent/assent forms for readability in both English and Spanish, and for cultural sensitivity. Required to enroll in PROMISE, every parent/guardian signed consent forms for their own and their child(ren); all youth signed assent forms.
SSA lists contained parent/guardians mailing addresses and telephone numbers; it was not possible to determine whether telephone numbers were landlines, cell phones, or a mix of the two. If mailing addresses and/or telephone numbers were out of date, programs were permitted to search available records of participating partners (e.g., Medicaid or Vocational Rehabilitation), or from other commercial sources such as LexisNexis, PeopleFinders, or Accurint, to obtain updated contact information.
In-person meetings
Programs determined where research staff met face-to-face with youth and parents/guardians, including their homes, PROMISE offices, and/or other public places such as libraries or coffee shops. Meetings might involve youth and parents/guardians individually or both together. Content presented by staff at the start of meetings adhered closely to that of initial mailings, telephone calls, and emails. Given the very large number youth and families that met in person with staff, a comprehensive accounting of the specific content of each in-person meeting was not possible.
Financial incentives for recruitment meeting and enrollment
Incentives, typically provided as gift cards, were calibrated to offset the time youth and/or parents/guardians spent associated with recruitment and enrollment to minimize chances that incentives would coerce participation. Three programs offered youth and families a financial incentive: Wisconsin ($30.00 total only to youth consisting of $15.00 for completing all enrollment forms, and $15.00 for completing the baseline survey), ASPIRE Consortium ($40.00 only to youth upon attending the required in-person enrollment meeting with a staff member). From the available data, we cannot assess the effectiveness of these incentives as a recruitment and enrollment strategy.
Results
Our cross-program profile of recruitment and enrollment outcomes compiled from data and narrative descriptions extracted from Mathematica’s process analysis reports of each of the six PROMISE programs covers (a) recruitment and enrollment rates by PROMISE program, (b) number of contacts necessary to enroll youth and parents/guardians by PROMISE program, (c) mix of recruitment strategies by enrollment status by PROMISE program, and (d) enrollment by both number of contacts and recruitment strategies by PROMISE program.
Recruitment and enrollment rates by PROMISE program
All six programs met the PROMISE cooperative agreement’s enrollment requirement within the two-year recruitment window. On all metrics of recruitment and enrollment, the programs varied considerably (Table 1). First, the cumulative number of all eligible youth on the SSA lists ranged from a low of 7,828 (Maryland) to a high of 21,939 (California). Second, the percentage of eligible youth specifically targeted for recruitment ranged from a low of 51.4 percent (California, 11,271 out of 21,939) to a high of 92.6 percent (Wisconsin, 8,570 out of 9,249). Third, the percentage of all eligible youth from the SSA lists who ultimately enrolled into the study ranged from a low of 10.3 percent (New York State, 2,090 out of 20,290) to a high of 25.6 percent (Maryland, 2,006 out of 7,828), and fourth, the percentage of eligible youth specifically targeted for recruitment who enrolled into the study ranged from a low of 22.3 percent (ASPIRE Consortium, 2,051 out of 9,196) to a high of 43.2 percent (Maryland, 2,006 out of 4,644). On a geographic basis, the ASPIRE Consortium, Maryland, and Wisconsin programs recruited statewide, Arkansas recruited from four counties, and California and New York State recruited from local education agencies located in four and three regions, respectively.
Recruitment and enrollment rates by PROMISE program
Recruitment and enrollment rates by PROMISE program
Sources: Mathematica Policy Research’s process analysis reports for each of the six PROMISE programs.
Notes:
1 Percentage of youth recruited from list = (Number of youth targeted for recruitment / Number of youth on SSA lists *100).
2 Percentage of youth enrolled into study = (Number of youth enrolled into study / Number of youth on SSA lists *100).
3 Percentage of youth recruited and enrolled into study = (Number of youth enrolled into study / Number of youth targeted for recruitment *100).
Mathematica reported the number of contacts made to targeted eligible youth in two ways: (a) average number of contacts per youth by enrollment status (yes/no) (Table 3), and (b) number of contacts per youth by four rank-ordered categories (one, two to five, six to ten, and eleven or more) by enrollment status (yes/no) (Table 4). Across the six programs, the average number of contacts to targeted eligible youth who enrolled ranged from a low of 2.5 (Arkansas) to a high of 6.2 (New York State); the average number of contacts for youth who did not enroll also ranged from a low of 2.5 (Arkansas) to a high of 6.2 (Maryland) (Table 3). By rank-ordered categories, the modal number of contacts to targeted eligible youth for five of six programs was two to five (Arkansas, 71.5%, ASPIRE Consortium, 76.1%, California, 63.3%, Maryland, 53.3%, Wisconsin, 43.2%), and six to ten for one program (New York State, 41.2%) (Table 4). For five of six programs, the bulk of youth (enrolled or not) were contacted between two and ten times (77% to 96%). Only two programs reported eleven or more contacts for more than a few percent of youth (Table 4). A very low percentage of eligible youth recruited and enrolled were contacted only once: less than ten percent for three programs (ASPIRE Consortium, Maryland, New York State), and between 20 and 41 percent for the other three programs (Arkansas, California, and Wisconsin) (Table 4). Similarly, a very low percentage of eligible youth recruited but not enrolled were contacted only once.
Mix of recruitment strategies by PROMISE program
All programs used different mixes of the six quantifiable strategies to recruit eligible youth from their SSA lists (Table 2). Five of six programs reported sending an initial mailing to more than 90 percent of eligible youth, inclusive of those who enrolled and those who did not. Two states (California, Maryland) reported that percentages of youth and families receiving an initial mailing enrolled at considerably lower rates, in both absolute and relative (California, 55%, Maryland, 89%) terms, compared to those who did not enroll (California, 71%, Maryland, 99%), indicating that multiple contacts of differing strategies would be necessary to meet enrollment targets (Table 2). Follow-up mailings varied sharply across the programs ranging from less than 10 percent (ASPIRE Consortium) to greater than 50 percent (Arkansas, New York State, Wisconsin). For all six states, youth and families receiving these follow-up mailings enrolled at considerably lower rates, both in absolute and relative terms, compared to those who did not enroll (Table 2). All programs relied heavily on telephone calls; all exceeded 50 percent and three exceeded 80 percent inclusive of those who enrolled and those who did not (ASPIRE Consortium, Maryland, New York State). Email was used infrequently by all programs (less than 5%). Text messages were used only by two programs (Maryland, Wisconsin). In-person meetings between one or more project staff members and youth and families who eventually enrolled in the study clustered into two groups: relatively lower absolute and relative (less than 17%, Arkansas, New York State, Wisconsin) frequency, and relatively higher absolute and relative (ASPIRE Consortium, 100%, California, 49.1%, Maryland, 62.2%) frequency (Table 2). For in-person meetings between project staff members and youth and families in these two clusters of three states who did not enroll, absolute and relative rates of meeting in person were lower, but similar in pattern (Table 2).
Recruitment strategies by enrollment status by PROMISE program
Recruitment strategies by enrollment status by PROMISE program
Sources: Mathematica Policy Research’s process analysis reports for each of the six PROMISE programs. Note: 1In the Mathematica process evaluation report, some of the California program contacts were assigned to an “other” category (not shown), which were not classifiable at the time of data input. These contacts likely fall into one of the categories in this table. 2Enrollment Status (Yes = number of eligible youth recruited who enrolled into the study. No = number of eligible youth recruited who declined to enroll into the study).
Three programs offered youth and families a financial incentive to enroll. Maryland offered both the youth and the parent/guardian a gift card for signing assent and consent forms, respectively. Wisconsin provided two gift cards to the youth, the first upon completing all enrollment forms, and the second upon completing the baseline survey. ASPIRE provided a gift card to each youth upon attending its required in-person enrollment meeting.
Importance of in-person meetings to enrollment
Three programs reported that multiple contacts by initial mailing, telephone calls, and text messages were necessary, but not sufficient to meet enrollment targets. Three states (Arkansas, California, New York State) reported that percentages of youth and families meeting in person with research project staff enrolled at much higher frequency (Arkansas, 12%, California, 49%, New York State, 89%) compared to those who did not enroll (Arkansas, 5%, California, 8%, New York State, 1%), indicating that even after multiple contacts of differing strategies with recruiters, it appear that in-person meetings were essential for persuading youth and parents/guardians to enroll in (Table 2) PROMISE programs.
Specifically the Maryland program reported that, although more than 90 percent of the eligible youth who received an initial mailing eventually enrolled, at least two to five follow-up contacts (second mailings, telephone calls, text messages) were made to 53 percent of eligible youth, and more than six contacts were made to 35 percent to youth who eventually enrolled (Table 4). Maryland staff reported that in-person visits with youth and parent/guardians, though time- and resource-intensive, were essential for enrolling youth even though a similar proportion who met with staff did not enroll (Table 2; 62.2% and 77%, respectively).
Second, although the California program sent initial and follow-up mailings to youth who eventually enrolled (55% and 17%, respectively) and followed-up with telephone calls to youth who eventually enrolled (83%), like Maryland, two to five contacts were made to 63 percent of youth who eventually enrolled 63 percent of youth, and six to ten contacts to enroll 14 percent (Table 4). Although California recruiters met in-person with only 20 percent of youth targeted for recruitment overall, enrollees were six times more likely to have met with a recruiter compared to youth who did not enroll (Table 2; 49.1% versus 8.1%, respectively). California program staff described in-person meetings with youth and parents/guardians as highly effective, but time-consuming, due to lengthy travel times over large geographic areas to meetings.
Third, the ASPIRE Consortium reported that, although nearly 98% percent of eligible youth and parents/guardians who received an initial mailing eventually enrolled (Table 2), at least two to five follow-up contacts (telephone calls, emails) were made to 76 percent of eligible youth who eventually enrolled (Table 2), and more than six contacts were made to another 20 percent (Table 4). The Consortium then required each youth who wanted to enroll to meet with a staff member; a very small percentage of youth declined to enroll after these meetings (Table 2; 100.0% versus 1.9%).
Assigning staff full-time versus part-time to recruitment activities
Three programs dedicated staff full-time to recruitment (Arkansas, Maryland, New York State). The Arkansas program designated four staff full-time to recruitment, one each to the program’s four geographic regions of operations, who were overseen by a full-time statewide recruitment coordinator. Among the six programs, the Arkansas program logged the lowest average number of contacts to both enrollees and non-enrollees (Table 3, 2.5 for both groups), and the lowest proportion six to ten contacts to both groups (Table 4, 2.9% and 3.6%, respectively).
Average number of contacts per youth by enrollment status by PROMISE program
Average number of contacts per youth by enrollment status by PROMISE program
Sources: Mathematica Policy Research’s process analysis reports for each of the six PROMISE programs.
Number of contacts per youth by rank-ordered categories by enrollment status by PROMISE program
Sources: Mathematica Policy Research’s process analysis reports for each of the six PROMISE programs.
At recruitment start-up, the Maryland program contracted with a third-party organization to field a team of full-time recruiters. The Maryland program logged an intermediate average number of contacts to both enrollees and non-enrollees (Table 3, 5.5, and 6.2, respectively), and an intermediate proportion two to five contacts (53.3%, Table 4) among the six programs, but achieved the highest rate of enrollment of youth targeted for recruitment among the six programs (Table 1, 43.2% versus a range of 15.6% to 29.0% for the other five programs).
During the first year of recruitment, the New York State program exhibited low rates of enrollment. For the second year of recruitment, among other adjustments, the program hired six community case managers, and a centralized telephone recruiter, to work full-time on recruitment. The pace of enrollment tripled, and the program met the required enrollment target on time.
For both enrollees and non-enrollees, the other three programs (ASPIRE Consortium, California, Wisconsin) logged an intermediate average number of contacts (Table 3), and an intermediate proportion two to five contacts (Table 4) among the six programs, allocating recruitment responsibilities to staff in a variety ofways.
Although the modal frequency of rank-ordered contacts made to youth who eventually enrolled across the programs is two to five (Table 4), there is considerable inter-program variation in the proportion of youth and families enrolled after just one contact (presumably in response to an initial mailing, Table 2). The Wisconsin, Arkansas, and, California, programs successfully enrolled more than 20 percent of youth and families with one contact (41%, 25%, 20%, respectively), a pattern different for youth who did not enroll after one contact (6.4%, 22%, 34%, respectively). The Wisconsin and Arkansas programs relied heavily upon initial mailings (Table 2, 95%, and 99%, respectively). The programs sent postcards to youth and families in advance of initial mailings to increase PROMISE brand recognition, and reduce the risk that mailings would be ignored. Nevertheless, despite these two programs’ modest success with using such a cost-efficient strategy, to meet their required enrollment targets, all PROMISE programs made multiple contacts using multiple strategies.
Discussion
The U.S. Department of Education funded six PROMISE programs, stipulating that, over a two-year recruitment window, five programs enroll a minimum of 2,000 youth and families, and one program a minimum of 3,000. This report profiled strategies used by each of the six PROMISE programs to meet their required enrollment targets on time, an impressive accomplishment given that less than half of randomized controlled trials of healthcare interventions meet pre-study specified recruitment goals (Treweek et al., 2018). However, we acknowledge that by providing each program with regularly updated lists of the entire population of eligible SSI youth residing within each program’s jurisdiction from which to recruit, SSA may have conferred a recruiting advantage lacking in most field-based trials by saving the programs considerable time and resources for identifying, locating, recruiting, and enrolling study participants. That said, we offer some observations about our cross-program PROMISE profile of recruitment and enrollment for consideration.
First, all six programs relied heavily upon initial mailings and follow-up telephone calls, the only two recruitment strategies determined to be effective with high certainty in randomized controlled trials (Treweek et al., 2018). Every program mailed initial recruitment packets to a vast majority of targeted eligible youth from SSA lists (>90%). Similarly, every program followed-up initial mailings with telephone calls to at least 50 percent of eligible youth and families; three programs exceeded 85 percent. In contrast, all programs infrequently communicated electronically with youth and families: emails (<5% across all programs) and text messages (used only by two of six programs). Second, in-person meetings between program staff, and youth and parents/guardians, varied sharply across the programs, ranging from 6.3 percent to 100 percent. Staff reported that in-person visits with youth and parent/guardians, though time- and resource-intensive, were essential for meeting enrollment targets.
Third, considering contacts and strategies together, we speculate that assigning staff full-time to recruitment activities may facilitate program enrollment. The Arkansas program designated four staff full-time to recruitment overseen by a full-time statewide recruitment coordinator. The program logged the lowest number of contacts among the six programs. The Maryland program contracted with a third party organization to conduct all recruitment activities, and achieved the highest rate of enrollment of youth targeted for recruitment among the six programs. In response to lagging enrollment rates, New York State hired full-time recruiters (six community case managers, and a centralized telephone recruiter), tripling their pace of enrollment, and meeting required enrollment on time.
Finally, because we do not have individual-level data of the type and amount exposure to recruitment strategies used with youth and parents/guardians, we cannot explore whether enrollment rates across programs might be associated with specific combinations of recruitment strategies interacting with frequency of contacts.
Lessons learned
We hope some of some lessons we learned about recruiting and enrolling youth and families with disabilities will be useful for transition researchers undertaking future field-based randomized control trials. Meeting ambitious enrollment targets requires multiple contacts using multiple recruitment strategies. For all but one PROMISE program, the modal number of contacts made to targeted eligible youth was two to five; one program made six to ten contacts. A very small proportion of potential participants enrolled after only one contact by initial mailing. Initial mailings and follow-up telephone calls are necessary, but not sufficient to meet enrollment targets. Although expensive in time and resources, recruiters meeting in-person with youth and parents/guardians, after multiple contacts using multiple strategies, may be essential to persuade some of those reluctant to participate to enroll. Assigning staff full-time to recruitment activities may facilitate program enrollment. Recruiting large numbers of participants dispersed over large geographic areas into randomized controlled trials is a time- and resource-intensive activity. Requiring program staff to multi-task—recruit persons while scaling-up new service programs—is a recipe for failing to meet enrollment targets and to ensure that service delivery adheres to program model standards.
In conclusion, recruiting and enrolling participants into randomized controlled trials requires considerable investment of time, money, and human resources. As the field of transition and special education moves toward testing the effectiveness of multicomponent service programs using experimental and quasi-experimental designs meeting What Works Clearinghouse (WWC) standards (e.g., Lemons et al., 2014), this descriptive study of a large-scale, field-based, and multi-site trial contributes to the limited existing knowledge on effective recruitment practices for enrolling youth with disabilities into future field-based trials.
Conflict of interest
None to report.
Footnotes
Acknowledgments
The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Office of Special Education Programs, associated with PROMISE Award #H418P140002 and a grant from the Department of Education (cooperative agreement #H418P130005). Selete Avoke served as the project officer. The views expressed herein do not necessarily represent the positions or policies of the Department of Education or its federal partners. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this publication is intended or should be inferred.
Arizona, Colorado, Montana, North Dakota, South Dakota, and Utah.
FR, Vol. 78, No. 98, p. 29738.
California was required to recruit at least 3000 youth.
The SSA lists were updated semi-annually throughout the duration of the recruitment period to reflect the fluid nature of the recruitment pool, in which youth would “age in” and “age out” of study eligibility.