Abstract
Natural experiments are often used for answering research questions in which randomization is implausible. Effective recruitment strategies are well documented for observational cohort studies and clinical trials, unlike recruitment methods for time-sensitive natural experiments. In this time-sensitive study of the impact of a minimum wage policy, we aimed to recruit 900 low-wage workers in Minneapolis, Minnesota, and Raleigh, North Carolina. We present our recruitment strategies, challenges, and successes for participant screening and enrollment of a difficult-to-reach population.
Introduction
Randomized controlled trials (RCTs) are the gold standard for studying causal relationships in epidemiology, as randomization ensures exchangeability between treatment and control groups (Hariton and Locascio 2018). However, RCTs are not useful for answering research questions in which random assignment is either impractical or unethical (Messer 2016). In these circumstances, natural experiments (NEs), in which naturally occurring events mimic random assignment of study subjects into treatment and control groups, are an alternate study design (Shadish et al. 2002). Events that mimic randomization in NEs may include policy changes, weather events, or natural disasters (Messer 2016).
NEs often capitalize on existing data for health-related evaluations, which can be cost effective and may offer broadly representative samples. However, primary data collection in which data are gathered prospectively from participants ensures that precise variables of interest are collected from the most appropriate samples. Recruiting participants for NEs poses a unique set of challenges. Data collection is often time sensitive because a defining characteristic of an NE is that the researcher does not control the randomizing event, which can leave little time for the execution of a rigorous evaluation (Taillie et al. 2017).
Effective recruitment strategies have been published for observational cohort studies (Toledano et al. 2015; Zucchelli et al. 2018) and clinical trials (Estabrooks et al. 2017; Frandsen et al. 2014), but few, if any, published studies have discussed recruitment methods relevant to the particular challenges of time-sensitive NEs. Additionally, recruitment of underrepresented populations, such as low-income households, remains a challenge in health research (Patel et al. 2003).
This article examines the successes and challenges of recruiting a sample of low-wage workers for the Wages Study, an NE evaluating the effect of a minimum wage increase ordinance on obesity and diet-related outcomes. The aims of the study are to test the effect of the minimum wage ordinance on change in body mass index (BMI), nutrition-related outcomes (healthy food purchasing, food security, participation in public assistance programs), and other health and household spending factors.
The Minneapolis City Council approved the Minneapolis Minimum Wage Ordinance on June 30, 2017. The ordinance raised the minimum wage to $15 per hour during an incremental implementation period, with a slower implementation for small businesses (seven years) compared with large businesses (five years) (City of Minneapolis 2017). This study aimed to recruit 450 low-wage workers in Minneapolis, Minnesota, and an additional 450 in a comparison community, Raleigh, North Carolina.
In this article, we describe the recruitment strategies used to reach a community-based sample of low-wage workers and the relative effectiveness of each strategy. The purpose is to inform future researchers aiming to recruit participants for time-sensitive NEs or cohort studies, particularly for studies focused on low-income populations. Maximizing recruitment successes for low-wage worker cohorts may be increasingly relevant as policies related to this segment of the workforce (e.g., Supplemental Nutrition Assistance Program work requirements, paid family and sick leave) become more prominent in the national policy discourse.
Methods
Selection of Comparison Community
First, we created a list of U.S. cities within 50% of the Minneapolis total population located in a state with a minimum wage preemption law as of October 2017. State preemption maximizes the potential for a consistent control setting. We then used U.S. Census data of these 41 cities to find matches (within 25% of Minneapolis) on 12 relevant demographics (Supplemental Table 1). Raleigh, NC, was a match on all criteria except percent poverty and percent Black. We next checked the “parallel trends” assumption for the primary outcome by looking at the 10-year BMI trend among residents with income <$35,000 in both cities (Supplemental Figure 1). Net BMI change in both cities was small and positive (an increase of 0.07 BMI units in Minneapolis and 0.02 in Raleigh). Finally, we compared weight-related outcomes and economic trends in relevant industries (Supplemental Table 2) before confirming Raleigh as the comparison site.
Target Sample and Eligibility
For this study, we aimed to recruit a sample of those likely to be affected by the minimum wage ordinance across all workforce sectors in Minneapolis, and a comparable sample in Raleigh. At the onset of the study, we set eligibility criteria to define this population as individuals working within the Minneapolis or Raleigh city limits and earning $10.00 per hour or less, the Minneapolis minimum wage for large businesses at the time recruitment began. Minimum wage at the start of the study was $7.87 for small businesses in Minneapolis, and $7.25 in Raleigh. Eligible participants also needed to expect to remain in the workforce for at least five years, verify wages with a paystub or letter from employer, and work at least 10 hours per week at a qualifying job. Participants could not be state or federal workers (exempted from the ordinance), students (who might expect a wage increase on graduation unrelated to the ordinance), or plan to move more than 100 miles away in the next five years. Subsequent changes were made to the eligibility criteria, though all were consistent with the intention of enrolling a representative sample of low-wage workers likely to be affected by the minimum wage in Minneapolis and comparable workers in Raleigh.
Participants enrolling in the study agreed to have their height and weight measured, complete a web-based survey, verify their wages with a paystub, and submit receipts for food purchases made over two weeks. Incentives for completing all baseline measures totaled $70. Participants also agreed to complete four additional annual follow-up visits.
Recruitment Strategies
We used both active and passive recruitment strategies to recruitment participants (Supplemental Table 3), characterizing active strategies as those with direct person-to-person interaction (Estabrooks et al. 2017). Figure 1 presents an overview of our recruitment timeline and strategies.
Recruitment timeline in Minneapolis (MN) and Raleigh (NC).
Initial Recruitment in Minneapolis
Research staff at the University of Minnesota began recruitment in Minneapolis in January 2018. The goal was to complete recruitment before July 2018, when the Minneapolis minimum wage increased from $10 to $11.25 for large businesses and $7.87 (state minimum wage) to $10 for small businesses. Due to challenges in recruitment and the phased ordinance implementation, the period of recruitment was extended through October 2018.
Initially, the primary recruitment strategies included posting fliers in high-traffic locations in the community, like laundromats and second-hand stores, and weekly online Craigslist advertisements. The online advertisements directed people to complete an online screening survey to determine their study eligibility. Study staff contacted those eligible to participate to schedule an appointment to complete the study activities at a centrally located and accessible university-based clinical research center.
The team modified processes to address barriers as they arose. During weekly study staff meetings, we reviewed trends in eligibility screenings, completed appointments, and no-shows and revised recruitment strategies accordingly.
Three specific barriers to recruitment emerged early on in Minneapolis. First, no-shows for scheduled appointments were common, suggesting that participants experienced barriers coming to the university location. Our community partners counseled us to enroll “on the spot” at a community location and offered space at their locations.
Second, the most common reason for ineligibility was earning more than $10 per hour. While a 2016 simulation study had suggested that many workers in sectors such as food service, health care, and retail would be making less than $10 an hour (Myers et al. 2016), by 2018, many Minneapolis employers appeared to be offering wages slightly higher at the time of enrollment. A strong local economy with low unemployment (2.9%), a vibrant public discourse about living wage, and wage changes for high-profile large employers may have contributed to the difficulty in identifying employers paying $10 or less (Federal Reserve Bank of St. Louis 2019).
Third, unemployment was the second most common reason for ineligibility. Low-wage workers also tend to have high job turnover and job insecurity (Saint-Paul 1997). Thus, many unemployed individuals in between minimum wage jobs would plausibly be affected by the ordinance during its years-long implementation.
In response to these challenges, we made several changes to eligibility criteria before 10% of the sample was enrolled. First, we allowed participants to enroll in the study “on the spot” at community sites and send in their paystub at a later time. We also increased the wage eligibility criteria from $10 per hour to $11.50 per hour (capturing workers at minimum wage for large employers and up to 15% above the minimum wage) (Dube 2019). Finally, we opened enrollment to those who were temporarily unemployed (had worked in the last six months at a job for $11.50 per hour or less) and were looking for work in Minneapolis or Raleigh. The research staff contacted any individual who completed an eligibility screen before these updates occurred to rescreen and subsequently enroll them in the study if they met the updated requirements.
The Minneapolis team also began to employ paid advertisement on Facebook with a link to complete an online eligibility screener and placed advertisements on high-use public transportation routes and attempted to build partnerships with local employers to promote the study. These strategies appeared to increase awareness about the study as indicated by a surge in the number of individuals screened for eligibility.
Initial Recruitment in Raleigh
Research staff at the University of North Carolina–Chapel Hill (UNC-CH) began recruitment in April 2018. Recruitment and enrollment activities concluded in September 2018. The Raleigh site began recruitment later and had more flexibility in enrollment dates as it was not bound by the same strict time constraints as the Minneapolis site (no minimum wage increase was anticipated).
The UNC-CH team identified a trusted community research partner based in Raleigh to serve as a community liaison and included funding for this liaison in the budget. Before beginning recruitment, this liaison convened a community advisory board with local leaders to guide recruitment efforts (see below in Community Location Recruitment). While the UNC-CH study team deployed similar passive recruitment strategies to Minneapolis (fliers, Craigslist), the team invested the most resources in community-based methods and relied on referral-based recruitment throughout the enrollment period based on lessons learned in Minneapolis prior to April 2018.
Recruitment in Minneapolis and Raleigh
The most effective strategies at both sites were active strategies: Community Location Recruitment and Friend Referral Recruitment, described in detail below.
Community location recruitment
The time-sensitive structure of the funding proposal and policy implementation did not lend itself to strict community-based participatory research principles during grant writing and start-up. However, both sites connected with community stakeholders during the grant submission period and intensified this outreach once support for the study appeared likely. The Raleigh team hired a community liaison at the outset, and the Minneapolis team connected with a community engagement lead shortly after recruitment began. The community liaisons were both well connected, trusted, and highly experienced in advocacy, research, and outreach. These liaisons identified community-based nonprofit and government organizations in both cities who serve low-wage workers through a variety of social services, including healthcare, employment, and emergency food.
In Minneapolis, the study team contacted over 80 local organizations to discuss a range of ways they could support recruitment, from posting fliers at their locations to actively promoting the study and hosting recruitment events. During the busiest months of recruitment in Minneapolis (April–September), the study staff engaged 35 organizations in recruitment activities. They spent several hours at five community locations every week to interact with and enroll potential participants.
In Raleigh, the study team connected with nearly 30 organizations to inform outreach and recruitment. Similar to the Minneapolis community network, these organizations supported the study team in both active and passive recruitment strategies, including disseminating flyers and electronic communications to their clients, connecting the team to organizations serving low-wage workers, and inviting the study team to events to screen participants for eligibility. Most notably, through this network, the study team partnered with a local nonprofit organization focused on connecting families with economic opportunities and affordable housing. This organization offered a comfortable, convenient space in one of their affordable housing apartment buildings to conduct study activities. This space was welcoming to participants with children and accessible via public transportation.
Friend referral recruitment
Both sites offered an incentive to enrolled participants who referred eligible people to the study. The enrolled participants received $5 for every screened and eligible person they referred. This strategy proved to be highly popular among participants, and to further incentivize referrals, the study team offered an opportunity to become a Super-Recruiter. Super-Recruiters were enrolled participants who referred more than 10 eligible people to the study; they received $10 per eligible person they referred beyond 10 people. In total, there were three Super-Recruiters in Minneapolis and four Super-Recruiters in Raleigh who, together, recruited 110 participants.
Recruitment Tracking and Analysis
While assessing eligibility, potential participants indicated how they learned about the study. We collected and managed data using REDCap electronic data capture tools hosted at the University of Minnesota (Harris et al. 2009, 2019). We exported data from REDCap into Stata 16 for analyses. We generated “Month of Screening” and “Month of Enrollment” variables for each participant based on the dates they completed their screening and data collection appointment. We identified the numbers and percent of people screened, found to be eligible, and enrolled by recruitment strategy at each site for individual recruitment strategies. We then calculated the number of people screened and enrolled by passive and active strategies in both sites by site and by month of enrollment.
Results
Both sites surpassed their target enrollment, but in Minneapolis it took longer than expected. In total, we screened 1,959 individuals for eligibility, of which 1,342 (68.5%) were eligible, and 974 (59.2% of eligible) enrolled in the study (n = 495 in Minneapolis, n = 479 in Raleigh). Participants drew from over a dozen job sectors in each city, with the top five sectors in both Minneapolis and Raleigh representing approximately two-thirds of the sample (Supplemental Table 4). Table 1 presents demographics of the final sample.
Demographic Information for Wages Participants in Minneapolis (n = 495) and Raleigh (n = 479).
Screening, eligibility, and enrollment data are presented in Table 2. Results demonstrate the success of active friend referrals and events, which accrued over three-quarters of participants in both cities. Costlier passive strategies implemented in Minneapolis, such as Facebook ads, were moderately successful for screening participants, but resulted in few enrollments.
Active (A) and Passive (P) Recruitment Methods in the Wages Study.
Figure 2 shows screening and enrollment by passive and active strategies in both cities. In Minneapolis, as recruitment progressed, a greater proportion of those screened and enrolled came from active strategies. In Raleigh, the proportion screened versus enrolled was fairly constant over time, with the majority using active strategies. Only 7% of enrolled Raleigh participants were recruited using passive strategies, compared to 14% in Minneapolis.
Number screened for eligibility versus enrolled by site, month, and method.
A post-hoc comparison of statistically significant differences in characteristics of participants (p <0.05) recruited by a friend versus those recruited via other methods was performed by study site (Supplemental Table 5). In both cities, those recruited by friends were less likely to be Latinx than those not recruited by friends. Additionally, in Minneapolis, those recruited by friends were more likely to be Black, and in Raleigh, those recruited by friends were on average 3.8 years younger.
A crude comparison with data from the 2018 American Community Survey (ACS) on city residents with annual earnings that are less than $15/hour based on 52 weeks of work and their average reported hours worked (“low-wage city residents”) shows that approximately 40% of low-wage city residents are in their 20s in both cities. Low-wage city residents are 21% Black in Minneapolis, and 31% Black in Raleigh. This comparison indicates that samples in both Minneapolis and Raleigh may disproportionately represent older workers and those who are Black.
Discussion
We used an iterative recruitment process to enroll a large sample of low-wage workers into a study evaluating the impact of a minimum wage ordinance on health. The initial recruitment plan relied on passive recruitment such as digital and print advertisements (Sikkens et al. 2017). Due to the initial low enrollment success, we added active recruitment strategies, such as attending community events and encouraging friend referrals. Friend referral recruitment was the most successful method at both sites. Ultimately, active recruitment strategies helped us reach the target study population.
This study, while focused on low-wage workers, sought to enroll a community-based sample rather than an employer-based sample. This approach maximized representation across employers in the low-wage workforce and circumvented the likely bias of only including “ordinance-friendly” employers. Indeed, a significant challenge in Minneapolis was the political atmosphere surrounding the minimum wage policy. Most employers and business-support organizations contacted by the University of Minnesota team did not agree to assist with participant outreach, presumably due to opposition to the policy. During study planning, several entities brought lawsuits against the City of Minneapolis challenging its minimum wage ordinance. Developing new relationships with labor advocacy organizations was also challenging; these organizations may have been hesitant to use valuable political capital endorsing a partnership with university research. Even though the study team was an independent evaluation team, some local partners may have viewed endorsing the study as an endorsement of the policy. The use of friend referrals was likely to have contributed to more successful recruitment of Black participants in Minneapolis (making it more comparable to the Raleigh site), while at the same time, friend recruitment may have contributed to a slightly younger sample.
Ultimately, we surpassed the enrollment goal in Minneapolis, but it required an extended baseline period and more flexible eligibility criteria. Under optimal circumstances, data would have been collected in a “clean” baseline period, and eligibility would have remained consistent after enrollment had begun to minimize issues with selection of different populations over time. In the decision to extend and expand enrollment, investigators weighed the quantifiable impacts on statistical power with the less quantifiable effects of extending and expanding enrollment on the study’s internal validity. It is notable that some elements in the design of the policy and the study made it possible to consider the baseline period with some flexibility.
First, the Minneapolis minimum wage increase has a phased implementation of multiple increases over several years, and the minimum wage for large businesses will not reach $15 until July 2022. Second, our analysis plan was based on net difference in wage growth between Minneapolis and Raleigh over the study period, and we still anticipate adequate differences in wage growth between the cities. Finally, and perhaps most importantly, the originally defined baseline period (before July 1, 2018) was perhaps an illusory “clean” baseline period because in a real-world setting, employer wage increases were likely to be staggered over a longer period of time. The start of ordinance-attributable wage increases may have begun as early as the period in which the ordinance was under consideration by City Council, because of concurrent public discussion urging large employers to consider a living wage. First wage increases may have also happened after July 2018 if employers were slow to comply with the policy. Thus, under optimal conditions, policy evaluation planning would begin while policies were still under consideration, and prospective data collection could start earlier to encompass a longer baseline period and minimize the challenges of collecting time-sensitive data collection.
Recruitment in Raleigh was generally less challenging than in Minneapolis, likely attributable to: (1) the troubleshooting period in Minneapolis before Raleigh began recruitment; (2) fewer political barriers to discussing the study; (3) initial involvement of a community-based liaison; and (4) potentially greater appeal of the participant incentive due to greater financial need among low-wage workers. North Carolina has a lower minimum wage and less access to safety net programs as compared to Minneapolis.
While we included funds for passive recruitment in the study budget, the more expensive passive strategies like Facebook and bus ads were not cost effective. Ultimately, community engagement had the greatest impact on enrollment. Budgeting fair compensation at the study outset for a community liaison or partners is imperative to develop effective and trustworthy communication and engagement strategies. Both sites employed a community liaison as a study team member and not simply an advisor. Both teams benefited from their existing networks and positive relationships with community organizations and participants. In Minneapolis in particular, navigating the political atmosphere required a seasoned community leader to suggest and execute new problem-solving approaches.
Community engagement also meant clearly and consistently conveying the purpose of the study and how results may be utilized to inform or recommend policy in the future to community-based organizations and participants. Because these relationships established community buy-in, they are also likely to bolster retention in the follow-up phases of data collection and contribute to more effective dissemination plans.
The Wages Study recruitment methods and results are similar to results from other studies that have used a community-engaged approach to recruit low-income participants. For example, the Southern Community Cohort Study (Signorello et al. 2010) partnered with community health centers to recruit underserved populations using a variety of active and passive strategies to enroll 32,632 participants with household incomes less than $15,000 per year. No formal comparison was made to determine whether active or passive recruitment strategies were more successful, but, like the Wages Study, the authors did conclude that working with community organizations was effective in recruiting low-income populations that are often difficult to reach. Additionally, the Talking Health Trial (Estabrooks et al. 2017) leveraged partnerships with community organizations to recruit 1,056 and enroll 301 participants from rural Virginia. Similar to the recruitment results from the Wages Study, the authors reported that active recruitment strategies yielded more enrolled individuals than passive. Additionally, they found passive recruitment strategies may yield a less representative sample in terms of sex (more women), education, and income.
Strengths and Limitations
Several limitations should be acknowledged. While we did not ask about immigration status and allowed participants to verify wages with a letter from their employers, the political climate surrounding immigration and the real or perceived threat of deportation limited our ability to engage with and subsequently enroll low-wage workers in both sites. Additionally, this study was conducted in two specific settings originally matched on demographics from the Census and ACS. Except for the difference in racial distribution (it was expected that Raleigh would have more African Americans in the study), the cohort that enrolled in the study differs somewhat more than expected between cities on key demographics including age and sex. It is unclear the extent to which these differences could be due to underlying differences between the two cities in the subpopulation of low-wage workers, but it appears plausible that these differences are mostly attributable to friend referral patterns at each site. In the second data collection period, assuring adequate retention of the sample will be critical to minimizing risk of bias (Bennett et al. 2018; Emmons et al. 2018).
Our study design offers several strengths. A community-based sample of workers offers considerably more generalizability than the alternate approach of recruiting a sample of workers from only one or two employment sectors, which has frequently been the approach of previous minimum wage studies and may have contributed to conflicting results about the effects of minimum wage ordinances (Otten et al. 2018; Reich et al. 2017; Zipperer and Schmitt 2017). Moreover, our analysis will be strengthened by collecting five data points from this cohort at different phases of policy implementation (Bennett et al. 2018). The study also offers several advantages over existing data sets that are likely to rely on self-reported weight outcomes and using proxy measures like education status to approximate the likelihood of being affected by minimum wage increases. Our design will allow us to collect data on individual wages and thereby calculate the precise “wage dose” received, it will measure the primary health outcome (BMI) objectively to eliminate self-report bias, and it will capture a range of plausible mediators to further test causal mechanisms specific to our research questions. Finally, by surpassing the targeted sample size, we have increased our statistical power and provided some insurance against both a higher-than-expected loss to follow-up rate and effect attenuation that may result from recruiting after the first wage increase. With our increased sample size, we are now well powered to detect our originally anticipated effect size of a BMI unit difference of 0.44 kg/m2, even with a 40% loss to follow-up, higher than our anticipated 25% rate. And with 25% loss to follow-up, we will have 80% power to detect a smaller BMI unit difference of 0.40 kg/m2 (i.e., we retain our original power for an effect that is approximately 10% smaller than anticipated).
Recommendations
Time-sensitive NEs affecting low-wage workers may be particularly relevant to researchers in the coming years as a range of policies likely to disproportionately affect low-wage workers are high on the political agenda, including proposed SNAP work requirements, paid family and sick leave, health insurance upheavals, and universal basic income. While in this study we found that active recruitment strategies were more successful than passive ones, we also recognized the importance of an iterative and flexible approach to recruitment.
Lessons learned for future NEs evaluating work-related policies include: (1) identifying well-connected community liaisons during the planning phases of the study and budgeting appropriate compensation to hire them; (2) collecting and monitoring recruitment screening metrics to identify successful and cost-effective strategies; (3) adopting a continuous learning approach by tracking reasons for ineligibility and attending to participant and community needs; (4) maintaining consistent communication with community organizations and offering multiple locations for recruitment; (5) implementing a mix of passive and active recruitment strategies; and (6) budgeting for regular study staff visits to community locations.
A funder’s perspective on the challenges faced for evaluations that are both prospectively designed and rapid response suggests additional recommendations. For example, strategies for mitigating the risks of such evaluation designs might include: (1) greater investment in pilot projects that build relationships that can capitalize on subsequent natural experiment opportunities, including planning grants; (2) setting budget parameters to ensure appropriate compensation for essential community partners; and (3) building capacity to ensure that reviewers have the relevant experience for gauging and metrics for evaluating the likelihood of successful community–researcher endeavors.
Conclusion
Utilizing a variety of active and passive recruitment methods, hiring a community liaison to build community rapport, and applying a continuous learning approach, we successfully enrolled 974 low-wage workers into a time-sensitive natural experiment to adequately power our study aims. Findings suggest that active recruitment methods, most notably being present at community locations and incentivizing friend referrals, may be key to enrolling low-wage workers.
Supplemental Material
Supplemental Material, sj-pdf-1-fmx-10.1177_1525822X20985966 - Recruitment of Low-wage Workers for a Time-sensitive Natural Experiment to Evaluate a Minimum Wage Policy: Challenges and Lessons Learned
Supplemental Material, sj-pdf-1-fmx-10.1177_1525822X20985966 for Recruitment of Low-wage Workers for a Time-sensitive Natural Experiment to Evaluate a Minimum Wage Policy: Challenges and Lessons Learned by Amy Shanafelt, Claire Sadeghzadeh, Leah Chapman, Molly De Marco, Lisa Harnack, Susan Gust, Melvin Jackson and Caitlin Caspi in Field Methods
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (1R01DK118664-01); NIH Grant UL1TR002494 from the National Center for Advancing Translational Sciences (NCATS) supported data management. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funding agencies had no role in the design, analysis, or writing of this article.
Supplemental Material
Supplemental Material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
