Return on Investment for Adult Basic Education: Existing Evidence and Future Directions

The National Evaluation of Adult Education Programs

NEAEP, conducted in the 1990s, examined ABE programs funded under the basic grants provision of the National Literacy Act in terms of participants’ educational gains (Beder, 1999; Ziegler & Sussman, 1996). An experimental design was not used for the evaluation. Instead, it surveyed and conducted pretest and posttest involving more than 20,000 ABE participants enrolled in 116 programs (Beder, 1999). Beder highlighted multiple methodological critiques of this study that render its analysis unusable for ROI purposes. One key area in which the rigor of the evaluation was compromised was a high attrition rate that left only 614 cases, not representative of the entire population, and too small for data analyses (Beder, 1999, p. 28). Most important, however, there was no direct relationship found between students’ persistence in their programs and learning gains as measured by test scores. NEAEP cannot provide an empirical basis for ROI estimation because of its research design as well as its findings (Beder, 1999).

Tennessee Longitudinal Study of Adult Literacy Programs

Tennessee Longitudinal Study was conducted from 1991 to 1995 by the Center for Literacy Studies (Bingman, 2009) and was designed to assess the long-term impacts of participation in adult literacy programs on quality of life. It employed a mixed-method approach consisting of at least three annual follow-ups, standardized interviews after baseline data collection, administration of a self-esteem assessment, and life history interviews conducted with just 10 participants. The study focused on 450 adult participants from three enrollment cohorts (1991-1992, 1992-1993, 1993-1994) whose initial scores on the Adult Basic Learning Exam reading test were below the sixth-grade level. The study participants were drawn from one of nine Tennessee programs selected to be representative of state demographic characteristics. Approximately 1 year after their initial enrollment in the program, the remaining 199 adults from the three cohorts who took part in a follow-up interview responded to a series of 116 questions on employment, literacy practices, involvement with children’s schooling, community awareness, self-esteem, and life satisfaction. Analyses of their responses showed mixed results at best. Although designed to study the impact of ABE participation, like the NEAEP, it did not employ an experimental design and could not determine the impacts of ABE programs on participants’ outcomes as compared with nonparticipants’ outcomes (Bingman et al., 1999).

The National Evaluation of the Even Start Family Literacy Program

This research focused on a federally funded intervention intended simultaneously to improve early childhood outcomes and adult basic skills as a way to break the so-called intergenerational cycle of illiteracy and poverty (Cooter, 2006; Daisey, 1991). To be eligible for the Even Start program, a family was required to have an adult eligible for adult literacy education who was also the primary caregiver of a child younger than 8 years of age living in a Chapter 1 elementary school attendance area. This program was distinctive in its assumption that bundling educational services for children and adults would have a greater impact than offering separate programs to each population. Its core elements consisted of ABE, early childhood education, and parenting education, along with relevant support services.

As required by statute, three national evaluations were conducted, but they did not demonstrate a positive impact. The evaluations were critiqued on many methodological grounds (Soliman, 2018; Tao et al., 1998) such as improper data use, incomplete and inconsistent data, nonrepresentative sampling, control group contamination, and poor data quality (Soliman, 2018). When experimental designs were employed in the first and third evaluations, the experimental group failed to show better outcomes than the control group in literacy, parenting skills, family resources, income or employment, and there was no relationship between the duration of program participation and gains in adult literacy (Soliman, 2018). Regardless of whether the methodology was appropriate, the findings render this evaluation also unusable for ROI estimation purposes.

National Evaluation of the Welfare-to-Work Programs

The most robust evidence on the impact of ABE participation comes from the 1989-2002 National Evaluation of the WTW Programs. The evaluation of GAIN WTW programs is one of the few studies in the literature that used a longitudinal randomized experimental design to study the impact of ABE participation on adult’s literacy gains, employment, earnings, and welfare use (Beder, 1999; Hamilton, 2002). In this study, program participation was mandatory because the 1988 Job Opportunities and Basic Skills Training program, authorized by the Family Support Act, required ABE participation for welfare recipients deemed in need of it. The WTW programs assumed that human capital deficiency is a major barrier to individual economic success. The participating programs randomized welfare recipients into treatment and control groups and offered ABE, GED, and ESL to the treatment groups but not to the control groups for at least the first 3 years of a 5-year follow-up period (although control group participants were allowed to access other local programs). Amid many reports and journal articles on the findings of these experiments, at least six are worth noting as they documented crucial 3-year, 5-year, and 9-year program impacts along with cost-benefits estimations (Boudett & Friedlander, 1997; Friedlander & Martinson, 1996; Greenberg et al., 2010; Hamilton et al., 2001; Hotz et al., 2006; Riccio et al., 1994).

Researchers found mixed 3-year follow-up results depending on location. The ABE participants who attained GEDs in Alameda County or achieved literacy gains in San Diego County did not experience a significant increase in earnings compared with the control groups (Riccio et al., 1994). Moreover, the experimental group in Riverside achieved earnings increases without improving literacy skills. When basic skill gains were measured by baseline test scores, there were no differences in literacy skills between the treatment and control groups although the treatment group made the largest increase in hours of ABE under GAIN (Friedlander & Martinson, 1996).

These disappointing findings were reinforced by a subsequent longitudinal study reported by Manpower Demonstration Research Corporation, which conducted a 5-year experimental design evaluation of 11 WTW programs. Of the seven programs that focused on ABE, only three had any impact on participants’ employment rate, and the impact was small. Just two programs had earnings impacts among those without high school education. That is, contrary to expectations, ABE did not significantly improve the earnings of welfare recipients, especially for those with limited education (Hamilton, et al., 2001). Participation in some education-focused programs rather significantly decreased earnings (Riccio et al., 1994). According to Greenberg et al. (2010), mandatory participation in ABE programs was not cost-beneficial, and sometimes had negative effects on both participants’ earnings and government budgets. At the time of the evaluation, the unexpectedly negative findings cast doubt on the classic human capital theory and endangered ABE funding in federal and state antipoverty programs. Consequently, as the Personal Responsibility and Work Opportunity Reconciliation Act indicated, policy makers largely abandoned ABE as an antipoverty strategy in the 1990s. They showed little interest in offering ABE opportunities to poorly skilled welfare recipients in the new policy statute.

Because these evaluations did not find a positive impact, the National Evaluation of WTW programs cannot be used as a basis of an ROI estimation. Critiques of the National Evaluation of the WTW program, however, pointed out that the evaluation did not take into account the quality of the intervention regarding variation in program delivery and instruction, which could have explained the inconsistent and negative impacts (Hotz et al., 2006). As discussed below, later longitudinal studies that examined the impacts over a longer period and controlled for program variations did show positive program impacts and cost-benefits of the program participation.

The Longitudinal Study of Adult Learning

LSAL was conducted from 1998 to 2007 and examined the impacts of ABE on various participant outcomes using a quasi-experimental longitudinal design (Reder, 2014a, 2014b, 2014c, 2014d). The study was unique in the literature as it was not an intervention study. It drew a statistically representative sample derived from random-digit dialing and enrollment forms from three major adult education programs for adults (aged 18 to 44 years) who did not have a high school diploma, a GED, or English proficiency in the Portland metro area. The impacts of ABE participation were analyzed while statistically controlling for differences in unobservable characteristics between self-selected participants and nonparticipants of the study. Because LSAL seems to be the only major evaluation study that finds positive impacts of ABE participation on a variety of outcomes, a closer examination is warranted to gauge if the impacts can serve as the basis of ROI estimation for ABE. The panel of 940 people who were initially interviewed was retained in the study and interviewed and assessed six times over 9 years (until the year 2007) with about a 90% retention rate. The study also obtained 11 years of information on their state unemployment insurance hours and wages from Oregon and Washington state (because of its proximity to Oregon) employment agencies.

Reder (2014a, 2014b, 2014c, 2014d) examined the impacts of ABE program participation on participants’ long-term literacy growth, GED pass rates, engagement in postsecondary education, and earnings. He found statistically significant differences (approximately $9,600) in the earnings of ABE participants and nonparticipants after controlling for observable differences between the two groups. According to his analysis, the intensity of program participation and elapsed time since the onset of participation explained these earnings gains. Reder (2012) observed that while participation in ABE programs did not have an immediate impact on literacy proficiency, it did increase the frequency of reading and writing practices, which, in turn, was related to improved literacy proficiency later. As with earnings gains, the improvement took substantial time, 5 to 6 years, to accrue. Although LSAL demonstrated positive impacts of ABE program participation, it did not provide cost data for the programs, which limits its usefulness from an ROI perspective.

The Evaluation Window

One of the major evaluation challenges has to do with the evaluation window. Program impacts as demonstrated above in LSAL typically take many years to emerge. Thus a long-term follow-up evaluation is necessary to truly capture the effects of participation in ABE programs. Assessing program impacts depends critically on how soon they appear and how long they last for the participants (Trutko & Barnow, 2010). The timing and the duration of program evaluations are also crucial to properly assess program impacts because program benefits would be smaller if outcomes take time to emerge but fade quickly; they would be much larger if they appear quickly and last a long time. The duration of the evaluations was typically short-term and was inappropriate in observing for long-term program impacts to emerge. Long-term evaluations are, in fact, essential to determining whether higher investments in ABE produce greater eventual impacts and the extent to which the impacts increase or diminish over time.

A team of researchers led by Hotz et al. (2006) reanalyzed the WTW program evaluation data and provides an excellent piece of supporting evidence for the importance of long-term follow-up. They noted that various experimental WTW programs differed in the mix and assignment of types of training for the participants and argued that those differences in the experimental data should be adjusted with multiple regressions. Using data from four of the six participating California counties for 9 years after random assignment, they tested the effects of mandatory participation in ABE focused programs on participants’ outcomes while controlling for the participants’ personal/family characteristics, pretreatment earnings and employment, and posttreatment local labor market conditions. As their data allowed examination of the long-term impact of ABE, their findings led to substantively different conclusions publicized in the previous decade with initial evaluations. The impacts of ABE on the rate and duration of employment and annual earnings did indeed take time to appear, and the impacts grew sizably with time, especially after about 5 to 6 years (Hotz et al., 2006). Although these new findings came too late to influence the policy movement away from mandatory ABE participation, they certainly highlight the importance of long-term evaluations for program impact.

Matching Program Goal to Program Offering

It is also important to point out that when an ABE program does not show a positive impact, it does not necessarily mean that the human capital theory that undergirds ABE is flawed, nor does it mean that it would not work under different circumstances. Failure to show impacts might be caused by challenges in matching program goals to program offerings and participants. As critics of the WTW evaluations argued, program impact was measured by increased earnings, but the remedial ABE courses did not lead to serious vocational training, a better match with the program’s employment-focused goal. Furthermore, many participants did not achieve a level of skill mastery that enabled them to earn a GED or a job training certificate, which might have had a payoff in the labor market (Greenberg et al., 2010).

More recently, programs such as Washington State’s Integrated Basic Education and Skills Training (I-BEST) have been addressing this type of mismatch by integrating ABE with occupational and career training in a community college setting to move adult students into a career pathway (Zeidenberg et al., 2010). A nonexperimental design study by the team led by Zeidenberg et al. (2010) found that the program had a positive impact on basic skills as compared with nonparticipants. A 2-year experimental design study of I-BEST found that it had positive impacts on the number of academic and workforce credits the students earned and their enrollment in occupational training courses. Furthermore, the Washington State Board for Community and Technical Colleges conducted an ROI analysis of I-BEST and found that the program had a positive ROI ratio of 0.425 (i.e., a 42.5 cent return on a $1 investment (Washington State Board for Community and Technical Colleges, 2013).

I-BEST was not a typical ABE program, however, participants appeared to be more highly skilled than typical ABE participants and their tuitions were covered by the colleges participating in the evaluation if they could not secure other funding. The cost of books, tools, and other course materials, as well as transportation, was also covered. Nevertheless, positive program impacts and ROI estimation highlight the importance of matching the program goals and program offerings through the integration of training and education (Washington State Board for Community and Technical Colleges, 2013).

Focusing on the Correct Change Mechanism

Carpentieri (2019) suggests that one reason ABE program evaluations have not generally shown effectiveness is not that ABE is ineffective, but rather because the evaluations did not focus on the mechanisms through which the desired outcomes are achieved; “The failure lies in the repeated focus on collecting . . . evidence on basic skills gain, before collecting sufficient evidence on the mechanisms through which basic skills gains might be achieved” (p. 649). He argues that when complex policy problems fail to produce expected results, it may be due to insufficient knowledge about the mechanisms that bring them about. Perhaps one reason that the LSAL showed effectiveness when other evaluations did not, Carpentieri argues, is that it was theoretically framed around a change mechanism (practice engagement theory), which hypothesizes that increased engagement in literacy practices leads to skill gain. Thus, it tracked the mechanism for change (increased practices), skill gain, and benefits of participation, which were not observable during or soon after program participation. Presumably, engagement with increased literacy practices is not the only change mechanism. Carpentieri suggests that evaluations should focus on change mechanisms and the outcomes they lead to. In this way, there will be a better match between program inputs and outcomes that can, in turn, more accurately describe program impacts.

Limitations in Benefit Valuation and Monetization Methods

Even with well-designed and well-implemented program evaluations that find evidence of positive impacts, ROI estimation can be challenging and even controversial because of the need to valuate and monetize all program costs and benefits. While direct costs and tangible benefits related to the intervention program may be easy to observe and monetize, indirect costs and nontangible benefits are not. For example, indirect or nontangible benefits of ABE programs such as improved health and well-being, more active participation in children’s education, or increased civic engagement and social cohesion are difficult to quantify without creating many measurement errors or controversy regarding subjectivity (Phillips, 1994, 2012). Nevertheless, how indirect costs and nontangible benefits are addressed may have implications for whether a given program has a positive ROI (Karoly, 2012).

Klees (2016) points to other nonmonetizable benefits of education programs when he argues that they go beyond the individual participants to include

their family, friends, coworkers; better health, lower crime, less welfare dependency; more technologically sophisticated workplace; greater international competitiveness; a common core of values; a literate, democratic society; and more. Externalities [such as these] are difficult to measure, especially in the monetary terms needed . . . [However,] without measuring externalities, we cannot accurately compute the ROI to education. (p. 650)

According to Moonen (2003), these kinds of extended benefits of educational program participation are often hidden, slow to emerge, and difficult to quantify (pp. 148-149). Klees and Moonens caution against calculating ROI for educational interventions. They remind us, instead, that although ROI estimates can be one of many useful tools that guides public investment in ABE, alternatives should be considered.

Limitations Due to Diversity in Programs and Learners

Although an important first step of ROI estimation is to establish a positive program impact, it is quite challenging to do so because of the enormous diversity in both learners and programs in ABE (Tamassia et al., 2007). In some of the aforementioned national evaluation studies, evidence of program effectiveness could have been masked as the studies examined the existing ABE system as a whole, rather than evaluating a small-scale intervention in a controlled setting. Given so little standardization in service provision, delivery, learners as well as most volunteer instructors with little professional development, the evaluation studies sought evidence of effectiveness from programs of vastly different quality and learners whose skills, needs, and interests diverge in significant ways. The evaluation studies used data collection instruments too blunt to distinguish between different levels of exposure (intensity and duration of participation), different program interventions, and different learners. In other words, there was no mechanism for distinguishing highly effective programs from less effective ones.

As much as the existing diversity in ABE programs and learners helps contextualize the disappointing results of earlier national evaluations, it also sends us a clear warning about the applicability and usefulness of ROI estimations for existing programs and learners on the ground. While typically ROI estimations are conducted on a single, small scale intervention where the program elements are tightly controlled, ABE in the United States is quite the opposite; it is not any one single intervention for any one single population (Tamassia et al., 2007). An estimation could, at best, only indicate the value of participation in a particular ABE program without reference to participants and programs in other contexts.

Limitations in Replicability and Scaling Up

The above discussion is directly relevant to the final, yet one of the most important limitations of ROI estimation. As repeatedly aforementioned, ROI estimation should be based on a small, bounded intervention program of limited duration and at a designated geographic location (or a series of multiple yet tightly integrated small scale programs) rather than a large scale, diverse, and ongoing program. This suggests that even if there is convincing evidence for positive ABE impacts and ROI estimates based on a specific program intervention, it could be difficult to scale up and replicate the success of small experimental ABE programs on a national scale given the vast diversity in providers, program delivery models and goals, learners, and local economic conditions. A weak and underfunded professional development system in ABE, for one thing, creates a further scaling challenge because it can hamper the potential for substantive changes in instructional practices that might be needed. Even if it were possible to create a small-scale, cost-effective program, it may not necessarily yield a high ROI ratio when it is scaled up and implemented in diverse settings nationally (Greenberg et al., 2010). ROI ratios may change dramatically depending on the scale of a program and diversity in participants and program settings.

Building on LSAL

In the section above, we discussed that LSAL was the only evaluation study for general ABE populations, albeit quasi-experimental, that demonstrated long-term positive impacts of the ABE program. Because the purpose of the LSAL did not extend to a cost-benefit analysis, costs and benefits, both direct and indirect, associated with program participation were not provided. It is, however, imperative to conduct an experimental design study that also collects the related cost data if sound arguments are to be made about ROI (Office of Career, Technical, and Adult Education, 2015; Shi & Tsang, 2008). Given limited evaluation funding and high costs involved in experimental design evaluation, however, it may be realistic to accept LSAL findings (despite its quasi-experimental design) and use it as the basis of ROI estimation. The costs of ABE participation for LSAL participants may be obtained through creative collaborations among researchers and stakeholders. As most LSAL participants were enrolled in community college programs in Oregon funded by the Adult Education and Family Literacy Act, it may be possible to guesstimate the per-capita program costs from the federal (and matching state) funds allocated to the programs. Additional costs that might have been born by the community colleges involved in LSAL could also be estimated through collaboration with those community colleges. The participating students’ opportunity costs could be estimated through collaboration with state agencies that report wage data. When the per-capita cost estimates for LSAL participants can be compared with their gains in earnings and incomes, it may be possible to shed some light on the ongoing ROI debates for ABE.

Systematic Yet Subjective Approaches

The problem of ROI with assigning a monetary value to nonmonetary outcomes for social programs has been addressed through what is known as Social Return on Investment (SROI). SROI was first developed to conduct a cost-benefit analysis for social enterprise endeavors to demonstrate wider value creation for social and economic returns (Rotheroe & Richards, 2007). It aims for systematicity and rigor of program evaluation but incorporates subjectivity of cost-benefit analysis (Rotheroe & Richards, 2007). It uses an approach known as “financial proxies” to estimate the value of nonmonetary social outcomes by identifying the economic cost or benefit of proxy outcomes and activities that may accrue as the result of a given social outcome. Identifying and combining all financial proxies can contribute to an ROI estimate that is informed by nonmonetizable social outcomes of an intervention. These estimations of proxies, however, are inherently subjective for those involved in the intervention.

Moonen (2003) doubles down on the subjectivity of SROI in a model he calls “simplified ROI.” This six-step approach begins with finding consensus among stakeholders about what should be considered as measurement items in the calculation and then organizing items into categories (economic, quality, efficiency), measuring change, calculating, transforming, and combining. His model recognizes that limiting benefits to those that can be monetized may unduly decrease actual returns and that trying to monetize nonmonetary benefits is generally a very subjective process. He suggests instead that monetary and nonmonetary costs and benefits be translated into numerical values on an “assumed interval scale” (p. 150). While he acknowledges that doing so requires substantial subjective judgments, he argues that this very subjectivity is reflective of reality. He proposes thinking of simplified ROI as less a financial calculation and more a process that can stimulate systematic thinking about change and improvement. It seems that this approach has stronger implications for practitioners than policy makers, but its focus on improvement and effectiveness suggests the potential for identifying and valuing the benefits of ABE programs that are agreed on by all stakeholders.

Values-Based Alternatives

Klees (2016) offers a nonmonetary, values-based alternative to ROI and suggests that human rights, human capabilities, and human agency can be used as lenses for conceptualizing and assessing educational policies. In contrasting the human capital logic which fuels ROI research with a human rights view, Klees (2016) states, “Instead of economists’ instrumentalist, a human capital approach using rates of return to establish investment trade-offs, a human rights approach does not consider these tradeoffs as a legitimate basis for policy” (p. 661). He also argues that each of these proposed lenses offers clear policy implications. Klees admits that these frameworks do not provide concise economic data and argues that the economic data is problematic, at best, anyway. When it comes to determining which policy is better than another (or where investments are best made), “better is embodied in a convoluted and ultimately empty concept of economic efficiency, and the impacts measured are partial, inaccurate, and misleading” (Klees, 2016, p. 666). His argument suggests that we should not get bogged down in a purely economic argument, which is unlikely to ever be made empirically in a reliable way. Instead, ABE advocates and policy makers should go beyond the narrow values embedded in ROI methods and embrace broader social values manifested in ABE policy and programs.