Evaluation Review

This article assesses several recent studies in the manpower training evaluation literature claiming that (1) nonexperimental methods of program evaluation produce unreliable estimates of program impacts and (2) randomized experiments are necessary to produce reliable ones. We present a more optimistic statement about the value of nonexperimental methods in analyzing the effects of training programs on earnings. Previous empirical demonstrations of the sensitivity of estimates of program impact to alternative non experimental procedures either do not test the validity of the testable assumptions that justify the nonexperimental procedures or else disregard the inference from such tests. We reanalyze data from the National Supported Work Demonstration experiment (NSW) utilized by LaLonde and Fraker and Maynard and reexamine the performance of nonexperimental estimates of the net impact of the NSW program on the posttraining earnings of young high school dropouts and adult women. Using several simple strategies for testing the appropriateness of alternative formulations of such estimators, we show that a number of the nonexperimental estimators used in these studies can be rejected. Although we eliminate a number of nonexperimental estimators by such tests, we are able to find estimators that are not rejected by these tests. Estimators not rejected by such tests yield net impact estimates that lead to the same inference about the impact of the program as the experimental estimates. The empirical results from our limited study provide tangible evidence that the recent denunciation of nonexperimental methods forevaluating manpower training effects is premature.

This article summarizes the results from two studies that used experimental data to evaluate nonexperimental methods of program evaluation. These studies compared the experimental estimates—of an employment and training program that used random assignment to allocate its participants into training positions—to estimates from nonexperimental procedures. Both studies indicate that the nonexperimental methods may not accurately replicate the experimental estimates, and that recently developed methods for constructing comparison groups are no more likely to yield accurate estimates of the impact of training than the more conventional econometric procedures.

This article examines the sensitivity of CETA net impact estimates to alternative methodological procedures to reconcile the diverse findings from various CETA studies that are based on the same basic data sets. Our results indicate that estimated net impacts are quite sensitive to whether individuals without recent labor market attachment are included in the comparison group and to how close to the date of enrollment one measures preprogram earnings. Also, impact estimates are quite sensitive to the choice of the estimation models used and to assumptions about the period in which the decision to enroll in CETA is made. However, net impact estimates are robust to the matching technique used. Our results indicate that the wide diversity of CETA net impact estimates reported in the literature can be explained by differences in the methodological procedures used.

This article summarizes estimates of CETA net impact on labor market outcomes using multivariate matching and weighting. The approach produces a comparison group from the Current Population Survey similar to the treatment sample on higher order interactions of match variables and displays only limited sensitivity to model specification. Unlike other approaches that rely on a priori assumptions, the estimates do not require explicit assumptions concerning the temporary or permanent nature of the "preentry earnings dip. "Statistical tests show the resulting comparison groups to be superior both to random samples of eligibles from the Current Population Survey and to comparison groups derived by statistical matching without replacement. Consistent with program changes, FY 1977 impact estimates were higher than FY 1976 impact estimates.

Recently, the Job Training Partnership Act has been implemented with the explicit purpose of moving economically disadvantaged individuals toward economic indepen dence through the provision of training. Ideally, we wish to know whether this type of training is a cost-effective expenditure.

This article presents estimates of program-induced, post-program earnings gains for men and women who enrolled in CETA between January 1975 and June 1976. Data for the analysis were obtained from the Continuous Longitudinal Manpower Survey for CETA participants with corresponding comparison group information for respondents from the March 1976 Current Population Survey. Although subject to methodological limitations, the findings suggest that CETA-sponsored classroom training, on-the-job training, and subsidized work experience noticeably increased the future earnings of female participants, but had virtually no impact for men. Impacts for women were roughly the same for all three program activities and were due primarily to increased employment rather than increased wage rates.

This study presents estimates of net earnings impact for adult-oriented CETA programs. The data bases used are the Continuous Longitudinal Manpower Survey and a matched comparison group derived from the Current Population Survey. After linking net impact to the human capital model, we estimate fixed effects regressions for numerous subgroups. The estimated equations were used to decompose earnings differentials to provide net impact estimates and to evaluate the quality of the matching procedure. Results are compared with related studies. We also examine broader issues comparing the potential for quasi-experimental versus experimental design in national evaluations of employment and training programs.