This article demonstrates a novel application of propensity score matching techniques: to estimate nonexperimental impacts on program participants within the context of an experimental research design. The author examines the relationship between program participation, defined as qualifying for an earnings supplement by working full-time, and marital union formation among low-income mothers in two Canadian provinces. The author finds that receipt of an earnings supplement substantially increased union formation in one province but not the other. A subgroup analysis based on propensities of program participation revealed that the positive effect on unions was concentrated among relatively disadvantaged participants. The techniques demonstrated in the article are broadly applicable to studies in which take-up is less than 100% among those randomly assigned to a program group.
Agodini, R.
, and
M. Dynarski.2004.
Are experiments the only option? A look at dropout prevention programs
. Review of Economics and Statistics86 (1):
180-194
.
2.
Angrist, J. D.
,
G. W. Imbens
, and
D. B. Rubin.1996.
Identification of causal effects using instrumental variables
. Journal of the American Statistical Association91(434):
444-455
.
3.
Becker, G.1981. A treatise on the family.
Cambridge, MA: Harvard University Press
.
4.
Berk, R. A.
, and
P. J. Newton.1985.
Does arrest really deter wife battery? An effort to replicate the findings of the Minneapolis spouse abuse experiment
. American Sociological Review50(2):
253-262
.
5.
Bloom, D.1997. After AFDC: Welfare-to-work choices and challenges for states.
New York: Manpower Demonstration Research Corporation
.
6.
Bloom, H.
,
C. Michalopoulos
,
C. Hill
, and
Y. Lei.2002. Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? MDRC Working Papers on
Research Methodology, Manpower Demonstration Research Corporation, New York
.
7.
Bloom, H. S.1984.
Accounting for no-shows in experimental evaluation designs
. Evaluation Review8:
225-246
.
8.
Bloom, H. S.
,
L. L. Orr
,
S. H. Bell
,
G. Cave
,
F. Doolittle
,
W. Lin
, and
J. M. Bos.1997.
The benefits and costs of JTPA title II-a programs: Key findings from the national Job Training Partnership Act study
. Journal of Human Resources32 (3):
549-576
.
9.
Cain, G. G.
, and
D. A. Wissoker.1990.
A reanalysis of marital stability in the Seattle-Denver income-maintenance experiment
. American Journal Of Sociology95 (5):
1235-1269
.
10.
Dehejia, R.
, and
S. Wahba.1998. Propensity score matching methods for non-experimental causal studies. Working Paper 6829, National Bureau of Education Research Working Paper Series, Cambridge, MA.
11.
Dickert-Conlin, S.
, and
S. Houser.2002.
EITC and marriage
. National Tax Journal55 (1):
25-40
.
12.
Dickinson, K. P.
,
T. R. Johnson
, and
R. W. West.1986.
An analysis of the impact of CETA programs on participants' earnings
. Journal of Human Resources21 (1):
64-91
.
13.
Dion, M. R.2005.
Healthy marriage programs: Learning what works
. Future of Children15 (2):
139-156
.
14.
Ellwood, D.
, and
C. Jencks.2004. The spread of single-parent families in the United States since 1960. In The Future of the Family, ed.
D. P. Moynihan
,
T. M. Smeeding
, and
L. Rainwater
.
New York: Russell Sage
.
15.
England, P.
, and
G. Farkas.1986. Households, employment, and gender: A social, economic, and demographic view.
New York: Aldine
.
16.
Fein, D.
,
R. A. London
, and
J. Mauldon.2002. Welfare reform and family formation: Assessing the effects.
Bethesda, MD: Abt Associates
.
17.
Fraker, T.
, and
R. Maynard.1987.
The adequacy of comparison group designs for evaluations of employment-related programs
. Journal of Human Resources22 (2):
194-227
.
18.
Friedlander, D.
,
D. H. Greenberg
, and
P. K. Robins.1997.
Evaluating government training programs for the economically disadvantaged
. Journal of Economic Literature35 (4):
1809-1855
.
19.
Gennetian, L. A.
, and
V. Knox.2003. Staying single: The effects of welfare reform policies on marriage and cohabitation. Next Generation Working Paper no. 19,
Manpower Demonstration Research Corporation, New York
.
20.
Gennetian, L. A.
, and
V. Knox.2004.
The effects of a Minnesota welfare reform program on marital stability six years later
. Population Research And Policy Review23 (5-6):
567-593
.
21.
Gibson, C. M.2003.
Privileging the participant: The importance of sub-group analysis in social welfare evaluations
. American Journal of Evaluation24 (4):
443-469
.
22.
Hannan, M. T.
,
N. B. Tuma
, and
L. P. Groeneveld
. 1978.
Income and independence effects on marital dissolution—results from Seattle and Denver income-maintenance experiments
. American Journal Of Sociology84 (3):
611-633
.
23.
Harknett, K.
, and
L. A. Gennetian
. 2003.
How an earnings supplement can affect union formation among low-income single mothers
. Demography40 (3):
451-478
.
24.
Heckman, J.
,
H. Ichimura
,
J. Smith
, and
P. Todd
. 1998.
Characterizing selection bias using experimental data
. Econometrica66 (5):
1017-1098
.
25.
Heckman, J.
,
H. Ichimura
, and
P. E. Todd
. 1997.
Matching as an econometric evaluation estimator: Evidence from evaluating a job training program
. Review of Economic Studies64 (4):
605-654
.
26.
Imbens, G. W.2004.
Nonparametric estimation of average treatment effects under exogeneity: A review
. Review of Economics and Statistics86(1):
4-29
.
27.
Katz, L. F.
,
J. R. Kling
, and
J. B. Liebman
. 2001.
Moving to opportunity in Boston: Early results of a randomized experiment
. Quarterly Journal of Economics116 (2):
607-654
.
28.
Kemple, J. J.
, and
J. C. Snipes
. 2001. A regression-based strategy for defining subgroups in a social experiment. MDRC Working Papers on
Research Methodology, Manpower Demonstration Research Corporation, New York
.
29.
Knab, J. T.
,
J. M. Bos
,
D. Friedlander
, and
J. W. Weissman
. 2000. Do mandates matter? The effects of a mandate to enter a welfare-to-work program.
New York: Manpower Demonstration Research Corporation
.
30.
LaLonde, R.1986.
Evaluating the econometric evaluations of training programs with experimental data
. American Economic Review76 (4):
604-620
.
31.
Lewin, A. C.2005.
The effect of economic stability on family stability among welfare recipients
. Evaluation Review29 (3):
223-240
.
32.
Lichter, D. T.
,
G. Kephart
,
D. K. McLaughlin
, and
D. J. Landry
. 1992.
Race and the retreat from marriage: A shortage of marriageable men?American Sociological Review57 (6):
781-799
.
33.
Lin, W.
,
P. K. Robins
,
D. Card
,
K. Harknett
, and
S. Lui-Gurr
. 1998. When financial incentives encourage work: Complete 18-month findings from the Self-Sufficiency Project.
Ottawa, Canada: Social Research and Demonstration Corporation
.
34.
Luellen, J. K.
,
W. R. Shadish
, and
M. H. Clark
. 2005.
Propensity scores: An introduction and experimental test
. Evaluation Review29 (6):
530-558
.
35.
Mijanovich, T.
, and
D. Long
. 1995. Creating an alternative to welfare: First-year findings on the implementation, welfare impacts, and costs of the Self-Sufficiency Project.
Ottawa, Canada: Social Research and Demonstration Corporation
.
36.
Rosenbaum, P.1995. Observational studies.
New York: Springer-Verlag
.
37.
Rosenbaum, P.
, and
D. Rubin
. 1983.
The central role of the propensity score in observational studies for causal effects
. Biometrika70:
41-55
.
38.
Rosenbaum, P.
, and
D. Rubin
. 1985.
The bias due to incomplete matching
. Biometrics41 (1):
103-116
.
39.
Rubin, D. B.
, and
N. Thomas
. 1996.
Matching using estimated propensity scores: Relating theory to practice
. Biometrics52 (1):
249-264
.
40.
Smith, H. L.1997.
Matching with multiple controls to estimate treatment effects in observational studies
. Sociological Methodology27:
325-353
.
41.
Wilde, E. T.
, and
R. Hollister
. 2002. How close is close enough? Testing nonexperimental estimates of impact against experimental estimates of impact with education test scores as outcomes. IRP Discussion Paper no. 1242-02,
Institute for Research on Poverty, University of Wisconsin–Madison
.
42.
Winship, C.
, and
S. L. Morgan
. 1999.
The estimation of causal effects from observational data
. Annual Review of Sociology25:
659-706
.
