Outcomes and outputs: Long-term effects of Temporary Assistance for Needy Families on caseload enrollment and poverty

Abstract

The Welfare Reform Legislation of 1996 is often cited as one of President Clinton’s most notable achievements, as this law was followed by sizable reductions in states’ welfare loads. Did this policy devolution lead to lower state poverty—as was suggested by reform advocates? We re-examine the effects of the new welfare regime on state-level poverty and welfare enrollment between 1996 and 2012. This is important to complement existing studies of individual-level experience with the welfare system. Our analysis confirms that the federal-to-state welfare transition eased the states’ caseload burden and poverty rate. We also find evidence that the relationship between welfare restrictiveness and caseload burden was strongest in the period before the recession, and with the inclusion of post-recession years, higher level restrictiveness may have little to no effect on reducing caseload. While state decisions to increase welfare restrictiveness did reduce poverty, our results show no added benefit to those with the highest levels of welfare restrictions. These findings reinforce the need to match policy goals to social outcomes, rather than relying on output measures such as caseload reduction.

Keywords

Devolution intergovernmental relations service delivery state poverty Temporary Assistance for Needy Families (TANF)welfare reform

When President Bill Clinton fulfilled his promise to “end welfare as we know it,” many critics feared the legislation would be a disaster for the poor (Blank, 1997). The new law, Personal Responsibility and Work Opportunity Reconciliation Act, ended the old system of direct cash assistance to the poor, Aid to Families with Dependent Children (AFDC), and replaced it with Temporary Assistance for Needy Families (TANF). No longer could welfare recipients benefit from aid in perpetuity. The federal government placed a five-year lifetime restriction on public assistance, in addition to a two-year limit before the beneficiary must obtain employment. Furthermore, the new legislation devolved substantial design and implementation authority to the states. Thus, many states chose to place even stricter time limits on assistance. Policy centers warned that the uptick in restrictive welfare legislation might condemn up to a million new children to poverty (Zedlewski et al., 1996). Many perceived the new legislation as another step in the campaign to blame the impoverished for their poverty, particularly single, low-income mothers (Handler and Hasenfeld, 2007; Lichter and Crowley, 2002; Soss and Schram, 2007).

The greatest fears of the critics did not occur in the years immediately following implementation. The number of people on welfare rolls was in decline in the three years prior to the legislation and continued to fall at an even greater pace following the transition to TANF. Between 1993 and 1997 state caseloads declined on average by 30% across the country (Schram et al., 1998). By 2000, caseload reductions had reached nearly 50% of their pre-TANF levels (Bell, 2001). Nor did the new welfare system seem to cause great increases in poverty, as opponents of TANF had predicted. The poverty rate for female-led households with children reached an all-time low of 32.5% in 2000 (Lichter and Crowley, 2002). Despite these findings, it is unclear whether the declines in poverty and caseloads can be attributed to TANF and state-level decisions or to propitious timing. TANF went into effect prior to an unprecedented economic boom in the US, and many states’ caseloads were already on a downward trajectory prior to the state takeover of welfare. Thus, it is difficult to determine whether the TANF transition was a true cause of declining poverty in the states. Researchers are now in a position to investigate the long-term effects of TANF, revealing important implications for public policy.

We examine how state-level variations in welfare restrictiveness shape a state’s welfare caseload and poverty rate. To reform proponents, TANF’s potential to reduce welfare caseloads is a victory for limited national government and local control, as individual states are free to implement program reform with varying degrees of restrictiveness (Blank, 2002). Opponents doubt the conventional wisdom of welfare reform’s success and fear that decentralization of welfare begets a “race to the bottom” in which states compete to offer minimal benefits (Brueckner, 2000; Peterson and Rom, 1990).

The primary goal of this study is to determine whether welfare reform impacted state poverty rates. We also extend the time period and scope of previous works, which have primarily focused on the impact of welfare reform on outputs (e.g., caseload burden) rather than outcomes (e.g., poverty rate) with some notable exceptions (Rodgers et al., 2006). In order to truly understand the influence of state-level decisions under TANF, research cannot focus on outputs alone. Output performance measures are indicative of the work produced by a policy, such as the number of individuals receiving welfare. Outcome performance measures, however, link the performance of a policy to the goal, mission, or benefit of that policy to society. In the instance of welfare, a key policy outcome is the ability of the policy reform to improve the state of the impoverished and reduce the overall level of poverty in the states.

We focus on performance measures that are related to both policy outputs and policy outcomes. Our analysis takes advantage of the sizable variation in welfare restrictiveness between states over time. By extending the time period of previous studies, we are able to show how welfare reform impacts caseload and poverty during both positive and negative fluctuations in the economy, including both the economic boom of the early 2000s and the crash of 2008.¹ Further, we employ propensity score weighting to account for previously un-modeled selection effects that lead states to adopt particular types of welfare reform over others (Guo and Fraser, 2014). We find moderate levels of welfare restrictiveness succeed in reducing caseload and all levels of restrictiveness tended to reduce poverty, all else being equal.

Increased TANF restrictiveness exerts an intuitive direct effect in the reduction of caseloads. Our findings show that enrollment numbers decrease as restrictiveness goes up, indicating that higher restrictiveness produces fewer caseloads. However, when the time period includes the recession, higher levels of restriction no longer seem to reduce caseloads. Unlike caseloads, restrictiveness reduces poverty, but the relationship is not monotonic. The impact on poverty is relatively the same in magnitude and significance across restrictiveness categories, indicating that while restrictiveness does influence state poverty, it does not do so in a strictly linear manner.

The shift to TANF

TANF is a federal-state government program that gives support to families with dependent children who are in need of financial assistance. States are charged with allocating money from a federally issued block grant to provide housing, clothing, and utilities to qualifying families. Additionally, TANF funds programs such as job training, child care, and temporary financial assistance. It is a form of social welfare with several outlined objectives; the most prominent of which is the reduction of poverty in the population (Office of Family Assistance, 2015). The other stated goals of the 1996 legislation are: (1) provide assistance to needy families so that children may be cared for in their own homes or in the homes of relatives; (2) end the dependence of needy parents on government benefits by promoting job preparation, work, and marriage; (3) prevent and reduce the incidence of out-of-wedlock pregnancies and establish annual numerical goals for preventing and reducing the incidence of these pregnancies; and (4) encourage the formation and maintenance of two parent families. However, these goals are considered broad in nature, and allow states to concentrate the block grant where they see fit, thus producing substantial variation between states in the types of policies adopted.

While TANF is noted for its aim to assist families in particular, the second goal is shared with almost all welfare programs: to reduce the likelihood of needing governmental assistance and thus reduce poverty in the United States. Auxiliary research has found that several resulting benefits of welfare reform, such as the reduction of caseloads, may also be considered a tacit goal of the program (Cho et al., 2005).

TANF provides a unique opportunity to study policy variation within the context of the United States. In 1996, control over welfare monies to support families with dependent children shifted from an entitlement program controlled by Washington to a federal block grant where states have considerable flexibility to implement and distribute benefits. The shift in oversight led to sizable variation across states over time in welfare eligibility rules, as states were able to use their local knowledge to administer the program in theory, with greater efficiency and effectiveness (Riccucci, 2005). While the transition to TANF enabled states to make their own judgment calls on program eligibility, the underlying objective of reducing poverty remained the same (Fellowes and Rowe, 2004).

The federal-to-state power shift provided not only the ability for states to differentiate in the implementation of the program but also opportunity to assess the relative success of varying welfare policy choices across the states. While there is an abundant literature analyzing the efficacy of welfare policy in the states, findings from these studies are frequently in conflict. For instance, earlier work on AFDC assessed the ideological assumptions of two opposing camps. One largely asserted that AFDC, along with other benefits such as Food Stamps (SNAP), discouraged citizens from working and undermined the goal of alleviating poverty (Murray, 1984; Niskanen, 1996). Others downplayed the existence of work disincentives and argued that AFDC was an essential program for easing the burdens of those temporarily caught in hard economic times (Marmor et al., 1990). Still others argued both perspectives are true to some extent but the key question is which outweighs the other; has the transition to TANF engendered a net loss or gain in poverty (Blank, 1997; Scholz and Levine, 2001)?

Studies on the impact of AFDC on poverty have been mixed. Earlier work found expanding welfare eligibility increased poverty (Murray, 1984; Niskanen, 1996; Peterson and Rom, 1989), which motivated a number of studies in rebuttal, asserting that these early models failed to control for key demographic factors and larger economic trends (Jencks, 1992; Schram, 1991). Other scholars found welfare spending to have no discernible effect on poverty (Blank and Blinder, 1986; Haveman and Schwabish, 2000). Fording and Berry (2007) explicitly modeled the work disincentive and found evidence for both sides of the story, concluding the net loss or benefit in terms of poverty depends on other factors such as wage level and the manner of cash assistance delivery.

Research in the wake of TANF has been similarly mixed. Studies show TANF cases (i.e., number of families subscribing to welfare assistance) have been falling since the state takeover of welfare (Bell, 2001; Cancian et al., 2002; Fremsted et al., 2004). However, this caseload reduction is not necessarily due to a reduction in state poverty. Ewalt and Jennings (2004) demonstrate that a state’s caseload burden is not always indicative of local poverty levels – only of policy choice and design within the state. There is no clear picture of how, or whether, the transition to TANF impacted state poverty (Rodgers et al., 2006). While some studies show TANF has helped to reduce poverty along with caseload burden (Ellwood, 2000), others demonstrate null effects (Cancian et al., 2002). A number of studies at the individual level suggest welfare restrictiveness under TANF adversely affected the population it intended to help. Pavetti et al. (2004) find those who are forced off TANF due to sanctions are more likely to reapply for benefits again later. Wu et al. (2014) argue those who leave the TANF rolls due to sanctions are less likely to find employment. In terms of income, Lee et al. (2004) argue leaving TANF to find work leads to less lucrative employment than leaving for other reasons. Finally, Wu (2008) finds harsher TANF sanctions are causes of increasing unemployment and poorer wages for those fortunate enough to find work.

It is important to keep in mind that these studies of individuals with personal TANF experience do not describe a state’s overall experience in terms of caseload and poverty. This would be a case of the individualistic fallacy. Part of how proponents hope welfare restrictiveness works is that it reduces the incentive for people to go on welfare to start. While it may be true that those individuals who are removed from welfare due to sanctions fare worse in terms of wages and employment opportunities, other citizens may be more inclined to find alternatives to welfare before applying. It also may be possible that the policy shift to TANF has led to other less measurable changes, such as the growth of hidden job markets or undocumented work (Kalil et al., 2002). Nevertheless, the view from the state level of analysis may be quite different. Overall, there are few studies of TANF’s effect on poverty at the state level, and there is conflicting evidence as to how the state takeover of welfare impacts both policy outputs (e.g., reduction in state caseload burden) and desired policy outcomes (e.g., reduction in state poverty). Moving forward, it is important to disentangle the impact of TANF from broader economic trends and policy selection effects within the states.

Policy choices under TANF

While minimum standards for welfare eligibility are set by the federal government, states can choose to include additional or stricter requirements. These restrictions generally come in three forms: work requirements for welfare recipients, time limitations on benefit duration, and punitive sanctions for non-compliance (Avery and Peffley, 2005; Soss et al., 2011). The federal government requires TANF recipients to show proof of work within 24 months of receiving assistance. Many states choose to require employment prior to the federal 24 month deadline. While the federal government has a lifetime eligibility cutoff maximum of 60 months, states may choose to shorten that horizon. States also have the freedom to decide if they put family caps in place, which deny increasing benefits to families as women give birth to new children. Finally, states may select punitive arrangements for recipients who commit some infraction. The stringency of the sanctions varies from state to state, some representing weaker sanctions (restricting partial funds for an adult recipient) and others stronger sanctions (revoking all funds immediately from the entire family unit).

Welfare restrictiveness in the states is often measured using an index created by Avery and Peffley (2005). There are three components to the restrictiveness index: whether the state requires a stricter work requirement than the federal standard (1 if yes, 0 if no), whether the state places shorter time limits on receiving benefits than does the federal standard (1 if yes, 0 if no), and whether the state has weak (0), moderate (1), or strong (2) punitive sanctions in place for non-compliance.² We add another dimension that accounts for the variation in family caps. These four dimensions are added to form an index that ranges from 0, indicating a state has adopted the minimal federal standards for welfare, to 5, which suggests that the state has adopted highly restrictive welfare rules. Since only 30 state-years (out of 850 possible) have the maximum value of 5, we collapse the two most restrictive categories into one. An index value of 1 denotes less restrictive welfare eligibility rules, whereas values of 2 and 3 indicate somewhat restrictive and more restrictive rules, respectively. A value of 4 indicates that a state’s welfare eligibility rules are among the most restrictive in the nation.

Figure 1 offers an overview of state choices regarding restrictiveness between 1996 and 2012.³ Most states have opted for moderate to more restrictive welfare eligibility rules. Over time, states appear to have adopted increasingly restrictive rules governing welfare. It is exceedingly rare for states to adopt rules that liberalize welfare eligibility.⁴ The greatest upswing in restrictiveness came during the economic boom of 1999–2000. During this year, seven states adopted the most restrictive welfare eligibility rules.⁵
Figure 1.
Welfare restrictiveness in the states.

Certain states will gravitate towards more restrictive welfare rules than others. In order to assess the influence of TANF, researchers must first understand the factors associated with the variation in policy choice across states. State preferences for welfare restrictiveness are traditionally attributed to key environmental, economic, and political factors. Two major theories frame the impact of these key factors on states’ program choices: policy typologies (Lowi, 1972; Peterson, 1981) and the sorting hypothesis (Tiebout, 1956). State and local expenditures can be divided into three policy categorizations that help explain what policy choices or even what programs a government will seek to offer: developmental, redistributive, and allocational (Peterson, 1981).

TANF is redistributive in nature. Redistributive policies include all programs and expenditures which are intended to enhance the lives of the poor or produce a benefit to tax ratio that is less than one for a majority of the community. Redistributive policies are thought to be best handled by the federal government because of the benefit to tax ratio it imposes on the citizenry (Peterson, 1995). As state-level variation in welfare services enables a purported race to the bottom, state and local governments will curtail the generosity of their welfare programs relative to neighboring states to encourage welfare recipients to move elsewhere (Brueckner, 2000; Rom, 1999). These concerns highlight the longstanding question of whether the states will actually work to reduce poverty or merely seek to ease their caseload burden.

Theory and hypotheses

Does state-level variation in welfare restrictiveness shape a state’s welfare caseload? More importantly, does restrictiveness impact the poverty rate within a state as well? We examine whether increased restrictiveness in welfare rules is capable of reducing statewide poverty (outcome) or whether it is simply producing fewer caseloads within a state (output).

We offer three hypotheses that address concerns of policy effectiveness. The first hypothesis pertains to a critical goal of the state takeover of welfare: reducing families’ dependence upon welfare services. Advocates of the new welfare regime argue that the states are equipped with greater local knowledge than the national government and will use that knowledge to successfully transition residents off welfare (Kim and Fording, 2008). Critics of the takeover assert that state and local governments are ill suited to handle redistributive expenditures, as state and local governments will compete with one another for a strong tax base. Thus, states are incentivized to curtail welfare benefits in a “race to the bottom” (Peterson, 1981). In fact, many proponents of TANF saw the reduction in caseloads as the first step in the fight against poverty. If states are able to reduce caseload while simultaneously avoiding increases in state-level poverty, many voters would consider the policy devolution a significant success (Berg, 2011). While there are competing theories of state motivations, there are comparable expectations of how states’ welfare restrictiveness will impact its caseload burden. States that opt for greater welfare restrictiveness should see a significant reduction in their welfare caseload.
Hypothesis 1: As welfare restrictiveness within a state increases, the state’s welfare caseload will decrease.
It is unclear from previous work, however, whether welfare restrictiveness influences a state’s poverty rate. Some studies have found that restricting access to welfare services encourages potential applications to obtain employment—reducing a state’s overall poverty rate. In contrast, if states are focusing on the output performance measure of caseload reduction, without a mind for reducing their residents’ need for welfare, then increased restrictiveness may actually work to increase a state’s poverty rate. These contradicting expectations lead to two distinct hypotheses:
Hypothesis 2: As welfare restrictiveness within a state increases, the state’s poverty rate will decrease.

Hypothesis 3: As welfare restrictiveness within a state increases, the state’s poverty rate will increase.
Support for Hypothesis 2 will bolster the argument that policy choices should achieve one of the key objectives of the program (e.g., reducing families’ need for welfare services). However, if states are racing to the bottom (e.g., working simply to reduce the number of welfare beneficiaries), we should observe a positive relationship between increased program restrictiveness and state poverty. In sum, the direction of the relationship between welfare restrictiveness and poverty is highly indicative of states’ motivations behind their policy choice.

Data and methods

We examine the influence of welfare restrictiveness on both a state’s caseload burden and its poverty rate from 1996 to 2012, a total of 850 state-years.⁶ The key explanation variable is the restrictiveness of a state’s welfare program.

Measuring the dependent variables: Caseload burden and poverty

The first dependent variable, a state’s caseload burden, is the natural log of the total number of welfare caseloads within a state divided by the state’s population in a given year.⁷ These data are publicly available from the Office of Family Assistance via the U.S Department of Health and Human Services. The second dependent variable is a policy outcome, state poverty. Poverty is measured by the percentage of a state’s population with a household income that falls below the federal poverty standard as measured by the U.S. Census Bureau.⁸ While scholars have criticized the poverty measure for not capturing other social welfare inputs such as food stamps (Duncan and Rogers, 1991) and for failing to capture poverty alleviation beneath the official poverty rate (Fording and Berry, 2007), this measure is able to serve as a stable benchmark to study the impact of welfare reform on a performance-based outcome measure.⁹ Additionally, we replicate our analysis using an alternate measure for state-level poverty—an anchored supplemental poverty measure available at the state level (Wilmer et al., 2016). This measure utilizes five year moving averages for out-of-pocket expenses on items such as food, clothing, shelter, and utilities. It also is adjusted for inflation using the consumer price index research series (Wilmer et al., 2016). This approach allows us to compare two different measurement strategies for poverty (a pre-tax/pre-transfer measure of resources and a post-tax/post-transfer measure of resources). The findings are robust to our original specification, in both sign and significance, and can be found in supplementary Appendix B along with a description of measurement and a detailed discussion of comparative results.

Measuring the independent variable: Welfare restrictiveness

As noted above, the key independent variable is a modified Avery and Peffley Index of Welfare Restrictiveness (2005). There are three key dimensions to the Welfare Restrictiveness Index, whether: (1) work requirements are more restrictive than the federal standard, (2) time limits on the receipt of benefits are more restrictive than the federal standard, (3) a family cap is in place, and (4) the extent beneficiaries are penalized for non-compliance with the state’s eligibility rules. The index ranges from 0 to 4, with higher values denoting greater welfare restrictiveness. Between 1996 and 2012, the mean index value is 2.37 with a standard deviation of 1.18.

Control variables

In addition to welfare restrictiveness, we control for other factors that may influence welfare restrictiveness or welfare caseload and poverty within a state. These variables include controls for economic conditions, political factors, and population demographics.¹⁰ There are three economic control variables:
Unemployment (+): the percentage of a state’s workforce that is currently unemployed as reported by the U.S. Census Bureau. As unemployment within a state increases, the state’s caseload burden and poverty rate should both increase (Handler and Hasenfeld, 2007).Unemployment levels may also influence the level of benefit offered to welfare recipients (Fording and Berry, 2007).¹¹

Annual Inflation (+): the gross domestic product implicit price deflator reported annually from the BEA. The GDP price deflator is the ratio of the current-dollar value of GDP to its corresponding chained-dollar value; this ratio is subsequently multiplied by 100. As inflation increases, states may see an uptick in welfare caseload and poverty (Berry et al., 2003; Volden, 2002). Inflation may also put pressure on states to adjust their level of restrictiveness.

Federal Medical Assistance Percentages (FMAP) (−): indicates the percentage of a state’s Medicaid expenditures that will be matched by the federal government in a given year as calculated by the DHHS. As FMAP increases, indicating a rise in federal-to-state aid, state poverty and welfare caseloads may decrease (Ziliak et al., 2000), though they may also increase as greater aid is a response to greater need.

We also include four control variables that pertain to the demographics of state’s population. Each is measured by data obtained from the U.S. Census Bureau:
Education (−): the percentage of state residents with a high school degree or higher. States with educated residents should typically have lower welfare caseloads and poverty than states with a less educated populace (Coelli et al., 2007). Education may also be correlated with restrictiveness as a more educated state may also have a more efficient, well-trained bureaucracy focused on the most disadvantaged citizens.

Urbanization (+): the percentage of state residents residing in a metropolitan area. Urbanized states typically have higher welfare caseloads and poverty than rural states (Mingione, 2008).

Percent Minority (+): the percentage of a state’s population that is nonwhite. States with a diverse population tend to have higher welfare caseloads and poverty than relatively homogenous states (Filindra, 2013); governments may be inclined to restrict benefits more in states with majority white conservative political power that perceive beneficiaries as coming from largely undeserving minority populations.

Out-of-Wedlock Births (+): the percentage of total births that are out-of-wedlock in each state-year as estimated by the National Center for Health Statistics. As out-of-wedlock births within a state increase, it is likely that this will result in an increase in welfare benefits and that the state’s caseload burden and poverty rate will increase as well (Fording and Berry, 2007).

We also control for two key political factors:
Government Ideology (+): government ideology index ranging from 0 to 100, with higher values denoting greater liberalism among a state’s legislative parties. It is alternatively labeled the NOMINATE measure of state-level ideology and uses “Common Space” ideology scores available at: http://voteview.com/basic/htm (Berry et al., 2010).

Citizen Ideology (+): citizen ideology index ranging from 0 to 100, with higher values denoting greater liberalism among a state’s population (Berry et al., 1998). States with a more liberal population should be supportive of welfare services, which may lead to an expanding welfare caseload; states that are liberal also tend to be coastal states with lower overall poverty.

Finally, we control for another aspect of TANF reform:
Maximum Eligibility (+): This is a state-year measure of the maximum income a family can declare and still be eligible for receiving TANF assistance.

Model

We estimate the effect of welfare restrictiveness on a state’s Caseload Burden and Poverty rate using a fixed effects regression, which allows us to control for state-level heterogeneity (Kristensen and Wawro, 2003).¹² We also include a one-year lagged dependent variable in order to account for auto-correlation typical of time-series estimations (Kennedy, 2008).¹³ Additionally, scholarship has demonstrated the influence that prior levels of poverty have on the ability of a state to reduce poverty, noting that states with higher initial poverty levels are able to reduce poverty at greater rates (Rodgers et al., 2006).

One obstacle to causal inference is that some states may be more inclined to choose more restrictive policies than others, introducing potential endogeneity. We therefore estimate propensity score weighted regressions. There are two steps to performing this analysis. First, we create a dependent variable indicating the highest level of restrictiveness for each state at any time over all years, and using data from 1995, we estimate the predicted values of the maximum welfare restrictiveness for each state through a multinomial logistic regression.¹⁴ These are the predicted probabilities that each state will choose the most restrictive category based on its pre-TANF characteristics. The predicted values for the level of restrictiveness are then inverted to obtain the probability weights for each state in any given year (Guo and Fraser, 2014). This allows us to account for differences in each state’s propensity to adopt more restrictive welfare policy by identifying states more likely to choose restrictive welfare policy at the outset. Mechanically, propensity score weights operate as sampling weights (Guo and Fraser, 2014).

Results

Findings for Hypothesis 1 can be found in Table 1, which shows the regression results for the dependent variable Caseload Reduction. In the weighted fixed effects regression, we see that the Index of Welfare Restrictiveness is both negative and significant for all levels of restrictiveness. We characterize these levels of restrictiveness as moderate, more, and most, relative to the base category of zero. Additionally, the magnitudes appear to grow in size for the fixed effects regression, suggesting that more restrictive policies monotonically decrease the number of TANF recipients within a state. However, what is most worth noting is the fact that significance for the caseload reduction is lost for the two most restrictive categories when the time series is expanded to include the recession years of 2009–2012 in the United States. It would seem that the state-level choices regarding restrictiveness only significantly reduced caseloads in more economically prosperous times. Only the two less restrictive categories continue to be significant and negative in the face of economic recession, potentially indicating that those states that were less restrictive in nature were better suited to reduce cases throughout the entire time series.
Table 1.
State Burden Models (1996–2012)

DV: Caseload Burden Fixed effects 1996–2008 Fixed effects 1996–2012 Weighted fixed effects

Welfare Restrictiveness

Somewhat Restrictive −0.561^* −0.534^ −0.362^

(0.071) (0.269) (0.212)

Moderately Restrictive −0.634^* −0.760^ −0.714^*

(0.070) (0.422) (0.237)

More Restrictive −0.826^* −0.343 −0.526^

(0.089) (0.456) (0.246)

Most Restrictive −0.950^*** −0.539 −0.462^*

(0.100) (0.570) (0.290)

Burden _t _{− 1} 0.583^* 0.866^* 0.872^***

(0.052) (0.030) (0.017)

Maximum Eligibility −0.000 0.002 0.001

(0.000) (0.002) (0.001)

Gov’t Ideology 0.001 0.003 0.006^*

(0.002) (0.013) (0.005)

Unemployment −0.122^* −0.046 −0.086

(0.040) (0.122) (0.069)

Out-of-Wedlock Births 0.076^* 0.170^* 0.155^*

(0.020) (0.106) (0.050)

Citizen Ideology 0.014^* 0.062^ 0.033^

(0.005) (0.036) (0.016)

Education 0.003 0.054^* 0.020^*

(0.006) (0.037) (0.015)

Urbanization 0.078^ −0.347 −0.122

(0.035) (0.357) (0.134)

FMAP −0.021^ 0.103^* 0.073^**

(0.012) (0.069) (0.033)

Percent Minority −0.007 −0.242 −0.090^*

(0.020) (0.249) (0.064)

Per Capita Income 0.005 −0.014 0.005

(0.005) (0.021) (0.018)

Population Change −0.207^* 0.089 −0.024

(0.056) (0.131) (0.069)

Annual Inflation −0.201^* −0.589^ −0.438^*

(0.043) (0.271) (0.169)

Year 0.002 0.021 −0.017

(0.033) (0.084) (0.060)

Constant −9.068 −28.574 31.846

(63.111) (153.138) (115.119)

N 650 850 850

R ² 0.583 0.718 0.710

Standard errors in parentheses. FMAP: Federal Medical Assistance Percentages.

*
p < 0.10; p < 0.05; p < 0.01.

The results confirm the predictions of Hypothesis 1: welfare restrictiveness, perhaps unsurprisingly, works to decreases a state’s caseload on average. The signs and magnitude of the coefficients suggest that each level of restrictiveness results in additional caseload reduction. In the period between the initial implementation of TANF and the onset of the recession, this effect was monotonically increasing, as each level of greater restrictiveness indicates a larger caseload reduction. In the full model incorporating four years of the recession, the two most restrictive categories indicate a null effect on caseload burden. Incorporating the propensity score weights, all levels of restrictiveness have a negative and statistically significant effect on caseloads, with moderate restrictiveness having the greatest substantive effect. As predicted by Hypothesis 1, and reinforced by the literature, policy choices by local officials to increase the restrictiveness of state rules and regulations serve to reduce the welfare caseload in the state, a finding which seems to hold when taking into account both pre- and post-recession years. As such, many states would consider the policy devolution a success, in that it limited the scope of the program and reduced the overall number of recipients.

In analyzing the results of the control variables, there are some unexpected findings. Surprisingly, Education* is significant in the full models but has the opposite sign of expectation. As the percentage of citizens with a high school diploma increases in a state so too does the welfare caseload burden ceteris paribus. Given the complexity of the new welfare rules, and the cumbersome and complicated nature of administrative organizations (Brodkin and Majumundar, 2010), it may be that potential beneficiaries are discouraged or unable to navigate the bureaucracy necessary to receive benefits. It may also be the case, however, that states with a more educated workforce are able to better target needy citizens for assistance. The unanticipated influence of education may also be tied to the fact that educated populations tend to be more supportive of social welfare, which in turn may increase the burdens in those states. Conforming to expectation, Out-of-wedlock births is positive and significant in all three models, but Unemployment is only statistically significant in the pre-recession model and has a negative effect on Caseload Reduction. Given model specification this may be unsurprising, as lagging the dependent variable and including a weighting measure that has already taken into account unemployment, likely absorbs much of the explanatory power of Unemployment on caseloads.

The results for the estimation of Poverty, using both the fixed effects estimation with propensity scores as well as the weighted fixed effects regressions with lagged dependent variables are reported in Table 2. The results suggest a consistent effect of welfare restrictiveness on shaping poverty within the states.
Table 2.
State Poverty Models (1996–2012)

DV: State Poverty Rates Fixed effects 1996–2008 Fixed effects 1996–2012 Weighted fixed effects

Welfare Restrictiveness

Somewhat Restrictive −0.222^ −0.488^* −0.597^*

(0.131) (0.133) (0.191)

Moderately Restrictive −0.227^ −0.565^* −0.687^*

(0.118) (0.141) (0.253)

More Restrictive −0.180^* −0.472^* −0.918^

(0.133) (0.173) (0.390)

Most Restrictive −0.182^* −0.516^* −0.850^

(0.138) (0.207) (0.433)

Poverty _t _{− 1} 0.560^* 0.425^* 0.383^*

(0.032) (0.037) (0.067)

Maximum Eligibility −0.000^ −0.000^* −0.000

(0.000) (0.000) (0.000)

Gov’t Ideology 0.001 −0.004^* −0.007^*

(0.002) (0.002) (0.002)

Unemployment 0.298^* 0.171^* 0.099^*

(0.021) (0.028) (0.075)

Out-of-Wedlock Births 0.024^* −0.013 0.006

(0.018) (0.021) (0.031)

Citizen Ideology 0.002 0.023^* 0.024^*

(0.004) (0.004) (0.007)

Education 0.008 −0.002 −0.006

(0.007) (0.006) (0.011)

Urbanization 0.072^* 0.069^ 0.099^

(0.046) (0.038) (0.051)

FMAP 0.005 0.030^ 0.064

(0.012) (0.018) (0.050)

Percent Minority −0.028 −0.030^* −0.029^

(0.024) (0.009) (0.013)

Per Capita Income −0.003 −0.004 −0.006

(0.004) (0.006) (0.009)

Population Change −0.123^* −0.171^* −0.234^*

(0.027) (0.039) (0.056)

Annual Inflation 0.146^* 0.147^* 0.138^*

(0.031) (0.040) (0.087)

Year 0.015 0.046^* 0.059^

(0.026) (0.019) (0.030)

Constant −31.463 −91.102^* −118.637^

(49.220) (36.669) (60.926)

N 650 850 850

R ² 0.699 0.600 0.559

Standard errors in parentheses. FMAP: Federal Medical Assistance Percentages.

*
p < 0.10; p < 0.05; p < 0.01.

In all three models, all levels of restrictiveness are negative and significant, suggesting that restrictiveness introduced under TANF is reducing poverty. Interestingly, the size of the effect appears to be more modest in the pre-recession period. The propensity score weighted model indicates that each category of restrictiveness results in nearly a percentage point decrease in the state-level poverty rate. Findings such as these indicate that states appear to be capable of achieving a primary goal of TANF administration: reduction of poverty.

From these models, we find that the poverty story is more nuanced than that of caseloads, especially when considering the significance and magnitudes of the restrictiveness variables. Unlike the results for TANF cases in Table 1, where increases in restrictiveness appear to amplify the magnitude of case reduction, Table 2 shows that restrictiveness does not seem to have the same pattern of impact on poverty. While increased restrictiveness appears to reduce poverty at the state level, the rate of reduction does not appear to be consistently increasing. States with less restrictive policies seem to reduce poverty at rates as high and sometimes higher than states with the Most Restrictive welfare laws, compared to the base category. This is particularly true in the pre-recession period. While most states reduce poverty by increasing welfare restrictiveness, the Somewhat Restrictive and Moderately Restrictive states appear to be just as effective at reducing poverty than the More Restrictive and* the Most Restrictive states in the across all models. The results suggest that increasing restrictiveness does not decrease poverty incrementally.

For policy makers, these findings indicate that increasing the restrictiveness of rules and regulations placed on recipients does aid in the reduction of poverty, but continuing to increase restrictiveness does not increase the degree to which poverty is reduced. The results indicate that, while increases in restrictiveness may lower both poverty and caseloads, it does not produce similar patterns in reduction. Care should be taken in assessing the policy needs of the state, specifically how much restrictiveness is optimal.

In all of these models, the control variables are consistent with expectations. At the state level, Previous poverty levels and Unemployment are the most consistent predictors of Poverty, and the coefficients are as expected, positive and highly significant. Further, Citizen Ideology, and shifts in Population appear to be strongly correlated with Poverty, all else being equal. Other variables are not as consistent in their significance levels across models, and surprisingly Education fails to reach significance across the models.

Conclusion

Understanding the consequences of policy choices is important, particularly for welfare. Millions of dollars are spent each year in an attempt to meet the goals of this program and to assist US states in reducing the level of poverty. Many lessons can be learned from the substantial variation in implementation that occurred within the US. It is the responsibility of policymakers to ensure the program’s goals and objectives are being achieved. One of the best ways policymakers can ensure program funds are used both effectively and efficiently is through the use of performance measures. However, care must be taken in the selection of evaluation measures. As explained above, a reliance on output performance measures, such as welfare caseload reduction, in lieu of outcome performance measurements, such as change in a state’s poverty rate, may offer an incomplete assessment of a program. Furthermore, policy evaluations that only address individual-level outcomes of those already in the welfare system ignore aggregate-level effects that are also intended by policymakers.

This study contributes to the practice of performance measurement by analyzing the effectiveness of the state takeover of welfare over a sizable time period that includes both an economic boom and a great recession. We address the influence of states’ policy choice restrictiveness under TANF on both caseload burden and poverty. Our findings provide evidence that state-level decisions on welfare restrictiveness influence both the number of participants enrolled in the program (state welfare caseload) and the overall influence of the program on poverty, arguably a key social metric for welfare.

Increased restrictiveness is associated with a decrease in both a state’s caseload and poverty rate, and we therefore conclude that these state policy choices do appear to work toward the fulfillment of the programs outlined goal. These findings demonstrate the ability of states to handle redistributive expenditures; in this case showing state choices to increase restrictiveness of TANF procedures decreased the number of enrollees in a program and the overall poverty levels.

However, the relationship is nuanced; increases in restrictiveness do not uniformly impact either poverty or caseload reduction. This is especially true for the longer time period we estimate. Our findings suggest that increases in restrictiveness contributed to the reduction of state-level poverty across the board between 1996 and 2012, but additional degrees of restrictiveness do not appear to yield substantially higher reductions in poverty. Regarding caseloads, our results demonstrate a decrease in number of enrollees as restrictiveness increased, but the most restrictive categories failed to consistently reach significance when the recession was included in the temporal domain.

Further, this study demonstrates the necessity to measure program performance using varied approaches and to pair appropriate measures with intended policy outcomes. The substantial variation within the context of the United States as well as a temporal domain that includes both economic prosperity and recession, allows for a detailed and broad discussion about the influence public policy choice has on poverty.

DV: Caseload Burden	Fixed effects 1996–2008	Fixed effects 1996–2012	Weighted fixed effects
Welfare Restrictiveness
Somewhat Restrictive	−0.561^***	−0.534^**	−0.362^**
	(0.071)	(0.269)	(0.212)
Moderately Restrictive	−0.634^***	−0.760^**	−0.714^***
	(0.070)	(0.422)	(0.237)
More Restrictive	−0.826^***	−0.343	−0.526^**
	(0.089)	(0.456)	(0.246)
Most Restrictive	−0.950^***	−0.539	−0.462^*
	(0.100)	(0.570)	(0.290)
Burden _t _{− 1}	0.583^***	0.866^***	0.872^***
	(0.052)	(0.030)	(0.017)
Maximum Eligibility	−0.000	0.002	0.001
	(0.000)	(0.002)	(0.001)
Gov’t Ideology	0.001	0.003	0.006^*
	(0.002)	(0.013)	(0.005)
Unemployment	−0.122^***	−0.046	−0.086
	(0.040)	(0.122)	(0.069)
Out-of-Wedlock Births	0.076^***	0.170^*	0.155^***
	(0.020)	(0.106)	(0.050)
Citizen Ideology	0.014^***	0.062^**	0.033^**
	(0.005)	(0.036)	(0.016)
Education	0.003	0.054^*	0.020^*
	(0.006)	(0.037)	(0.015)
Urbanization	0.078^**	−0.347	−0.122
	(0.035)	(0.357)	(0.134)
FMAP	−0.021^**	0.103^*	0.073^**
	(0.012)	(0.069)	(0.033)
Percent Minority	−0.007	−0.242	−0.090^*
	(0.020)	(0.249)	(0.064)
Per Capita Income	0.005	−0.014	0.005
	(0.005)	(0.021)	(0.018)
Population Change	−0.207^***	0.089	−0.024
	(0.056)	(0.131)	(0.069)
Annual Inflation	−0.201^***	−0.589^**	−0.438^***
	(0.043)	(0.271)	(0.169)
Year	0.002	0.021	−0.017
	(0.033)	(0.084)	(0.060)
Constant	−9.068	−28.574	31.846
	(63.111)	(153.138)	(115.119)
N	650	850	850
R ²	0.583	0.718	0.710

DV: State Poverty Rates	Fixed effects 1996–2008	Fixed effects 1996–2012	Weighted fixed effects
Welfare Restrictiveness
Somewhat Restrictive	−0.222^**	−0.488^***	−0.597^***
	(0.131)	(0.133)	(0.191)
Moderately Restrictive	−0.227^**	−0.565^***	−0.687^***
	(0.118)	(0.141)	(0.253)
More Restrictive	−0.180^*	−0.472^***	−0.918^**
	(0.133)	(0.173)	(0.390)
Most Restrictive	−0.182^*	−0.516^***	−0.850^**
	(0.138)	(0.207)	(0.433)
Poverty _t _{− 1}	0.560^***	0.425^***	0.383^***
	(0.032)	(0.037)	(0.067)
Maximum Eligibility	−0.000^**	−0.000^*	−0.000
	(0.000)	(0.000)	(0.000)
Gov’t Ideology	0.001	−0.004^***	−0.007^***
	(0.002)	(0.002)	(0.002)
Unemployment	0.298^***	0.171^***	0.099^*
	(0.021)	(0.028)	(0.075)
Out-of-Wedlock Births	0.024^*	−0.013	0.006
	(0.018)	(0.021)	(0.031)
Citizen Ideology	0.002	0.023^***	0.024^***
	(0.004)	(0.004)	(0.007)
Education	0.008	−0.002	−0.006
	(0.007)	(0.006)	(0.011)
Urbanization	0.072^*	0.069^**	0.099^**
	(0.046)	(0.038)	(0.051)
FMAP	0.005	0.030^**	0.064
	(0.012)	(0.018)	(0.050)
Percent Minority	−0.028	−0.030^***	−0.029^**
	(0.024)	(0.009)	(0.013)
Per Capita Income	−0.003	−0.004	−0.006
	(0.004)	(0.006)	(0.009)
Population Change	−0.123^***	−0.171^***	−0.234^***
	(0.027)	(0.039)	(0.056)
Annual Inflation	0.146^***	0.147^***	0.138^*
	(0.031)	(0.040)	(0.087)
Year	0.015	0.046^***	0.059^**
	(0.026)	(0.019)	(0.030)
Constant	−31.463	−91.102^***	−118.637^**
	(49.220)	(36.669)	(60.926)
N	650	850	850
R ²	0.699	0.600	0.559

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Avery

Peffley

(2005) Voter registration requirements, voter turnout, and welfare eligibility policy: Class bias matters. State Politics & Policy Quarterly 5(1): 47–67.

Bell

(2001) Assessing the New Federalism: An Urban Institute Program to Assess Changing Social Policies, Washington, DC: The Urban Institute.

Berg J (2011) Welfare reform’s forgotten goals. The Washington Post. Available at: www.washingtonpost.com/opinions/welfare-reforms-forgotten-goals/2011/08/15/gIQAmycWOJ_story.html?utm_term=.79405aa8c132 (accessed 22 December 2016).

Berry

Fording

Hanson

(2003) Reassessing the ‘Race to the Bottom’ in State Welfare Policy. Journal of Politics 65(2): 327–349.

Berry

Ringquist

Fording

et al. (1998) Measuring citizen and government ideology in the American States, 1960–93. American Journal of Political Science 42(1): 327–348.

Berry WD, Ringquist EJ, Fording RC, et al. (2010) Measuring citizen and government ideology in the US States: A reappraisal. State Politics and Policy Quarterly 10: 117–135.

Blank

(1997) It Takes a Nation: A New Agenda for Fighting Poverty, Princeton, NJ: Princeton University Press.

Blank

(2002) Evaluating welfare reform in the United States. Journal of Economic Literature 40(4): 1105–1166.

Blank

Blinder

(1986) Macroeconomics, income distribution, and poverty. In: Danziger

Weinberg

(eds) Fighting Poverty: What Works and What Doesn’t, Cambridge, MA: Harvard University Press, pp. 180–208.

10.

Brodkin

Mujmundar

(2010) Administrative exclusion: Organizations and the hidden costs of welfare claiming. Journal of Public Administration Research and Theory 20(4): 827–848.

11.

Brueckner

(2000) Welfare reform and the race to the bottom: Theory and evidence. Southern Economic Journal 66(3): 505–525.

12.

Cancian M, Hayeman R, Meyer DR, et al. (2002) Before and After TANF: The Economic Well-Being of Women Leaving Welfare. IRP Discussion Paper No. 1244-02. Madison, WI: University of Wisconsin Institute for Research on Poverty.

13.

Cellini

McKernan

S-M

Ratcliffe

(2008) The dynamics of poverty in the United States: A review of data, methods, and findings. Journal of Policy Analysis and Management 27(3): 577–605.

14.

Cho

C-L

Kelleher

Wright

et al. (2005) Translating National Policy objectives into local achievements across planes of governance and among multiple actors: Second-order devolution and welfare reform implementation. Journal of Public Administration Research and Theory 15(1): 31–54.

15.

Coelli

Green

Warburton

(2007) Breaking the cycle? The effect of education on welfare receipt among children of welfare recipients. Journal of Public Economics 91(7): 1369–1398.

16.

Duncan

Rogers

(1991) Has children’s poverty become more persistent? American Sociological Review 56(4): 538–550.

17.

Ellwood

(2000) Anti-poverty policy for families in the next century: From welfare to work and worries. Journal of Economic Perspectives 14(1): 187–198.

18.

Ewalt

Jennings

(2004) Administration, governance, and policy tools in welfare policy implementation. Public Administration Review 64(4): 449–462.

19.

Fellowes

Rowe

(2004) Politics and the New American Welfare States. American Journal of Political Science 48(2): 362–373.

20.

Filindra

(2013) Immigrant social policy in American States: Race politics and state TANF and Medicaid Eligibility Rules for Legal Permanent Residents. State Politics and Policy Quarterly 13(1): 26–48.

21.

Fording

Berry

(2007) The historical impact of welfare programs on poverty: Evidence from the American States. The Policy Studies Journal 35(1): 37–60.

22.

Fremsted

Parrott

Sherman

(2004) Unemployment Insurance Does Not Explain Why TANF Caseloads Are Falling as Poverty and Need Are Rising, Washington, DC: Center on Budget and Policy Priorities.

23.

Gujarati

(1995) Basic Econometrics, New York, NY: McGraw-Hill.

24.

Guo

Fraser

(2014) Propensity Score Analysis: Statistical Methods and Applications, Thousand Oaks, CA: Sage Publications.

25.

Handler

Hasenfeld

(2007) Blame Welfare, Ignore Poverty and Inequality, Cambridge, MA: Cambridge University Press.

26.

Haveman

Schwabish

(2000) Has macroeconomic performance regained its antipoverty bite? Contemporary Economic Policy 18(4): 415–427.

27.

Jencks

(1992) Rethinking Social Policy: Race, Poverty, and the Underclass, Cambridge, MA: Harvard University Press.

28.

Kalil

Seefeldt

Wang

H-C

(2002) Sanctions and material hardship under TANF. Social Services Review 76(4): 642–662.

29.

Kennedy

(2008) A Guide to Econometrics, 6th ed. Malden, MA: Wiley-Blackwell.

30.

Kim

Fording

(2008) Second-order devolution and the implementation of TANF. In: Presentation at the annual meeting of the midwest political science association, Chicago, IL, 2008.

31.

Krause GA and Melusky BF (2012) Concentrated powers: Unilateral executive authority and fiscal policymaking in the American states. The Journal of Politics 74(1): 98–112.

32.

Kristensen

Wawro

(2003) Lagging the dog? The robustness of panel corrected standard errors in the presence of serial correlation and observation specific effects. In: Presentation at the annual meeting of the society for political methodology, University of Minnesota, Minneapolis, MN, 2003.

33.

Lee

Slack

Lewis

(2004) Are welfare sanctions working as intended? Social Service Review 78(3): 370–403.

34.

Lichter

Crowley

(2002) Poverty in America: Beyond Welfare Reform. Population Bulletin 57(2): 3–36.

35.

Lowi

(1972) Four systems of policy, politics, and choice. Public Administration Review 32(4): 298–310.

36.

Marmor

Mashaw

Harvey

(1990) America’s misunderstood welfare state: Persistent myths, enduring realities, New York, NY: Basic Books.

37.

Mingione

(2008) Urban Poverty and the Underclass, Cambridge, MA: John Wiley & Sons.

38.

Murray

(1984) Losing Ground: American Social Policy, 1950–1980, New York, NY: Basic Books.

39.

Niskanen

(1996) Welfare and the culture of poverty. Cato Journal 16(1): 1–15.

40.

Office of Family Assistance (2015) U.S Department of Health and Human Services. Available at: www.acf.hhs.gov/programs/ofa/programs/tanf (accessed 3 November 2015).

41.

Pavetti LA, Derr MK, Kirby G, et al. (2004) The Use of TANF Work-Oriented Sanctions in Illinois, New Jersey, and South Carolina: Final Report. Report to the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (contract no. 100-01-0011). Mathematica Policy Research, Washington, DC.

42.

Peterson

(1981) City Limits, Chicago, IL: The University of Chicago Press.

43.

Peterson

(1995) The Price of Federalism, Washington, DC: The Brookings Institution.

44.

Peterson

Rom

(1989) American federalism, welfare policy and residential choices. American Political Science Review 83(3): 711–728.

45.

Peterson PE and Rom MC (1990) Welfare Magnets: A New Case for a National Standard. Washington, DC: The Brookings Institute.

46.

Riccucci

(2005) Street-level bureaucrats and intrastate variation in the implementation of Temporary Assistance for Needy Families policies. Journal of Public Administration Research and Theory 15(1): 89–111.

47.

Rodgers

Payne

Chervachidze

(1996) State poverty rates: Do the new welfare policies make a difference? Review of Policy Research 23(3): 657–679.

48.

Rom

(1999) Transforming State Health and Welfare Programs. In: Gray

Jacobs

(eds) Politics in the American States, Washington, DC: CQ Press, pp. 117–131.

49.

Scholz

Levine

(2001) The evolution of Income Support Policy in recent decades. In: Danziger

Haveman

(eds) Understanding Poverty, Cambridge, MA: Harvard University Press, pp. 193–228.

50.

Schram

(1991) Welfare spending and poverty: Cutting back produces more poverty, not less. American Journal of Economics and Sociology 50(2): 129–141.

51.

Schram

Nitz

Krueger

(1998) Without cause or effect: Reconsidering welfare migration as a policy problem. American Journal of Political Science 42(1): 210–230.

52.

Soss

Schram

(2007) A public transformed? Welfare Reform as policy feedback. American Political Science Review 101(1): 111–127.

53.

Soss

Fording

Schram

(2011) Disciplining the poor: Neoliberal paternalism and the persistent power of race, Chicago, IL: University of Chicago Press.

54.

Tiebout

(1956) A pure theory of local expenditures. Journal of Political Economy 64(5): 416–424.

55.

Volden

(2002) The politics of competitive federalism: A race to the bottom in welfare benefits? American Journal of Political Science 46(2): 352–363.

56.

Wilmer

Fox

Garfinkel

et al. (2016) Progress on poverty? New estimates of historical trends using an anchored supplemental poverty measure. Demography 53: 1–12.

57.

C-F

(2008) Severity, timing, and duration of welfare sanctions and the economic well-being of TANF families with children. Children and Youth Services Review 30: 26–44.

58.

C-F

Cancian

Wallace

(2014) The effect of welfare sanctions on TANF exits and employment. Children and Youth Services Review 36: 1–14.

59.

Zedlewski

Clark

Meier

et al. (1996) Potential Effects of Congressional Welfare 49 Reform Legislation on Family Incomes, Washington, DC: The Urban Institute.

60.

Ziliak

Figlio

Davis

et al. (2000) Accounting for the decline in AFDC caseloads. Journal of Human Resources 35(3): 570–586.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.23 MB