When Claimant Characteristics and Prior Performance Predict Bureaucratic Error

Abstract

The public administration literature has paid scant attention to bureaucratic errors as performance measures. This has largely been due to a lack of data. Unlike most programs, the unemployment insurance (UI) program has systematically collected performance data and has independently audited those data to determine error responsibility (employer, employee, and agency error). In the first comprehensive analysis of these data, we examine the probability that a bureaucrat makes an error involving nonpayment of UI benefits and theorize about the reasons for these errors. Our findings indicate that the previous UI office error rate is a good predictor of current error rates, demonstrating that poorly performing offices remain poor performers. In addition, local offices with high error rates account for a disproportionate percentage of the errors, indicating a need to examine agency management. Second, errors are more commonly made on cases involving White UI claimants and claimants with a college education. Finally, we find that claimants who have higher self-valuation, are less likely to experience agency errors. Taken together, these results point to systematic agency errors. Public managers and the unemployed would be better served if training efforts and performance targets were developed with these systematic error effects in mind.

Keywords

agency error wrongful denials unemployment insurance performance management

In 2010, during the worst sustained labor market downturn in nearly 80 years, the unemployment insurance systems of the states paid out more than US$140 billion in benefits to the eligible unemployed (U.S. Department of Labor, 2011).¹ This expenditure constituted 3.8% of the unified Federal budget, or US$435 per capita. These historically large expenditures are the consequence of high unemployment rates resulting in a record number of people applying for unemployment insurance benefits.

Meanwhile, bureaucrats implementing the program struggle to keep up with this avalanche of cases. For each person filing for unemployment insurance (UI) benefits, a bureaucrat assesses the validity of the applicant’s claim and determines whether the claimant is eligible to receive payments. A large percentage of claimants are denied eligibility, either because they did not work enough hours or earn enough income to qualify, or more likely, claimants quit or are fired for cause and were not eligible to receive benefits. However, sometimes these denials occur because the bureaucrat does not carefully adjudicate the claim.

Even under better labor market conditions, a surprising number of eligibility determinations are incorrectly made. In nearly 13% of denied cases the claimant was wrongfully denied benefits. Errors come in a number of guises: UI agents can fail to apply proper procedures, fail to collect adequate information, and fail to complete necessary forms. Surprisingly, looking across all of the parties who could be responsible for errors, fully 57% of the errors were made solely by the agency, as opposed to the claimant, employer, or some combination. Indeed, if we count errors where the agency was at least partly responsible or fully responsible, nearly three fourths (74%) of the errors were due to the agency.

In this article we investigate the factors related to these agency errors. We focus exclusively on errors that are solely the agency’s fault and result in the claimant being wrongfully denied. We analyze data collected by the U.S. Department of Labor for the Benefit Accuracy Measurement program (BAM). Our data consist of more than 69,000 person-level, audited UI claims from 2003 to 2009. Because we have state-level data with a unique UI office identifier, we are able to track the performance of individual UI offices, but not individual bureaucrats. We also have demographic and employment information about the claimant, including age, race, sex, education, industry, occupation, and wage history. Uniquely, we also have the lowest hourly wage that an unemployed person would find acceptable to return to work, also known as the reservation wage.

We find some disturbing relationships in the data that should help guide public managers target their limited resources and enable them to improve overall program performance. One of our major conclusions is that a minority of UI offices is responsible for a significant majority of the UI errors. This implies that a targeted effort in improving these offices’ performance would have a disproportionate impact on the system. Second, errors are more common for claimants who are White and those with a college education. Finally, we note that those who report that their lowest acceptable wage is greater than their previous wage are less likely to experience errors, leading us to hypothesize about the presentation of these cases and the role of self-confidence in the determination of eligibility. Ultimately, we conclude that when the case (or situation) appears normal bureaucrats in UI offices are less likely to spend the extra time needed to scrutinize a claim, leading to errors.

Predicting Agency Error

By determining program eligibility, bureaucrats have a profound impact on the lives of individuals seeking government services. While performing these critical tasks, bureaucrats face multiple and often conflicting objectives. There is a requirement to adjudicate claims in a timely fashion, make accurate decisions, and do all of this on a limited budget. In this environment, where the decisions are important and the demands are often conflicting, there is concern about the possibility of errors. The probability of an error increases as bureaucrats face cognitive limitations and rely on shortcuts to facilitate decision making (Lipsky, 1980).

A large body of public management research examines the predictors of agency performance. However, these studies almost exclusively analyze the determinants of successful performance by agencies.² Many of these studies of agency success have relied on the model posited by O’Toole and Meier (1999, 2000, 2003; Meier & O’Toole, 2001) to explore the relationship between the role of management and performance. This model predicts that organization performance is a function of the stability, management strategies, and environmental factors. Studies in Texas public schools (Fernandez, 2005; Goerdel, 2006; Hicklin, 2004; Hill, 2005; Johansen, 2007; Juenke, 2005; Meier & O’Toole, 2001, 2002, 2003; Pitts, 2005), British local authorities (Andrews, Boyne, Moon, & Walker, 2010; Walker, Andrews, Boyne, Meier, & O’Toole, 2010; Walker, O’Toole, & Meier, 2007), and law enforcement agencies (Nicholson-Crotty & O’Toole, 2004) have concluded that these factors (stability, management, and environment) have a significant impact on agency success. While this work provides insights into the success of public organizations, it is important to note that the adoption of the O’Toole and Meier model may not be sufficient in the context of agency error. However, O’Toole and Meier are correct to point out the critical role of management in the context of policy implementation and its influence on agency performance. Typical measures of performance in this literature are only partially determined by bureaucratic behavior. In this research, we examine errors made solely by the agency, allowing us to avoid attributing bureaucratic errors to other actors and factors. Consequently, this allows us to develop a framework where causal paths are more direct and are testable.

To predict agency errors, we need a theory to understand how the bureaucrats working in public organizations make decisions (Jones, 2001; Simon, 1947, 1997; Stinchcombe, 1990). Bounded rationality posits that decision makers are unable to make flawless decisions due to cognitive constraints and the uncertainty of information (Jones, 2001; Simon, 1947, 1997). Given the informational requirements of an eligibility determination, it is important to understand how the acquisition, interpretation, and analysis of information influence the likelihood of an error. Information overload and asymmetries leave bureaucrats to depend on “rules of thumb” or heuristics when making decisions. It may be the case that these shortcuts are the manifestations of psychological biases held by the bureaucrat (Bazerman, 2008). These limitations in the processing of information require that bureaucrats satisfice rather than optimize when handling claims (Simon, 1947, 1997). Ultimately, this satisficing behavior can lead to systematic errors.

When bureaucrats satisfice, suboptimal decisions are likely being made. These deviations from optimality may produce errors in the handling of cases. The interesting question is whether these errors are random or systematic. If the errors are random, then inquiries into the causes of these errors will be unfruitful. Conversely, errors may arise due to the processing mechanisms adopted by the individual bureaucrat, which could include “rules of thumb” borne out of psychological biases (Bazerman, 2008). Biases may lead to the favoring of one group of clients over another, introducing partiality into the claims processing system. If bureaucrats systematically adopt shortcuts or rely on biases then we should see persistent error making.

We hypothesize that several factors are likely to influence the likelihood that bureaucrats will commit systematic errors in the determination of program eligibility. These factors include the organizational culture, organizational (or task) environment, complexity of the policy area, the degree of coproduction involved in the determination, and the characteristics of the claimants, including how the claimant presents him- or herself. Organizational culture is defined in terms of beliefs, values, and artifacts (Ott, 1989; Schein, 1985). Within organizations, culture is often expressed as beliefs that define—“the way we do things here” (McCurdy, 1992, p. 189). Studies have explored the effect of organizational culture, among other variables, on employees making errant decisions. Research shows that organizational culture played a role in errors made at NASA (McCurdy, 1992), in the aviation industry (Klinect, Wilhelm, & Helmreich, 1999), in health care outcomes in hospitals (Shortell et al., 2001), on performance in medical group practices (Kralewski, Wingert, Knutson, & Johnson, 1999), and at the Chernobyl nuclear plant (Medvedev, 1991). Recent investigations concentrate on organizational attributes that serve as precursors to accidents. An organization’s culture reflects its attitudes and policies regarding the punishment of those who commit errors, the openness of communications between management and frontline worker, and the level of trust between individuals and senior management. Organizational culture also influences norms regarding adherence to regulations and procedures (Klinect et al., 1999; Kralewski et al., 1999; McCurdy, 1992; Shortell et al., 2001). This leads us to hypothesize that bureaucrats working in organizations where mistakes go unnoticed or unaddressed, where training is not used to improve accuracy, where employees are unwilling to report problems, or where there is poor communication between managers and the frontlines, are more likely to make errors when processing cases.³

Beyond the culture of any individual office, the tolerance and reaction to errors likely varies across different agencies. How error making is viewed within an agency is linked to the mission and function of the agency, specifically who is harmed by the error and who pays the cost for an error. In the case of agents wrongfully denying benefits, harm falls on the individual claimant—in this case the claimant is denied a payment they should have received. In the case of wrongful payment (overpayment) costs are diffused across taxpayers. Because of the concentrated costs of a wrongful denial bureaucrats are less likely to wrongfully deny a claimant benefits. However, much of this bias against wrongful denials is due to the social service mission of the agency. For bureaucrats working in social service agencies the cost of overpayment is shifted to the taxpayer, while wrongful denial errors fall on the claimant directly. On the other hand, a bureaucrat working in an agency where both the harm and cost of an error are diffused would be less likely to experience this type of systematic bias. Consequently, it is likely that our findings are more applicable to social service agencies.

Next, not all programs are equally complex to administer. There exists tremendous variation both between different programs and within a single program. Within programs, rules may differ between jurisdictions, altering the complexity of administration. Consider the Social Security (OASI) program where the overwhelming majority of eligibility determinations rely on nothing more than the age and earnings of the claimant. This is in stark contrast with eligibility decisions for the Social Security Disability Insurance program where an examiner has to decide whether a claimant is able to work at a hypothetical job given the claimant’s education and experience (Berkowitz, 1987). The complexity of the policy being administered, at a minimum, is related to the following factors: eligibility rules, standards of proof, number of actors, application venues, the opportunities to appeal. In addition, complexity of the bureaucracy’s task environment plays a role in influencing bureaucratic behavior (Wilson, 1989). The task environment refers to the conditions that bureaucrats face when completing their work (Thompson, 1967). For social service agencies the caseload is a major part of the task environment. The size and composition of the workload influences a bureaucrat’s ability to collect and process the necessary information to properly evaluate cases. Bureaucrats with large caseloads will have less time and resources to commit to each case, which may increase the probability of an error being made.

Third, the implementation of most social policies involves the exchange of information between bureaucrats and the clients resulting in the coproduction of a claim determination. This coproduction is complicated by the fact that much of the information is private, resulting in an information asymmetry. Resolving the information asymmetry is central to the bureaucratic function since the interaction between bureaucrats and clients ultimately affects program eligibility (Wenger & Wilkins, 2009). Implementation is difficult when policies are technically complex or require costly information to execute, and these difficulties are likely to increase bureaucratic errors (Hill & Hupe, 2002).

In the application process, Whitaker (1980) argues that bureaucrats deal with the claimant’s application based on the administrative rules and procedures. As a result, their job is limited to sorting applications into the appropriate categories and bureaucrats are discouraged from developing or using their own judgment (Whitaker, 1980). However, Whitaker posits that a bureaucrat’s personal experiences and biases may pervert the administrative process. When bureaucrats perceive claimants positively they are more likely to advocate on the claimant’s behalf, and vice versa. During the application process, bureaucrats not only assess the information the claimant is providing but also evaluate how the claimant is presenting herself. Factors influencing this evaluation may include attire, appearance, weight, use of language, and confidence. Maynard-Moody and Musheno (2003) argue that bureaucrats first assess who their clients are and then find and apply rules or procedures that fit the clients. A bureaucrat’s personal bias toward a particular group of claimants may introduce error in cases when the bureaucrat relies on shortcuts, fueled by bias, to process the claim.

Errors in the Context of Unemployment Insurance

Since the passage of the Social Security Act in 1935, the unemployment insurance system of the United States has provided temporary income replacement for experienced workers who became unemployed through no fault of their own. Each state operates its own UI system and independently determines benefit eligibility, generosity, financing, and taxation. The federal government oversees the program ensuring compliance with broad-based insurance principles, collecting taxes for the administration of the system, making loans available to insolvent state programs, and collecting data on program performance.

Unemployment insurance offers an excellent opportunity to examine error making. The task environment, program complexity, coproduction and claimant characteristics all differ dramatically over the business cycle and across and within UI state offices. As discussed above, this is a large and expensive program and understanding the performance of bureaucrats administering it is important. Many claimants who are wrongfully denied benefits suffer hardships resulting from this loss of income. There are also societal costs associated with wrongful denials, since the UI program helps provide countercyclical economic stimulus. The wrongful denial of benefits short-circuits this stimulative effect.

The size and composition of a UI office’s caseload is linked to macroeconomic conditions. When the economy is in recession, caseloads increase, and dramatically alter the task environment that UI bureaucrats face. As caseloads rise, bureaucrats may rely on rules of thumb to manage the increased amount of information they must process. This response may lead to more errors. Analyzing the tradeoffs between speed and accuracy in the processing of UI claims, Wenger, O’Toole, and Meier (2008) find that this tradeoff is exhibited only in the worst performing offices. Offices with more timely claims processing demonstrated higher levels of accuracy. This finding suggests that increasing caseloads may not be associated with increased errors. While in theory we expect that as caseloads increase bureaucrats would be more likely to make errors, empirically it may be difficult to fully model the complex dynamics of this relationship, due to the lack of explicit managerial performance measures.

By its very nature the administration of unemployment insurance is complicated because of previous employment conditions, and the reasons for unemployment matter in the determination of eligibility. Initially the bureaucrat must assess the claimant’s “attachment” to the workforce. This involves a review of the claimants’ amount and distribution of earnings over the previous year. Next, the bureaucrat must verify the reason(s) for job separation. Typically workers who voluntarily leave a job are ineligible to receive UI benefits for the duration of that unemployment spell. Finally, the bureaucrat must ensure that the claimant actively searches for work; if the claimant stops actively searching for work she no longer meets the definition of unemployed and is ineligible to receive benefits. When compared to other social insurance programs, like OASI discussed above, the determination of the UI eligibility is far more complex than programs that do not have means testing or when eligibility is determined by transparent criteria.⁴

The administration of unemployment insurance is further complicated by the fact that there are 53 separate programs (including DC, Puerto Rico, and the Virgin Islands) and each program has the freedom to adopt different policies for eligibility. For example, some states allow claimants who leave a job for a good cause to still be eligible for benefits after a specified waiting period. Another example of variation is in the treatment of misconduct; if you are discharged from work due to “misconduct” the definition of misconduct and the associated penalties (e.g., benefits postponed, benefits cancelled for the duration of the unemployment spell, must requalify for benefits) vary significantly across states. States infrequently change their policies; however, there is a lot of variation in the complexity of policies from one state to another and this complexity variation must be accounted for, as greater complexity raises the likelihood of making an error. When estimating a model that has 50 different sets of rules it is important to control for this state-to-state variation in eligibility policies to avoid conflating the varying levels of complexity with the likelihood of making an error.

The UI system is both complex and the claim process requires coproduction. The coproduction in UI is somewhat unique because it is driven by the technology used to file a claim. There are numerous methods available for filing a claim (in person, by mail, online, employer-filed claim, by phone, at the workplace), each of which is qualitatively different and requires varying levels of coproduction. For example, filing a claim in person allows for the interaction between claimant and bureaucrat and presents opportunities for clarification, nonverbal cues, and collaboration. In this case, bureaucrats are able to use face-to-face interaction to obtain more accurate information needed to avoid errors. Conversely, filing a claim online allows for no interaction and limits the ability of the bureaucrat to clarify issues that arise during the claim and may lead to errors.

Bureaucrats make an assessment of the claimant and a related evaluation of the quality of a claim based on the characteristics of the claimant. These assessments may introduce bias into the eligibility determination process and increase the likelihood that an error occurs. The bureaucrats use their experience to develop rules of thumb about the difficulty and veracity of certain cases. Bureaucrats may also decide which cases are going to require additional attention. These judgments may overlap with certain demographic characteristics leading the bureaucrat to cut corners when dealing with cases involving certain groups. As discussed above, the bureaucrat may also make an assessment regarding how the claimant presents himself or herself. When corners are not cut, and additional attention is paid, the bureaucrat does the necessary work to properly adjudicate the claim. Given this, we should observe systematic difference in error rates for certain groups of claimants.

Data

To test our hypotheses we analyze data collected by the U.S. Department of Labor for the Benefit Accuracy Measurement program (BAM). Since 2002, the Department has tracked the extent to which UI claims are wrongfully denied through the Denied Claims Accuracy (DCA) program (U.S. Department of Labor, 2006).⁵

Under the BAM program, each state randomly samples a predetermined number of UI claims each week (between 9 and 35, depending on the size of the state) and subjects each of those claims to an intensive field investigation (audit) to determine whether the denial was proper. Audits are performed by a team of experts in each state and consist of telephone and in-person interviews with the claimant, employers, and third parties to determine whether the denial complied with the laws and policies of the state. Errors in these cases are not simply matters of interpretation of the law. Often errors are associated with improper procedure, failure to collect adequate information, wrong information, and failure to properly follow procedure or complete forms. Of course, there are cases when procedures fail, or different judgments indicate an error. However, states assign a panel of expert adjudicators to review these cases. Consequently, it seems prudent to treat cases were these experts disagree with the bureaucrat’s judgment or interpretation of the law as errors.

Once a wrongful denial is identified, the investigator then goes on to record how the error was detected, who was responsible for the error, and what previous behavior (actions) were the likely cause of the error. The DCA data include extensive information about the characteristics of the individual claimant as well as information on his or her employment, earnings history, and complete information on the UI claim. DCA records include both the original information of the claims and the postaudit (correct) information; discrepancies between the two allow us to pinpoint the source of the error. For example DCA records include the reason the claimant separated from his prior employer both as recorded when the initial claim was made and as revealed by the field investigation.

a. Sample Definition

Although we have data on the errors made by all actors involved in this UI claims process, in this analysis we focus on errors that were made solely by the agency.⁶ Since the agency is expected to have the most expertise and experience, we believe that explaining widespread errors by the agency to be the most important. This is especially true since the agency is a repeat player in that they process so many claims. Equally important is the fact that the agency makes more errors than either the claimant or the employer.

Of key importance is our measure of organizational performance. In each state we have a unique identifier that allows us to determine the local office that processed the UI claim. Aggregating for the full year we are able to measure the error rate in each local office. Importantly, some small local offices had very few cases represented in the BAM data. In cases where we could not get consistent measures over time for a local office, we dropped these observations from our analysis. This reduced our sample by 23% and represents 26,864 cases. Consequently, we can only infer the effects of larger local offices on the likelihood of having an error made on a case.

As with many survey data sets, the BAM data contains a large number of missing values. However, unlike most other nationally representative and government sponsored data sets, the BAM data do not include imputations for missing values.⁷ Approximately 29.9% of claimants did not provide reservation wage information and 22.9% did not report a previous wage. One option under these circumstances is to treat the missing values as missing completely at random and use listwise deletion—essentially estimating results only for those observations that report complete information. This would be an acceptable strategy if the missing values were randomly determined. In this case the differences between missing and nonmissing populations should be statistically similar on other observable characteristics, and sample statistics will be unbiased estimates of the population parameters. Unfortunately, this is not the case in the BAM data. Cases that fail to report lowest acceptable hourly wage or previous wage information are statistically different from those who reported such information. Overall, those with missing values were more likely to be female, non-White, and less educated. Since the default option of using listwise deletion is likely to generate biased estimates, we opted to use a regression imputation technique.⁸

In these cases we imputed reservation wage using gender, age, race, education, vocational or technical training, seeking a different job, total earnings for the week(s) before investigation, year, and region. Previous wage was imputed using education, citizenship, race, age, and occupation.

Approximately 11% of the unemployed reported reservation wages above their previous wage.⁹ In general, these responses are difficult to interpret. These claimants have demonstrated their willingness to work at lower wages and now claim that wages that are above their previous wage are necessary to induce them into employment. In cases where claimants provided answers that indicate some reevaluation of their time has occurred, we create a dummy variable for them and include that measure in our analysis.

b. Evidence of Erroneous Denials

Table 1 offers a first look at the denied claims accuracy data for 2003-2009 for all 50 states. The second and third columns show error counts and percentages on UI denied claims sampled for DCA investigations. The estimates show that, over the period from 2003 to 2009, 7.5% of denied claims were wrongfully denied. It is interesting to note that the number of errors (and the percentage of cases that resulted in errors) both fell as we moved into the most severe part of the recession from 2007 to 2008; in 2007 the error rate falls to a low of 7.14%, the same rate as 2003 when the economy was expanding and unemployment was considerably lower. These findings offer the first test of how the task environment influences error making and demonstrates an unexpected finding. It appears that as we moved from economic expansion to recession, error rates remained virtually unchanged. While there are lots of reasons why this may be the case (the mix of cases and the ease of processing being two obvious examples, or the region that experienced unemployment being a less obvious example) it is nevertheless intriguing that the recession is not an obvious predictor of error rates.

Table 1.

Percentage of Wrongfully Denied Claims Due to Agency Errors

Year	Count	Percentage
2003	708	7.14%
2004	829	8.34%
2005	727	7.30%
2006	799	7.95%
2007	737	7.14%
2008	725	7.20%
2009	700	7.46%
Average	746	7.50%

In general, the fact that agency errors appear somewhat common and that they are not obviously correlated with the recession, suggests that eligibility and payment determinations are complex. Other programs, of similar size and complexity, such as food stamps and SCHIP (state children’s health insurance program) have comparable error rates.¹⁰

In Figure 1 we show state-by-state estimates of error rates. In these cases, the unit of analysis is the percentage of cases that were wrongfully denied in any particular week for each state. In most cases the minimum number of cases erroneously denied was zero. However, 15 states had minimum error rates greater than zero. This implies that in these 15 states there was never a week when a mistake wasn’t randomly sampled. The shaded box represents the standard deviation of the error range. The line in the center of the box represents the median. Finally the upper “whisker” represents the maximum error percentage in the state for any week during the 2003-2009 period. The states are arranged by median error rate with Florida and Mississippi having the lowest median error rates—both of which are zero.¹¹ The variation in error rates is considerable with Iowa, New Hampshire, Louisiana, Missouri, Minnesota, Tennessee, and Nebraska having median error rates in excess of 20%. To be clear, this implies that half the time, the field investigation determines that state UI bureaucrats wrongfully denied 20% of cases. Figure 1 demonstrates large differences in error rates by states.

Figure 1.

Distribution of wrongfully denied UI claims by state 2003-2009

By using a unique identifier, we are able to calculate the error rate in the local office for each year. In cases where we do not observe an error rate in the local offices (that is, when too few—or no—cases were sampled from that local office in the entire year) we drop that office from our analysis. In Ohio, Virginia, and Utah the local office designations changed during our study and none of the data could be matched; consequently, this analysis does not include these states. In Figure 2 we show our estimates of the concentration of errors in the local office. We can identify “consistent poor performance” when the local office’s average error rate was above the state median for all 8 years. In some states (Kansas and Nebraska) there are no local offices that fit this description. In addition, some states (Connecticut, Delaware, New Hampshire, Nevada, and Rhode Island) had very few offices that were consistent poor performers—consequently the percentage of total errors made by these offices is quite small. In this case, the consistent poor performers made only a small fraction of the state errors. For the remainder of these states we can see that more than half of all errors are made by a small fraction of the local offices.¹² The interesting part of our analysis shows that seven states have concentration ratios of above 90%. That is, consistently poor performing local offices (those with averages above the state median in each year) account for a massively disproportionate number of errors.

Figure 2.

Error concentration: Proportion of errors made by poor-performing local offices, 2003-2009

The implication of this finding is important. In most states, claimants file unemployment insurance claims via the telephone or the Internet, and these claims are processed in the “local” office. Our analysis indicates that bureaucratic performance is not solely due to the decisions made by an individual bureaucrat at a particular time, but instead, there is a level of path dependence (or hysteresis) that predicts outcomes, including the likelihood that a claim is wrongfully denied. In the multivariate analysis that follows we test this further by analyzing whether last year’s local office error rate is a significant predictor of whether an individual’s claim was wrongfully denied in the current period.

Results

In the remainder of our analysis, we investigate how last year’s error rate in the local office, task environment, complexity, coproduction, and personal characteristics of the claimant alter the probability that a bureaucrat makes an error. Our dependent variable is dichotomous, taking a value of one when a claim is wrongfully denied (error) and a zero when the field audit finds no error in the processing of the denied claim (descriptive statistics are provided in appendix). The mean for this variable is 7.6 with a standard deviation of 26.5. This indicates that 7.6% of cases resulted in a wrongful denial.

We hypothesize that a local office’s error rate in the previous year will have a large significant effect on the probability of an error being made in processing a UI claim. Some local UI offices may deal with a greater number of, and more complex cases, owing to the nature of the population and work histories of the unemployed in their area. We control for this by including the volume of initial claims handled by the office. To control for complexity across states, we include a set of state dummy variables. To capture coproduction we include whether the claimant filed in person, by phone, via Internet, by mail, or had his claim filed by the employer. We also include a set of demographic controls of the claimant: sex, race, age, and education.

Finally, we think there are good reasons that a person’s sense of self-worth may influence the bureaucrat’s behavior. To control for this we include two measures of wages in our model. Our first measure is simply the difference between the previous wage and the lowest wage a claimant would accept. We divide this difference by the previous wage measure and arrive at the percentage wage reduction a claimant is willing to accept. For example, if a claimant was previously working for US$15 per hour and was willing to accept a job that paid US$10 per hour, we would calculate a percentage wage reduction of 33%, (15 – 10)/15. Our second measure is a dummy variable for claimants who state that their lowest acceptable wage is above their previous wage. These claimants have reassessed the value of their time and are no longer willing to accept a job at their previous wage or do not understand the question. In the case where the claimant has reassessed his or her time, we believe this measure serves as a proxy for a sense of self-worth.

In Table 2 we present beta estimates, standard errors, and average marginal effects based on our probit estimates. Our first result is that the previous year’s error rate in the local office significantly raised the probability of an error being made on a claim. Using the estimates from Model 3 that include the full set of controls, the effect of having a claim processed in an office with a 40% error rate last year (compared 10% error rate) raises the probability of having an erroneous denial by 1.2 percentage points. Given that the baseline probability of an error is 7.6%, an increase of 1.2 percentage points represents more than a 15% increase in the likelihood of an error being made.

Table 2.

Probit Estimates: Probability of Wrongful Denial

	(Model 1)		(Model 2)		(Model 3)
Variables	β	Marginal effects	β	Marginal effects	β	Marginal effects
Lagged error rate (%)	0.003***	0.0004	0.003**	0.0004	0.003**	0.0004
	(0.001)		(0.001)		(0.001)
Reservation wage> Previous wage	−0.130**	−0.0182	−0.131**	−0.0183	−0.125**	−0.0174
	(0.055)		(0.055)		(0.055)
Acceptable wage (% previous wage)	−0.000	−0.0001	−0.000	−0.0001	−0.001	−0.0001
	(0.001)		(0.001)		(0.001)
Initial claims (in thousands)	−0.325	−0.0455	−0.320	−0.0447	−0.338	−0.0471
	(0.389)		(0.387)		(0.390)
Filing in person			−0.061	−0.0086	−0.067	−0.0094
			(0.052)		(0.051)
Filing by mail			−0.069	−0.0096	−0.073	−0.0102
			(0.105)		(0.105)
Filing by employers			0.175*	0.0245	0.175*	0.0244
			(0.098)		(0.098)
Filing by other methods			0.090	0.0126	0.077	0.0108
			(0.139)		(0.143)
Filing online			0.023	0.0032	−0.011	−0.0016
			(0.037)		(0.038)
HS/GED					0.039	0.0054
					(0.043)
Some college					0.123***	0.0171
					(0.032)
BA/BS					0.141***	0.0197
					(0.048)
Graduate degree					0.215**	0.0300
					(0.086)
Non-White					−0.076***	−0.0106
					(0.029)
Female					0.023	0.0031
					(0.027)
Age (/10)					0.009	0.0013
					(0.015)
Constant	−1.351***		−1.355***		−1.413***
	(0.060)		(0.061)		(0.090)
State	Yes		Yes		Yes
Year	Yes		Yes		Yes
Observations	69,648		69,648		69,648
Pseudo R-squared	.026		.027		.030

Note: Robust standard errors in parentheses.

p < .1. **p < .05. ***p < .01.

Even after controlling for the local office’s error rate, we find that White claimants and claimants with more education (relative to high school dropouts) were more likely to have their claim wrongfully denied. Finally, in our preferred specification (Model 3), we find that a claim filed by an employer is significantly more likely to result in an error being made. As we hypothesized, relative to filing by the telephone, having your claim filed by your employer increases the probability of an error by 2.4%.When UI claims are employer-filed the bureaucrat may fail to scrutinize these claims leading to more errors.

The complexity of the policy area influences the performance of the agency as indicated by the joint significance of the state dummy variables in our regression. In our preferred specification (Model 3) the chi-squared statistic for the joint significance of the state dummies is 532.99, indicating that we can reject the null hypothesis that the state effects are jointly equal to zero. While it is clear that our state dummy variables fail to capture all the complexity of a state’s UI program, these dummy variables will capture all the time invariant effects that express a correlation between a state and the likelihood of making an error. Given the relative stability in the error rates over the business cycle and the fact that our measures capture time-invariant policy complexity we believe that we have sufficiently controlled for the policy environment. Finally, the American Reinvestment and Recovery Act of 2008 did create some incentives to alter UI programs. However, since the ARRA had a similar impact on most states,¹³ our year dummy variables should adequately control for this effect.

We also find, in agreement with our main hypothesis, that claimants with reservation wages above their previous wage have a reduced likelihood of having an error made. It seems clear to us that claimants who report that they will not return to employment unless they are paid a wage above their previous wage are either (a) not telling the truth or (b) have a significantly inflated sense of self-worth. Consequently, claimants with reservation wages that exceed their previous wage are likely to attract a bureaucrat’s attention. This additional attention is likely to reduce errors—and that is what our analysis indicates. Claimants who have reservation wages above their previous wage are 1.7% less likely to have an error made in their case. Interestingly, we also include the percentage decline in wages that a claimant would be willing to accept to return to work. Claimants with reservation wages equal to their previous wage have a zero for this measure while those who would return to work for no pay have a one for this value.¹⁴ The acceptable percentage decline in wages has no impact on error making. It appears that the entire effect of the reservation wage is determined by the presentation of self that is associated with an overestimation of what the market will pay (Goffman, 1955). Claimants willing to accept large percentage reduction in wages do not signal to the bureaucrat that she must take notice of the claimants’ behavior; in fact it is only those with absurdly high reservation wages that draw additional attention and result in reduced error making.

Our findings regarding the characteristics of the claimants suggest that the bureaucrat’s preconceived notions about whose application needs additional scrutiny influence the likelihood of a wrongful denial. It is likely that most of time bureaucrats in UI offices perform relatively routinized tasks when evaluating claims. When claims conform to the expectations of the bureaucrats (i.e., when the situation is “normal”) the bureaucrat can safely rely on shortcuts and easily adjudicate the claim. In many cases this is optimal; bureaucrats must process many claims and paying strict attention to details cannot be sustained over the course of months and years. In cases that are outside the norm, the bureaucrat is less likely to rely on rules of thumb and more heavily scrutinize the claim, which, in turn, lowers the likelihood that the claim will be wrongfully denied. Our results indicate that when an agent processes a claim from a non-White claimant, the agent is significantly less likely to make an error. A non-White claimant is approximately1% less likely to have his or her claim wrongfully denied than a similar claim made by a White claimant. Clearly, fewer erroneous denials is good, however, unequal scrutiny based on race is not. While we have no direct evidence of differential scrutiny, our statistical evidence is robust to model specification, sample, and time period.¹⁵

Apart from the effects of race and ethnicity, we see significant effects of education on error making. In these cases, it appears that better educated claimants have more errors made (wrongful denials) than do less educated claimants. This result seems counterintuitive. Obviously, the claimant does not want to be wrongfully denied (or denied at all), and yet more education is associated with a higher level of wrongful denials. However, we note that we cannot distinguish different effects across higher education levels (a joint F test of some college, BA/BS, and graduate degree levels of education found that these estimates were not different from each other—although the estimates were significantly different from the parameter estimate for claimants who dropped out of high school). We hypothesize that bureaucrats spend more investigative effort on claimants with less education because the bureaucrat sees fewer of these cases and these cases are often more complicated to adjudicate. Claimants with more education may provide confident and certain responses during adjudication. The sophistication that these claimants bring to the process reduces the probing and clarification needed by the bureaucrat to properly make a determination. This consequently leads to more errors. An alternate hypothesis, that better educated claimants are able to dupe the bureaucrat into wrongfully providing benefits is not supported empirically. Those with college education are more likely to have errors made in their claims and in this case the error is to wrongfully deny benefits.

Conclusion

Optimal decision making is difficult when information is technically complex and/or costly to interpret; bureaucrats in this situation end up making decisions based on cognitive limitations or psychological biases (Bazerman, 2008; Simon, 1947). In this article we investigate whether systematic errors are made by bureaucrats who work in unemployment insurance offices. In general we find that bureaucrats rely on rules about which cases require additional scrutiny.

Recent research in public administration focuses on modeling agency behavior and trying to develop consistent measures of performance. We build on this work to develop an understanding of when agencies are unsuccessful and commit errors in the processing of claims. We identify several factors correlated with wrongful denials. Of course, more work is needed to test and understand the relationships our results suggest. However, we argue that concepts such as task and environment complexity, coproduction, and the presentation of self are important (and often overlooked) components of agency error and performance.

Taken together our findings demonstrate that previous office performance, complexity, method used to file the claim (level of coproduction), demographic characteristics of the claimant (race and education), and presentation of self all influence the likelihood that a claim will be wrongfully denied. Although we are primarily capturing a street-level bureaucrat affect, there are definitely lessons here for managers. Wenger et al. (2008) find that UI managers use training to address accuracy concerns. Managers argue that a well-crafted training program is a key component in office performance (Wenger et al., 2008). At a minimum our results indicate that management should focus more attention on claimants with higher levels of education since bureaucrats are systematically making more errors in those cases. Similarly, equal attention should be paid to White and non-White claimants since non-White claimants are significantly less likely to have their claim wrongfully denied. Ultimately, in a fair system race and education would not be predictive of the likelihood of an error, and instead the merits of the case (along with random error) would be the only factors that determine outcomes.

Finally, we find that the impact of local office error rates on the probability of an error being made confirms the existence of hysteresis in the administration of the UI program. Consistently poor-performing offices account for a significant share of the errors made, and these offices continue to make errors each year at significantly higher rates than other well-performing offices. It seems that managers in these local offices have a systemic problem with wrongful denials and our results suggest that managerial reform and efforts are needed to address these consistently poor-performing offices.

With the passage of the Improper Payments Information Act of 2002 (IPIA P.L.107-300) programs at a high risk of making errors (e.g., TANF, Medicare/Medicaid, SCHIP, and foster care) are required to report annually on the extent of the incorrect payments and the actions the agencies are taking to reduce these errors. This legislation will vastly increase the amount of data available to researchers to evaluate bureaucratic errors. The extent to which wrongful denials change a program’s outcomes is an understudied area in public administration. The optimal number of errors is not zero—public managers need to weigh the cost of an error (denying benefits to an eligible claimant) against the benefits (processing more claims in a timely manner). This article is an important first step in developing and testing a framework to understand the impact of satisficing on eligibility determinations. In many cases bureaucrats lack the resources necessary to make optimal decisions and are left to rely on rules of thumb that can result in claimants being wrongfully denied unemployment benefits.

Footnotes

Appendix

Descriptive Statistics

Variable	Observations	Mean	SD	Min	Max
Agency error	69,648	.076	.026	0	1
Lagged local office error rate (%)	69,648	15.05	12.32	0	100
Reservation wage>Previous wage (dummy)	69,648	0.11	0.32	0	1
Acceptable wage reduction (%)	69,648	14.28	21.20	−100	100
Initial claims (in thousands)	69,648	0.04	0.05	0	0.333
Filing in person	69,648	0.10	0.29	0	1
Filing by mail	69,648	0.01	0.11	0	1
Filing by employers	69,648	0.01	0.10	0	1
Filing by other methods	69,648	0.02	0.14	0	1
Filing online	69,648	0.36	0.48	0	1
HS/GED	69,648	0.16	0.36	0	1
Some college	69,648	0.29	0.45	0	1
BA/BS	69,648	0.10	0.29	0	1
Graduate degree	69,648	0.03	0.16	0	1
Non-White	69,648	0.49	0.50	0	1
Female	69,648	0.49	0.50	0	1
Age (/10)	69,648	3.79	0.85	2.5	5.5
Year 2003	69,648	0.15	0.36	0	1
Year 2004	69,648	0.14	0.35	0	1
Year 2005	69,648	0.13	0.34	0	1
Year 2006	69,648	0.13	0.34	0	1
Year 2007	69,648	0.13	0.33	0	1
Year 2008	69,648	0.15	0.35	0	1
Year 2009	69,648	0.18	0.38	0	1

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Bios

Sangyub Ryu is a PhD candidate in the Department of Public Administration and Policy at the University of Georgia. His main research interests are public management networks and network management behavior. In addition, his research interests include organizational performance, decision making, policy diffusion, policy implementation, and emergency management.

Jeffrey B. Wenger is an associate professor in the Department of Public Administration and Policy, School of Public and International Affairs, at the University of Georgia. His research focuses on unemployment insurance policy, administration, and financing.

Vicky M. Wilkins is an associate professor in the Department of Public Administration and Policy, School of Public and International Affairs, at the University of Georgia. Her primary research interests are in representative bureaucracy and the exercise of bureaucratic discretion.

References

Acock

A. C.

(2005). Working with missing values. Journal of Marriage and Family, 67, 1012-1028.

Allison

P. D.

(2001). Missing data. Thousand Oaks, CA: Sage.

Allison

P. D.

(2009). Missing data. In Millsap

R. E.

Maydeu-Olivares

(Eds.), The Sage handbook of quantitative methods in psychology (pp. 72-89). Thousand Oaks, CA: Sage.

Andrews

Boyne

G. A.

Moon

M. J.

Walker

R. M.

(2010). Assessing organizational performance: Exploring differences between internal and external measures. International Public Management Journal, 13, 1-25.

Bazerman

(2008). Judgment in managerial decision making. New York, NY: Wiley.

Berkowitz

E. D.

(1987). Disabled policy. Cambridge, UK: Cambridge University Press.

Bollinger

Hirsch

(2006). Match bias from earnings impuation in the current population survey: The case of imperfect matching. Journal of Labor Economics, 24, 483-520.

Center for Medicare and Medicaid Services. (2006). Payment error rate measurement pilot project (Final Report). Baltimore, MD: Author.

Fernandez

(2005). Developing and testing an integrative framework of public sector leadership: Evidence from the public education arena. Journal of Public Administration Research and Theory, 15, 197-217.

10.

Goerdel

H. T.

(2006). Taking initiative: Proactive management and organizational performance in networked environments. Journal of Public Administration Research and Theory, 16, 351-367.

11.

Goffman

(1955). On face-work: An analysis of ritual elements in social interaction. Psychiatry: Journal for the Study of Interpersonal Processes, 18, 213-231.

12.

Hicklin

A. K.

(2004). Network stability: Opportunity or obstacle. Public Organization Review, 4, 121-133.

13.

Hill

(2005). The public policy process. Harlow, UK: Pearson Longman.

14.

Hill

Hupe

(2002). Implementing public policy. Los Angeles, CA: Sage.

15.

Johansen

M. S.

(2007). The effect of female strategic managers on organzational performance. Public Organization Reivew, 77, 269-279.

16.

Jones

B. D.

(2001). Politics and the architecture of choice: Bounded rationality and governance. Chicago, IL: University of Chicago Press.

17.

Juenke

E. G.

(2005). Management tenure and network time: How experience affects bureaucratic dynamics. Journal of Public Administration Research and Theory, 15(1), 113-131.

18.

Keiser

L. R.

(2008, August). The impact of politics on the administration of the welfare state. Paper presented at annual meeting of the American Political Science Association, Boston, MA.

19.

Klinect

J. R.

Wilhelm

J. A.

Helmreich

R. L.

(1999). Threat and error management: Data from line operations safety audits. In Jensen

R. S.

(Ed.), Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus: The Ohio State University.

20.

Kralewski

J. E.

Wingert

T. D.

Knutson

D. J.

Johnson

C. E.

(1999). The effects of medical group practice organizational factors on physicians’ use of resources. Journal of Healthcare Management, 44, 167-182.

21.

Lipsky

(1980). Street-level bureaucracy. New York, NY: Russell Sage.

22.

Logan

Rhodes

Sabia

(2006). Food stamp program costs and error rates, 1989-2001 (Under contract to the U.S. Department of Agriculture, Economic Research Service). Retrieved from www.ers.usda.gov/Publications/CCR15/ccr15.pdf

23.

Maynard-Moody

Musheno

M. C.

(2003). Cops, teachers, counselors: Stories from the front lines of public service. Ann Arbor: University of Michigan Press.

24.

McCurdy

H. E.

(1992). NASA’s organizational culture. Public Administration Review, 52, 189-192.

25.

Medvedev

(1991). The truth about Chernobyl. New York, NY: Basic Books.

26.

Meier

K. J.

O’Toole

L. J.

(2001). Managerial strategies and behavior in networks: A model with evidence from U.S. public education. Journal of Public Administration Research and Theory, 11, 271-295.

27.

Meier

K. J.

O’Toole

L. J.

(2002). Public management and organizational performance: The impact of managerial quality. Journal of Policy Analysis and Management, 21, 629-643.

28.

Meier

K. J.

O’Toole

L. J.

(2003). Public management and educational performance: The impact of managerial networking. Public Administration Review, 63, 689-699.

29.

Nicholson-Crotty

O’Toole

L. J.

Jr. (2004). Testing a model of public management and organizational performance: The case of law-enforcement agencies. Journal of Public Administration Research and Theory, 14(1), 1-19.

30.

O’Toole

L. J.

Meier

K. J.

(1999). Modeling the impact of public management: Implications of structural context. Journal of Policy Analysis and Management, 9, 505-526.

31.

O’Toole

L. J.

Meier

K. J.

(2000). Networks, hierarchies, and public management: Modeling the nonlinearities. In Heinrich

Lynn

(Eds.), Governance and performance: New perspectives (pp. 239-291). Washington, DC: Georgetown University Press.

32.

O’Toole

L. J.

Meier

K. J.

(2003). Plus ça change: Public management, personnel stability, and organizational performance. Journal of Public Administration Research and Theory, 13(1), 43-64.

33.

Ott

J. S.

(1989). The organizational culture perspecive. Chicago, IL: Dorsey Press.

34.

Pitts

D. W.

(2005). Diversity, representation, and performance: Evidence about race and ethnicity in public organizations. Journal of Public Administration Research and Theory, 15, 615-631.

35.

Schein

E. H.

(1985). Organizational culture and leadership. San Francisco, CA: Jossey-Bass.

36.

Shortell

S. M.

Zazzali

J. L.

Burns

L. R.

Alexander

J. A.

Gillies

R. R.

Budetti

P. P.

. . . Zuckerman

H. S.

(2001). Implementing evidence-based medicine: The role of market pressures, compensation incentives, and culture in physician organizations. Medical Care, 39, 162-178.

37.

Simon

H. A.

(1947). A comment on “the Science of Public Administration.” Public Management Review, 7, 200-203.

38.

Simon

H. A.

(1997). Administrative behavior. New York, NY: Free Press.

39.

Stinchcombe

A. L.

(1990). Information and organizations. Berkeley: University of California Press.

40.

Thompson

J. D.

(1967). Organizations in action. New York, NY: McGraw-Hill.

41.

Walker

R. M.

Andrews

Boyne

G. A.

Meier

K. J.

O’Toole

L. J.

(2010). Wakeup call: Strategic management, network alarms and performance. Public Administration Review, 70, 731-741.

42.

Walker

R. M.

O’Toole

L. J.

Meier

K. J.

(2007). It’s where you are that matters: The networking behaviour of english local government officers. Public Administration, 85, 739-756.

43.

Wenger

J. B.

O’Toole

L. J.

Meier

K. J.

(2008). Trading speed for accuracy? Managing goal conflict and accommodation in the U.S. unemployment insurance program. Policy Studies Journal, 36, 175-198.

44.

Wenger

J. B.

Wilkins

V. M.

(2009). At the discretion of rogue agents: How automation improves women’s outcomes in unemployment insurance. Journal of Public Administration Research and Theory, 19, 313-333.

45.

Whitaker

G. P.

(1980). Coproduction: Citizen participation in service delivery. Public Administration Review, 40, 240-246.

46.

Wilson

J. Q.

(1989). Bureaucracy: What government agencies do and why they do it. New York, NY: Basic Books.