What is policy failure? A primer to help navigate the maze

Abstract

The discipline of public policy has struggled to come to terms with how we may conceive of ‘policy failure’. It tends to assume either that failure is self-evident or that it can be assessed by means of examining the gap between government goals and outcomes. Often, there are multiple caveats that seem too difficult to address – particularly the role of perceptions, which in turn are often dependent on whether or not the policy is supported. This ground-breaking article builds on and refines existing literature. It turns on its head the multiple methodological challenges surrounding what constitutes policy failure (such as competing goals and variance over time) and suggests that such seemingly impenetrable challenges actually help illuminate our understanding. In doing so, it argues that once we conceive of studying policy failure as ‘art and craft’, we are better placed to navigate the messy realpolitik of types and degrees of failure, as well as ambiguities and tensions between them. The groundwork for doing so is based on a working definition of failure, namely that a policy fails, even if it is successful in some minimal respects, if it does not fundamentally achieve the goals that proponents set out to achieve, and opposition is great and/or support is virtually non-existent.

Keywords

Policy failure policy disasters policy fiascos policy evaluation

Introduction

Policy failures seem pervasive, with no policy sector or country appearing immune to the operational challenges and political pitfalls of failure. A simple internet search for news stories at the time of writing produced headings such as ‘Obama’s Foreign-Policy Failures Go Far Beyond Iraq’, ‘Homelessness Crisis Show “Social Policy Failure”’ and ‘Prisons Face Overcrowding Due to Policy Failure’. We can also add to this list, other prominent attributions of policy failure, such as the Poll Tax and Child Support Agency (UK), home insulation program (Australia), response to the global financial crisis (Iceland), ‘No Child Left Behind’ (USA) and Guantanamo Bay detention camp (USA). Yet despite the apparent confidence and conviction of many that certain policy outcomes constitute ‘failure’, the more that researchers delve into the topic and seek to define failure, the more they seem trapped in a maze of methodological difficulties, such as multiple goals, failure for whom, and not least varying perceptions. Disputes over whether a policy has actually ‘failed’ are commonplace. Allegations of policy failure from opposition parties, the media and others, typically produce counter discourses from supporters attempting to shore up support for policies by claiming that the alleged failure is in fact a success.

The lack of a means of navigating the issue of what constitutes policy failure is a significant one. It is a barrier to comparing individual case studies, undertaking comparative research and indeed to understanding the causes of failure, which often gets mired in post-failure blame games. Unless we can find a way of navigating the maze of what constitutes policy failure, we risk repeating the mistakes of the past, both in terms of producing policy failures and learning from failure (Howlett, 2012). In this context, this article provides an original primer to assist analysts in navigating the maze of issues surrounding what constitutes policy failure. First, it examines the paucity of public policy writings on policy failure – from those touching on the subject tangentially, to those tackling it directly. Second, it identifies a number of key methodological difficulties involved in any attempt to comprehend the nature of policy failure, including perceptions, grey areas, ambiguities and variations over time. Third and finally, it argues that once we accept the messy realpolitik of failure, we are better primed to approach the subject, armed with awareness of the ‘art and craft’ of understanding failure, types of failure, degrees of failure and tensions between failure and success.

While some of us might consider failure to be a simple phenomenon that ‘just is’ because it breaches a sense of common morality, I refer to the realpolitik of policy failure for an important reason. It indicates that to engage in a more meaningful way with the real world complexities of policy failure, we need to accept that failure is bound up with issues of politics and power, including contested views about its existence, and the power to produce an authoritative and accepted failure narrative.

The public policy literature on defining policy failure

Failure is more newsworthy than success, and political analysis is generally more interesting when it includes (at least to some degree) strong aspects of failure: whether it is perverse policy outcomes, corrupt practices, or public officials in the spotlight for misjudgement. Policy analysis itself is a diverse practice, with varying traditions and purposes from the provision of advice to policy makers in the interests of societal betterment (Lasswell, 1971) to highlighting social injustices (Dryzek, 2006). Surprisingly, there is a relative paucity of writings on policy failure. Perhaps one main reason is that deep analysis requires confronting the near intractable methodological issues mentioned previously. Nevertheless, to varying degrees there are broad strands of policy literature that help inform us. In some, the concept of failure is tackled in relatively ‘light touch’ fashion while in others there are direct attempts to focus on the nature of policy failure.

One of the most prevalent is single case studies of failure. Examples include Hurricane Katrina, anti-money laundering (AML) policies, the ‘war on drugs’, Australian asylum seeker policy, childcare provision, US Department of Homeland Security, housing reform in Scotland, nuclear regulation in Japan and British security policy after the Cold War (see e.g. Buchanan, 2010; Dyson, 2006; Kearns and Lawson, 2009; Kettl, 2004; Kingston, 2011; Lewis, 2012; Sharman, 2011; Walsh, 2006). The study of single cases often assumes a priori that the case is a ‘failure’ and does not provide a definition. Even some comparative case studies are prone to assuming that failure is self-evident and needs no definition, e.g. see Grossman (2013) on economic policy disasters. Others provide ad hoc definitions (or make ad hoc assumptions appropriate to the case study), such as costs outweighing benefits (Sharman, 2011) and the non-achievement of policy making goals. As an example of the latter, Walsh (2006: 495) suggests that ‘Policy failure occurs when the decision makers responsible for initiating the consideration of and approving new policies conclude that current policy is no longer achieving the political and program goals they prefer’. The inclusion of ‘politics’ is important here, because it alerts us to the possibility of a disjunction between political and programme outcomes.

Public policy literature also concerns itself with policy outcomes. The literature on policy evaluation is a major contributor in this regard, because it focuses explicitly on understanding the methods, tools and techniques of how we assess whether policies work or not. For the rationalist, scientific tradition, policy outcomes can typically be measured and assessed against original goals, using a variety of techniques such as cost–benefit analysis, multi-attribute value functions and a wide range of statistical methods from regression to multivariate time series modelling (see e.g. Argyrous, 2009; Gupta, 2001). It is assumed that failure can be observed and measured against whichever standard we choose – typically programme goals. Yet the pluralistic nature of the social sciences has also produced post-positivist methods and frameworks, emphasising the role of perceptions in ‘constructing’ failure as a political act. Therefore, by implication, ‘failure’ is a construction by those whose social power allows them to articulate and succeed in securing a dominant failure narrative (Taylor and Balloch, 2005). Some of the more nuanced literature on evaluation straddles this methodological divide. For example, Vedung (2013) identifies six policy evaluation models, where assessments are conducted respectively against goals, goals plus side effects, client need, views of multiple stakeholders and the judgement of independent peer reviewers. The very fact that such diversity exists seems to compound the difficult of understanding the nature of failure when the ‘goalposts’ keep moving.

The literature on policy implementation is also important in terms of understanding outcomes. The ground-breaking study of implementation by Pressman and Wildavsky (1973) was produced amid a climate that many post-war policy aspirations and policies did not always translate into workable policy. The prime focus of a burgeoning literature (see e.g. Barrett and Fudge, 1981; Hood, 1976; Mazmanian and Sabatier, 1981) was on a variety of ‘top down’, ‘bottom up’ and hybrid explanation for ‘implementation gaps’, with little attention paid to the nature of failure per se. Pressman and Wildavsky (1973) themselves did not define policy failure, other than seeing its roots just as much in flawed ideas, as in the inefficiencies or adverse chance circumstances of implementation. The near default position of subsequent implementation studies was captured by Hogwood and Gunn (1984) who describe policy failure as either non-implementation where the policy is not put into effect as intended, or unsuccessful implementation where, the policy is enacted but circumstances are such that the policy fails to achieve the desired outcomes or results.

Beyond evaluation and implementation, the discipline of public policy has rich and varied foci, but only tangential focus on policy failure. The general tendency is to make the occasional passing commentary on policy failure, but not doing so in any systematic or conceptually structured manner – largely because studying failure is a means to an end. For example, the study of policy formation and design processes addresses issues such as why some policy options are included in decision making processes while others are excluded (Schneider and Ingram, 1997; Stone 2012). Failure here is assumed in a general way to be policies which don’t solve problems, don’t act in the public interest or contribute to a more democratic polity.

In contrast to ‘light touch’ or tangential discussion of failure, there exists a handful of works explicitly on policy failure – or to be precise, an extreme form of failure that can be described as ‘policy fiascoes’, ‘policy disasters’ and ‘policy blunders’. They have varying foci, covering typically the causes of failure rather than the nature of failure per se. Some are exceptionally vague on what constitutes failure, assuming it to be a general inability to achieve effective outcomes (Wallis, 2011). Ingram and Mann’s (1980) writing in the context of the economic and political pressures of the mid to late 1970s do not define failure but point to many of the difficulties of measuring failure, including varying perceptions, unrealistic goals and isolating a policy from its interlinkages with other policies. Hogwood and Peters (1985) use a medical metaphor to equate policy failure with the types of pathologies which beset the human body such as starvation and overindulgence, although there is no definition of failure, other than a general assumption that it is a condition that is ‘not healthy’ and may be subject to varied diagnoses from different experts. A subsequent debate by Dunleavy and Gray on UK policy disasters epitomises many of the difficulties in defining policy failure. Dunleavy (1995: 52) defines policy disasters as ‘significantly and substantially costly failures of omission or commission or by government’, while acknowledging pragmatic difficulties of producing a suitable definition. Gray (1998) in his critique adds a number of criteria that should be taken into consideration (including being considered as a disaster by a wide range of opinion, and producing extensive disruption to social and political processes) but stops short of defining disasters and is clearly of the view that they are more severe than ‘mere failures’ (p.76). A parallel work by Gray (1998: 8) goes further and – while not defining failure – defines policy disasters as ‘… policies which have failed against nearly every possible criteria of evaluation, caused considerable disruption which was foreseeable and/or avoidable, and triggered complex trails of unintended consequences’.

Bovens and ‘t Hart, writing together and in combination with others (Bovens and ‘t Hart, 1996; Bovens et al., 2001a; see also Gray and ‘t Hart, 1998), directly address issues of failure. In their study of policy fiascos, they focuses on the interpretation and construction of failures by different actors (Bovens and ‘t Hart, 1996). The contention is that ‘Failure is not inherent in policy events themselves. ‘“Failure” is a judgement about events’ (Bovens and ‘t Hart, 1996: 21) that is contingent on who is judging, the measures they use, the information basis and the time period they assess. A particular contribution here is to dissect the framing of failure into four main sub frames (severity, causes, motivations and blame). The first of these is particularly important for understanding the nature of failure but it suggests that failure cannot be determined by a fixed measure. As Brändström and Kuipers (2003) indicate in a subsequent work inspired by Bovens and ‘t Hart, failures may be framed as localised, marginal and even routine, or deeper rooted in the core values of society. According to this argument, ascertaining the scale of failure requires a strong element of subjectivity.

A more extended treatment of failure – especially in empirical terms – is contained in an edited volume by Bovens et al. (2001a) examining success and failure in public governance, spanning four policy sectors and six countries. Crucially they separate and recognise the potential for conflict between programme and political outcomes, while tailoring the criteria for each individual case study. For example, the decline of the steel sector includes a measure related to the financial costs of restructuring, while the criteria for the blood transfusion sector makes no mention of costs and focuses instead on criteria such as the effectiveness of donor selection. Although not developed conceptually, Bovens et al. (2001b) allude to the complex relationship between success and failure, especially in describing reform of the steel sector as a case of ‘non failure’ rather than ‘straightforward success’ (p.596).

More recently, a major book on ‘policy blunders’ in the UK by King and Crewe (2013) is an empirically-focused examination of many policy failures in the UK, from the Child Support Agency to reforms of the London Underground. Their definition of a ‘blunder’ alludes to degrees of failure while also touching on the political repercussion of failure. They define a blunder as ‘… an episode in which a government adopts a specific course of action in order to achieve one or more objectives and, as result largely or wholly of its own mistakes either fails completely to achieve these objectives, it does achieve some or all of them but at a totally disproportionate cost, or else does achieve some or all of them but contrives at the same time to cause a significant amount of “collateral damage” in the form of unintended and undesired consequences’ (King and Crewe 2013: 4). Yet the difficulty in using this definition for ‘policy failure’ is that it attributes causality principally to government, rather than separating failure and causality, and seeing them as related but independent. It is possible that a policy may fail because it is knocked off course by unforeseeable circumstances but remains a failure nevertheless, e.g. public infrastructure projects cancelled because of the global financial crisis.

One final area of study to highlight is recent writings on policy success (Marsh and McConnell, 2009; McConnell 2010a, 2010b, 2012). They examine failure insofar as it is the ‘mirror image of success’ (McConnell, 2010b: 356). McConnell (2010b: 356–357) suggests that ‘A policy fails if it does not achieve the goals that proponents set out to achieve, and opposition is great and/or support is virtually non-existent’. These works confront many of the methodological difficulties in comprehending issues such as varying perceptions, multiple benchmarks and the issue of failure for whom. They also build on Bovens et al. (2001a) by recognising different types of successes (process, programme and politics), reflecting the complexity of policy outcomes and the realities that policies may succeed in some respect but not others. However, in these works, policy failure is tackled tangentially as something of a by-product to understanding policy success. Furthermore, the discussion by McConnell of a spectrum of policy outcomes from success to failure (five categories in all) lacks a degree of parsimony that might better serve the study of this fledgling field of study.

In sum, literature on policy failure is remarkably thin on the ground and often lacks explicit conceptualisation. Even many of the works focused explicitly on failure, assume failure to be self-evident or struggle to provide a usable definition. In order to develop our understanding much further, I now present in detail for the first time, a series of methodological difficulties in comprehending what constitutes policy failure. As will be argued, their very existence – often considered to be crucial impediments to our understanding of policy failure – actually helps advance our comprehension of this understudied phenomenon.

The methodological difficulties of defining policy failure

Differing perceptions

In essence, what one individual perceives as a failure may be viewed by another as ‘not a failure’ or even a ‘success’. Such issues get to the heart of the methodological diversity of political science and the social sciences more generally (Hay, 2002; Marsh and Stoker, 2010). Political life, whether it be the health of democratic systems or changing policy agendas, is studied from an array of assumptions around ontology (what is the true nature of the phenomena we are studying?), epistemology (how do we know that this is the true nature?) and methodology (how should we study this phenomenon?). Needless to say the multiplicity of fine-grained debates and philosophical reflections cannot be dwelt on here, but for present purposes we should recognise two counter tendencies – as reflected in debates within the evaluation literature. The first, we can call the rational scientific tradition, which in terms of ‘failure’ translates into the assumption that failure is an objective fact (see e.g. Davidson, 2005 and Gupta, 2001 on evaluation). A counter-tendency is the interpretivist, constructivist and discursive tradition, which views the world very much as contingent on individual perceptions, which typically vary, depending on who is ‘perceiving’ (see e.g. Edelman, 1988; Fischer, 2003; Stone, 2012). In any quest to understand policy failure, therefore, there is a real difficulty in reconciling two competing phenomena with seemingly equal plausibility. It would be difficult to dispute the fact that a government failing to implement a controversial ‘rendering’ policy (terrorist suspects being sent overseas for interrogation) constitutes a failure when matched against originally government intentions, but equally the outcome may be seen as a success for those arguing that rendering poses high risks of human rights violations.

Differing benchmarks

The word ‘failure’ has negative connotations (even if we think some positive benefits might ensue) and brings to the fore a relational issue, i.e. failure in relation to what? Once we unpack this issue, relying on many of the struggles of the direct and indirect attempts to write about policy failure, there are a host of different, non-mutually exclusive possibilities. They include failure to:

meet original objectives, e.g. to reduce alcohol-related crime by 10%,

be implemented as intended, e.g. establish a new agency for food safety,

benefit the intended target group, e.g. women over the age of 55,

provide benefits that outweigh the costs, e.g. lasting peace vs. loss of lives in military intervention,

satisfy criteria that are highly valued in that policy sector, e.g. national security in the intelligence sector,

meet legal, moral or ethical standards, e.g. protecting human rights,

garner support from key stakeholders in that sector, e.g. farmers, small businesses,

improve on the previous state of affairs, e.g. incidences of corruption are less in number since creation of new anti-corruption watchdog and

improve by comparison to a similar jurisdiction, e.g. one provincial government improving educational performance relative to a neighbouring provincial government.

Understanding failure would be straightforward if there was universal agreement on failures being defined by breach of a universally agreed benchmark of X, but this is simply not possible, given the existence of multiple and often conflicting evaluation measures (Vedung, 2013) and also the propensity of policy opponents to emphasise those aspects that have failed to be achieved, and for policy supporters to emphasise those that have. For example, a government’s defence that its investment in education has increased by 10% over the previous five years, could be countered by critics who argue that funding is still lower than competitors and is failing to translate into improved educational standards for students.

Grey areas

Differing perceptions aside, failure is rarely ‘all or nothing’. Typically there are shades of grey, where judgement is needed in terms of the interpretation and significance that should be given to shortfalls, lack of evidence and conflicts. Ambiguity is not only part of the policy formation process (Zahariadis, 2003) (indeed Stone, 2012, argues that ambiguous and ‘feelgood’ language is necessary for the purpose of alliance building) but it is also a recurring theme in policy implementation (Matland, 1995) as well as how we evaluate complex policy outcomes. There is a certain logic, even without adopting a rational-scientific perspective, to the view that we should identify what goal or objective was set, and then ascertain if it was in fact met. But what if targets constantly altered, such as the case of shifting mandates for forestry policy in Canada (Rayner, 2012)? One can conceive of original goals being met at the same as a failure to meet goals that have been ‘added on’. Or what if a goal was only partially fulfilled? If for example a government’s anti-drink driving campaign aims to reduce offences by 50% but the reduction is only 40%, does this mean the policy has failed? Do the shortfalls negate the success, or should we weigh up each. Of course, the issue then becomes one of where we draw the line. There is no scientific formula for making such decisions.

We may also not have sufficient evidence to make a judgement on policy failure. Appropriate information may simply not be available (on patient care or the extent of child abuse) or may even be hidden from view in the sense that a policy may have failed against a hidden agenda goal, but we will never know because that goal is not in the public domain. A hidden goal of public policy, to some degree, may be to manage a difficult issue down or off the policy agenda, through a ‘placebo’ policy which may be more successful in terms of political agenda management because there is the appearance that an issue is being addressed, but may do little to actually address complex and ‘wicked’ causes and symptoms (Gustafsson, 1983; McConnell, 2010a). Arguably, many social issues such as poverty, drug abuse and homelessness are ‘wicked issues’ but even the creation of the US Department of Homeland Security has been considered a move to present a unified approach to public and political fear of terrorist threats, rather than a proportionate response to actual threats (Friedman, 2011). Such issues will continue to be debated but the key point of relevance here is that – albeit difficult to ascertain with absolute certainty – policies may fail in some respects but succeed against latent political goals.

Furthermore, policies often have multiple goals, and so a further and exceptionally difficult issue is how we weigh up and prioritise failure in one goal, against success in another. For example, New York’s Family Rewards Scheme which provided financial incentives for the very poorest families subject to them undertaking certain activities and attaining particular outcomes failed to make any difference to school attendance or academic performance but was successful in increasing families’ use of medical care and reducing hardship (Miller and Riccio, 2011). Such grey areas pose serious difficulties for analysts in terms of whether they can say with any degree of comprehensiveness that a policy has ‘failed’.

Failure for whom?

Public policies often have ‘target groups’ (such as smokers or young drivers under-25) and policy makers hope that the circumstances and/or behaviour of these groups will be altered by the requisite policy. At times a policy may be designed to limit the rights/rewards of a target group (those convicted of fraud being unable to keep the proceeds of crime) or provide/expand the benefits to a particular group (the right to same-sex marriage). Crucially, therefore, the issue of ‘failure for whom?’ adds further complexity. A policy that failed to encourage parents to provide healthier lunches for their children when attending school would make little difference to families with no school-age children. In fact, a policy that failed to deliver benefits for one group may be successful for another. A failed attempt by a local government to build a waste facility next to a local community, would also be a failure for the commercial contractor specialising in waste removal, but a success for local residents who campaigned against the project. If a policy fails some groups/stakeholders but brings successes for others, there is a real difficulty in weighing-up these complex outcomes and ascertaining which matters most.

Variance over time

Attributing the word ‘failure’ to a particular set of policy outcomes seems definitive, as though policy is irredeemable from that point onwards. Yet in addition to the ‘grey areas’ identified previously, a further challenge in ascertaining policy failure is that there are multiple points in the policy cycle when an evaluation may occur, leading potentially to different outcomes. For example, projections of failure at the policy making stage may differ from an evaluation of outcomes after implementation. Furthermore, policy that failed in the short term may yield successes in the long term. An often cited example is the planning disaster (Hall, 1982) of Sydney Opera House which was over a decade late and 14 times over budget, but has since become one of the world’s top tourist attractions. Even policies that ‘failed’ can help open policy windows for further reform. The highly controversial poll tax in the UK between 1989 and 1993 (in effect a per capita tax on the right to vote) paved the way for a reformed property tax (Council Tax), addressing many of the deficiencies of a local tax system that had been resistant to serious reform over the previous century. Weighing up a period of ‘failure’ against a period of ‘success’ seems an almost intractable issue, taking us even further away from the idea that failure is defined by a clear and constant set of undesirable circumstances, ascertained only at a fixed point in time.

Overall, therefore, understanding what constitutes policy failure is beset by a series of methodological difficulties that seem to make the challenge insurmountable. However, I would argue that we can advance our understanding by embracing its vagaries and recognising that ‘policy analysis in the real world’ (to coin Hogwood and Gunn) does not need perfect answers to advance our understanding. Policy sciences, as a discipline, has always proceeded incrementally (deLeon, 1988).

Moving forward: A primer to help cope with the ‘maze’ of policy failure

In order to help make sense of the messiness and contestability of failure, I modify the earlier definition by (McConnell, 2010b: 357) and suggest we consider that:

A policy fails, even if it is successful in some minimal respects, if it does not fundamentally achieve the goals that proponents set out to achieve, and opposition is great and/or support is virtually non-existent.

There are several implications of this wording that require discussion, as well analytical issues that need built up from the definition. Doing so will also illustrate the value in modifying McConnell’s work on success in order to help develop our preparedness to enter the ‘maze’ of policy failure.

Understanding failure as ‘art and craft’

In his classic book Speaking Truth to Power: The Art and Craft of Policy Analysis, Wildavsky (1987: 16) wrote that the ‘… technical basis of policy analysis is weak. In part its limitations as those of social science: innumerable discrete propositions, of varying validity and uncertain applicability, occasionally touching but not necessarily related, like beads on a string’. A theme throughout is that in the face of complex and uncertain environments with innumerable potential problems and differing moral views on the way forward, policy analysis requires inventiveness, imagination, judgements, art and craft. I would argue that any search for a scientific, unambiguous and value-free definition of policy failure would face serious difficulty in being able to cope with the complex, contested and often ambiguous realities of policy outcomes. The definition of policy failure above, and key issues that can be built-up from its foundations, is provided in the spirit of analysis as ‘art and craft’.

First, the definition can help us cope with quasi-scientific, objectivist aspects of failure (e.g. policies failing to meet their objectives, not being implemented as intended) with the reality that political actors may have different views on that failure, depending on their core values and their view on the best means of achieving them through particular policy initiatives. Imagine a hypothetical example where a government rescinds minimum statutory minimum wage levels, and puts in place a new policy of non-intervention, leaving wage levels entirely to the laws of supply and demand. The key goal is to ensure that wage levels are economically sustainable for employers. One year later, against the wishes of employers, it backtracks are reintroduces statutory minimum wage requirements, albeit in a slightly modified form. The reasons for the change are many, including the unpopularity of the original policy and a looming national election. Did the government’s free market policy fail? In all likelihood the government would have pragmatically accept its policy as a failure, and the trade unions would also accept it as failure. But it is likely that many employers would view outcomes as successful, because they were able to pay to affordable ‘market’ rates. It is often the case that regardless of the multiple facts and figures produced in relation to policy outcomes, differing attributions of ‘failure’ (or success) will often depend on the extent to which a policy is supported by an array of policy actors, which in turn depends on underlying values and what is considered to be the best means of achieving them. We should not shy away from the realpolitik of differing views on policy outcomes but the definition of failure given above can accommodate differing views. A policy may abandoned and/or not achieve the goals that its proponents set out to achieve, but we have the capacity analytically to separate outcomes from the extent or support or otherwise for those outcomes.

Second, the definition goes beyond McConnell’s original definition by tempering the implication that failure resides at the extreme end of a success–failure spectrum where ‘failure’ is marked by an absolute non-achievement. In reality, even policies that have become known as classic policy failures also produced small and even quite modest successes. For example, the ill-fated Millennium Dome in London attracted some six million visitors and an 84% satisfaction rating (King and Crewe, 2013: 123–124). By adding the words ‘even if it is successful in some minimal respects’ we are able to grasp the reality that failure is rarely unequivocal and absolute. Also, by introducing the words ‘does not fundamentally’ in relation to the non-achievement of goals allows us to grasp that small pockets of success do not detract from a policy failing to achieve goals. Of course commentators may differ on what constitutes ‘fundamental’ but such differences are indicative of the contested nature of failure and should be recognised rather than shunned as ‘unscientific’. A similar issue arises with the matter of what constitutes circumstances when ‘opposition is great and/or support is virtually non-existent’. There is no scientific gauge that suddenly indicates ‘danger zones’ in terms of support and indeed protagonists may still read the signs differently if there were. Rather, including lack of support and opposition into a definition of policy failure allows us to recognize that policy is not produced and implemented in a vacuum. Policies failing fundamentally ‘on the ground’ may be kept alive by sufficiently strong coalitions of support or terminated because of lack of support/opposition that makes continuation politically unsustainable.

Third, the definition does not specifically mention the crucial issue of ‘time’, but its ability to accommodate differing perspectives also allows it to cope with temporal aspects of failure. The policy cycle/stages approach to understanding policy has its weaknesses, but it nevertheless has value in helping us understand that policy processes involve different activities such as problem definition, options appraisal, decision making, implementation and evaluation. Assessments of policies may occur at any of these stages. Policy can fail ex ante in the process of policy formation, for example by being withdrawn because of an assessment that it would be too risky (Spain’s withdrawal of a bill that would have imposed stricter limits on abortion), or defeated during legislative passage (Obama’s gun control legislation). Or policy may fail at the crucial decision stage (Scottish First Minister Alex Salmond’s failure to secure a ‘Yes’ vote in the referendum on independence). Or policy may fail in an ex-post evaluation of implementation outcomes (Royal Commission into the home insulation scheme in Australia). Failure may manifest differently at multiple stages throughout the policy cycle but the common denominator – returning to the definition – is that government is unable to do what it set out to do (e.g. failure to produce legislation, failure to obtain approval for a decision, failure to achieve outcomes) and fails to garner the requisite support. Policy processes are in perpetual states of evaluation, not only often informal but also formal through means such as risks assessments, cost–benefit analysis, scenario plans and post-hoc evaluation exercises. The ‘failure’ of policy is highly temporal in the sense that there are variations in terms of when the assessment is conducted in the policy cycle and the time period (past, present and future) that is being evaluated.

Three forms of failure: Process, programmes and politics

Definitions of public policy abound, but that of Dye (2012: 12) encapsulates the essence of most: ‘whatever governments choose to do or not do’. If we know what governments ‘do’, we are well placed to understand ‘failures to do’. There are certainly many ways of dissecting the roles of government. I argue here for three types of ‘doing’. They are inextricably linked, but it is useful to separate them analytically because they draw on different traditions within the policy sciences and provide us with an understanding of different ways in which government may fail to do what it intended. This in turn opens the door for us to develop an understanding of some of the internal tensions of policy failure, with governments failing in some respects but succeeding in others. An important caveat is that while I fully recognise the multi-faceted nature of modern governance systems and the capacity of market-based, hierarchical and network governance (Bell and Hindmoor, 2009), such complexity is beyond the scope of one short article which should be seen as a first step on the road, rather than the end of a journey. Using ‘government’ as a reference point, therefore, Table 1 summarises usable criteria for (outright) policy failure.

Table 1.

Degrees of policy failure across process, programme and politics.

Criteria	Tolerable failure	Conflicted failure	(Outright) Failure
Policy as process
1. Preserving goals and policy instruments	Policy goals and instruments preserved, despite minor failure to achieve goals.	Preferred goals and instruments proving controversial and difficult to preserve. Some revisions needed.	Government unable to produce its desired policy goals and instruments.
2. Securing legitimacy	Some challenges to legitimacy but of little or no lasting significance.	Difficult and contested issues surrounding policy legitimacy, with some potential to taint the policy in the long-term.	Policy process illegitimate.
3. Building sustainable coalition	Coalition intact, despite some signs of disagreement.	Coalition intact, although strong signs of disagreement and some potential for fragmentation.	No building of a sustainable coalition.
4. Attracting support for process	Opposition to process is low level and outweighed by support.	Opposition to process and support are equally balanced.	Opposition to process is virtually universal and/or support is virtually non-existent
Policy as programme
5. Implementation in line with objectives	Implementation objectives broadly achieved, despite minor failures and deviations.	Mixed results, with some successes, but accompanied by unexpected and controversial failings.	Despite minor progress towards implementation as intended, programme is beset by chronic failures, proving highly controversial and very difficult to defend.
6. Achieving desired outcomes	Outcomes broadly achieved, despite minor shortfalls.	Some successes, but the partial achievement of intended outcomes is counterbalanced by unwanted results, generating substantial controversy.	Some small outcomes achieved as intended, but overwhelmed by controversial and high profile failure to produce results.
7. Benefitting target group(s)	A few shortfalls and possibly some anomalous cases, but intended target group broadly benefits.	Partial benefits realised, but not as widespread or deep as intended because of substantial failings.	Small benefits are accompanied and overshadowed by damage to the very group that was meant to benefit. Also likely to generate high profile stories of unfairness and suffering.
8. Satisfying criteria highly valued in policy domain	Not quite the outcome desired, but despite flaws, close enough to lay strong claim to fulfilling the criteria.	Partial achievement of goals, but accompanied by failures to achieve, with possibility of high profile examples.	A few minor successes, but plagued by unwanted media attention.
9. Attracting support for programme	Opposition to program aims, values and means of achieving them is stronger than anticipated, but easily outweighed by support.	Opposition to program aims, values and means of achieving them is equally balanced with support for same.	Opposition to program aims, values and means of achieving them outweighs small levels of support.
Policy as politics
10. Enhancing electoral prospects/reputation	Favourable to electoral prospects and reputation enhancement, despite minor setbacks.	Policy obtains strong support and opposition, working both for against electoral prospects and reputation in fairly equal measure.	Despite small signs of benefit, policy proves an overall electoral and reputational liability.
11.Easing the business of governing	Despite some problems in agenda management, capacity to govern is unperturbed.	Policy proving controversial and taking up more political time and resources in its defence than was expected.	Clear signs that the agenda and business of government struggles to suppress a politically difficult issue.
12. Promotion of government’s desired trajectory	Some refinements needed but broad trajectory unimpeded.	Direction of government very broadly in line with goals, but clear signs that the policy has promoted some rethinking, especially behind the scenes.	Entire trajectory of government in danger of being compromised.
13. Providing political benefits for government	Opposition to political benefits for government is stronger than anticipated, but outweighed by support.	Opposition to political benefits for government is equally balanced with support for same.	Opposition to political benefits for government outweighs small levels of support.

Note: Original table, substantially adapted from McConnell (2010a) Tables 3.1, 3.2 and 3.3; McConnell (2010b): Tables 1, 2 and McConnell (2012).

First, governments engage in processes to produce policy decisions. The policy sciences since its origins in Lasswell and others in the aftermath of World War II, and with ambitions for a ‘better society’, have been preoccupied with policy making processes, focusing on factors ranging from problem definition and consultation, to options appraisal and policy design (see e.g. Lasswell, 1956, 1971; Lerner and Lasswell, 1951) as well as deeper political issues on the issue of who holds power in policy formation processes (Bachrach and Baratz, 1970; Cobb and Ross, 1997). During the policy making process, therefore, governments may fail to achieve their intended goal of gaining authoritative approval for a particular policy initiative. Process failure can be dissected further in order to help us grasp key aspects of such failures, as well as providing us with standard criteria for assessment. Therefore, policy making process failure can comprise of policymakers to varying degrees being unable to fashion the type of policy they had hoped for, being considered illegitimate in terms of the processes used, being unable to build a sustainable coalition of support and attracting widespread criticism (and little or no support) for the process itself.

Second, governments produce policies (here I call them programmes, simply to avoid clashing with the broader phenomenon of policy failure). Such programmes, designed to address goals and underpinned by assumptions about appropriate levels of government intervention in society, may range from persuasive policy instruments such as public information campaigns, to financial subsidies, incentives and penalties, as well as the regulation of behaviour (Hood and Margetts, 2007; Howlett, 2010). Programme failure can be characterised by varying degrees of failure to be implemented as intended, achieve desired outcomes, benefit target groups, meet criteria which are highly valued in that policy domain (e.g. efficiency in public budgeting) and attract opposition to, and attract little or no support, for either the policy goals and/or the means of achieving them.

Third, governments ‘do’ politics, because amid the multiple conflicts in society over the making, shaping and enacting of public policies, they play powerful roles in inter alia shaping debates, managing conflicts, attending to the business of governing and establishing visions. Public policies can shape and be shaped by politics, from the careerism of public officials to the pursuit of ideologies. Governments, therefore, can fail to achieve their intended political outcomes, with impacts including reputational damage, out of control agendas, damage to core governance values and opposition to any small political benefits that may remain.

Although the process, programmatic and political aspects of policies are inextricably linked, it is useful to separate them analytically because doing so helps develope our understanding of some of the internal tensions of policy failure, with governments failing in some respects but not others. To explore this issue further, we need to explore the relationship between success and failure.

Degrees of failure

We know already that success is not ‘all or nothing’. Failure can occur in some of the three realms mentioned above but not others and/or can be a matter of degree, as well as being interspersed with success(es). We cannot capture such complex and intertwined phenomenon in scientific formula, given the role of judgement and interpretation in policy analysis (Wildavsky, 1987). However, Table 1 specifically identifies criteria (across the process, programme and political dimensions) to help us think about degrees of failure and their relationship to success. It is a refinement and adaption of the earlier McConnell (2010a, 2010b) framework and is more parsimonious in its categories (three sets of outcomes rather than five). I have also supplemented the original categories with ‘failure’ terminology. Doing so helps us grasps the real politick of failure, that some failures are survivable and others not, while failure in some realms may actually be a consequence of success in others. We can conceive of:

Tolerable Failure (=Resilient Success): Failure is tolerable when it does not fundamentally impede the attainment of goals that proponents set out to achieve, and opposition is small and/or criticism is virtually non-existent. In essence, tolerable failures are marginal features – a politically realistic ‘second best’ – of dominant and resilient successful outcomes.

Conflicted Failure (=Conflicted Success): Failures to achieve goals are fairly evenly matched with attainment of goals, with strong criticism and strong defence in roughly equal measure. In essence, conflicted failures are dogged by periodic controversy that is never quite enough to act as a fatal blow to the policy, but insufficient to seriously damage its defenders.

(Outright) Failure (=Marginal Success): A policy fails, even if it is successful in some minimal respects, if it does not fundamentally achieve the goals that proponents set out to achieve, and opposition is great and/or support is virtually non-existent. In essence, failures outweigh success and the policy is a political liability.

As indicated by the ‘art and craft’ argument, placing aspect of failure in these categories should be considered something of an intellectual mapping exercise involving judgement in order to get a sense of the forms, strengths and interconnections of failure. Very few policies will fit neatly into the same category but the weighing up what factors are/are not important, is part of the ‘art and craft’ of analysis.

Tensions between failures

There will always be unique configurations and elements of apparent randomness, but in the disciplines of political science and policy analysis, some of the most studied phenomena (such electoral preferences, path dependency, incremental policy change) concern patterns of behaviour, often stable over time. It would not be particularly surprising to find patterns of policy failure or tensions between them. As an analytical starting point, three such patterns can be flagged. All involve policy makers trading off different types of failure over time (prospective, present and retrospective) and at times preparing to ‘risk’ or tolerate one form of failure (or its possibility) in pursuit of ‘success’ in another realm of policy (Althaus, 2008).

Process success vs. programme/political failure

Government succeeds in the policy making phase of the policy cycle by getting authoritative approval for the decision or decisions it sought, but the very means of doing so (such as rushing a bill through a legislature, ignoring consultation feedback about potential implementation problems, marshalling evidence to legitimise the proposed policy) can create risks – which may come to fruition, that the programme fails in the implementation stage to achieve its goals, resulting in political backlash which proves unmanageable. Peace agreements are an example. Only about 50% survive five years and more, because the very circumstances that are conducive to compromise (lack of detail, ambiguity, vague aspirations) are the stuff of implementation nightmares and create huge space for each faction to withdraw if they feel politically and militarily vulnerable in comparison to the other (Bekoe, 2008).

Political success vs. programme failure

Colloquially, this would refer to ‘good politics but bad policy’. For example, government may succeed in perpetuating its governance ideas by initiating policy with a high placebo content, demonstrating that a policy is in place to tackle a particular ‘wicked problem’, but which fails to actually deliver on programme goals because of the complexity and intractability of problems with multiple individual, institutional and societal causes. Sharman (2011) in his study of AML policies demonstrates precisely this issue, arguing that a diffusion of westernised norms and policies in AML have spread rapidly, not because they solve the problems of criminals abusing financial systems, but because weaker, developing nations must appear modern and progressive in the face of international donor communities, the World Bank and the IMF. The small island of Naura (pop. 11,000) has adopted state-of-the-art AML policies, despite having no financial sector and no banks!

Programme success vs. political failure

Government succeeds in implementing unpopular programme measures, but leads to political failure. Austerity measures such as those in Greece, Spain and Ireland are a classic example, implemented with efficiency but producing damage to governments’ key political success aspirations, e.g. reputational protection, control of policy agendas and promotion of governance ideas.

Conclusion

Despite the standard assumptions of media headlines and much academic analysis, policy failures do not come neatly packaged with clear definitions. Policy failures are intensely political because of conflict over whether a particular set of policy outcomes constitutes failure, and what (if anything) caused failure in the first place. The very messiness and ambiguities of policy failure create space for the construction of often wildly different narratives throughout different stages of the policy cycle. Yet this does not mean to say that failures are purely a matter of perception. While it is important to recognise that disentangling ‘objective’ aspects of failure from ‘constructed’ factors, speaks to broader and ongoing debates at the heart of social and political sciences, it is difficult to avoid the conclusion the ‘real’ factors may also be at work in defining policy failure, such as promises that simply did not happen. Such matters cannot be resolved in one short paper, or perhaps not at all in the plural discipline of political science with multiple and legitimate assumptions and methodologies. Nevertheless, it is hoped that the framework and arguments presented in this article are useful primers to help analysts navigate the mysterious and unsteady ‘maze’ policy failure.

In this spirit I offer some pointers. Detailed examination of individual cases is clearly the way forward from here. The framework presented in this article can be used to map the characteristics of different policy outcomes. This exercise helps us dig deeper into single and multiple cases on issues such as causes, blame, decision making, learning and evaluation.

In terms of individual cases, the framework helps divide aspects of failure into meaningful analytical categories (process, programme, political) as well as degrees of failure (outright, conflicted, tolerable) that help provide the basis for further examination of dynamics. Questions could include: Did conflicted process failure (where government modified many of its original goals) lead to outright programme failure? Or why did outright programme failure lead only to tolerable political failure?

In terms of multiple cases, the framework allows for structured comparison. Questions could include: why did two ‘most similar’ programme failures lead to outright political failure in one case but only tolerable political failure in another? Or why did two ‘most different’ programme failures (outright and tolerable) both lead to similar political failure (ministerial resignation)?

Comparison over time is possible, within and across sectors. For example, why in policy sector X do different policy making processes with varying degree of failure throughout (modification to government goals, legitimacy of the process), always produce conflicted programme failure?

There are innumerable possibilities for applying the framework but an important point is that while failures pervade all policy sectors, they vary in type, magnitude and impact. Studying failure, therefore, is not simply a matter of studying outright failure. Studying failure through further empirical work means we need to find an analytical place for marginal and outright failures alike, across all aspects of what government chooses to do (or not to do). Only by doing so we can move further beyond the assumption that failure is a unitary, self-evident phenomenon that either does or doesn’t exist.

References

Althaus C (2008) Calculating Political Risk. Sydney: University of New South Wales Press.

Argyrous

(2009) Evidence for Policy and Decision-Making, Sydney: UNSW Press.

Bachrach

Baratz

(1970) Power and Poverty: Theory and Practice, New York, NY: Oxford University Press.

Barrett S and Fudge C (eds) (1981) Policy and Action: Essays on the Implementation of Public Policy. London: Methuen.

Bekoe

(2008) Implementing Peace Agreements: Lessons from Mozambique, Angola, and Liberia, New York, NY: Palgrave MacMillan.

Bell

Hindmoor

(2009) Rethinking Governance: The Centrality of the State in Modern Societies, Cambridge: Cambridge University Press.

Bovens M, ‘t Hart P and Peters BG (eds) (2001a) Success and Failure in Public Governance: A Comparative Analysis. Cheltenham: Edward Elgar.

Bovens M, ‘t Hart P and Peters BG (eds) (2001b) Patterns of governance: Sectoral and national comparisons. In: Success and Failure in Public Governance: A Comparative Analysis. Cheltenham: Edward Elgar, pp.593, 640.

Bovens

`t Hart

(1996) Understanding Policy Fiascoes, New Brunswick, NJ: Transaction.

10.

Brändström

Kuipers

(2003) From “Normal Incidents” to political crises: Understanding the selective politicization of policy failures. Government and Opposition 38(3): 279–305.

11.

Buchanan

(2010) Drug policy under new labour 1997–2010: Prolonging the war on drugs. Probation Journal 57: 250–262.

12.

Cobb RW and Ross MH (eds) (1997) Cultural Strategies of Agenda Denial: Avoidance, Attack and Redefinition. Lawrence, KS: University Press of Kansas.

13.

Davidson

(2005) Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation, Thousand Oaks, CA: Sage.

14.

deLeon

(1988) Advice and Consent: The Development of the Policy Sciences, New York, NY: Russell Sage Foundation.

15.

Dryzek

(2006) Policy analysis as critique. In: Moran

Rein

Goodin

(eds) The Oxford Handbook of Public Policy, Oxford: Oxford University Press, pp. 190–203.

16.

Dunleavy

(1995) Policy disasters: Explaining the UK's record. Public Policy and Administration 10(2): 52–70.

17.

Dye

(2012) Understanding Public Policy, 12th ed. Upper Saddle River, NJ: Pearson Prentice Hall.

18.

Dyson

(2006) Come Hell or High Water: Hurricane Katrina and the Color of Disaster, New York, NY: Basic Books.

19.

Edelman

(1988) Constructing the Political Spectacle, Chicago, IL: University of Chicago Press.

20.

Fischer

(2003) Reframing Public Policy: Discursive Politics and Deliberative Practices, Oxford: Oxford University Press.

21.

Friedman

(2011) Managing fear: The politics of homeland security. Political Science Quarterly 126(1): 77–106.

22.

Gray P (1998) Policy disasters in Europe: An introduction. In: Gray P and ‘t Hart P (eds) Public Policy Disasters in Western Europe. London: Routledge, pp. 3–20.

23.

Gray P and ‘t Hart P (1998) (eds) Public Policy Disasters in Western Europe. London: Routledge.

24.

Grossman

(2013) Wrong: Nine Economic Policy Disasters and What We Can Learn From Them, New York, NY: Oxford University Press.

25.

Gupta

(2001) Analyzing Public Policy: Concepts, Tools, and Techniques, Washington, DC: CQ Press.

26.

Gustafsson

(1983) Symbolic and pseudo policies as responses to diffusion of power. Policy Sciences 15(3): 269–287.

27.

Hall

(1982) Great Planning Disasters, Berkeley, CA: University of California Press.

28.

Hay

(2002) Political Analysis, Basingstoke: Palgrave Macmillan.

29.

Hogwood

Gunn

(1984) Policy Analysis for the Real World, Oxford: Oxford University Press.

30.

Hogwood

Peters

(1985) The Pathology of Public Policy, Oxford: Clarendon Press.

31.

Hood

(1976) The Limits of Administration, London: John Wiley.

32.

Hood

Margetts

(2007) The Tools of Government in the Digital Age, Basingstoke: Palgrave Macmillan.

33.

Howlett

(2010) Designing Public Policies: Principles and Instruments, Abingdon: Routledge.

34.

Howlett

(2012) The lessons of failure: Learning and blame avoidance in public policy-making. International Political Science Review 33(5): 539–555.

35.

Ingram HM and Mann DE (eds) (1980) Why Policies Succeed or Fail. Beverly Hills, CA: Sage.

36.

Kearns

Lawson

(2009) (De)constructing a policy ‘Failure': housing stock transfer in Glasgow. Evidence & Policy 5(4): 449–470.

37.

Kettl

(2004) System under Stress: Homeland Security and American Politics, Washington, DC: CQ Press.

38.

King

Crewe

(2013) The Blunders of Our Governments, London: Oneworld.

39.

Kingston J (2011) Ousting Kan Naoto: The Politics of Nuclear Crisis and Renewable Energy in Japan. The Asia-Pacific Journal: Japan Focus. Available at: http://japanfocus.org/-Jeff-Kingston/3610 (accessed 19 November 2014).

40.

Lasswell

(1956) The Decision Process, College Park, MD: University of Maryland Press.

41.

Lasswell

(1971) A Pre-View of Policy Sciences, New York, NY: American Elsevier.

42.

Lerner

Lasswell

(1951) The Policy Sciences: Recent Developments in Scope and Method, Stanford: Stanford University Press.

43.

Lewis

(2012) The failure to expand childcare provision and to develop a comprehensive childcare policy in Britain during the 1960 and 1970s. Twentieth Century British History 24(2): 249–274.

44.

Marsh

McConnell

(2009) Towards a framework for establishing policy success. Public Administration 88(2): 564–583.

45.

Marsh D. and Stoker G (eds) (2010) Theories and Methods in Political Science. 3rd ed. Basingstoke: Palgrave Macmillan.

46.

Matland

(1995) Synthesizing the implementation literature: The ambiguity – Conflict model of policy implementation. Journal of Public Administration Research and Theory 5(2): 145–174.

47.

Mazmanian DA and Sabatier PA (eds) (1981) Implementation and Public Policy. Lexington, MA: Lexington Books.

48.

McConnell

(2010a) Understanding Policy Success: Rethinking Public Policy, Basingstoke: Palgrave Macmillan.

49.

McConnell

(2010b) Policy success, policy failure and grey areas in-between. Journal of Public Policy 30(30): 345–362.

50.

McConnell

(2012) Learning from success and failure? In: Araral

Fritzen

Howlett

(eds) Routledge Handbook of Public Policy, Oxon: Routledge, pp. 484–494.

51.

Miller C and Riccio C (2011) Toward reducing poverty across generations: Early findings from New York City's Conditional Cash Transfer Program Society for Research on Educational Effectiveness. Available at: http://files.eric.ed.gov/fulltext/ED517881.pdf (accessed 29 January 2014).

52.

Pressman

Wildavsky

(1973) Implementation: How Great Expectations in Washington are Dashed in Oakland: Or, Why It's Amazing that Federal Programs Work At All, this Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes, Berkeley, CA: University of California Press.

53.

Rayner

(2012) Shifting mandates and climate change policy capacity: The forestry case. Canadian Political Science Review 6(1): 75–85.

54.

Schneider

Ingram

(1997) Policy Design for Democracy, Lawrence, Kansas: University Press of Kansas.

55.

Sharman

(2011) The Money Laundry: Regulating Global Finance in the Criminal Economy, Ithaca, NY: Cornell University Press.

56.

Stone

(2012) Policy Paradox: The Art of Political Decision Making, 3rd ed. New York, NY: W.W. Norton.

57.

Taylor D and Balloch S (eds) (2005) The Politics of Evaluation: Participation and Policy Implementation. Bristol: The Policy Press.

58.

Vedung

(2013) Six models of evaluation. In: Araral

Fritzen

Howlett

(eds) Routledge Handbook of Public Policy, Oxon: Routledge, pp. 387–400.

59.

Wallis

(2011) Avoiding Policy Failure: A Workable Approach, Litchfield Park, AZ: Emergent Publications.

60.

Walsh

(2006) Policy failure and policy change British security policy after the cold war. Comparative Political Studies 39(4): 490–518.

61.

Wildavsky

(1987) Speaking Truth to Power: The Art and Craft of Policy Analysis, 2nd ed. New Brunswick, NJ: Transaction.

62.

Zahariadis

(2003) Ambiguity and Choice in Public Policy: Political Decision Making in Modern Democracies, Washington, DC: Georgetown University Press.