Impact-based planning evaluation: Advancing normative criteria for policy analysis

Abstract

Planning decisions have considerable impacts on both natural and built environments. The impacts of these decisions may remain for many decades and many are irreversible. In order to gain a better understanding of these long-standing impacts, planners require a systematic approach to evaluate the planning policy instruments utilised. The literature on planning evaluation shows that most studies have taken a conformance-based evaluation approach, where the success of a planning policy instrument is based on the degree of conformity between the policy outcomes and its intended objectives. While evaluating such criteria is necessary, it is hardly ever sufficient largely because of unintended effects. This paper proposes an impact-based approach to planning evaluation that incorporates all the impacts, intended and otherwise, that a planning policy instrument may bring about, irrespective of the initial objectives of the policy. Using a number of economic and planning theories, this paper argues that, in addition to conformance and performance, other normative evaluation criteria, such as, efficiency, equity, social and political acceptability, and institutional arrangements, should be included to emphasise the importance of planning decisions and their substantial impacts on quality of life, social justice, and sustainability.

Keywords

Planning evaluation policy analysis welfare economics New Institutional Economics normative evaluation criteria

Introduction

Making plans for future activities is one of the basic human characteristics. Nonetheless, ‘it is difficult to forecast, particularly about the future’, according to Niels Bohr, Nobel laureate (Ellis, 1970: 431). Attempting to be certain about the future, which is inherently uncertain, seems to be a contradiction that lies at the heart of planning. Based on its nature, planning faces different social, political and economic uncertainties and planners hope to confront these uncertainties with their decisions. Using different instruments, planning policy can be applied and implemented in order to address planning objectives. Planning policies are the cluster of initiatives aimed at dealing with urban problems (Cochrane, 2007). These policies should be selected among various potential and possible alternatives. It is crucial for decision makers to know the effects that different potential policies might bring about (Alexander, 2012; Laurian et al., 2010), especially considering the fact that resources are not infinite and decisions about the use of them should be made wisely. Policy analysis and evaluation has been an established and inseparable concept in the planning tradition and literature (Alexander, 2002; Guyadeen and Seasons, 2016b; Khakee, 1998; Lichfield, 2001a). Evaluation has always been an intrinsic part of the planning process and it can be applied to any planning decision: Which policy instrument should be chosen in order to preserve historic landmarks? How to design a ‘good’ policy instrument to redirect future development potentials to more preferable areas? Planning as a decision-making process should be able to convincingly answer these types of questions.

To answer such questions, planning needs a comprehensive process and methodology for evaluation of its plans and policies from both theoretical and practical perspectives. Such an evaluation methodology essentially requires a set of normative criteria to enable planners to make ‘a real judgment’ and reduce the degree of subjectivity in the policy analysis process (Alexander and Faludi, 1989; Laurian et al., 2010; Lichfield, 2001b; Oliveira and Pinho, 2010a; Seasons, 2003). Alexander and Faludi (1989) argue that, in order to promote the credibility of planning as a discipline or a profession, planners have to use evaluation criteria. ‘The planning profession needs normative criteria to shape the plans it produces’ (Alexander, 2002: 191). Normative evaluation criteria enable planners to distinguish ‘good’ from ‘bad’ and to answer the normative questions concerning what make a plan/policy, a good plan/policy? Or how would we know a good plan/policy if we saw one? Without such normative criteria, policy analyses might be merely based on some subjective value judgments and planners cannot properly justify and validate the outcomes of their evaluations. Although some researchers have offered different criteria (Alexander, 2002; Alexander and Faludi, 1989; Baer, 1997; European Commission, 1999; Norton, 2005; Oliveira and Pinho, 2011), thus far, there has been no consensus about a set of appropriate criteria to assess the quality of planning and its products and processes (Oliveira and Pinho, 2010a). Baer (1997) believes that criteria formulation is not merely a checklist design, rather, it is a necessary skill of planning, as a profession, which any planners should develop, along with their other plan-making skills.

Through considering specified policy objectives as evaluation criteria, several studies on planning evaluation have conducted a conformance-based evaluation approach (Baer, 1997; Brody and Highfield, 2005; Brody et al., 2006; Laurian et al., 2004; Talen, 1997). According to this ‘conventional’ approach (Faludi, 1989), a policy is considered successful, if the outcomes of policy applications adhere and meet its specified objectives. However, the conformance-based evaluation fails to incorporate all the impacts a policy instrument might bring about. While the impacts of a policy instrument can be divided into intended effects (i.e. the premeditated intervention goals) and unintended effects (i.e. the serendipitous results or unanticipated side-effects), the ‘goal-attainment model’ or conformance-based evaluation approach does not take account of the latter type of effects (Vedung, 1997). Likewise, the performance-based evaluation, as an alternative approach, mainly focuses on the helpfulness of the policy in decision-making processes (Faludi, 2000), rather than considering its impacts. Despite the shortcomings of the conformance- and performance-based approaches, there has been little analysis on how planners can extend their evaluations to cover all the impacts of policy instruments, including both intended and unintended effects.

The objective of this paper is to address this gap by proposing an impact-based evaluation approach, whereby planners can evaluate the impacts of planning policy instruments using a set of normative criteria. Informed by the planning literature and economic theories (e.g. welfare economics and New Institutional Economics (NIE)), this paper suggests effectiveness, performance, efficiency, equity, acceptability, and institutional arrangements, as six normative criteria, for planning policy analysis and evaluation. The paper aims to enhance the planning evaluation theory and practice by proposing a more holistic approach that incorporates both intended and unintended impacts of policy instruments. To this end, firstly we will review the literature on planning policy analysis and evaluation, before giving particular consideration as to how planning evaluation can benefit from the extensive literature of welfare economics and NIE. The paper, then, goes on to propose the impact-based evaluation approach. Finally, it discusses the implications of suggested normative criteria for planning policy analysis and evaluation.

Planning policy analysis and evaluation

Similar to any other public policy discipline, planning has developed its own literature concerning policy analysis and evaluation over recent decades (Guyadeen and Seasons, 2016a). However, some researchers argue that this literature faces a number of challenges or shortcomings, including, inter alia, the increasing gap between evaluation theory and practice (Khakee, 2003), the dominance of ex-ante evaluations over ongoing and ex-post evaluations (Carmona and Sieh, 2004; Lichfield, 2001b), the need for a set of normative evaluation criteria (Alexander, 2002), and the dominance of evaluating planning outputs (i.e. policies, plans and programs) rather than impacts and outcomes (Baum, 2001; Laurian et al., 2010). In this section, we review this literature in terms of the evolution, approaches, and methodologies of planning evaluation.

Evolution of planning evaluation

Planning evaluation has changed during the last few decades. It is argued that these changes have been closely associated with the development of planning theory and practice (Khakee, 1998; Oliveira and Pinho, 2010a) as well as public policy literature. Nonetheless, there is no consensus about the exact nature of this process of shift. Innes (1995) describes eight different theoretical planning positions, namely, rational-comprehensive planning, incremental planning, advocacy planning, implementation-oriented planning, strategic planning, transactive planning, negotiative planning, and communicative planning. Khakee (2003) categorises these positions into two distinct paradigms of rational planning and communicative planning and discusses the relevance of these paradigms to both planning theory and evaluation. He analyses the first five positions within the rational paradigm and considers the last three positions, as alternatives leading to communicative planning theory. In line with instrumental rationality, rational planning seeks a relationship between goal achievement and the usage of resources. On the other hand, within an institutional context, communicative planning aims to obtain a consensus among all stakeholders and shape a platform to promote a learning process (Khakee et al., 2008). Similar paradigm shifts are introduced by Forrester (1989), Healey (1996), and Sager (1994). Despite the differences between these paradigms, some authors highlight the complementary nature of them, rather than their conflicts (Alexander, 1998, 2000; Oliveira and Pinho, 2010b). Faludi (2000, 2006) believes that planning can be seen as both a technical exercise and a learning process. Rational paradigm is a suitable model for the technical approach, while for the learning process, new approaches are required. Lichfield (1998, 2001a), however, argues that there is no need to substitute the rational approaches since the rationality will always be associated with planning theory and evaluation.

Planning evaluation approaches: Conformance vs. performance

In the planning evaluation literature, there are two conceptions of success in plan implementation, namely, the conformance-based and performance-based evaluations (Berke et al., 2006; Faludi, 1989; Fudge and Barrett, 1981; Oliveira and Pinho, 2010a). The divergence between these two evaluation approaches has its roots in their different assumptions about the plan/policy functions and purposes (Laurian et al., 2004). Rationality in planning is the basic assumption of the conformance-based approach and plans/policies are seen as blueprints for future development. This approach defines success or failure of a policy instrument using the degree of conformity between policy outcomes and its specified goals. Focusing on the linkage between policy and actual development, this approach considers a policy instrument to be successfully implemented, only if the development patterns of the targeted area adhere to the intended policy objectives (Berke et al., 2006; Guyadeen and Seasons, 2016a; Laurian et al., 2004). This approach can be considered consistent with Sabatier’s (1986) ‘top-down’ evaluation, in which the outputs of a plan/policy should precisely conform to its objectives. In contrast, the performance-based approach focuses on planning processes and considers plans/policies as decision frameworks (Alexander, 2009). Plan/policy in this approach is a guide for the future decision-making process, rather than a blueprint. The performance-based approach considers planning as an ever-changing process which faces significant uncertainties. Thus, it defends the departure from the plan/policy, if it is necessary and rational. In this situation, if the implemented policy instrument deviates from its proposals, the instrument still might be considered implemented, subject to have rational reasons. In fact, a plan/policy is considered successful, if it is frequently used or consulted in decision-making processes (Alexander and Faludi, 1989; Baer, 1997; Faludi, 1987; Laurian et al., 2004; Mastop and Faludi, 1997).

Planning evaluation methodologies and criteria

In order to evaluate planning and plan implementation in a systematic way, some researchers have proposed different evaluation methodologies. Among those, we can refer to Policy-Plan/Program-Implementation-Process (PPIP) Model (Alexander and Faludi, 1989), Means for Evaluating Actions of a Structural Nature (MEANS) (European Commission, 1999), Plan Implementation Evaluation (PIE) (Laurian et al., 2004), Norton’s Evaluation Model (Norton, 2005), and Plan-Process-Results (PPR) (Oliveira and Pinho, 2011). The key elements of these methodologies are their proposed evaluation criteria. Nonetheless, the proposal of such criteria does not appear to have a strong theoretical foundation and their selection is not always justified properly by researchers based on their underpinning theories. In other words, despite the attempts to propose a range of criteria, these evaluation methodologies do not make sufficient reference to the origins and theoretical underpinnings of evaluation criteria in order to appropriately locate them in the evaluation process. Table 1 presents these evaluation methodologies along with their proposed criteria for evaluating plans and policies.

Table 1.

Planning evaluation methodologies and criteria.

Researcher(s)	Methodology	Evaluation criteria
Alexander and Faludi (1989)	Policy-Plan/Program-Implementation-Process (PPIP) Model	Conformity – Rational process – Optimality ex-ante – Optimality ex-post – Utilisation
European Commission (1999)	Means for Evaluating Actions of a Structural Nature (MEANS)	Relevance – Effectiveness – Efficiency – Utility Clarity of the objectives – Internal coherence of the objectives – External coherence of the objectives
Laurian et al. (2004)	Plan Implementation Evaluation (PIE)	Implementation breadth – Implementation depth
Norton (2005)	Norton’s Evaluation Model	Local elected officials’ commitment to planning – Overall plan quality – Plan implementation (plan use) – Local elected officials’ policy trade-off preferences – Local plan policy emphasis – Plan use emphasis
Oliveira and Pinho (2011)	Plan-Process-Results (PPR)	Internal coherence (of the plan) – Plan relevance to the city needs and ambitions – (Plan) interpretation of the planning system – External coherence – Public participation in plan making and in plan implementation – Plan utilisation in decision making – Commitment of human and financial resources – Effectiveness (plan – results) – Direction for the urban development process

Impact-based planning evaluation: Towards a more comprehensive policy analysis

The planning evaluation literature indicates that: (1) the ongoing and ex-post evaluations have had a rather ‘reduced expression’ and a ‘marginal role’ in planning, in comparison to the ex-ante evaluation (Lichfield, 2001b; Oliveira and Pinho, 2010a), (2) the main focus in the ex-post evaluation researches has been on evaluating planning outputs (i.e. plans, policies, and programs), rather than planning outcomes (i.e. impacts of implemented plan/policy) (Baum, 2001; Laurian et al., 2010), and (3) most outcome evaluation studies have been based on the conformance-based approach (e.g. Berke et al., 2006; Laurian et al., 2004; Talen, 1997). The conformance-based approach has been criticised for providing a rather biased and tunnel-vision image of real impacts of an implemented policy instrument. ‘Were the evaluators to confine themselves exclusively to researching the achievement of premeditated intervention goals, any serendipitous results or unanticipated side-effects would not be included in the main evaluation process’ (Vedung, 1997: 45). In other words, a conformance-based approach fails to evaluate all impacts that a policy instrument might bring about, and instead, it focuses merely on the predetermined policy objectives, which might be only partial effects of an implemented policy instrument. According to Mickwitz (2013), although incorporating policy objectives into evaluation is helpful and necessary, because of their importance in the decision-making process, they are not always sufficient. He discusses that this insufficiency is to some extent due to the existence of side-effects, and also the fact that the policy environment has a dynamic and ever-changing nature which can make defined policy objectives ‘irrelevant at a later stage’. In addition, the policy objectives might be drafted based on incorrect assumptions (Hoogerwerf, 1990) and incomplete information.

Some scholars have attempted to provide responses to the shortcomings of the conformance-based evaluation approach. Vedung (1997), for instance, proposes a ‘side-effects evaluation’ approach, which divides the effects of a policy into the main effects and side-effects. A policy effect can either occur inside or outside the intervention target area. The effects inside the target area are called intended main effects, whereas the effects outside the target area are considered side-effects. In this approach, while the main effects are positively valued by policy analysts, the side-effects are not necessarily negative or positive. Although this approach attempts to complement the conformance-based evaluation approach by searching outside of the target area for side-effects, it retains the policy objectives as the main evaluation criteria. On the other hand, Strategic Environmental Assessment (SEA) is an evaluation methodology that does not limit itself to the policy objectives, and instead, attempts to incorporate all consequences of a proposed project or policy on the environment. However, SEA is mainly focused on the environmental effects of a project/policy and does not provide a comprehensive understanding of all impacts (Short et al., 2013; Therivel, 2012). Despite these discussions, there have been a lack of studies on how planners can conduct a more comprehensive policy analysis.

Acknowledging this gap in the planning evaluation literature, we identify the need for a more holistic approach in planning policy analysis. To address this gap, we propose an impact-based evaluation approach which aims to incorporate all the impacts that a planning policy instrument may bring about. Proposing a set of normative evaluation criteria, which are based on their underpinning theories, is central in this impact-based evaluation approach. Besides conformance and performance, as two criteria widely used in the planning evaluation literature, we argue that four normative evaluation criteria, namely, efficiency, equity, acceptability, and institutional arrangements, should also be included in any planning evaluations. In proposing these evaluation criteria, we pay particular attention to their underpinning theories, in which the first two criteria (i.e. efficiency and equity) are derived from the welfare economic theory and the last two criteria (i.e. acceptability and institutional arrangements) have their roots in NIE theory. We believe that applying such normative criteria enables planners and policy analysts to comprehensively evaluate the impacts of planning policy instruments on people, and both natural and built environments. Before discussing the suggested criteria in details, the theories of welfare economics and NIE, and their insights into policy analysis, will be presented in the following sections.

Welfare economics insights into policy analysis

As one of the main foundations of policy analysis, welfare economic theory can offer considerable insights into planning evaluation. It is argued that applying welfare economic theory is essential for conducting a useful and appropriate policy analysis (Just, 1988; Just et al., 2004). Welfare economics addresses issues surrounding how best to allocate resources to maximise societal well-being (Ng, 2004). It applies mainly microeconomic techniques to evaluate well-being. The formulated propositions of welfare economics enable economists and policy analysts to determine the level of social welfare in different economic situations, as well as to examine effects of various policy proposals on the well-being of individuals or groups. It elucidates the role of planning and clarifies that the objective of any interventions is to improve quality of life. The welfare economic theory consists of two fundamental theorems. The first theorem of welfare economics states that the price system within a perfectly competitive market brings the economy to a socially optimal state through inducing self-interested individuals to independently maximise their private well-being (Feldman, 2008). A socially optimal or Pareto-efficient situation is a state that is impossible to make anybody better off without making any other individual worse off (Johansson, 1991). Due to market failure, however, the free market outcome is not necessarily socially optimal. These market failures include, inter alia, externalities, public goods, information asymmetries, and inadequate assignment of property rights. Acknowledging the market failures, the second theorem offers a rationale for the necessity to intervene in the market. Addressing these problems through the second theorem, welfare economic theory supports intervention to improve the efficiency and equity of outcomes. Planning literature sometimes refers to welfare economics as a classic justification of planning (Alexander, 2001; Klosterman, 1985; Mills, 1976; Moore, 1978). In other words, the welfare economic theory justifies the existence of planning and the necessity for interventions where market failure is a reality. Intervention for efficiency and equity is discussed as the main reasons for intervention in welfare economics (Evans, 1985). Therefore, presented by the welfare economic theory, efficiency and equity can be considered as two fundamental criteria that can provide a theoretical basis for planning evaluation. Such a theoretical basis may ultimately create opportunities to improve the design and implementation of planning policy instruments and achieve more efficient and fairer outcomes. Planners are required to evaluate their policy instruments in terms of efficiency and equity, given the implementation of different policy instruments results in different degrees of efficiency and equity.

New Institutional Economics insights into policy analysis

While welfare economic literature provides policy analysis with a theoretical basis, it has a number of weaknesses. In particular, welfare economic theory does not pay adequate attention to the institutional aspects of policy design and analysis (Adams et al., 2008; Furubotn and Richter, 2005). New institutional economists criticise welfare economics literature for ignoring transaction costs, property rights, and institutional systems (Rutherford, 1996). Emerging from Coase’s (1937) seminal article ‘The Nature of the Firm’, NIE is a relatively new branch of economics which focuses and acknowledges the important role of institutions in social and economic activities. ‘The central message of the NIE is that institutions matter for economic performance’ (Furubotn and Richter, 2005: 1). NIE is concerned with defining, understanding and explaining institutions and their effects on economic effectiveness, efficiency, and distribution (Nabli and Nugent, 1989). Although NIE acknowledges the necessity for government interventions, it is sceptical about the ability of government to address market failures in an efficient and equitable manner. In line with public choice theory, NIE argues that state intervention can be distorted by power holders, lobby groups, and actors who know how to ‘work the system’. In other words, government officials may show opportunistic behaviours or might attempt to work for the interests of pressure groups, rather than maximising social welfare or quality of life. Moreover, the knowledge and information of decision makers are severely limited and people are rationally bounded (Adams et al., 2008; Williamson, 1993). Taking government failure into account, NIE suggests that government might be able to resolve market failures in a more effective and efficient manner through creating stronger property rights and designing better institutional arrangements to reduce uncertainties and transaction costs within markets. This view provides policy analysts with useful insights on the role of institutions in the process of collective decision-making and the use of ‘transaction costs’ concept as a mediating tool (Samuels, 1995). This way, NIE highlights the importance of institutional design and arrangement and social and political acceptability in the application of any public policy.

Normative evaluation criteria

As discussed above, planners and policy analysts are required to apply normative evaluation criteria, if they aim to make ‘a real judgment’ about their actions and policies. In this paper, we posit that ‘efficiency’ and ‘equity’, as two fundamental criteria suggested by welfare economic theory, can be used as a basis for planning evaluation and policy analysis. However, we argue that welfare economic literature on policy analysis has a number of shortcomings. First, it does not offer a strong framework to incorporate the issues surrounding processes and procedures of policy design and implementation. In other words, this literature does not identify any process through which policy instruments can be implemented to achieve efficient and equitable outcomes. We suggest that ‘performance’, as a criterion presented by planning evaluation literature, can be used to address the first shortcoming of welfare economics. We discuss that the planning theory has paid considerable attention and has a strong tradition on these very issues of process (Alexander and Faludi, 1989; Faludi, 2000; Healey and Barrett, 1990). Second, welfare economic theory fails to realise the reality of transaction costs and other institutional dimensions. To address this shortcoming, we discuss that NIE provides policy analysis with helpful insights from institutional perspectives. Incorporating these institutional aspects into planning policy analysis is particularly important, since one of the major tasks of planners is argued to be minimising transaction costs (Alexander, 1992; Buitelaar, 2007; Dawkins, 2000; Shahab et al., 2017). Therefore, as outlined in Table 2, this paper presents a set of normative criteria for evaluating planning policy instruments. These evaluation criteria will be discussed in more details in the following sections.

Table 2.

A set of normative criteria for evaluating planning policy instruments.

Underpinning theory	Evaluation criteria		Description
Planning evaluation	Effectiveness (conformance)		The degree to which the outcomes of a policy meet the predetermined objectives.
Planning evaluation	Performance		The degree to which a policy is used or considered in decision-making processes.
Welfare economics	Efficiency	Static	Concerned with the best allocation of resources at a given point in time.
	Efficiency	Dynamic	Concerned with the best allocation or the most beneficial use of resources over a period of time.
	Equity	Intragenerational	Concerned with equity between people of the same generation.
	Equity	Intergenerational	Concerned with equity between people of the different generations.
New institutional economics	Acceptability	Social	The degree to which the design and implementation of a policy is supported by affected individuals and groups.
	Acceptability	Political	The degree to which the design and implementation of a policy is supported by decision-makers and decision-takers.
	Institutional arrangements	Administrative feasibility	Concerned with the administrative capacity and the ability of policy administrators to implement and administer a policy.
	Institutional arrangements	Transaction costs	Concerned with the costs that are involved in exchanges or transactions, other than the production costs.

Effectiveness (Conformance)

Effectiveness or conformance¹ can be defined as ‘attaining the specific objectives set and achieving the intended results’ (CEC, 2008: 40). Effectiveness concerns the degree to which the objectives formulated in policy instruments were achieved or are expected to be achieved. There are a number of reasons for using effectiveness as an evaluation criterion. First, it promotes the links between planning policy analysis and intentions of planners in proposed policy instruments. In other words, incorporating effectiveness into planning evaluations reflects the basis and principles of policy instruments and intents of practitioners (Laurian et al., 2004). Second, proving the effectiveness of a policy instrument enables planners to defend the ‘credibility of planning’ as a discipline or a profession, especially when decision-makers are required to show the positive results of their activities. Third, it reflects the accountability and responsibility that planners have regarding their actions and policies. And finally, it provides planners with a more applicable evaluation method for day-to-day planning decisions (Oliveira and Pinho, 2010a). In fact, effectiveness can be used as a helpful indicator of success or failure, particularly in ongoing evaluations, where planners have the opportunity to review and modify their policy instruments and strategies.

Performance

Performance can be defined as the usefulness of a policy in decision-making process. It concerns whether, how, and in what conditions the policy was consulted or used in planning processes (Alexander and Faludi, 1989; Mastop and Faludi, 1997). Performance refers to the ‘application’ of plans or policies, rather than their implementation. According to Faludi (2000: 306), a plan or a policy instrument can be considered as ‘performed’, ‘if and only if it plays a tangible role in the choices of the actors to whom it is addressed.’ Incorporating performance into planning policy analysis provides planners with a basis to evaluate the policy instruments as part of a learning process and not merely a technical exercise. In other words, it adds the concept of ‘process’ in policy analysis and acknowledges the dynamic process of achieving socially preferable outcomes. It also helps to gain a better understanding of what happens to the policy. However, unlike effectiveness, evaluating the performance of policy instruments that focus on structural and long-term decisions is not always applicable in day-to-day planning decisions.

Efficiency

Efficiency is associated with maximisation of the result and minimisation of the waste. Efficiency concerns whether the outcomes of a policy are achieved at the lowest cost or whether better outcomes could be obtained at the same cost. Effectiveness should not be confused with efficiency. In simple terms, the former is doing the right things, the latter concerns doing things right. Among the various meanings of efficiency, the definition suggested by Vilfredo Pareto is one of the most widely used definitions (Kirman, 2008; Lockwood, 2008). He argues that an allocation of resources is optimal, if it is impossible to make anyone better off without making someone else worse off. Although Pareto uses the word optimal, this definition is considered as a definition of efficiency that is often called Pareto-efficiency (Lockwood, 2008; Reiter, 2008). There are two types of efficiency, namely, static and dynamic (Clinch and O'Neill, 2010). Static efficiency is concerned with the best allocation of the resources at a given point in time. This type of efficiency, which is sometimes called cost-efficiency, addresses the best use of given resources or achieving optimality at least cost. On the other hand, dynamic efficiency concerns the best allocation or the most beneficial use of resources over a period of time. Instead of focusing on a short-term optimality, a dynamic approach is concerned with a long-term efficiency. Therefore, dynamic efficiency is essentially associated with continuous innovation and investment to improve systems and processes. In selecting, designing and implementing policy instruments, planners need to consider not only statically efficient solutions (least cost), but also the decisions that promote dynamic efficiency through providing incentives to encourage continuous improvements.

Equity

Efficiency and equity are two fundamental concepts that can be seen as twin criteria in policy analysis from the perspective of welfare economic theory. Due to the strong connections between these two concepts, they should not be evaluated separately. Paradoxically, many policies, supposedly designed to increase the level of efficiency, often focus on equity aspects and vice versa (Lesser et al., 1997). Efficiency, as discussed, is associated with maximising the outcomes, subject to availability of resources, whereas equity concerns the distribution of resources, goods, and services among individuals. Given a certain level of resources, policy makers may choose a very efficient distribution of resources and incomes. However, this might not necessarily be very equitable in the way its outcomes are distributed. Equity should not be confused with equality. While the former refers to the qualities of justness, fairness, impartiality, and evenhandedness, the latter concerns sameness and equal sharing (Bronfenbrenner, 1973). In other words, equitable means fair, but does not necessarily means equal. Most societies view equity as an important objective, due to its links with social justice. While there is an agreement to decrease inequity in any society, equity has a wide variety of perceptions among different people and societies. Despite some scholars having attempted to answer what a fair distribution is (Nozick, 1974; Rawls, 1971), there is no consensus about the exact nature of equity, largely because as an ethical and subjective concept, it is highly dependent on and intertwined with the norms, values, and cultures of a society (Bronfenbrenner, 1973; Coleman, 2008).

There are two principal types of equity, namely, intragenerational and intergenerational. While intragenerational equity concerns equity between people of the same generation, intergenerational equity addresses equity between both members of the present generation and other generations (i.e. past and future) (Weiss, 1990). Both types of equity are inherently related to the concept of sustainable development, which is one of the fundamental objectives of planning. The implementation of any policy may have different effects on different people of the same generation. Nevertheless, although some groups might be winners and others might be losers of an intervention (Clinch and Murphy, 2001), policy makers cannot always properly identify who will benefit from a given policy instrument and who will not, largely due to imperfect information (Just, 1988; Runge and Myers, 1985). On the other hand, planning policy decisions might have considerable effects on generations that are not involved in the decision-making process. Therefore, it is inevitable for planners to take different aspects of equity into account in designing and analysing their policy instruments.

Acceptability

NIE claims that Pareto optimality is an artificial construct that cannot normally be achieved in real life. Lipsey and Lancaster (1956) argue that in a situation which all the necessary conditions to achieve Pareto optimality do not exist, a ‘second-best’ approach is a more likely outcome. Many factors account for the failure to implement first-best approaches, however one of the main reasons concerns the social and political acceptability of interventions (Clinch et al., 2008). In other words, the low level of acceptance or unpopularity of a policy can be considered a constraining factor (Burton et al., 2013; Schade and Schlag, 2003; Wüstenhagen et al., 2007) in applying the first-best solutions. Acceptability can be defined as the degree to which the design and implementation of a policy is supported by affected people (social acceptability), as well as decision-makers (political acceptability). The issues surrounding equity, perceived effectiveness, coerciveness, and social norms are argued in some studies to be the determining factors in the level of social and political acceptability of any policy (de Groot and Schuitema, 2012; Eriksson et al., 2006; Jaensirisak et al., 2005). The degree of social and political acceptance that is required for a policy instrument to be selected or implemented in different jurisdictions might vary widely, depending on their different legal and political systems, and institutional arrangements.

Institutional arrangements

Institutions, as the ‘rules of the game’, are created to decrease uncertainties in human interactions (North, 1990). The creation, use and change of institutions are associated with transaction costs and require proper arrangements. An institutional arrangement can be defined as an arrangement among different agents ‘that governs the ways in which [they] can cooperate’ (Davis and North, 1970: 133). Different institutional design and arrangements of a policy might lead to different policy outcomes, in terms of its effectiveness, efficiency, equity, and transaction costs. In analysing the institutional arrangements of any policy instrument, policy analysts need to pay particular attentions to their administrative feasibility and transaction costs. While administrative feasibility concerns the administrative capacity and the ability of policy administrators to implement and administer a policy, transaction costs are concerned with the costs that are involved in exchanges or transactions, other than the production costs. Considering institutional aspects of planning policy instruments enables planners to gain a better understanding of policy design and administration practicalities, such as, how to design a policy that is easier to administer, and more easily understandable and less costly for those who wish to participate. It also highlights the importance of the cultural, political and legal contexts in which a policy is implemented and the impact of these specificities on the policy outcomes (Coggan et al., 2010; Falconer and Saunders, 2002; McCann et al., 2005).

Summary and conclusions

Despite an increasing number of studies and contributions in planning evaluation (Alexander, 2012; Khakee et al., 2008; Talen, 1997), it is argued that planning policy analysis and evaluation is still a rather ‘unexplored’ area in some respects (Guyadeen and Seasons, 2016b; Oliveira and Pinho, 2011) and that there are considerable shortcomings. Since plans and plan-making processes have a special place in the professional practices of planners (Balsas, 2012), there has been a dominance of the ex-ante evaluations in planning literature (Lichfield, 2001b). For example, planners are particularly interested in devising methodologies in order to compare different alternatives and scenarios before their implementation. As a result, there has been reluctance to conduct ongoing or ex-post evaluations. In addition, emphasis is given to the conformity between the policy outcomes and its intended objectives. Such a conformance approach fails to study all the impacts a policy instrument might bring about. As the well-known philosopher Popper (2003: 104–105) puts it, ‘although we may learn to foresee many of the unintended consequences of our actions … there will always be many we did not foresee … [thus,] the main task of the social sciences … is the task of analysing the unintended social repercussions of intentional human actions.’ Furthermore, in an uncertain world and without an extraordinary amount of information, policy objectives are unlikely to be set in a totally accurate, precise, and relevant manner. Therefore, there is a clear need for planners to depart from using conformance as the sole criterion for the evaluation of their policy instruments. Rather, a set of normative evaluation criteria, informed by their underpinning theories, would facilitate a more comprehensive study of the impacts of their actions and policies.

This paper is an attempt to address this gap in planning evaluation through proposing an impact-based evaluation approach, which can be applied as an alternative to existing approaches. The impact-based approach can be used in different types of evaluation, namely, ex-ante, ongoing and ex-post evaluations. A set of normative evaluation criteria, which are informed by their underpinning theories, lies at the heart of this proposed impact-based evaluation approach. These criteria include effectiveness, performance, efficiency, equity, acceptability, and institutional arrangements. These evaluation criteria are interrelated, in which each criterion has an effect on, or depends on, the other criteria. For example, re-allocation or re-distribution of resources can have considerable effects on efficiency. Stiglitz (1985: 31) posits that because of imperfect information ‘the separation between equity and efficiency considerations is no longer generally valid.’ On the other hand, evaluating efficiency in isolation from equity considerations is almost impossible in democratic systems, largely because of the issues surrounding social and political acceptability (Just, 1988). Likewise, institutional design and arrangements of a policy might have significant impacts on effectiveness, efficiency, and equity of any policy instruments. Therefore, this paper advocates for planners and policy analysts to take the interdependence of these normative criteria into account in order to gain a better understanding of policy impacts as well as potential trade-offs between criteria. While this study attempts to provide a theoretical basis for a more comprehensive planning policy analysis, more work will need to be undertaken to empirically apply the proposed normative evaluation criteria so as to evaluate different planning policy instruments.

Footnotes

Acknowledgements

We would like to thank the two anonymous reviewers for their positive comments. The second and third co-authors are listed alphabetically.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Irish Research Council (IRC) Government of Ireland Postgraduate Research Scholarship Fund.

Note

Sina Shahab is a PhD candidate in Planning and Environmental Policy at University College Dublin where he is also an occasional lecturer for several modules in urban planning and environmental policy. During his PhD studies, Sina was a visiting scholar for six months at the University of Maryland, USA. He previously graduated from the University of Tehran with a Master’s degree in Urban and Regional Planning. Sina’s main research interests lie in planning and environmental policy analysis, specifically, market-based policy instruments, transaction costs, and new institutional economics. Prior to his research career, Sina spent some years in professional planning consultancy.

J Peter Clinch is Jean Monnet Professor and Chair of Public Policy at University College Dublin, Chair of National Competitiveness Council of Ireland and Member of the National Advisory Council on Climate Change. He previously was chief policy adviser to the Irish Prime Minister and Vice-President. He has authored over 100 publications including books, book chapters and journal articles as well as 100 other publications. Peter’s principal research contributions relate to the measurement of, and factors determining, societal welfare; definition and measurement of ‘value' in this context; how markets fail and how policy instruments can be used to correct them.

Eoin O'Neill is a tenured Lecturer in Environmental Policy at University College Dublin and a Fellow and executive member of the UCD Earth Institute. He was awarded a PhD in planning and economics in 2007. He previously worked as Technical Specialist in Flood Risk Management in the UK Environment Agency as well as gaining experience in planning practice. Eoin is a chartered planner of the Royal Town Planning Institute (MRTPI). His research interests concern human relations with the natural and built environments, under the broad themes of incentives and regulation of land-use externalities, and risk perception and behaviour.

References

Adams D, Dunse N and White M (2008) Conceptualising state–market relations in land and property: The growth of institutionalism – Extension or challenge to mainstream economics? In: Planning, Public Policy & Property Markets. Oxford: Blackwell Publishing Ltd.

Alexander

(1992) A transaction cost theory of planning. American Planning Association. Journal of the American Planning Association 58: 190–200.

Alexander

(1998) Where do we go from here: Evaluation in spatial planning in the post-modern future. In: Nathaniel Lichfield

Dino Borri

Anna

(eds) Evaluation in Planning: Facing the Challenge of Complexity, Dordrecht, Netherlands: Kluwer.

Alexander

(2000) Rationality revisited: Planning paradigms in a post-postmodernist perspective. Journal of Planning Education and Research 19: 242–256.

Alexander

(2001) A transaction-cost theory of land use planning and development control: Towards the institutional analysis of public planning. The Town Planning Review 72: 45–75.

Alexander

(2002) Planning rights: Toward normative criteria for evaluating plans. International Planning Studies 7: 191–212.

Alexander

(2009) Dilemmas in evaluating planning, or back to basics: What is planning for? Planning Theory & Practice 10: 233–244.

Alexander

(2012) Evaluation in Planning: Evolution and Prospects, Hampshire: Ashgate Publishing, Ltd.

Alexander

Faludi

(1989) Planning and plan implementation: Notes on evaluation criteria. Environment and Planning B: Planning and Design 16: 127–140.

10.

Baer

(1997) General plan evaluation criteria: An approach to making better plans. Journal of the American Planning Association 63: 329–344.

11.

Balsas

(2012) What about plan evaluation? Integrating evaluation in urban planning studio's pedagogy. Planning Practice & Research 27: 475–494.

12.

Baum

(2001) How should we evaluate community initiatives? Journal of the American Planning Association 67: 147–158.

13.

Berke

Backhurst

Day

et al. (2006) What makes plan implementation successful? An evaluation of local plans and implementation practices in New Zealand. Environment and Planning B: Planning and Design 33: 581–600.

14.

Brody

Highfield

(2005) Does planning work? Testing the implementation of local environmental planning in Florida. Journal of the American Planning Association 71: 159–175.

15.

Brody

Highfield

Thornton

(2006) Planning at the urban fringe: An examination of the factors influencing nonconforming development patterns in southern Florida. Environment and Planning B: Planning and Design 33: 75–96.

16.

Bronfenbrenner

(1973) Equality and equity. The Annals of the American Academy of Political and Social Science 409: 9–23.

17.

Buitelaar

(2007) The Cost of Land Use Decisions: Applying Transaction Cost Economics to Planning and Development, Oxford, UK: Blackwell Publishing.

18.

Burton

Jenks

Williams

(2013) Achieving Sustainable Urban Form, London: Routledge.

19.

Carmona

Sieh

(2004) Measuring Quality in Planning: Managing the Performance Process, London: Routledge.

20.

CEC (2008) Financial Regulation and Implementing Rules Applicable to the General Budget of the European Communities, Luxembourg: Office for official Publications of the European Communities.

21.

Clinch

Murphy

(2001) Modelling winners and losers in contingent valuation of public goods: Appropriate welfare measures and econometric analysis. The Economic Journal 111: 420–443.

22.

Clinch

O'Neill

(2010) Designing development planning charges: Settlement patterns, cost recovery and public facilities. Urban Studies 47: 2149–2171.

23.

Clinch

O'Neill

Russell

(2008) ‘Pure’ and ‘impure’ Coasian solutions in planning. TPR: Town Planning Review 79: 623–649.

24.

Coase

(1937) The nature of the firm. Economica 4: 386–405.

25.

Cochrane

(2007) Understanding Urban Policy: A Critical Approach, Oxford: Blackwell Publishing.

26.

Coggan

Whitten

Bennett

(2010) Influences of transaction costs in environmental policy. Ecological Economics 69: 1777–1784.

27.

Coleman

(2008) Equality. In: Durlauf

SNAB

Lawrence

(eds) The New Palgrave Dictionary of Economics, London: Palgrave Macmillan.

28.

Davis

North

(1970) Institutional change and American economic growth: A first step towards a theory of institutional innovation. The Journal of Economic History 30: 131–149.

29.

Dawkins

(2000) Transaction costs and the land use planning process. Journal of Planning Literature 14: 507–518.

30.

De Groot

JIM

Schuitema

(2012) How to make the unpopular popular? Policy characteristics, social norms and the acceptability of environmental policies. Environmental Science & Policy 19–20: 100–107.

31.

Ellis

(1970) Teaching and Learning Elementary Social Studies, Old Tappan: Allyn and Bacon.

32.

Eriksson

Garvill

Nordlund

(2006) Acceptability of travel demand management measures: The importance of problem awareness, personal norm, freedom, and fairness. Journal of Environmental Psychology 26: 15–26.

33.

European Commission (1999) MEANS Collection – Evaluating Socio-Economic Programs, Luxembourg: Official Publications Office of the European Communities.

34.

Evans

(1985) Urban Economics: An Introduction, Oxford: Blackwell.

35.

Falconer

Saunders

(2002) Transaction costs for SSSIs and policy design. Land Use Policy 19: 157–166.

36.

Faludi

(1987) A Decision-Centred View of Environmental Planning, Oxford, UK: Pergamon.

37.

Faludi

(1989) Conformance vs. performance: implications for evaluations. Impact Assessment 7: 135–151.

38.

Faludi

(2000) The performance of spatial planning. Planning Practice & Research 15: 299–318.

39.

Faludi

(2006) Evaluating plans: The application of the European spatial development perspective. In: Alexander

(ed.) Evaluation in Planning: Evolution and Prospects, Aldershot, UK: Ashgate.

40.

Feldman

(2008) Welfare economics. In: Durlauf

Blume

(eds) The New Palgrave Dictionary of Economics, 2nd ed. London: Palgrave Macmillan.

41.

Forrester

(1989) Planning in the Face of Power, Berkeley: University of California Press.

42.

Fudge

Barrett

(1981) Reconstruction of the field of analysis. In: Barrett

Fudge

(eds) Policy and Action: Essays on the Implementation of Public Policy, London: Methuen.

43.

Furubotn

Richter

(2005) Institutions and Economic Theory: The Contribution of the New Institutional Economics, Ann Arbor: University of Michigan Press.

44.

Guyadeen

Seasons

(2016a) Evaluation theory and practice. Journal of Planning Education and Research. 1–13.

45.

Guyadeen

Seasons

(2016b) Plan evaluation: Challenges and directions for future research. Planning Practice & Research 31: 215–228.

46.

Healey

(1996) The communicative turn in planning theory and its implications for spatial strategy formations. Environment and Planning B: Planning and Design 23: 217–234.

47.

Healey

Barrett

(1990) Structure and agency in land and property development processes: Some ideas for research. Urban Studies 27: 89–103.

48.

Hoogerwerf

(1990) Reconstructing policy theory. Evaluation and Program Planning 13: 285–291.

49.

Innes

(1995) Planning theory's emerging paradigm: Communicative action and interactive practice. Journal of Planning Education and Research 14: 183–189.

50.

Jaensirisak

Wardman

May

(2005) Explaining variations in public acceptability of road pricing schemes. Journal of Transport Economics and Policy 39: 127–153.

51.

Johansson

P-O

(1991) An Introduction to Modern Welfare Economics, Cambridge: Cambridge University Press.

52.

Just

(1988) Making economic welfare analysis useful in the policy process: Implications of the public choice literature. American Journal of Agricultural Economics 70: 448–453.

53.

Just

Hueth

Schmitz

(2004) The Welfare Economics of Public Policy: A Practical Approach to Project and Policy Evaluation, Cheltenham: Edward Elgar.

54.

Khakee

(1998) Evaluation and planning: Inseparable concepts. The Town Planning Review 69: 359–374.

55.

Khakee

(2003) The emerging gap between evaluation research and practice. Evaluation 9: 340–352.

56.

Khakee

Hull

Miller

et al. (2008) New Principles in Planning Evaluation, Hampshire: Ashgate Publishing, Ltd.

57.

Kirman

(2008) Pareto, Vilfredo. In: Durlauf

SNAB

Lawrence

(eds) The New Palgrave Dictionary of Economics, 2nd ed. London: Palgrave Macmillan.

58.

Klosterman

(1985) Arguments for and against planning. Town Planning Review 56: 5.

59.

Laurian

Crawford

Day

et al. (2010) Evaluating the outcomes of plans: Theory, practice, and methodology. Environment and Planning B: Planning and Design 37: 740–757.

60.

Laurian

Day

Berke

et al. (2004) Evaluating plan implementation: A conformance-based methodology. Journal of the American Planning Association 70: 471–480.

61.

Lesser

Dodds

Zerbe

(1997) Environmental Economics and Policy, Boston: Addison-Wesley Educational Publishers.

62.

Lichfield

(1998) Trends in planning evaluation: A British perspective. In: Lichfield

Barbanente

Borri

et al. (eds) Evaluation in Planning: Facing the Challenge of Complexity, Dordrecht, Netherlands: Kluwer.

63.

Lichfield

(2001a) The philosophy and role of community impact evaluation in the planning system. In: Voogd

(ed.) Recent Developments in Evaluation, Groningen: Geo Press.

64.

Lichfield

(2001b) Where do we go from here? In: Voogd

(eds) Recent Developments in Evaluation, Groningen, Netherlands: Geopress.

65.

Lipsey

Lancaster

(1956) The general theory of second best. The Review of Economic Studies 24: 11–32.

66.

Lockwood

(2008) Pareto efficiency. In: Durlauf

Blume

(eds) The New Palgrave Dictionary of Economics, 2nd ed. London: Palgrave Macmillan.

67.

McCann

Colby

Easter

et al. (2005) Transaction cost measurement for evaluating environmental policies. Ecological Economics 52: 527–542.

68.

Mastop

Faludi

(1997) Evaluation of strategic plans: The performance principle. Environment and Planning B: Planning and Design 24: 815–832.

69.

Mickwitz

(2013) Policy evaluation. In: Jordan

Adelle

(eds) Environmental Policy in the EU: Actors, Institutions and Processes, London: Routledge.

70.

Mills

(1976) Economic analysis of urban land use controls. In: Stein

(ed.) Classic Readings in Real Estate and Development, Washington, DC: Urban Land Institute.

71.

Moore

(1978) Why allow planners to do what they do? A justification from economic theory. Journal of the American Institute of Planners 44: 387–398.

72.

Nabli

Nugent

(1989) The new institutional economics and its applicability to development. World Development 17: 1333–1347.

73.

Y-K

(2004) Welfare Economics: Towards a More Complete Analysis, London: Palgrave Macmillan.

74.

North

(1990) Institutions, Institutional Change and Economic Performance, Cambridge: Cambridge University Press.

75.

Norton

(2005) More and better local planning: State-mandated local planning in Coastal North Carolina. Journal of the American Planning Association 71: 55–71.

76.

Nozick

(1974) Anarchy, State and Utopia, New York: Basic Books.

77.

Oliveira

Pinho

(2010a) Evaluation in urban planning: Advances and prospects. Journal of Planning Literature 24: 343–361.

78.

Oliveira

Pinho

(2010b) Measuring success in planning: Developing and testing a methodology for planning evaluation. The Town Planning Review 81: 307–332.

79.

Oliveira

Pinho

(2011) Bridging the gap between planning evaluation and programme evaluation: The contribution of the PPR methodology. Evaluation 17: 293–307.

80.

Popper

(2003) The Open Society and its Enemies, London: Routledge.

81.

Rawls

(1971) A Theory of Justice, Cambridge, MA: Belknap.

82.

Reiter

(2008) Efficient allocation. In: Durlauf

Blume

(eds) The New Palgrave Dictionary of Economics, 2nd ed. London: Palgrave Macmillan.

83.

Runge

Myers

(1985) Shifting foundations of agricultural policy analysis: Welfare economics when risk markets are incomplete. American Journal of Agricultural Economics 67: 1010–1016.

84.

Rutherford

(1996) Institutions in Economics: The Old and the New Institutionalism, Cambridge: Cambridge University Press.

85.

Sabatier

(1986) Top-down and bottom-up approaches to implementation research: A critical analysis and suggested synthesis. Journal of Public Policy 6: 21–48.

86.

Sager

(1994) Communicative Planning Theory, Aldershot, UK: Avebury.

87.

Samuels

(1995) The present state of institutional economics. Cambridge Journal of Economics 19: 569–590.

88.

Schade

Schlag

(2003) Acceptability of urban transport pricing strategies. Transportation Research Part F: Traffic Psychology and Behaviour 6: 45–61.

89.

Seasons

(2003) Monitoring and evaluation in municipal planning. Journal of the American Planning Association 69: 430–440.

90.

Shahab S, Clinch JP and O’Neill E (2017) Timing and distributional aspects of transaction costs in transferable development rights programs. Habitat International, Forthcoming.

91.

Short

Baker

Carter

et al. (2013) Strategic Environmental Assessment and Land Use Planning: An International Evaluation, London: Routledge.

92.

Stiglitz

(1985) Information and economic analysis: A perspective. The Economic Journal 95: 21–41.

93.

Talen

(1997) Success, failure, and conformance: An alternative approach to planning evaluation. Environment and Planning B: Planning and Design 24: 573–587.

94.

Therivel

(2012) Strategic Environmental Assessment in Action, London: Routledge.

95.

Vedung

(1997) Public Policy and Program Evaluation, New Jersey: Transaction Publishers.

96.

Weiss

(1990) In fairness to future generations. Environment: Science and Policy for Sustainable Development 32: 6–31.

97.

Williamson

(1993) Opportunism and its critics. Managerial and Decision Economics 14: 97–107.

98.

Wüstenhagen

Wolsink

Bürer

(2007) Social acceptance of renewable energy innovation: An introduction to the concept. Energy Policy 35: 2683–2691.