The quality and implementation of local plans: An integrated evaluation

Abstract

Local plans are key components of municipal planning and scholars have developed methodologies to evaluate them. However, the purpose and use of local plans may vary, making their evaluation challenging, especially in comparative studies. In this context, existing evaluation methodologies that target local plans do not always account for the purpose of plans when evaluating their quality. Moreover, it is still unclear whether high-quality plans are better implemented than plans of lower quality. In this paper, we propose an integrated evaluation approach that links plan quality to implementation. To assess plan quality, we developed a flexible framework that accounts for the fact that local plans may present different characteristics according to their purpose. The framework rests on the analysis of the communication- and action-oriented dimensions of plans, and a system of coordinates that enables the classification of plans into four types (i.e. visions, blueprints, communicative policy acts, and basic plans) according to the score on each dimension. Plan implementation, assessed as conformance and performance, is determined based on the perception of local planners. Our approach and framework are applied to a set of 37 Swiss local plans. The approach is presented in a transparent way and requires basic numeracy skills, making it accessible to scholars and planning practitioners. Findings may be used to guide municipalities towards enhancing the quality of plans and identify best practices.

Keywords

Planning evaluation performance conformance communicative policy act plan content analysis

Introduction

Plans represent a core element of planning activity, both as the output of the plan-making process and as the input for subsequent discussions and land use-related decisions (Norton, 2008). In this context, plan evaluation becomes a crucial step to ensure that planning yields expected results. Moreover, evaluation helps increase the accountability of public institutions and strengthens public confidence in planning decisions (Guyadeen and Seasons, 2016). In this sense, scholars have developed analytical approaches to evaluate plan quality (e.g. Baer, 1997; Stevens, 2013) and their implementation (e.g. Berke et al., 2006; Lyles et al., 2016; Talen, 1996).

Effective plan evaluation acknowledges that plans may have different purposes and uses (Lyles et al., 2016). Indeed, Baer (1997) identifies eight types of plans for distinct situations, among them: blueprints, land use guides, administrative requirements for federal funds, pragmatic actions and responses to state and federal planning mandates. Similarly, Hopkins (2001) refers to plans as agendas, policies, visions, designs, and strategies. With regard to local plans, Norton notes that: “… researchers working on local planning and development management appear to have settled on a consensus view of the role played by a plan, whether doing so expressly, implicitly, or for the sake of critique” (Norton, 2008: 436). Under this consensus view, local plans are more than mere visions but less rigid than blueprints. Visions can be defined as communication-oriented documents that aim at defining common goals, guiding future development, and inspiring people for action (e.g. strategies or strategic plans) (Hopkins, 2001), and blueprints as action-oriented, focusing on which precise tasks should be pursued to reach specific outcomes (Baer, 1997).

Indeed, local plans – also known as “comprehensive”, “general”, “master”, and “community” plans in the international literature – aim to (a) provide a vision to steer the long-term development of municipalities and (b) define policies to steer local development towards achieving this vision (Norton, 2008; Randolph, 2004; Stevens, 2013). They should serve as a guideline in the face of changing local conditions, such as political turnovers affecting municipality goals, or new higher-level requirements, which may, for example, restrict the issuance of building permits. Therefore, local plans should provide a repository where the needs and the resources of the municipality are detailed and where the decisions and alternatives discussed or agreed upon during the planning process are described. In this respect, Norton (2008) suggested viewing local plans as communicative policy acts.

Interestingly, little effort has been put into investigating whether plan quality influences implementation (Guyadeen and Seasons, 2016; Lyles and Stevens, 2014). Among the few authors who approached this issue at a local level, Brody and Highfield (2005) observed that plans containing specific implementation provisions, such as monitoring programs, succeeded at better containing wetland development in Florida (U.S.), while Berke et al. (2006) concluded that the quality of local sustainability plans affected their implementation in New Zealand. In contrast, Norton (2005a) found that plan quality did not influence the use of plans by local officials for decision-making processes in North Carolina (U.S.). The limited attention directed towards the influence of plan quality on implementation is a major gap in planning evaluation and restricts the understanding of the impact and relevance of plans.

To contribute to recent debates on planning evaluation, this paper proposes a framework to evaluate the quality of local plans and an integrated approach that allows the linking of plan quality to plan implementation as assessed by local planners. The approach and framework are applied to a set of Swiss local plans (Richtplans) to answer the following research questions. First, what is the quality of local plans? Second, are local plans successfully implemented according to the perception of local planners? Third, is there a relationship between the quality of the local plans and their implementation?

Current methods for evaluating plan quality and implementation

Measuring plan quality

Irrespective of whether plans are implemented at a state, regional, or local scale, their evaluation should be adapted to their purpose because evaluation criteria must be targeted towards the specific type of plan under consideration (Lyles et al., 2016). To evaluate the quality of plans viewed mainly as visions, it is recommended to use communication-oriented dimensions of plan quality and assess whether their design is accessible to the wider public and whether they entail a narrative storyline to motivate stakeholders and improve their commitment towards the goals of the plans (Bunnell and Jepson, 2011). In contrast, plans conceived as blueprints entail a list of policies to guide policy-making, and their evaluation generally implies using action-oriented dimensions to check whether the plans contain provisions to ensure consistent implementation (Baer, 1997; Hopkins, 2001), i.e. precisely describing who is in charge of implementing the policies and over what timescale.

Because local plans entail a long-term vision as well as precise implementation provisions, they are both communication- and action-oriented, and their evaluation implies assessing both dimensions of plan quality. However, with the exception of two recent studies (Berke et al., 2013; Lyles et al., 2014), previous research efforts have not specified whether the dimensions of plan quality they examined were communication- or action-oriented. Early scholars who developed plan-evaluation protocols examined three main dimensions: a “fact base” to describe the local context at the time of the plan’s development (e.g. geographical and socio-economic conditions); “goals” to identify future desired conditions; and “policies” to determine practical strategies aimed at attaining the goals (Berke and French, 1994; Deyle and Smith, 1998). Godschalk et al. (1999) added the following four dimensions: “implementation” to describe how policies should be implemented, “monitoring” to detail how policy implementation should be monitored, “interorganizational coordination” to specify how policies should be coordinated with other plans or agencies, and “participation” to document the public participation process set up during the development of the plan. Recently, Stevens (2013) built on the dimensions mentioned above to assess the quality of local plans in British Columbia (Canada) and added an “organization and presentation” dimension to judge the user-friendliness of plans.

Over recent decades, more than 47 peer-reviewed studies (Lyles and Stevens, 2014) have used all or part of these dimensions of plan quality to evaluate how local, state, and regional plans address specific policy issues, such as the mitigation of natural hazards (Berke and French, 1994) or the promotion of affordable housing (Hoch, 2007), smart growth (Edwards and Haines, 2007), and sustainable development (Conroy and Berke, 2004). However, the evaluation protocols used in most of these studies are contingent on the purpose of the plans under study, and they are not adapted for comparisons across policy issues. For example, Edwards and Haines (2007: 55) evaluated whether local plans entailed policies to “create walkable communities” and “provide a variety of transportation choices”. While these items are suitable to the analysis of smart growth, they would not apply to other policy issues such as affordable housing or hazard mitigation. Hence, it would be useful to have protocols applicable across a broad range of plans. Norton (2008) recognized this need and suggested distinguishing between the policy focus and formal quality of plans. According to his definition, policy focus relates to the policy message conveyed by the plan, such as the mitigation of natural hazards or the management of urban sprawl (i.e. “policies” dimension in Stevens’ protocol), whereas formal quality relates to how the policy message is expressed, justified, and implemented (i.e. other plan quality dimensions in Stevens’ protocol). Following this distinction, protocols designed for the evaluation of the formal quality of plans should be independent of the policy issue at stake in order to be used across a broad range of plans.

Measuring plan implementation

The assessment of plan implementation is the subject of ongoing debate in the planning evaluation field. Generally, scholars follow either a performance or a conformance approach (Alexander and Faludi, 1989; Guyadeen and Seasons, 2016; Oliveira and Pinho, 2010). However, as Kinzen (2016) notes, scholars have not always used the terms conformance and performance consistently, and there is no clear consensus regarding their definition.

The performance approach is often applied when plans are considered rather as visions, and it focuses on the role of plans in decision-making (Faludi, 2000). It considers plans to be successfully implemented if they are useful in supporting decision-making regardless of whether they influence planning outcomes such as urban densification or hazard reduction (Laurian et al., 2004; Mastop and Faludi, 1997). For example, at the local level, Norton (2005b) followed this approach to study whether local plans shaped the land-use decisions of locally elected officials in North Carolina.

The conformance approach generally considers plans as blueprints, with their provisions eventually reflected in actual spatial development (Laurian et al., 2004; Mastop and Faludi, 1997; Talen, 1997). Consequently, this approach assumes that plans are successfully implemented if (1) their policies are executed (Lyles et al., 2016) and/or (2) if outcomes on the ground correspond to plan provisions (Alexander and Faludi, 1989; Grădinaru et al., 2017; Loh, 2011; Oliveira and Pinho, 2010; Talen, 1996). Recently, Lyles et al. (2016) suggested using the term conformance when assessing whether policies are carried out, the term influence when assessing whether a plan is used in decision-making and performance when addressing whether a plan influences outcomes. Particularly to address the relationship between outcomes and plan provisions, several other terms have been proposed. For example, Abrantes et al. (2016) used the term compliance when analysing whether land-use dynamics in the Lisbon metropolitan region relates to land use allocation in the municipal master plans. The term effectiveness was proposed by Long et al. (2012) to address the spatio-temporal heterogeneity of Beijing’s master plans’ effect on urban expansion, when the impact of planning is analysed along with other forces (e.g. economic).

Considering the ongoing academic debate on the approaches for assessing plan implementation, for the remainder of this paper we refer to performance to address the usefulness of plans in decision-making, and to conformance when analysing whether policies included in the plan are carried out, and whether outcomes on the ground correspond to plan provisions.

Some recent studies have argued for combining performance and conformance approaches (e.g. Oliveira and Pinho, 2010). Guyadeen and Seasons (2016) referred to this pragmatic line of research as an integrative approach. For example, in the context of local planning, Lyles et al. (2016) relied on a survey of local officials to evaluate the performance and conformance of local hazard mitigation plans in the United States. At a regional scale, Feitelson et al. (2017) combined both performance and conformance approaches to evaluate a regional land use plan in Israel and noted that high conformance is not necessarily linked to high performance. Regarding recreation areas, for example, they observed a high level of conformance because the plan was successful at limiting housing development in areas dedicated to recreational activities. However, they discovered that the plan’s performance was poor, since the plan had not been effective at improving recreational resource planning or supporting decisions to provide appropriate amenities. Similarly, at a national scale, Altes (2006) studied national urbanization policies in the Netherlands and also showed that performance and conformance are independent because the plan under study did not influence decision-making regarding housing stagnation (i.e. poor performance) but succeeded at limiting urban outward expansion as expected (i.e. high conformance).

Integrated approach towards assessing the quality and implementation of local plans

Based on the concepts presented in sections above, we developed an integrated approach resting upon two elements:

The framework for assessing plan quality (for details, see next section “Framework for assessing plan quality”) comprising the following:

an analysis of the communication- and action-oriented dimensions of plan quality

a classification of local plans according to their scores on communication- and action-oriented dimensions into four types of plans (i.e. basic plans, visions, blueprints, and communicative policy acts)

The assessment of the relationship between plan quality and plan implementation: this is accomplished by relating communication- and action-oriented scores of the plans (plan quality) with their performance and conformance as assessed by local planners.

Framework for assessing plan quality

Stevens’ protocol (2013) was used as the starting point of framework development because it assesses both the communication- and action-oriented dimensions of plan quality. His protocol was adapted to the Swiss planning context and to the purpose of the study. To allow a comparison across plans addressing different policy issues, we focused on the formal quality of the plans and did not include any protocol item related to the policy focus of the plans. Consequently, we removed from Stevens’ protocol (2013: 485) all “policy” items (e.g. “Does the plan contain at least one specific policy or action related to food and/or agriculture?”) and summarized several of the detailed “fact base” and “goal” items (e.g. “Does the plan include a descriptive statement about air quality/water bodies in the community?”) into more general statements applicable across policy issues (e.g. “Does the plan include a section that precisely describes the municipality’s general situation?”). After the adjustments made to Stevens’ (2013) protocol, our final evaluation protocol contained 36 items grouped into seven dimensions of plan quality. A list of the plan quality dimensions as well as the corresponding protocol items are reported in supplementary material A.

Dimensions that aim at describing the local context, detailing the long-term goals, and documenting and justifying the planning process were classified as communication-oriented (Figure 1). Dimensions providing provisions for effective implementation were categorized as action-oriented. The “interorganizational coordination” dimension was divided into two groups because three of its items are communication-oriented (i.e. 5.a.1–5.a.3), whereas the three others are action-oriented (i.e. 5.b.1–5.b.3).

Figure 1.

Dimensions of plan quality.

Finally, we used a system of coordinates to visually represent the quality of the plans according to their overall score on communication- (x-axis) and action-oriented dimensions (y-axis), both measured on a scale from 0 (low quality) to 100 (high quality) (see supplementary material B for an illustration). The coordinate system defines four quadrants, of which quadrants I, II, and IV can be related to three types of plans described in the literature:

Quadrant I: Plans that are both highly communication- and action-oriented can be related to Norton’s (2008) definition of communicative policy acts.

Quadrant II: Plans presenting a low quality along the communication-oriented dimension but a high quality along the action-oriented dimension can be related to the notion of blueprint described by Baer (1997).

Quadrant III: Plans that have both low quality in the communication- and action-oriented dimensions represent examples of plans that provide only basic provisions to guide local development. These are named basic plans.

Quadrant IV: Plans with high quality along the communication-oriented dimension but low quality along the action-oriented dimension can be related to visions in the sense of Baer (1997) and Hopkins (2001).

This coordinate system makes it possible to visualize the overall quality of local plans and their position relative to each other.

To evaluate the quality of the local plans, we followed the recommendations from Berke and Godschalk (2009) and Lyles and Stevens (2014) regarding the administration of the protocol and the reliability of the coding process. Two coders were trained, and then tested a draft version of the protocol on five plans that were not included in the sample. After this trial phase, the protocol was refined and enhanced. The two coders worked independently and assigned each item a score of 1 when it was present and a score of 0 otherwise. We assessed the reliability of the measurements by calculating Krippendorff’s alpha (Krippendorff, 2013) for each item (see supplementary material A) and by applying the upper and lower standards recommended for each dimension of plan quality by Stevens et al. (2014). Items with alpha scores above upper standards were included in subsequent analyses once the differences between the coders were reconciled. For items with alpha scores between upper and lower standards, the differences between coders were reconciled after a clarification and reassessment of the protocol. The same procedure was followed for items with alpha scores below lower standards. We chose not to drop items with a low alpha score from the analysis because their exclusion would have considerably lowered the thematic breadth of the study. However, we identified the reasons for these low alpha scores and carefully reassessed the corresponding items. To ensure transparent results, we provide the initial alpha scores in supplementary material A. The analysis yielded overall robust results because 83% of the protocol items presented alpha scores above upper standards and could be reconciled without a reassessment of the protocol.

For each plan and each dimension of plan quality, we computed index scores in three steps (Berke et al., 2013: 454). First, the scores of the protocol items were summed within each dimension. Second, the summed scores were divided by the total possible score for each dimension. Finally, this fractional score was multiplied by 100 to place each index on a scale ranging from 0 to 100. To determine the extent to which plans are communication- and action-oriented, we summarized the corresponding plan quality dimensions as presented in Figure 1 and used the same standardization process as described above.

A series of non-parametric statistical tests were conducted to understand what the possible factors are that shape the communication- and action-oriented dimensions of plans. Namely, whether plans for larger cities are more likely to be in a certain quadrant than plans from smaller municipalities, whether comprehensive plans are more likely to fall in a given quadrant than sectorial plans, or whether the existence of a planning mandate might cause plans to be placed in a certain quadrant. In this way, we inspected the relationship between the position of a plan within the four quadrants and: (a) the population size of the municipality (Kruskal–Wallis test), (b) the thematic breadth of the plan, namely whether it is comprehensive or sectorial (Pearson’s chi-squared test), and (c) plan development under a cantonal planning mandate (yes or no) (Pearson’s chi-squared test).

Assessing performance and conformance

To assess the performance and conformance of the plans, we sent a questionnaire to the main planning officer of the sampled municipalities. To assess performance, local planners were asked to evaluate the usefulness of their plan for day-to-day planning practice on a 0 to 10 scale (0 = not useful at all; 10 = very useful, helping to steer the development of the municipality in daily practice). To assess conformance, the respondents were asked to estimate the proportion of policies they expected to be completed or at least further examined before the end of the plan’s lifespan on a 0 to 100% scale. We did not assess whether local plans influenced the outcome of planning on the ground (e.g. urban densification, hazard mitigation) because most of the sampled plans have been adopted too recently to show such impacts.

This method yields useful but by no means perfect measures of plan performance and conformance since it is not based on direct measurements (i.e. actual proportion of implemented policies) but on the perception of local planners. The tendency of local planners to deliver socially desirable answers may have introduced some bias into the analysis. Also, to illicit a high response rate, the survey questions were rather general and short, resulting in multiple levels of ambiguity. For example, respondents were asked to assess the general usefulness of the local plans, leaving it to the respondent to decide which specific aspects to take into account. Furthermore, respondents’ predispositions as optimistic or pessimistic may have influenced their estimation of the proportion of policies expected to be completed or at least further examined before the end of a plan’s lifespan. Despite these limitations regarding data reliability and validity, the method provided data that proved to be valuable for exploring the relationship between plan quality and plan implementation.

We used Spearman’s correlation coefficients to assess whether the communication- and action-oriented dimensions of the plans were statistically related to their performance and conformance (Conover, 1999). Additionally, several statistical tests were performed to understand whether factors related to the planning context could be correlated with plan implementation. By means of Mann–Whitney U tests and Spearman’s correlation coefficients, we explored the relationships between plan performance and conformance and: (a) the population size of the municipalities, (b) the thematic breadth of the plans (namely, whether they are comprehensive or sectorial), and (c) plan development under a cantonal planning mandate.

Study area and sample selection to test the integrated approach

To test our proposed framework, we selected municipalities in federalist Switzerland, which is organized in 26 cantons. Federal, cantonal and local governments are jointly responsible for spatial planning (Newman and Thornley, 1996), but municipalities hold the greatest decision-making power regarding local planning (Mann, 2009). While they are required by federal law to develop a land-use plan (Nutzungsplan) binding to land owners, federal legislation also grants them the freedom to decide whether they want to develop a local plan (Richtplan). However, several cantonal governments impose planning mandates to oblige their municipalities to develop local plans. Irrespective of whether they are mandated or not, local plans are binding for municipal and cantonal authorities, and their policies are meant to be applied within the plans’ lifespan. Depending on cantonal legislation, local plans may be sectorial—in which case they focus on a specific policy issue, such as energy or landscape protection—or comprehensive, which allows them to coordinate several goals (Gilgen, 2012). Otherwise, cantonal regulations hardly differ among themselves, and largely lack precise rules regarding the content of local plans, the level of detail and enforcement mechanisms. Recently, Kaiser et al. (2016) conducted a survey of local planning instruments and found that 53.0% of Swiss municipalities had a local plan in place in 2014. In addition, their results indicated that the use of local plans increased steadily between 1970 and 2014 and that municipalities with a large number of inhabitants are more likely to develop local plans than their smaller counterparts.

Through its position at the intersection of Germanic and Romance Europe, Switzerland spans over four linguistic and cultural regions (i.e. German, French, Italian, and Romansh). The present study is limited to German-speaking municipalities to ensure the comparable terminology among the studied plans. The analysis of plans from other language areas would have introduced ambiguities regarding terminology and required a considerable additional resources for translations. The German-speaking region spatially covers 70% of the country (28,971 km²), comprises 71% of the total Swiss population (5,758,699 inhabitants) and is organized in 1470 municipalities (Bundesamt für Statistik, 2014).

To select the local plans, we used stratified sampling (Gregoire and Valentine, 2007) according to the population size of the municipality (<1000 inhabitants; 1000–4999; 5000–9999; >9999) because previous studies (e.g. Edwards and Haines, 2007; McDonald and McMillen, 2004) revealed that population size may influence local planning practices. In each stratum, we randomly selected ten municipalities and collected their plans. After initial analysis, three plans were removed from the sample because they did not cover the entire municipal territory. The final sample included 37 local plans, a large majority of them (65%) being comprehensive, while the remaining ones (35%) focused on selected issues such as transportation, utilities provision, energy supply, and landscape protection (see supplementary material C). Most of the plans (65%) were developed under a cantonal planning mandate.

Results

Quality of local plans

The analysis confirmed that local plans were both communication- and action-oriented, but revealed that overall their action-oriented dimensions were stronger (Figure 2). The graph shows that most plans clustered in quadrant II (13 plans), followed by quadrant I (11) and III (6). In contrast, no plan was classified in quadrant IV. These results imply that the sampled local plans related most closely to blueprints and comprehensive policy acts but less to basic plans and visions.

Figure 2.

Quality of the sampled local plans as measured by their communication- and action-oriented dimensions. The plans situated in three of the four quadrants can be related to three types of plans described in the literature: visions (quadrant IV), blueprints (II), and communicative policy acts (I). The plans situated in quadrant III have both low levels of communication- and action-oriented content and represent examples of basic local plans.

The content analysis of the plans yielded complementary qualitative information that highlighted the differences among the main types of plans. Plans that most closely met the characteristics of communicative policy acts (e.g. P16, P14, and P6, which scored high in both dimensions of plan quality) were almost always divided into several sections that were clearly written and could be understood independently from each other. For example, plan P14 comprises four different sections. The first describes the purpose of the plan, details how it is embedded within other local planning activities and instruments, and provides an overall table of contents. The second section describes the plan-making process, while the third section entails an analysis of local conditions and a description of the development goals of the municipality for each planning topic (e.g. urban development, landscape protection, and mobility). The last section comprises a collection of policies with detailed implementation provisions assembled into an action plan.

The three plans in quadrant I with higher communication-oriented than action-oriented scores (i.e. plans P11, P17, and P7) had similar characteristics. However, they entailed much less detailed implementation provisions. For example, plan P7 also comprised an action plan; however, the agencies responsible for the implementation of the policies were not identified, and the plan entailed no timetable for implementation.

In contrast, the plans situated in quadrant II were overall much shorter, presented characteristics similar to blueprints, and usually comprised a single section. Most of these plans only entailed an action plan with concrete policies and detailed implementation provisions. Information related to the plan-making process or the local context of the municipality was sometimes included in other planning documents, namely an independent planning report (Planungsbericht) in the case of plans P35 and P2.

Finally, the plans situated in quadrant III had both low communication-oriented and implementation-oriented quality, and they were not very detailed. In general, they did not address long-term planning issues, such as energy provision or the coordination of urban development and mobility, but rather limited themselves to listing which plots of land should be assigned to specific land uses, such as development or conservation areas. As a result, they failed to provide a flexible guideline to steer municipal development in the face of changing conditions and rather acted like justifications for the preparation of binding land use plans.

The findings of the statistical tests demonstrate that there was no statistically significant relationship between the position of the plans within the four quadrants and municipality population (Kruskal–Wallis test = 3.908, p = 0.272). A similar non-significant relationship was found between the position of the plans within the four quadrants and their thematic breadth (X²= 2.540, p = 0.468), and the existence (yes or no) of a cantonal planning mandate (X²= 6.287, p = 0.098).

The mean scores per dimension provide more detailed indication regarding which aspects of plan quality have been considered in our sample (Supplementary material D). They reveal that local plans generally entail precise information regarding which actions should be coordinated to attain the stated goals (mean “inter-org. coord.” scores > 80). In contrast, development goals are sometimes unclear (mean “goals” score = 61), and some plans miss specific provisions regarding the practical implementation of the policies (mean “implementation” score = 59). In addition, numerous plans do not entail a description of the public participation process set up during their development (mean “participation” score = 42), and many of them are not designed to attract attention from a broad audience (mean “organization and presentation” score = 47). Finally, the lowest scores were reached by the dimensions “fact base” and “monitoring” (mean scores < 40), which indicates that local conditions, such as geographical and socio-economic characteristics, are often insufficiently described, and that many plans lack a section detailing long-term monitoring and evaluation of the policies. For example, less than one third of the plans include a description of the municipal population broken down by age or gender, entail population projections for the next fifteen years, quantify policies with measurable objectives or indicators, or identify organizations responsible for monitoring (see Supplementary material A for details).

Performance and conformance of local plans as assessed by local planning officers

Local plans had a rather high performance, as most local officers found their plans useful for day-to-day planning activities. The majority of local officers (67%) rated their usefulness between 5 and 8 (Supplementary material E: Figure A). A small proportion of respondents (12%) indicated that their plans were not useful to their municipality (scores below 5), while almost 25% of local planners attributed a score above 8 to their plan, indicating that it was very useful for steering the development of their municipality on a daily basis.

The conformance of the plans was also rather high because 40% of the planning officers expected that approximately half (41–60%) of the policies contained in their plan would be implemented before the end of its lifespan (Supplementary material E: Figure B). An equal proportion of officers expected that between 60% and 100% of the policies would be completed.

The findings of the statistical tests indicated no statistically significant relationship between the thematic breadth of the plans and their performance (Mann–Whitney U = 109.0, p = 0.542) or conformance (Mann–Whitney U = 149.5, p = 0.180). Similarly, the adoption of the plans under a cantonal mandate did not lead to a statistically significant difference in plan performance (Mann–Whitney U = 122.0, p = 0.897) or conformance (Mann–Whitney U = 134.0, p = 0.705). Furthermore, the tests showed no significant correlation between municipality population and plan conformance (Spearman’s ρ = 0.039, p = 0.834). However, they showed a moderate but significant correlation between population size and plan performance (Spearman’s ρ = 0.409, p = 0.018).

Linking communication- and action-oriented dimensions of plan quality with performance and conformance

The performance of the plans, i.e. their perceived usefulness for day-to-day practice, was positively correlated with their quality for communication-oriented (Figure 3A) as well as action-oriented dimensions (Figure 3B). These results were confirmed by statistical analyses, which revealed a moderate but significant correlation in the first case (Spearman’s ρ = 0.42) and a strong significant correlation in the latter case (Spearman’s ρ = 0.58).

Figure 3.

Correlation between the quality and implementation of plans. In the first row, the communication- (A) and action-oriented dimensions (B) are correlated with their performance. In the second row, the communication- (C) and action-oriented dimensions (D) are correlated with their conformance. Statistically significant Spearman’s correlation coefficients (ρ) are indicated by a star (p < .05).

The results were more ambiguous regarding conformance, i.e. the proportion of policies likely to be implemented (Figure 3C and D). While weak positive correlations were identified for both the communication- (Spearman’s ρ = 0.23) and action-oriented dimensions (Spearman’s ρ = 0.28) of the plans, these results were not statistically significant.

Discussion

The approach developed in this article is well embedded within existing theoretical concepts and analytical procedures, and it reveals that (a) the quality of local plans varies greatly, and (b) their purpose and use varies between basic plans, blueprints, communicative policy acts and visions. The analysis empirically confirms that Swiss local plans often combine communication and action-oriented dimensions. However, contrary to Norton’s assumption (2008), less than half of the local plans can qualify as communicative policy acts.

Indeed, policies and implementation provisions remain their core elements. One example for a successful combination of communication and action-oriented dimensions is plan P14: it entails a section describing the desire of municipal authorities to promote nearby recreation (among others) and concrete policies to show how this objective will be pursued, such as the realization of new walking and cycling trails.

Insights from communication- and action-oriented dimensions of quality

Whether our results indicate appropriate levels of plan quality depends on which purpose these plans are supposed to fulfil. For example, if a municipal government wishes for its local plan to act as a flexible development guideline in the sense of a communicative policy act, its plan should present a high quality on both the communication- and action-oriented dimensions. However, not all local plans may need to meet these characteristics. For example, large cities with a professional planning administration may prefer to limit their plans to a list of policies and implementation provisions similar to a blueprint. They may already have other planning documents detailing their local context and development goals (e.g. strategic plan, concept of development), and they may not want to overload their local plans. In contrast, small municipalities without professional planners may need a plan that is more in line with the notion of a communicative policy act in order to assemble all important planning-related information into a single document and facilitate policy continuity in case of political turnovers. In a study about the inclusion of smart growth principles in U.S. local plans, Edwards and Haines (2007) similarly acknowledged that small municipalities need different plans and policies than their larger counterparts. Although our analysis showed no statistically significant relationship between the position of the plans within the four quadrants and municipality population, future studies could use our approach to assess how other factors, such as local planning capacity, shape the communication- and action-oriented dimensions of plans and explore how to better adapt local plans to the needs of municipalities.

The analysis further reveals that many plans have only moderate scores for the communication- and/or action-oriented dimensions. The analysis of individual plan quality scores provides more insights into the main strengths and weaknesses of the local plans and allows recommendations to be made to improve their general quality. Overall, the scores for “fact base”, “goals”, “implementation”, “monitoring” and “participation” indicate low to moderate quality. There are some similarities between these findings and the results of a study by Lyles et al. (2014), on the quality of local hazard mitigation plans in six U.S. states, as in both studies the scores for “goals”, “implementation”, and “participation” were found to be higher than the scores for “fact base” and “monitoring”. However, the score for “inter-organizational cooperation” was much higher in our study than in the Lyles et al. (2014) analysis, and clearly surpassed all other dimensions of plan quality. To strengthen the communication-oriented dimensions of their plans, people in charge of plans could start by better describing the local context to strengthen sense of place and help local officials and stakeholders recognize the unique characteristics of their municipality. Additionally, the participation process set up during plan development could be made more transparent, and the organization and presentation of the plans could be enhanced to make them accessible to a broad audience. Regarding the action-oriented dimensions of plan quality, the people responsible for plans should mainly focus their efforts towards improving the level of detail of the implementation and monitoring provisions of the policies. To improve clarity, they could assemble these provisions into an action plan, as suggested by Stevens (2013).

Relationship between plan quality and implementation

Local plans had a rather high performance and conformance, indicating that local planning officers value their plans and use them in daily planning practice and that most municipalities are committed to implementing their plans. The quality of the plans was significantly correlated with their performance, i.e. their perceived usefulness for steering municipal development in day-to-day planning practice, when measured according to both the communication- and action-oriented dimensions of the plans. This reveals that local officers not only found their plans more useful if they entailed clear descriptions of the local context, the plan-making process and the goals of the municipality, but also if they encompassed detailed policies and implementation provisions. These results are very encouraging for the Swiss planning community, as they confirm the benefit of high-quality plans. However, our conclusions are limited to the assessment of the relationship between plans’ formal quality and implementation. Future studies could investigate whether quality measured according to plans’ policy focus is also correlated with performance and conformance.

In light of our results showing that the communication-oriented dimensions of local plans also seems to contribute to their perceived performance and conformance, we recommend that future plan evaluation studies take into account both the communication- and action-oriented dimensions of plan quality.

In addition, the present analysis suggests that the relationship between plan quality and plan implementation may vary according to whether implementation is measured in terms of performance or conformance. When implementation was defined and measured in terms of performance, the communication- and action-oriented dimensions of plan quality were moderately to strongly correlated with performance. In contrast, when implementation was defined in terms of conformance, the correlation was much weaker. A possible explanation for these results may be that numerous external factors can affect conformance. We found no statistically significant relationships between implementation measured in terms of plan conformance and (1) plan development with/without a cantonal mandate, or (2) its thematic breadth. It is possible that other factors – such as local political will or financial and planning capacity – could have an effect on this issue (Rudolf et al., 2017). Regarding plan implementation measured in terms of performance, tests showed a positive correlation with population size, indicating that plans for larger cities tend to be used in day-to-day planning practice more often. Consequently, high-quality plans may not be sufficient to ensure successful policy implementation, whereas plan performance – measured as the perceived usefulness of the plans in daily planning practice – may be less influenced by such additional factors.

It is a challenge to measure conformance and performance, and our conclusions are bound by the generality of the survey questions used to assess plan implementation. To increase data reliability and validity, stronger measures of performance and conformance could be obtained by consulting official reports documenting implementation progress, if available. For example, Berke et al. (2006) relied on development permits in their evaluation of local plans and implementation practices in New Zealand and concluded that plan quality had an important influence on conformance but not on performance. More in-depth evaluations in different local contexts and with a focus on multiple aspects of conformance (e.g. outcome of planning on the ground) and performance (e.g. importance of the plan to coordinate several planning issues) are necessary to increase our understanding of the relationship between plan quality and implementation.

Conclusions

Evaluation has gained much attention in the public sector since the 1990s in the context of increased demand for accountability by elected officials and local stakeholders (Bernstein, 2001) and the application of New Public Management (NPM) practices (Gerber, 2016; Guyadeen and Seasons, 2016), which call for more evidence-based policy-making and aims to increase the efficiency of public administrations (Mueller and Hersperger, 2015). The approach presented in this article distinguishes between the communication- and action-oriented dimensions of plans, which accounts for the fact that local plans may present different characteristics depending on their purpose. This conceptual distinction facilitates the evaluation of the quality and the perceived performance and conformance of plans in the context of local planning and provides an innovative and transparent scheme that could easily be applied by planners. The administration of the protocol is straightforward, and the calculation of the plan quality scores only requires basic numeracy skills. Planners could use this approach for in-house evaluation to support continuous learning, improve future plan-making processes and assess whether their plans have succeeded at steering local development as expected. Furthermore, the approach could provide planners with a solid basis to communicate the quality of their plans and legitimize their professional activity. State governmental agencies could use the proposed framework to systematically evaluate the quality of the local plans developed by municipalities within their jurisdiction. The visual representation could help to identify differences among plans, guide municipalities towards enhancing low-quality plans and identify high quality, best-practice plans. Because our approach is strictly assessing formal quality, it is suitable for the evaluation of local plans in different national planning contexts and across a wide range of policy issues.

Footnotes

Acknowledgements

We are grateful to Matthias Müller, who assisted with the content analysis of the local plans, to Felix Kienast and Anna M. Hersperger, whose invaluable comments considerably improved the manuscript, and to Curtis Gautschi for language proofing. We are grateful to the two anonymous referees for their valuable comments that helped to improve the quality of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Swiss National Science Foundation as part of the project “Controlling urban sprawl to limit soil consumption” (SPROIL, grant number 406840_142996) conducted within the National Research Programme “Sustainable Use of Soil as a Resource” (NRP68) and the project “From plans to land change” (CONCUR, ERC TBS Consolidator Grant number BSCGIO 157789).

Sophie C. Rudolf holds a master degree in environmental sciences and a PhD in spatial planning, both from the Swiss Federal Institute of Technology in Zurich. She has been a Ph.D. student at the Swiss Federal Research Institute for Forest, Snow and Landscape Research (WSL) from 2013 to 2017. Her main research interests include planning evaluation, urban studies, and urban sprawl.

Simona R. Grădinaru is a postdoctoral researcher at the Swiss Federal Research Institute for Forest, Snow and Landscape Research (WSL). Her research interests include evaluation of spatial planning instruments and outcomes, assessing the contribution of planning to land change and analysis of the role of spatial information in strategic spatial plans.

References

Abrantes

Fontes

Gomes

et al. (2016) Compliance of land cover changes with municipal land use planning: Evidence from the Lisbon metropolitan region (1990–2007). Land Use Policy 51: 120–134.

Alexander

Faludi

(1989) Planning and plan implementation: Notes on evaluation criteria. Environment and Planning B: Planning and Design 16(2): 127–140.

Altes

WKK

(2006) Stagnation in housing production: Another success in the Dutch ‘planner’s paradise’? Environment and Planning B: Planning and Design 33(1): 97–114.

Baer

(1997) General plan evaluation criteria: An approach to making better plans. Journal of the American Planning Association 63(3): 329–344.

Berke

Backhurst

Day

et al. (2006) What makes plan implementation successful? An evaluation of local plans and implementation practices in New Zealand. Environment and Planning B: Planning and Design 33(4): 581–600.

Berke

French

(1994) The influence of state planning mandates on local plan quality. Journal of Planning Education and Research 13(4): 237–250.

Berke

Godschalk

(2009) Searching for the good plan: A meta-analysis of plan quality studies. Journal of Planning Literature 23(3): 227–240.

Berke

Spurlock

Hess

et al. (2013) Local comprehensive plan quality and regional ecosystem protection: The case of the Jordan Lake watershed, North Carolina, U.S.A. Land Use Policy 31: 450–459.

Bernstein

(2001) Local government measurement use to focus on performance and results. Evaluation and Program Planning 24(1): 95–101.

10.

Brody

Highfield

(2005) Does planning work? Testing the implementation of local environmental planning in Florida. Journal of the American Planning Association 71(2): 159–175.

11.

Bundesamt für Statistik (2014) Regionale Porträts 2014: Gemeinden [Regional Portraits: Municipalities 2014] [Statistics]. Neuchâtel: Bundesamt für Statistik. Available at: www.media-stat.admin.ch/maps/profile/data/237/fr/pdf/Portraits-regionaux-2014-communes.pdf (accessed 9 November 2015).

12.

Bunnell

Jepson

(2011) The effect of mandated planning on plan quality: A fresh look at what makes “a good plan”. Journal of the American Planning Association 77(4): 338–353.

13.

Conover

(1999) Practical Nonparametric Statistics, New York: Wiley.

14.

Conroy

Berke

(2004) What makes a good sustainable development plan? An analysis of factors that influence principles of sustainable development. Environment and Planning A 36(8): 1381–1396.

15.

Deyle

Smith

(1998) Local government compliance with state planning mandates: The effects of state implementation in Florida. Journal of the American Planning Association 64(4): 457–469.

16.

Edwards

Haines

(2007) Evaluating smart growth: Implications for small communities. Journal of Planning Education and Research 27(1): 49–64.

17.

Faludi

(2000) The performance of spatial planning. Planning Practice and Research 15(4): 299–318.

18.

Feitelson

Felsenstein

Razin

et al. (2017) Assessing land use plan implementation: Bridging the performance-conformance divide. Land Use Policy 61: 251–264.

19.

Gerber

J-D

(2016) The managerial turn and municipal land-use planning in Switzerland – Evidence from practice. Planning Theory & Practice 17(2): 192–209.

20.

Gilgen

(2012) Kommunale Raumplanung in der Schweiz, 3rd ed. Zürich: VdF Hochsch.-Verlag an der ETH.

21.

Godschalk

Beatley

Berke

et al. (1999) Natural Hazard Mitigation: Recasting Disaster Policy and Planning, Washington, DC: Island Press.

22.

Grădinaru

Iojă

Pătru-Stupariu

et al. (2017) Are spatial planning objectives reflected in the evolution of urban landscape patterns? A framework for the evaluation of spatial planning outcomes. Sustainability 9: 1279.

23.

Gregoire

Valentine

(2007) Sampling Strategies for Natural Resources and the Environment, Boca Raton, FL: Chapman & Hall/CRC.

24.

Guyadeen

Seasons

(2016) Plan evaluation: Challenges and directions for future research. Planning Practice & Research 31(2): 215–228.

25.

Hoch

(2007) How plan mandates work: Affordable housing in Illinois. Journal of the American Planning Association 73(1): 86–99.

26.

Hopkins

(2001) Urban Development: The Logic of Making Plans, Washington, DC: Island Press.

27.

Kaiser N, Rudolf S, Berli J, et al. (2016) Raumplanung in den Schweizer Gemeinden: Ergebnisse einer Umfrage [Spatial Planning in the Swiss Municipalities: Results of a Survey] (WSL Bericht Nr 42). Birmensdorf: Eidg. Forschungsanstalt für Wald, Schnee und Landschaft WSL.

28.

Kinzen

(2016) Missed connections: A critical analysis of interconnections between public participation and plan implementation literature. Journal of Planning Literature 31(3): 299–316.

29.

Krippendorff

(2013) Content Analysis: An Introduction to Its Methodology, Los Angeles, CA: SAGE Publications.

30.

Laurian

Day

Berke

et al. (2004) Evaluating plan implementation: A conformance-based methodology. Journal of the American Planning Association 70(4): 471–480.

31.

Loh

(2011) Assessing and interpreting non-conformance in land-use planning implementation. Planning Practice and Research 26(3): 271–287.

32.

Long

Han

(2012) Spatiotemporal heterogeneity of urban planning implementation effectiveness: Evidence from five urban master plans of Beijing. Landscape and Urban Planning 108(2): 103–111.

33.

Lyles

Berke

Smith

(2014) A comparison of local hazard mitigation plan quality in six states, USA. Landscape and Urban Planning 122: 89–99.

34.

Lyles

Berke

Smith

(2016) Local plan implementation: Assessing conformance and influence of local plans in the United States. Environment and Planning B: Planning and Design 43(2): 381–400.

35.

Lyles

Stevens

(2014) Plan quality evaluation 1994–2012: Growth and contributions, limitations, and new directions. Journal of Planning Education and Research 34(4): 433–450.

36.

McDonald

McMillen

(2004) Determinants of suburban development controls: A Fischel expedition. Urban Studies 41(2): 341–361.

37.

Mann

(2009) Institutional causes of urban and rural sprawl in Switzerland. Land Use Policy 26(4): 919–924.

38.

Mastop

Faludi

(1997) Evaluation of strategic plans: The performance principle. Environment and Planning B: Planning and Design 24(6): 815–832.

39.

Mueller

Hersperger

(2015) Implementing comprehensive plans: Indicators for a task-sheet based performance evaluation process. Journal of Environmental Planning and Management 58(11): 2056–2081.

40.

Newman

Thornley

(1996) Urban Planning in Europe: International Competition, National Systems, and Planning Projects, London: Routledge.

41.

Norton

(2005a) Local commitment to state-mandated planning in Coastal North Carolina. Journal of Planning Education and Research 25(2): 149–171.

42.

Norton

(2005b) More and better local planning: State-mandated local planning in Coastal North Carolina. Journal of the American Planning Association 71(1): 55–71.

43.

Norton

(2008) Using content analysis to evaluate local master plans and zoning codes. Land Use Policy 25(3): 432–454.

44.

Oliveira

Pinho

(2010) Evaluation in urban planning: Advances and prospects. Journal of Planning Literature 24(4): 343–361.

45.

Randolph

(2004) Environmental Land Use Planning and Management, Washington, DC: Island Press.

46.

Rudolf

Grădinaru

Hersperger

(2017) Impact of planning mandates on local plans: A multi-method assessment. European Planning Studies 25(12): 2192–2211.

47.

Stevens

(2013) Evaluating the quality of official community plans in Southern British Columbia. Journal of Planning Education and Research 33(4): 471–490.

48.

Stevens

Lyles

Berke

(2014) Measuring and reporting intercoder reliability in plan quality evaluation research. Journal of Planning Education and Research 34(1): 77–93.

49.

Talen

(1996) After the plans: Methods to evaluate the implementation success of plans. Journal of Planning Education and Research 16(2): 79–91.

50.

Talen

(1997) Success, failure, and conformance: An alternative approach to planning evaluation. Environment and Planning B: Planning and Design 24(4): 573–587.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.53 MB