Better Than Good: Three Dimensions of Plan Quality

Introduction

What constitutes a good plan is the focus of plan quality evaluation. However, when one examines the meaning of “quality” in plan quality evaluation, one cannot find a clear sense of this defining concept. Most frequently, “quality” is used synonymously with “good” and “better.” For example, Berke and Godschalk (2009) searched for the “good plan”; Lyles and Stevens (2014) associate plan quality with a “better plan.” Presumably, a good or better plan is one that attains a higher score on a scale using prescribed criteria, thereby being of higher quality. However, to say that good or better is a measure of quality is not sufficient because “good” and “better” are used without specific regard for the purpose of a plan or scope of plan evaluation. What, then, does “quality” mean? What does “good” mean? And so on.

In this commentary, our aim is to critically examine the meaning of “quality” in plan quality evaluation. To frame our discussion, we first discuss where and how plan quality, as a distinct type of evaluation, fits within the broader field of plan evaluation. We then examine the meaning of quality by exploring its different dimensions. As a final step, we direct our attention to the matter of principles that define each dimension of plan quality.

Through our examination, we aim to clarify and thereby strengthen the field of plan quality evaluation by refining what it means to evaluate the quality of a plan. In particular, our examination extends current discussions in the field of plan quality prompted by Norton (2008), among others, and responds to the “challenge before plan quality researchers . . . to refine plan quality principles to facilitate application across plan types” (Lyles and Stevens 2014, 438).

Scope of Plan Evaluation

We begin our examination by situating plan quality within the broader field of plan evaluation. Evaluation is widely recognized as the systematic acquisition and assessment of information to provide useful feedback about the significance, worth, or condition of some object or intervention. Accordingly, the field of plan evaluation can be viewed as a systematic evaluation that takes plans, the planning process, planning outputs, or outcomes as the object or intervention.

Whereas the field of plan evaluation can be extended to public policy and related fields, in the following brief account of the field of plan evaluation we look inward in order to focus our discussion of plan quality as a subfield of plan evaluation. We will discuss how the diversity of the field of plan evaluation presents a challenge for understanding where plan quality fits. This diversity is due to different ways of viewing a plan, different approaches used to evaluate plans, and different stages in the planning process when evaluation takes place.

First, plans are viewed as different types of documents that serve different purposes. By way of contrast, plans are often seen as either strict blueprints that articulate clear goals and policies or as frameworks that guide decisions (e.g., Guyadeen and Seasons 2016; Laurian et al. 2004). Plans can also be viewed as a decision that should be made in light of other decisions, whereby a plan can work as an agenda, policy, vision, design, and strategy (Hopkins 2001). In broader terms, Baer (1997) identifies plans as a vision, blueprint, land use guide, remedy, administrative requirement, pragmatic action, and responses to senior government mandates.

That plans can be viewed as different things is not at issue; rather, it is important to recognize that one’s view of a plan is bound fundamentally to how it is evaluated. Correspondingly, as noted throughout the plan evaluation literature (e.g., Baer 1997; Norton 2008; Oliveira and Pinho 2010; Talen 1996), one cannot separate plan evaluation from one’s notion of success and its criteria. By necessity, this relationship is derived from and embedded in one’s view of a plan. An example helps to illustrate the relationship. Hopkins (2001) notes that the success of plans can be viewed with a focus on understanding how plans are used in decision-making situations, whereby he identifies four criteria: interdependence (the value of the results of a decision depends on other decisions), indivisibility (the decision cannot be made in infinitesimally small steps), and irreversibility (the decision cannot be reversed without cost) of decisions in the face of imperfect foresight (we lack complete knowledge of the future). This view of plan success is derived from Hopkins’s central focus on planning decisions as a series of actions, whereby, “in simplest terms, plans provide information about interdependent decisions” (2001, 5) and a plan “identifies a decision that should be made in light of other concurrent or future decisions” (2001, 5). Accordingly, “the Four Is define the circumstances in which plans can improve outcomes” (Hopkins 2001, 6 [emphasis added]). By this account, Hopkins evaluates the success of a plan based on the usefulness of a decision, which is an approach that, according to Lyles, Berke, and Smith (2016), aligns with performance evaluation during plan implementation. Through this example, we can see how one’s conception of success is embedded in one’s view of a plan. By extension, the relationship between “plan quality” is inseparable from how one views “quality.”

As a way of organizing the field of plan evaluation, one can also focus on different approaches. Laurian et al. (2010) describe three general approaches: a rational approach examines linkages between plans (as blueprints) and actual developments; a communicative approach examines the performance and use of plans (as guides) as well as the planning process; an integrative approach makes use of both rational and communicative approaches depending on the context and use of a plan (e.g., blueprint or guide). In a similar way, Alexander (2000) presents four complementary views of planning: rational planning, communicative practice, coordinative planning, and frame setting. All of these views involve different kinds of actors who are doing different kinds of planning.

One can also consider plan evaluation in relation to stages of the planning process. Three phases of the planning process during which evaluation takes place are commonly identified: during plan preparation (ex ante or a priori); during plan implementation (ongoing); and after the plan is implemented (ex post) (Alexander 2006a; Laurian et al. 2010; Oliveira and Pinho 2010). Baer (1997) identifies five stages in plan-making during which evaluation can take place: (1) plan assessment, (2) plan testing and evaluation, (3) plan critique, (4) comparative research and professional evaluations, and (5) post hoc evaluation of plan outcomes. Regardless of how the stages may be defined, the timing of the evaluation is strongly associated with the object of study, whereby one might focus on comparing possible alternatives during plan preparation, on results and use of resources during plan implementation, or on whether and to what degree plan policies are carried out (conformance) or their role in effecting change when the plan is implemented (performance).

Collectively, the different perspectives of and within plan evaluation have evolved considerably over the past fifty years during what is often described as a shift from a positivist to constructivist paradigm (Alexander 2002; Oliveira and Pinho 2010). Communicative characteristics of plans are premised on social interaction as a legitimate basis for reason, such that knowledge is not based solely on other individualistic and logical-empirical forms of rationality (Alexander 2006b). Therefore, when a plan is viewed as a communicative policy act in the context of uncertainty and pluralistic interests, it is viewed as a series of assertions that is part of an ongoing, deliberative decision-making process (Norton 2008). Thus, a plan can serve several purposes: advise local officials, explain decisions to local citizens, and justify decisions to the courts (Norton 2008).

Given the shifting perspectives within plan evaluation, what constitutes a “good” plan is “intrinsically relative and historical, and combines problematic issues of capacities, intentions and outcomes” (Alexander 2002, 191). This process of shifting perspectives continues, contributing to several contemporary issues identified by Oliveira and Pinho (2010, 343), as follows:

While these issues highlight important elements of plan evaluation, the breadth of these issues also suggests a field of inquiry and practice still under development.

In context of the diversity of plan evaluation and its ongoing development, several issues obscure the specific contribution of plan quality to plan evaluation. First, the term quality is used throughout the literature, and often in general terms. A more obfuscating factor relates to where plan quality fits among the stages of the planning process. As noted above, most discussions about types of evaluation are aligned with stages of the planning process, yet evaluating a plan as an object does not align well with the commonly used stages. General approaches to evaluation research, at a minimum, distinguish between evaluating plan development and plan implementation. However, evaluating a plan as an object of study does not align precisely with either plan development or implementation; it is something between the two. A plan is an outcome of a development process and the object used afterward in planning decisions. This problem of alignment arises regardless of whether types of evaluation are characterized as three stages (e.g., ex ante, ongoing, and ex post) (e.g., Alexander 2006a; Laurian et al. 2010; Oliveira and Pinho 2010; Talen 1996) or five stages (e.g., Baer 1997). Thus, although aligning types of evaluation with stages in the planning process helps to frame the scope of plan evaluation, this way of organizing approaches to plan evaluation contributes to ambiguity when it comes to understanding where plan quality fits.

One step to resolving the confusion about where plan quality fits in the field of plan evaluation is to more clearly articulate how plan quality evaluation can be seen as a distinct method. Working within the above definition of plan evaluation, we agree with the definition of plan quality evaluation presented by Lyles and Stevens (2014, 434): “the process by which plan content analysis data is linked to normative criteria of what constitutes a better plan.” This definition of plan quality has two features that separate it from other types of plan evaluation. One feature is that it takes place “after they [plans] have been developed.” The other feature is the use of content analysis. To underscore these two features of plan quality evaluation, we emphasize that plan quality evaluation has a primary focus on a plan as it is written as the object of evaluation (hereafter, for our purposes, shortened to “plan-as-object”).

To build upon an understanding of plan-as-object, we introduce efficacy as an additional feature unique to plan quality evaluation. Efficacy is “the power to produce a desired result or effect” (Merriam-Webster n.d.). By definition, the words efficacy and effective are very similar. To emphasize the difference, effective is about producing an effect; efficacy is about the power to produce an effect. When a plan is taken as the object of evaluation, it is the efficacy of a plan as it is written that is evaluated against the desired result (based on preestablished, normative criteria). That is, the power of a plan to produce an effect is embodied in its text, not in its application. It is this difference that sets efficacy apart from the other terms and serves our purpose to draw a clearer distinction between plan quality evaluation and other types of plan evaluation. As distinct from a focus on a plan’s development (efficiency of process) or implementation (effectiveness of application), an evaluation of a plan’s efficacy serves to focus attention on the plan itself. Thus, based on Alexander’s (2006a) means of organizing types of evaluation, we believe that plan quality can be defined clearly as a distinct type of plan evaluation with regard for both the object and timing of evaluation.

With this understanding of plan quality as a distinct type of evaluation, we divide the field of plan evaluation into three areas, as shown at the top of the chart in Figure 1: plan development, plan quality, and plan implementation. In light of the many approaches and types of plan evaluation mentioned above, this division is primarily heuristic; it is an attempt to simplify a diverse field of study as a means to focus on plan quality evaluation as distinct within the field of plan evaluation research.

OPEN IN VIEWER

Figure 1.

The scope of plan evaluation and dimensions of plan quality.

Dimensions of Plan Quality Evaluation

Having framed our focus on the evaluation of plan-as-object as a distinct type of evaluation, we now direct our attention to the dimensions of plan quality. The bulk of plan quality evaluation efforts, as summarized by Berke and Godschalk (2009) and Lyles and Stevens (2014), is synonymous with “plan quality.” However, as we noted at the outset, “quality” is most often used synonymously with “good” and “better.” As Berke and Godschalk (2009, 228 [emphasis added]) state, “Only systematic evaluation enables us to identify their specific strengths and weaknesses, to judge whether their overall quality is good, and to provide a basis for ensuring that they reach a desirable standard.” However, an undifferentiated concept of “quality” fails to reflect important differences identified within plan quality evaluation, which is problematic when it comes to refining the scope and dimensions of quality. That is, we need to move beyond this ambiguous relationship between quality and good/better.

We posit that different dimensions of “quality” can be identified within the current scope of plan quality evaluation. The “core principles” identified by Lyles and Stevens (2014) provide a starting point. These principles include goals, fact bases, policies, public participation in plan creation, and plan provisions for implementation and monitoring. The consensus on this set of principles is the outcome of a measured process of adding and refining elements of what constitutes a good plan. For example, early work focused on three general characteristics of quality: a strong factual basis, clearly articulated goals, and appropriately directed policies (Kaiser, Godschalk, and Chapin 1995). These three categories were soon expanded by Baer (1997), who developed an extensive set of sixty criteria that were organized under eight categories. Berke and Godschalk (2009) identified ten internal and external characteristics. However, as Lyles and Stevens (2014) note, the consensus that emerged from these ongoing developments is “largely focused on the rational comprehensive view of planning and prioritize criteria that conceive of a plan as a blueprint or agenda” (2014, 438). We agree with this association between these principles and a rational comprehensive approach. Correspondingly, we identify this area of plan quality evaluation as one dimension of quality. Consistent with a rational comprehensive approach, and as reflected in the range of criteria associated with this approach, we propose that this dimension of plan quality centers on the quality of documentation as measured by comprehensiveness. (The use of the term “documentation” is consistent with Norton’s [2008] characterization of analytical quality, as discussed below.)

To explore additional dimensions of plan quality evaluation, we now shift our attention to the communicative characteristics of plan quality evaluation. Specifically, Lyles and Stevens (2014) point to critiques by Norton (2008) and Bunnell and Jepson (2011). Both critiques reflect a discursive approach to planning, whereby plans are viewed as guides rather than blueprints, as frameworks for consensus building rather than means of control (Guyadeen and Seasons 2016). Norton’s (2008) critique centers on a concern that the criteria to measure overall plan quality do not account for conceptually distinct dimensions of plans associated with its different functions. As Norton (2008, 433) argues, “the resulting need [is] to separate analytically the policy message of the plan from the conveyance of that message.” Thereby, Norton focused on “policy focus” as a distinct dimension that was previously obscured by existing measures of and focus on “plan quality.” The latter refers to the analysis of a plan’s documentation, or “analytical quality,” which Norton operationalizes using criteria that are similar to the components used in other studies of plan quality.

For our purposes, it is important to emphasize that we agree with Norton that policy focus is a distinct dimension of evaluation. However, whereas Norton argues that policy focus is something different from and should not be conflated with “plan quality,” we take a different view. We see policy focus not as distinct from but as one dimension of plan quality. That is, in our view, Norton’s distinction between “plan quality” (analytical quality) and policy focus points to two dimensions of plan quality, which helps to resolve what Norton characterized as paradoxical implications of trying to separate policy focus from plan quality. In effect, we have realigned Norton’s terms without changing the foundation of his argument.

Isolating policy focus as a dimension of quality is a critical step to separate the means from the ends, but policy focus does not encompass other aspects of a communicative approach to plan quality evaluation. To fill this gap we suggest a third dimension of quality, which we identify as “discourse.” Adding this third dimension captures discursive elements associated with guides rather than blueprints, with consensus building rather than means of control, which are elements that reflect interactive and normative characteristics of a plan. In this regard, both Norton (2008) and Bunnell and Jepson (2011) directly associate this discursive dimension of plan quality with persuasion. Norton states that a plan, as a communicative policy act that makes assertions, is “especially useful for contemplating the extent to which the plan represents persuasive and undistorted communication” (2008, 443 [emphasis added]). Similarly, Bunnell and Jepson (2011, 338 [emphasis added]) argue, “the extent to which plans persuasively connect with readers, and elicit their positive participation, should constitute the core notion of what constitutes a good plan.” Persuasion—the ability to engage and motivate—is necessary, they argue, in order to garner the attention of policy makers and the public.

Thus, we distinguish three dimensions of plan quality, as follows: quality of documentation (comprehensiveness), quality of policy focus (strength), and quality of discourse (persuasion) (as shown above in Figure 1).

Three Sets of Core Principles

To consolidate our ideas, we propose sets of principles to more clearly distinguish each of the three dimensions of plan quality. Consistent with Lyles and Stevens (2014), principles refer to the characteristics that are considered to be widely applicable to each dimension of plan quality. Each set of principles frames the scope of analysis for each dimension of plan quality and can be used to guide the development of specific criteria according to the needs of analysis.

The first set of principles was discussed above, the core principles identified by Lyles and Stevens (2014). Consistent with Lyles and Stevens, we view this set of principles in relation to a rational comprehensive approach to plan quality evaluation.

With regard for a set of principles that inform the “discourse” quality of plans we turn to Norton’s (2008) characterization of a plan as a communicative policy act and his criteria for evaluating communicative action. As Norton explains, there are four key characteristics derived from Habermas’s notion of the ideal speech act. These characteristics are comprehensibility (can the assertion be comprehended or understood?), truth or accuracy (is the statement demonstrably correct?), legitimacy (does the person making the statement have a legitimate basis upon which to do so?), and sincerity (is the person making the statement sincere or is he or she attempting to be manipulative?). In Habermas’s ideal speech, these four factors are the basis for testing the acceptance of an assertion. In relation to plan quality, “the written assertions made by a plan can be ‘tested’ using criteria that parallel those used with the speech act” (Norton 2008, 443). These four principles, as operationalized by Norton, can be stated as follows:

With our attention directed at the plan-as-object, we present these four characteristics as the set of principles for the discursive dimension of plan quality.

To complete our examination of the dimensions of plan quality, we are left with one item to address: a set of principles that reflect the strength of policy focus. Although Norton (2008) prepared criteria to measure a policy focus on neotraditional landscapes, he does not offer a generic set of principles that guide measures of the strength of any policy focus. To fill this gap, we propose the following set of four principles: maximize stability, minimize uncertainty, integrate public priorities across jurisdictions, and accommodate flexibility.

The proposed set of principles for quality of policy focus is premised on a view of planning (Connell 2009) that is based on a general theory of society (Luhmann 2012, 2013). Our interest is in the function of planning for society as a basis to identify a set of generic principles that is independent of any particular policy focus and plan content. From a broad perspective, planning enables communication about a future that can never be fully visible to society. Correspondingly, planners seek stability by maximizing what we can know about the future and seek security by minimizing what we do not know, thereby establishing a domain of understanding within which to make the best possible decisions in the present. This theoretical perspective led us to two principles that can be analyzed separately: maximize stability; minimize uncertainty. The other two principles that we propose come from different sources. The principle of “integration of public priorities across jurisdictions” is evident among existing plan quality criteria, for example, interorganizational coordination (Berke and Godschalk 2009) and vertical consistency (Norton 2008). As well, Smith (1998) places significant emphasis on this principle for agricultural land use planning in British Columbia, Canada, which has a strong policy focus to protect farmland. The fourth principle of accommodating flexibility is a more pragmatic response to the goal of achieving a balance among competing interests. In more detail, each of the four principles can be used as a basis for evaluating the strength of a plan’s policy focus as follows.

Conclusion

While many authors have reflected on understanding and measuring the “good” plan, there has been far less discussion about the meaning of “quality.” Similar to concerns raised by Norton (2008), we believe that important differences are obscured by conceptual problems that define the field of plan quality evaluation. To address this concern, we questioned the presumption that quality is synonymous with “good” or “better,” which led us to three important outcomes. First, plan quality can be understood as one type of plan evaluation with a unique focus on efficacy, that is, evaluating a plan—as it is written—against its desired result. Second, the field of plan quality evaluation can be viewed as having three dimensions of quality: comprehensiveness as a measure of documentation quality, strength as a measure of policy focus, and persuasiveness as a measure of discourse quality. From these two outcomes we recognized a gap in the literature: the absence of a set of principles to guide the measure of policy focus. To fill this gap, we proposed a set of four principles: maximize stability, minimize uncertainty, integrate public priorities across jurisdictions, and accommodate flexibility. The outcome, as summarized in Figure 1, is a picture of the scope of plan quality evaluation and its three dimensions of quality, each with a corresponding set of principles.

Whereas a successful plan quality evaluation links content analysis methods with theoretical arguments and measures of plan characteristics (Lyles and Stevens 2014), a successful evaluation also requires a clear understanding of what constitutes the “quality” of a good plan. By articulating three sets of principles that correspond with three dimensions of quality, we help to resolve the paradoxical implications identified by Norton (2008) of conflating different ends of a plan by using a single measure of quality. By the same means, we also help address the challenge to refine plan quality principles to facilitate application across plan types.