Novel Methods,Novel Metrics: Using a Meta-Ethnography to Create a Plan Quality Framework for Sustainable and Resilient Social

Abstract

Social–ecological resilience (SER) and sustainable commons management (SCM) are process- and outcome-based paradigms for natural resource management. Resource management plans foster them, but traditional plan quality assessment typically omits process dimensions. We examined what constitutes an appropriate plan quality evaluation framework for resource management plans that reflect these SER and SCM paradigms. In order to create a plan quality instrument for assessing embedded SER and SCM processes, we employed a two-part methodology of a meta-ethnographic analysis and then used the results to augment current advances in plan quality evaluation to assess adaptation efforts. While methodologically novel in the academic planning canon, the meta-ethnographic approach has been used to inductively synthesize qualitative studies’ results in, for example, the fields of public health, medicine, and education. Our use advances plan quality evaluation with a new metric that combines the paradigmatic resource management strategies and addresses identified shortcomings in the more novel adaptive plan quality evaluation approach that builds on the former, traditional one.

Keywords

plan quality evaluation social–ecological resilience sustainable commons management meta-ethnography

Resource management has evolved to include the integral linkage between social and ecological systems, differentiating each as their own system but the two together as a functional whole. This linkage concept is emerging throughout the planning discipline and in resource management more generally (e.g., the food–water–energy nexus). Ostrom and Cox (2010) state that “complex social-ecological systems (SESs)…are defined by Anderies, Janssen, and Ostrom (2004) as social systems ‘in which some of the interdependent relationships among humans are mediated through interactions with biophysical and non-human biological units’” (p. 3). Resource managers can no longer assess only the ecosystem, while social scientists must consider the role of the ecosystem in responding to their governance systems, particularly with the rapid global and local ecosystem changes anticipated from climate change. Social–ecological resilience (SER) and sustainable commons management (SCM) have emerged as paradigms that include both process and outcomes to address perceived failures in traditional command and control (Ostrom 1990, 2009) and predict and control approaches for natural resource management (Pahl-Wostl et al. 2007). Consequently, resource plans ideally reflect these emerging management paradigms, and plan quality assessment instruments/frameworks follow in order to evaluate how well the plans reflect them.

While it may appear that the plan quality assessment literature is focused on local jurisdictions and that resource management plans are more regional in nature, there is a substantial body of work supporting the application of the local plan quality evaluation to other forms of regionally based plans (e.g., hazard mitigation, Berke et al. 2014 and coastal management plans, Norton 2005). Additionally, as Brody (2003) notes, “planners and managers increasingly recognize that while ecosystem management requires looking beyond specific jurisdictions and focusing on broad spatial scales, the approach will in part be implemented at the local level with local land use decisions. Furthermore, ecosystem approaches to management may not be realized solely by structural or engineering approaches to management, but by the coordination of local plans and policies across larger landscapes” (p. 512). And yet, there is no comparative framework through which to assess the resilience of resource management plans because of the scalar, process, and conceptual inadequacies of the current plan quality evaluation approach.

We also recognize that resource plans vary considerably in character, crossing a wide spectrum of natural resource types/features (e.g., open ocean and estuarine coastal areas, watershed/river basins, forests, parks, green infrastructure, ecological systems) and levels of both comprehensiveness and scale. The variability suggests that the conceptual elements in the plan quality assessment instrument must be carefully—but systematically and rigorously—chosen with the dual objectives of sufficient universality for broad application and sufficient depth to meaningfully evaluate multiple schools of thought simultaneously (e.g., common pool resource [CPR] management conditions, SER and SCM process and outcomes). While retaining the basic skeletal elements of the widely accepted local plan evaluation approach (Berke et al. 2006, 2014), we sought to scalarly, substantively, and procedurally adapt it by determining what constitutes an appropriate assessment framework for these variable resource management plans (whether common pool or not) using both a review of the existing frameworks and a meta-ethnography of SER and SCM studies to identify additional process indicators. Based on these findings, we developed a new and more universal framework to assess resource plans that foster the SES components to achieve a sustainable and resilient outcome for both social and environmental systems.

This work represents the first step of a larger natural resource-focused plan quality study. We conducted a meta-ethnography of the studies that articulate and measure the SER and SCM processes and outcomes in order to synthesize rather than summarize their results, introducing a widely accepted, rigorous methodology used in the public health and medical disciplines (as well as the education research field, more recently) to the planning methodological canon (France et al. 2019). A meta-ethnographic analysis is a synthesis approach for qualitative studies that “draw[s] on Geertz’s concept of thick description and Turner’s theory of sociological understanding as ‘translation’. It is unique among qualitative evidence synthesis methodologies in synthesizing conceptual data from primary studies and was designed specifically to take into account the unique research contexts in primary studies” (France et al. 2019, 36). It provides an analytic framework that exceeds a traditional literature review and qualitative content analysis (which categorizes themes and their frequency) by instead “building ‘comparative understanding’ rather than aggregating data…[through] three different methods of synthesis. One involves the ‘translation’ of concepts from individual studies into one another, thereby evolving overarching concepts or metaphors. [The second,] refutational synthesis, involves exploring and explaining contradictions between individual studies. [The third,] lines-of-argument synthesis involves building up a picture of the whole from studies of its parts.” (Barnett-Page and Thomas 2009, 60).

We begin with a review of the two primary literatures to inform the meta-ethnography: plan quality and SER and SCM (conceptually differentiated but highly related bodies of work). We then present the methodology used to analyze studies to address our fundamental question of what constitutes an appropriate evaluation framework for research management plans. We build on Agrawal’s (2003) and Anderies, Janssen, and Ostrom’s (2004) foundational assessments of the SCM studies and systematically identify both subsequent SCM and SER studies to isolate indicators of their efficacy. Expanding on the existing plan quality assessment approaches referencing environmental characteristics and adaptive capacity, including Brody’s (2003) environmental plan quality components and the Berke et al.’s (2014) effort to incorporate adaptive capacity into plan quality through dimensionality, we develop the basis of the new resource plan quality metric. Finally, we incorporate the chosen subset of SER and SCM process indicators culled from the meta-ethnography and the existing literature to create a new plan quality framework with sufficient breadth and depth to be applied to a variety of resource plans. Findings, including our proposed resource management plan review framework, follow. We conclude with a discussion of the implications of this framework and of potential future research directions.

Relevant Literature

Plan Quality

There is an established literature on plan quality assessment for comprehensive land use planning (Kaiser, Godschalk, and Chapin 1995), captured in Baer’s (1997) seminal distinctions of plan assessment, testing and evaluation, and professional evaluation. His questions for plan assessment regarding the adequacy of context, rational model considerations, adequacy of scope, and guidance for implementation helped to inform the now standard internal and external plan quality work from Berke et al. (2006) (Norton 2005; Brody 2003; Baer 1997). The established basic elements of plan quality metrics include the fact base, the goals, the policies/plan proposals, and both implementation and monitoring (Berke et al. 2006). Norton (2008) makes the important distinction of the plan as “a communicative policy act” (p. 433), and in conducting the content analysis of the plans as a way to determine their efficacy, there is a distinction between the “communicative content (i.e., the focus of its development management and substantive landscape goals and policies) [and] its quality (i.e., its ability to convey those goals and policies clearly and to provide evidence and analysis sufficient to justify them)…it is also necessary to frame that analysis in terms of the larger policy goals of interest to the research to ensure that the analysis is meaningful, policy relevant, and useful” (p. 433). This distinction is further supported by other scholars’ critiques regarding process and participation, which are often omitted in traditional plan quality assessments likely due to the inconsistent inclusion of such elements in the final plan document (Berke et al. 2006). However, it is important to note that Conroy and Berke (2004) analyzed the influence of process elements, including participation measures, on their evaluation of plan sustainability (quality) scores. Their work found that the variety of participants (participation breadth) was a significant contributor to higher plan scores, while the manner in which they participated (participation depth) was not.

Plan quality assessments are traditionally focused on achieving objectivity and measures of intercoder reliability rather than the public accessibility and appeal of the plan (Bunnell and Jepson 2011). There is a debate in the academy about planning mandates’ effects on plan quality, particularly the plan’s “communicative and persuasive qualities,” since it is challenging to incorporate process concepts into plan quality evaluation (Bunnell and Jepson 2011, 340; Pendall 2011; Berke et al. 2014; Berke and French 1994; Burby and May 1997; Baer 1997). Bunnell and Jepson (2011) found that when plans are assessed for these qualities, state-mandated plans are no better than nonmandated plans (p. 351). Pendall (2011) found that Maine’s mandates had little effect on either plan submission or compliance. Bunnell and Jepson (2011) suggest that other process aspects of plans (i.e., persuasiveness and the communicative nature of plans) cannot be easily mandated—although they also found that there needed to be flexibility and local tailoring/context-specific options available through the legislation.

While not easy to draft or legislatively conceptualize, mandates for resource management plans are beginning to reflect more challenging resource management concepts that involve process and participation (Dyckman 2016). Plan quality metrics are evolving as well, including assessing concepts of adaptive capacity and resilience (Berke et al. 2014), and environmental quality (Brody 2003). Brody’s (2003) work added two integral plan components to plan quality analysis, the second of which is now standard in plan quality assessment: “interorganizational coordination” and “capabilities and implementation;” the first “captures more accurately the aspects of collaboration and conflict management often required with ecosystem approaches to management” while “the implementation component measures how likely the goals, objectives, and policies in the plan are to be put in place (not if implementation actually occurred)” (p. 514). Implementation assessment is manifested through monitoring, training/technical assistance, sanctions, time frames, and updating requirements (Brody 2003, 515–17). But more importantly, Brody (2003) lists environmentally related facts, goals, and objectives that are arguably the basis of both social and ecosystem service assessments. These include resource inventories, ownership patterns, and human impacts in the fact base, which begin to document the linkage/integration (and potential resilience) between social and ecological systems. They lay the groundwork for the ecological resilience side of SER, though the focus on the ecological lacks balance with the social system and its needs. SESs do not conceptualize the ecosystem as an isolated entity, which means that resource plans must address both the dynamism of the combined SES and the linkage between the two systems (social and ecological).

In response to the uncertainty associated with planning for disaster recovery, Berke et al. (2014) sought to empirically evaluate the adaptive nature of disaster recovery plans. In doing so, they altered the plan quality metric, integrating plan quality principles with planning process dimensions. They divided the plan assessment metric into two sets of principles: direction-setting and action-oriented indicators. The direction-setting indicators included the standard goals, fact base, and plan proposals/policies, which were tailored to flexible recovery measures (Berke et al. 2014, 312). The action-oriented indicators included interorganizational coordination, participation, and implementation and monitoring (Berke et al. 2014, 312). The participation section is intended to “engage the public to build a knowledgeable constituency able to create a plan that reflects local values, needs, and capabilities, and enable ongoing public input throughout the recovery process” and has a “narrative on who was involved in pre-disaster planning, how they participated, and how they affected evolution of [the] plan” (Berke et al. 2014, 312). However, it does not contain metrics assessing the types of system users nor does it reflect the integrated nature of the social and ecological systems. These integrated systems are fundamental components of emerging resource management concepts, SER, and SCM processes. Together, Brody (2003) and Berke et al. (2014) lay the groundwork for altering the plan quality assessment metric to reflect actual practice in resource management, since these are the leading conceptual articles on both resource management and adaptive capacity in plan quality assessment.

SER and SCM

Since the introduction of resilience concepts in the biological sciences (Holling 1973), scholars have been addressing basic elements of the ecological system that respond to disturbance and retain their basic functions (Folke et al. 2005; Walker and Salt 2012). Scholars across both the physical and social sciences (Ostrom 1990) are increasingly recognizing the integrated relationship between social and ecological systems, and that studying one in isolation can be detrimental to realizing overall resilience. Consequently, studies of SESs as integrated, rather than isolated, systems have been emerging for the past fifteen years. They are focused on different aspects of the social system, the ecological system, or the combined SES, depending on the project. Some of the resource systems are common pool, while others are not.

As previously stated, SER and SCM have emerged as paradigms that include both process and outcomes to address perceived failures in traditional command and control (Ostrom 1990, 2009) and predict and control approaches for natural resource management (Pahl-Wostl et al. 2007). Researchers who are documenting and evaluating these paradigms have tried to identify the necessary preconditions that support a successful avoidance of the tragedy of the resource commons with increasing environmental and global pressures over natural resources. Walker and Salt (2012) generated a nuanced understanding of resilience thinking that they translated into measures of resilience practice, or “SER analytical tools,” which are “used to describe and measure the resilience of a system with very specific features such as the adaptive cycle” (Deppisch and Hasibovic 2013, 120). Ostrom (1990, 2009) and Agrawal (2003) have examined the SCM characteristics across several resource management settings, finding that, while highly contextual, a small group of CPR users, an allocated level of responsibility, and nested scales of governance are preconditions of improved resource management. Collaborative watershed management is a subset of SCM. Pahl-Wostl et al. (2007) have affirmed and examined the role of SER and SCM’s shared adaptive capacity features and social learning, which is also known as collaborative environmental decision-making, in water resources management. According to Pahl-Wostl (2007) citing Tompkins and Adger (2004), “community-based management enhances adaptive capacity in two ways: by building networks that are important for coping with extreme events and by retaining the resilience of the underpinning resources and ecological systems” (p. 5). Combined with the sustainable commons systems thinking and practice, “a resilience perspective, as a way of thinking about social-ecological systems, may serve as a common conceptual framework in transdisciplinary research processes aimed at facilitating adaptation to climate change” (Deppisch and Hasibovic 2013, 118). With climate change uncertainty and increasing variability in resource availability, resource management plans may foster the SER and SCM processes that are organically manifesting.

It is challenging for legislators/policy makers and researchers to codify resilience characteristics, but several can be captured in the mandated process structure associated with resource planning (Dyckman 2016). For instance, the characteristic of self-organization means that the SES is complex and adaptive; when a component of the system changes, the whole system will self-organize around the change (Holling and Gunderson 2002). The limits to self-organizational capacity are manifested through thresholds over which, once the system is changed, it can’t recover and potentially transforms. In practice, the thresholds (where identifiable i.e. an enduring drought) are prioritized as something for which to watch and plan. The linked social, economic, and biophysical domains of an SES are fairly self-explanatory; this linked relationship needs to be acknowledged and maintained (i.e., the basic definition of sustainability) in the plans and the legislation shaping the plans. Through their own internal processes, these systems move through adaptive cycles, which can also be addressed in the resource plans, and the assessment approach to evaluate their quality (Holling and Gunderson 2002). The “way that the components of the system interact causes the system to go through cycles in which the connections between its components tighten, loosen, and even break apart. As this happens, the capacity of the system to absorb disturbance (its resilience) also changes, as does the potential for people managing the system to make changes” (Walker and Salt 2012, 13).

There are also different dimensions of resilience in the system, including specified and general resilience, and transformability (Folke et al. 2010; Walker et al. 2004). With specialized resilience, there is a shock that affects part of or a component of the system and that part is able to absorb the effect and continue (Walker and Salt 2012). “General resilience is the capacity of a system that allows it to absorb disturbances of all kinds, including novel, unforeseen ones, so that all parts of the system keep functioning as they have in the past” (Walker and Salt 2012, 18). Maintaining one may come at the expense of the other. Finally, “transformability is the capacity of a system to become a different system, to create a new way of making a living” (Walker and Salt 2012, 20). The system actually shifts into another system after crossing irreversible thresholds that can lead to a new state of being, whether positive or negative (Holling and Gunderson 2002).

Context and social cooperation comprise the foundation of the SCM approach for SESs that have been evolving since Ostrom’s (1990) seminal work in which she presented the concept of the polycentric (i.e., multiple stakeholders) and multi-scale governance structure for commons management. As previously noted, some of the “critical enabling conditions for sustainability on the commons” (Agrawal 2003, 253) include the small size of the CPR user groups; the identity of the resource users; the level of responsibility allocated; and the concept of nested scales of responsibility, also known as polycentricity (Ostrom 2009, 2010; Dyckman and Paulsen 2012). Both Agrawal (2003) and Anderies, Janssen, and Ostrom (2004) examined the institutional preconditions for “long-enduring” sustainable resource management. In addition to Agrawal’s findings from a qualitative synthesis of SCM studies, Anderies, Janssen, and Ostrom included the following principles in place between the human actors in an SES: (1) clearly defined boundaries, (2) proportional equivalence between benefits and costs, (3) collective-choice arrangements, (4) monitoring, (5) graduated sanctions, (6) conflict-resolution mechanisms, (7) minimal recognition of rights to organize, and (8) nested enterprises (Anderies, Janssen, and Ostrom 2004, 8). The definitions associated with these concepts are included in the coding approach in the Appendix and in the coding spreadsheet template illustrating our coding process (concepts, definitions, approach) in Table 1. Effectively, these authors are focused on the social and institutional side of the SES in determining how the human actors manifest a resilient and sustainable resource outcome (for both the humans and the ecological system). These principles have been affirmed and supported in the subsequent literature, which is why we draw upon them.

Table 1.

Coding Spreadsheet Template.

Study Elements Being Coded		Definitions/Examples	Coding Approach
Author(s)		All named authors	OE
Year		Year the study was published	OE
Title		Title of the study	OE
Study’s main concepts	Unit(s) of analysis	What they’re studying/what they’re trying to measure	OE
	Research question(s)	Stated or inherent research question	OE
	Hypothesis/es	Stated or inherent hypothesis/es being tested	OE
	Quantitative outcomes (if any)	Increase in species populations, greater water quality/quantity, and so on	OE
	Qualitative outcomes (if any)	Better communication among the resource managers and users, a new element that needs to be considered, and so on.	OE
	Study setting	Study location	OE
	Study participants	Number, population group(s) of study participants	OE
	Method(s)	Methodological approach to answer the research question(s)	OE
	Sampling techniques	How the authors chose the unit of analysis	OE
	Governance system(s)	For the resource, including one or more of the following: government organizations, nongovernment organizations, network structure, property rights systems, operational rules, collective-choice rules, constitutional rules, and monitoring and sanctioning process (Ostrom, 2009, 421)	OE
	Resource units	Trees, fish, water quality, and so on	OE
	Kind of resource system	Broader version of the resource units (i.e., forest, ocean, and watershed)	OE
	Resource users	Who may be differentiated from the study participants	OE
Ostrom, Agrawal, and Anderies et al.’s sustainable commons management principles	Clearly defined boundaries	“The boundaries of the resource system (e.g., irrigation system or fishery) and the individuals or households with rights to harvest resource units are clearly defined.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Proportional equivalence between benefits and costs	“Rules specifying the amount of resource products that a user is allocated are related to local conditions and to rules requiring labor, materials, and/or money inputs.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Collective-choice arrangements	“Most individuals affected by harvesting and protection rules are included in the group who can modify these rules.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Monitoring	“Monitors, who actively audit biophysical conditions and user behavior, are at least partially accountable to the users or are the users themselves.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Graduated sanctions	“Users who violate rules-in-use are likely to receive graduated sanctions (depending on the seriousness and context of the offense) from other users, from officials accountable to these users, or from both.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Conflict resolution mechanisms	“Users and their officials have rapid access to low-cost, local arenas to resolve conflict among users or between users and officials.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Minimal recognition of rights to organize	“The rights of users to devise their own institutions are not challenged by external governmental authorities, and users have long-term tenure rights to the resource.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Nested enterprises	“Appropriation, provision, monitoring, enforcement, conflict resolution, and governance activities are organized in multiple layers of nested enterprises.” (Anderies, Janssen, and Ostrom 2004, 8)	LS
	Accommodation of norms/building social capital	“users of all types of resource systems who share moral and ethical standards regarding how to behave in groups they form, and thus the norms of reciprocity, and have sufficient trust in on another to keep agreements will face lower transaction costs in reaching agreements and lower costs of monitoring” (Ostrom 2009, 421)	LS
	Social learning	“collaborative environmental decision-making” using inter-directional information and data sharing, using cooperation (Freeman and Farber 2005, 836)	LS
	Number of users	“the impact of group size on the transaction costs of self-organizing tends to be negative given the higher costs of getting users together and agreeing on changes. If the tasks of managing a resource, however,…are very costly, larger groups are more able to mobilize necessary labor and other resources.” (Ostrom 2009, 421)	LS
Walker and Salt’s social-ecological resilience principles	Self-organizing systems	“complex, adaptive systems have emergent properties…It’s possible to control parts of the system for a time, but no one is in charge of the whole system. And because of all these features it’s virtually impossible to keep it in some (optimal) state.” (Walker and Salt 2012, 5); “you can change bits of the system, but the system will then self-organize around this change. Other bits will change in response to your control.” (Walker and Salt 2012, 4)	LS
	Linked social, economic, and biophysical domains	“many of the problems associated with managing natural resources relate to the fact that our approaches don’t acknowledge that we’re dealing with systems that have linkages between the social, economic, and biophysical domains that make them up.” (Walker and Salt 2012, 10)	LS
	Adaptive cycles	“the behavior of self-organizing systems changes over time, not because of external influences, but through internal processes. The way that the components of the system interact causes the system to go through cycles in which the connections between its components tighten, loosen, and even break apart. As this happens, the capacity of the system to absorb disturbance (its resilience) also changes, as does the potential for people managing the system to make changes.” (Walker and Salt 2012, 13)	LS
	Adaptive cycles across multiple scales	“self-organizing systems operate over a range of different scales of space and time, and each scale is going through its own adaptive cycle. What happens at one scale can have a profound influence on what’s happening at scales above it and on the embedded scales below.” (Walker and Salt 2012, 15)	LS
	Resilience dimensions (specified, general, and transformability)	“specified resilience, as its name suggests, is the resilience of some specified part of the system to a specified shock—a particular kind of disturbance. General resilience is the capacity of a system that allows it to absorb disturbances of all kinds, including novel, unforeseen ones, so that all parts of the system keep functioning as they have in the past.” (Walker and Salt 2012, 18); “transformability is the capacity of a system to become a different system, to create a new way of making a living…consider the system at multiple scales, because making the system resilient at a regional scale may require transformational changes at lower scales (Folke et al. 2010). Adapting and transforming are actually complementary processes, and adaptability and transformability are complementary attributes of a resilient system.” (Walker and Salt 2012, 18–19)	LS
	Thresholds	“limits to how much a self-organizing system can be changed and still recover. Beyond those limits it functions differently because some critical feedback process has changed. These limits are known as thresholds.” (Walker and Salt 2012, 4)	LS
Other concepts needed for sustainable and resilient outcomes		Other concepts study authors’ note for sustainable and resilient outcomes, in addition to the Walker and Salt, Ostrom, Agrawal, Anderies et al.’s principles	OE
Author(s)’ conclusions		Author(s)’ perceptions of their work’s import	OE
Notes/observations		Coders’ additional observations about the studies	OE

Note: Open-ended coding is abbreviated as “OE.” The three-point coding scale for degree of presence of particular concepts is abbreviated as “LS” and is defined as follows: 0 = not present, 1 = noted but not described, and 2 = carefully described.

Building on Berke et al. (2014), Brody (2003), Agrawal (2003), and Anderies, Janssen, and Ostrom (2004), our research asks how the traditional plan quality metric as presented in Berke et al. (2006) can be updated to incorporate these emerging resources management processes that reflect SER and SCM principles to objectively assess resource management plan quality. We seek to create an assessment instrument that has the breadth and flexibility—while maintaining objectivity and validity—to assess a diversity of embedded planning processes for their SER and SCM principles, as well as the plan quality of natural resource plans in the face of climate change.

Methods

We chose to inductively study SCM and SER studies in order to validate the use of the acknowledged SCM and SER principles in the literature and to identify any novel, emerging and potentially complimentary ones revealed in these studies’ findings. The process elements of SCM and SER studies—as well as their basic characteristics—are challenging to quantify because they are varied and contextually specific. There is no standard systematic assessment approach for SCM and SER studies because they have different foci (i.e., watershed management or forest lands) and different units of analysis. According to Agrawal (2003), “[t]he multiplicity of research designs, sampling techniques, and data collection methods means that there are few compelling analyses that systematically test findings, compare postulated causal connections across contexts, or carefully specify the contextual and historical factors relevant to success” (p. 246). There are numerous, highly contextual findings by study, which can be difficult or methodologically inaccurate to generalize. Using content analysis with key word counts and frequency measures for resource user groups in the studies would be problematic since grouping would lose context and essential nuance. Additionally, given the need to attempt to “build comparative understanding” (Barnett-Page and Thomas 2009, 60) of the SER and SCM concepts across the studies, we agree that in aspects of developing this plan assessment framework, “[s]tatistical methods for aggregating quantitative data are inapplicable” (Britten et al. 2002, 209). Therefore, we determined that a traditional meta-analysis of statistical results compared across several studies (Berke and Godschalk 2009) would not meet our research objectives.

Instead, we used a meta-ethnography, which is a synthesis approach for qualitative studies that originated in the medical and public health disciplines (Feder et al. 2006; Britten et al. 2002; Noblit and Hare 1988). A meta-ethnography is differentiated from a literature review, “[t]he traditional method of summarising a field of research” (Britten et al. 2002, 209), in that it conceptually synthesizes the studies it evaluates “to derive new concepts and theoretical insights” (Campbell et al. 2003, 672). It is also differentiated from a quantitative meta-analysis, which is “not transferable to qualitative research for a number of pragmatic and epistemological reasons” (Britten et al. 2002, 209). It is an inductive approach that is “more hermeneutic, seeking to understand and explain phenomena” (Walsh and Downe 2005, 204). Campbell et al. (2011), citing Noblit and Hare (1988), note that there is a “distinction between integrative reviews, in which data from different studies are pooled or aggregated, and interpretive reviews, which bring together the findings from different studies using induction and interpretation to gain deeper understandings of a particular phenomenon” (p. 5). There are multiple ways to conduct qualitative synthesis, including: “‘meta-ethnography,’ ‘meta-interpretation,’ ‘meta-analysis,’ ‘narrative synthesis,’ ‘meta-synthesis’” (Campbell et al. 2011, 6), but “meta-ethnography was the most widely cited method” (p. ix) in Campbell et al.’s (2011) evaluation of systematic analysis and synthesis of qualitative research.

Building on the original approach proposed by Noblit and Hare (1988) and modified by Britten et al. (2002) and Campbell et al. (2003) “in order to deal with relatively large numbers of studies” (Lee et al. 2015, 336), our meta-ethnography involved the following steps (see Figure 1): (1) determining the focus of interest, which is to systematically determine the elements of SER and SCM to include in a resource plan quality assessment approach through a literature review as noted above; (2) “deciding what is relevant to the initial interest” (Noblit and Hare 1988, 28), which is determining screening criteria for SER and SCM studies, as detailed below; (3) “reading the studies” (Noblit and Hare 1988, 29) to identify the main concepts, noting details of the context (i.e., setting, participants, etc.) and the methods and sampling techniques; (4) “determining how the studies are related” (Noblit and Hare 1988, 29) also known as interpreting the studies’ similarities based on coding for the presence of ideal SER and SCM principles, as noted in Anderies, Janssen, and Ostrom (2004), Agrawal (2003), Ostrom (2009), and Walker and Salt (2012); (5) “translating the studies into one another” (Noblit and Hare 1988, 29) by comparing the open-ended coding of the other concepts needed for sustainable and resilient outcomes and assigning descriptive and process codes to explain the deeper phenomena and processes; and (6) “synthesizing the translations” (Noblit and Hare 1988, 29) by incorporating the findings related to the principles into the plan quality framework following Berke et al. (2014) and Brody (2003).

Figure 1.

Meta-ethnographic process, in theory and application.

Primary and Secondary Study Selection Criteria and Coding

Our primary study selection criteria for SER and SCM studies utilized EBSCO’s Academic Search Premier and Academic Search Complete database, which is an accepted and generally accessible source of academic articles across multiple disciplines (Berke and Godschalk 2009). We limited our search to peer-reviewed journal articles in English, from 2003 through Fall 2016. The timing was selected to build on Agrawal (2003) and Anderies, Janssen, and Ostrom (2004)’s previous efforts. We further limited the criteria to SER and SCM studies that included measurement outcomes because those could provide evidence of resilience or transformation, where possible. Our search terms under abstracts using “or” between terms included “social-ecological resilience,” “socio-ecological resilience,” “sustainable commons management,” “collaborative watershed management,” and “institutional analysis and development framework.” This list was coupled with “measurement” and “outcome” as “ands” for those abstracts. The search generated 174 articles, 87 of which were duplicates, leaving 87 articles to narrow through our secondary study selection criteria.

Although it did not manifest, we agreed that if there were multiple publications on the same data set, we would only use one (Berke and Godschalk 2009). Second, we were focused on SER, SCM, collaborative watershed management, and the institutional analysis and development framework (also a form of institutional and resource user assessment in the SES) studies that produced quantifiable ecosystem outcomes. While we wanted studies that addressed both the social and ecological systems as an integrated unit (and narrowed accordingly), we also sought indicators of efficiency manifested through ecosystem outcomes. Originally, we further limited the studies to U.S. cases, since the governance system on the social side would be complimentary. However, this narrowed the studies to five, which was too small a sample from which to generalize. Consequently, we expanded the criteria to include studies outside of the United States, for a sample of twenty-three studies.

A graduate student and one author coded these studies for intercoder reliability after comparing our coding for the first study. We coded according to the first step in the instrument included in the Appendix and in Table 1, seeking to identify SER and SCM process indicators that (1) were universal across resource types/management issues and (2) were measures that would be included, either procedurally or substantively within a resource management plan, including the role(s) of resource user groups. For the already identified and regularly referenced SER and SCM institutional elements and resilience principles from prior literature, we used three-point coding scale of whether they were not present, noted but not described, or described in detail. We then open-ended coded the similarities in the studies’ characteristics and other concepts needed for sustainable and resilient outcomes. We acknowledge that this introduces both a mix of methods (i.e., traditional scalar coding and a meta-ethnography) and a mix of measurements (i.e., assigning ordinal scores for part of the study attributes while using open-ended meta-ethnographic coding to translate the more nuanced, contextual, and emerging SER and SCM characteristics and concepts). However, mixing methods is an accepted practice in planning scholarship (Gaber 2020), and we continue to rely heavily on the meta-ethnography results for the novel concepts needed for sustainable and resilient outcomes that were ultimately included within the framework. We compared our coding for these open-ended study characteristics and engaged in the meta-ethnographic approach of determining how the studies are related or the reasons for contradiction (interpretative assessment) to generate the final coding used for the translation of the studies into one another.

One author then compared these open-ended study characteristics and commonalities for the third form of synthesis using first cycle coding (descriptive and process coding, both elemental coding; Saldana 2016) to reveal the deeper phenomena and processes of the studies as a whole. According to Saldana (2016), “descriptive coding summarizes in a word or short phrase—most often a noun—the basic topic of a passage of qualitative data…not abbreviations of the content” (p. 102) and process coding “uses gerunds (“-ing” words) exclusively to connote action in the data” (Saldana 2016, 111). This coding allowed us to translate the studies into one another to generate the principles needed for sustainable and resilient resource outcomes. We were cognizant of the fact that the new framework would be used to assess resource plans for both the social and the ecological systems and their interrelationships, not just the social or institutional controls. Although Anderies, Janssen, and Ostrom (2004), Agrawal (2003), and Walker and Salt (2012) had indicators and characteristics of enduring sustainable resource management institutions and combined SES resilience characteristics, this work seeks to update their efforts to determine where and how additional information altered these characteristics.

Intercoder Reliability Results

For the three-point coding scale of already identified and accepted SER and SCM principles, we used Krippendorff’s α to test our intercoder reliability (Krippendorff 2011, 2013; Hayes and Krippendorff 2007), tempering the results interpretation with the categories and associated boundaries (on a scale of 0–1.0) using the guidelines from Stevens, Lyles, and Berke (2014). Stevens, Lyles, and Berke “recommend that the values of α that plan quality evaluation researchers select to assess intercoder reliability be adjusted in order to account for variation in the cognitive burden that is placed on coders by different categories of items typically included in a plan quality evaluation protocol” (p. 10). They define category 3 as “few items that are highly distributed,…includes ‘Inter-Organizational Coordination’” and the upper and lower boundary standards are “0.50 to 0.42” (Stevens, Lyles, and Berke 2014, 86–87). Given the level of coder complexity for SES studies and the associated “cognitive burden” (Stevens, Lyles, and Berke 2014, 86), half (eight of the sixteen) of our principles were classified as category 3. Even with the lower bounds, the coding for all of the principles except “clearly defined boundaries” exceeded the upper boundaries for reliability (see Table 2). “Clearly defined boundaries” was between the upper and lower boundaries, suggesting that the coding results were only somewhat reliable. However, the values exceeded the lower boundary, meaning that the results can still be used.

Table 2.

Study Characteristics/Commonalities.

Author(s)	Year	Unit(s) of Analysis	Method(s)	Resource Units	Other Concepts Needed for Sustainable and Resilient Outcomes
Author(s)	Year	Unit(s) of Analysis	Method(s)	Resource Units	Descriptive Coding Results	Process Coding Results
Balbo et al.	2016	Societies	Comparative case studies	Crops	Traditional ecological knowledge (“TEK”)	Contemporary global context impacts SES cohesion
Bardsley and Peck	2012	Farming	Case study	Agroecosystem	Regional land use policy Land use classification	Decreasing market-based reliance for climate risk reduction
Campbell et al.	2011	Water quality/collaborative processes	Comparative case study	Water quality	None	None
Cumming et al.	2005	Road paving in forests	Case study	Crops	Regional identity concept	Addressing SES uncertainties
Davidson et al.	2013	Marine sectors/fishing	Case studies	Aquaculture	Frameworks and measurable resilience indicators	Avoiding unintentionally crossing thresholds
De Vos et al.	2016	Pathogens in protected areas	Case studies	Reserve species	Beneficial pathogenic influence	Knowing forms of disturbance Anticipating disturbance’s effect on SES resilience
Delgado-Serrano et al.	2015	Natural resource management	Three case studies	Multiple ecosystem management	None	Localism increasing SES resilience
Duer-Balkind et al.	2013	Fish species	Modeling	Aquaculture	Self-governing rules for resource management Outside user limitations	Combining SES rules and the natural ecosystem function to improve resilience
Hornbeek et al.	2013	Water quality/collaborative processes	Content analysis	Water quality	Positive impacts of watershed group presence	Improving implementation through local nongovernmental organization actors
Ingalls and Dwyer	2016	Forest degradation	Habitat survey review	Forest management	Local controls to maintain biodiversity Nested governance	Challenging to make equitable interventions
Jacobi et al.	2015	Farming	Interviews and focus groups	Crops	Educational training improves adaptive capacity Self-organization in knowledge networks	Improving data management Multi-scalarity of the knowledge networks
Johnson and Becker	2015	Social-ecological systems	Case study	Specialized ecosystem management	Need for institutional cooperation with existing entities and multiple scales	None
Lopes et al.	2011	Forest reserves	Case studies	Aquaculture	Need for local context in adaptive comanagement Need interviews to generate measures for social resilience	Improving ecological resilience through adaptive comanagement
McDowell and Ford	2014	Oil and gas extraction	Case study	Arctic extraction	Need local user inclusion	Failing current governance Understanding trade-offs of economic gains and environmental impacts
Moller et al.	2009	Traditional ecological knowledge and science	Interviews and participant observation	Marine species	TEK Critical nature of relationship building	Complementarity of science and TEK for resilience
Moore	2015	Riverine resilience	Modeling	Aquaculture	Whole system assessment for resilience	None
Olsson et al.	2007	Social system in SES	Network theory analysis	Multiple ecosystem management	Social memory storage	Bridging through social networks increases SES resilience
Perz et al.	2010	Societies	Case study	Forest management	Role of collective memory in resilience	Negatively impacting rural resilience with external connectivity and mechanization
Rinkus et al.	2016	Collaborative watershed process	Case study	Water quality	Perceived effectiveness measures	Learning processes and associated funding
Rodriguez- Solorzaho	2014	Farming	Case studies	Crops	None	Local decision-making inclusion Understanding trade-offs of farming versus resource extraction
Scmitt and Brugere	2013	Aquaculture	Modeling	Multiple ecosystem management	None	Understanding trade-offs in economic gains and environmental impacts using modeling
Teegalapalli and Datta	2016	Farming	Case studies	Crops	None	Honoring existing cultural and management practices
Zimmerer	2010	Farming	Case study	Agroecosystem	Local-level resource practices improve agrobiodiversity Scalar importance Agrobiodiversity and irrigation interactions	None

Note: The deeper phenomena revealed in the descriptive and process coding results are grouped by font type as follows: italics denotes localism increasing SES resilience (through TEK, etc.); underline denotes the need for an understanding of trade-offs, both ecological and economic, in resource use; bold italics denotes the need to share local knowledge/information with higher levels of governance over the resources in a nested/scalar process; and bold denotes need for greater knowledge in the SES. SES = social-ecological system.

Since the open-ended questions were not numerically coded, we used percent agreement to determine the reliability. We assigned a frequency of agreement value (0 = no agreement, 1 = partial agreement, and 2 = full agreement) on the similarity between the graduate student and the author’s assessments of the other concepts needed for sustainable and resilient outcomes from the studies. We had 48 percent agreement. Although the standard is generally 75 percent or higher, given the challenge of these studies and the level of theoretical interpretation needed for these questions, we deemed this is an acceptable outcome based on the upper and lower boundaries established by Stevens, Lyles, and Berke (2014) as noted in the preceding paragraph.

Limitations of the Methodology

In gathering the initial eighty-seven studies from the journal sites, once the abstracts were identified through the search terms, we found that several additional articles (particularly in Ecology and Society) were equally as relevant to our search terms and were published within our time frame (2003–2016). However, EBSCO did not identify them. We decided, in the interest of methodological replicability, to continue to use the EBSCO results. This suggests that the results are a sample, rather than the population of the results that the search term algorithm yields, and they are likely not as comprehensive as we would otherwise choose. Thus, we accept replicability and more limited generalizability as limitations of the meta-ethnography’s comprehensiveness. We also acknowledge the bias associated with a single, penultimate coder for the first cycle coding of the open-ended questions and the previously identified lower percent agreement of their coding.

Findings: The New Framework

We found that the studies, despite their variability, revealed common deeper phenomena related to sustainable and resilient outcomes in SESs, including shared additional concepts and affirmation of existing SER and SCM principles. Table 2 summarizes the characteristics and commonalities of the reviewed studies. The studies were disparate in the units of analysis, and the resource units ranged considerably. Additionally, the majority of the studies do not note the number of resource users (only two of the twenty-three do so), which further confirms the difficulty in incorporating the user metrics in the plan quality assessment, and potentially impacts the institutional side of the SES. The primary methods in these studies are a single case study (seven), comparative case studies (two), or syntheses of case studies (six). They range from 2005 to 2016 in publication year, with the majority published in 2013 (four), 2015 (four), and 2016 (five).

When examining the additional requisite concepts for sustainable and resilient SES outcomes, the descriptive and process coding shows that there is a repeated need for local knowledge, which some studies identify more specifically as “traditional ecological knowledge,” local control, and maintaining a close relationship between the resource users and the resources (Table 2). While the studies suggest that localism increases SES resilience, there is an identified need to share that information with higher levels of governance over the resources in a nested/scalar process. There is also a need for greater knowledge in the SES (e.g., data, indicators, identifying uncertainties, disturbances and thresholds, additional education) and an understanding of trade-offs, both ecological and economic, in resource use.

These results also correspond to the basic characteristics of SCM and SER for which the studies were coded. As Table 3 shows, the studies had high scores for clearly defined boundaries, collective-choice arrangements, minimal recognition of rights to organize, nested enterprises, adaptive cycles, and linked and self-organizing systems.

Table 3.

Basic Study SCM & SER Characteristics.

SCM & SER Characteristics		Mean	Med.	Std. Dev.	Variance	Krippendorff’s α^a	Krippendorff α’s Reliability Interpretation per Stevens et al. (Upper Standard–Lower Standard)^b	Cronbach’s α
Ostrom, Agrawal, and Anderies et al.’s SCM Principles	Clearly defined boundaries	1.78	2	.6	.36	0.755	Somewhat reliable (0.80–0.67)	.742
	Proportional equivalence between benefits and costs	0.61	0	.839	.704	0.812	Reliable (0.50–0.42)
	Collective-choice arrangements	1.57	2	.788	.621	0.930	Reliable (0.50–0.42)
	Monitoring	1.35	2	.935	.874	0.883	Reliable (0.80–0.67)
	Graduated sanctions	0.22	0	.6	.36	0.866	Reliable (0.70–0.58)
	Conflict resolution mechanisms	0.48	0	.846	.715	1.000	Reliable (0.80–0.67)
	Minimal recognition of rights to organize	1.78	2	.6	.36	0.864	Reliable (0.70–0.58)
	Nested enterprises	1.83	2	.576	.332	0.817	Reliable (0.70–0.58)
	Accommodation of norms/building social capital	1.35	2	.885	.783	0.974	Reliable (0.50–0.42)
	Social learning	1.65	2	.714	.51	0.994	Reliable (0.50–0.42)
SER Principles (Walker & Salt, 2012)	Self-organizing systems	1.78	2	.518	.269	0.700	Reliable (0.40–0.33)	.913
	Linked social, economic, and biophysical domains	1.87	2	.458	.209	0.605	Reliable (0.50–0.42)
	Adaptive cycles	1.7	2	.703	.494	0.854	Reliable (0.50–0.42)
	Adaptive cycles across multiple scales	1.61	2	.783	.613	0.730	Reliable (0.50–0.42)
	Resilience dimensions (specified, general, transformability)	1.57	2	.788	.621	0.879	Reliable (0.50–0.42)
	Thresholds	1.57	2	.788	.621	0.760	Reliable (0.70–0.58)

Note: Coded 0 = not present; 1 = noted but not described; 2 = carefully described; SCR = sustainable commons management; SER = social-ecological resilience.

^a Krippendorff’s α was run on the ordinal data, with two coders, and bootstrapping set at 1,000.

^b Stevens, Lyles, and Berke (2014) categories (1 to 4, based on level of coding complexity) to interpret Krippendorff’s α for each principle in the studies. The category assigned to each principle is included in the Appendix following the definition of each of the principles in the coding instrument.

The New Resource Plan Quality Assessment Framework

Using these findings, we augmented the plan quality framework as follows. We adopted the Berke et al. (2014)’s distinction between direction-setting and action-oriented principles, augmented by Brody’s (2003) environmental quality elements and those from the SER and SCM literature, as well as the above findings. The following is an amalgam intended to promote stronger assessment of resource plan quality and the emerging management paradigms in each (Table 4). Although some of the principles were not as heavily used in the meta-ethnography results (i.e., conflict resolution mechanisms, graduated sanctions, etc.), we chose to include them since they were accepted in the SER and SCM long-standing studies (Agrawal 2003; Anderies, Janssen, and Ostrom 2004; Walker and Salt 2012) and were elements that were also discussed in Brody (2003). We established a 1–3 coding scale, with 1 = not mentioned, 2 = mentioned but not in detail, and 3 = detailed description. We recognize the 0–2 scale used by others (Berke et al. 2014; Berke and Conroy 2000) and in our study coding, but this presents potential analytic problems (e.g., having a zero in the denominator) that we seek to avoid with future work.

Table 4.

New Resource Plan Quality Framework.

Note: Text in blue indicates that it is an SCM principle from Table 1; text in green indicates that it is an SER principle from Table 1; text in red indicates that it is one of the emerging concepts from the meta-ethnography needed for sustainable and resilient outcomes from Table 2; text in orange is from Brody (2003); and text in purple is from Berke et al. (2014). Coding: 1 = not present; 2 = noted but not described; 3 = carefully described not mentioned; SER = social-ecological resilience; SCM = sustainable commons management; SES = social-ecological system.

* Indicates that this principle was also revealed through the meta-ethnographic open-ended results but was originally identified by the SER or SCM literatures, Brody (2003) or Berke et al. (2014) and color-coded accordingly.

** Indicates that this principle was also revealed through the meta-ethnographic open-ended results but was originally identified in a combination of the SER and SCM literatures.

Conclusions and Future Research

This work has two significant contributions: first, introducing an accepted medical and public health research methodology, the meta-ethnography, to the planning discipline in order to generate a new resource plan quality metric that combines existing resource management strategies (i.e., principles of SER and SCM) and additional concepts needed for sustainable and resilient outcomes with a combination of existing plan quality assessment frameworks. The second contribution is in addressing identified and acknowledged shortcomings in the existing plan quality assessment approach(es). Combining SER, SCM, and plan evaluation approaches allows evaluation of variability across institutional settings and kinds of managed resources, giving breadth and structure through which seemingly disparate plans can be assessed. We sought to create an assessment framework that addresses commonality and evaluation capacity where it has otherwise been impossible but is needed (e.g., examining the resource resilience of an entire region such as the Glacier National Park area or of the Chesapeake Bay recovery plans).

We also anticipate that given the theoretical and fluid nature of the SES principles, coupled with their measurement challenges, the ensuing plan evaluation assessment outcomes will likely score significantly lower than the already low plan quality scores often observed in the literature (Laurian et al. 2004; Berke and Conroy 2000; Spurlock 2017). We assert that this is precisely the purpose of the plan quality metrics more generally, namely, to create conceptual awareness and to ultimately improve plan quality and associated implementation rather than to generate the best possible plan quality score. In subsequent research, we intend to assess the framework with the novel metrics’ efficacy across disparate resource management plan typologies, including watershed-level section 319 nonpoint source water quality plans, and state and substate comprehensive water management plans.

The significance of this work is in the generation of a new framework, using a novel method to do so. With climate change, we have an imperative to create plans that reflect the uncertainty and the integrated nature of social and ecological systems as we address a rapidly changing physical/biological landscape. The plan assessment approach must and should now be able to follow.

Footnotes

Appendix

Acknowledgments

The authors would like to thank Katherine Amidon, MCRP, for her tireless intercoder reliability efforts, as well as the anonymous peer reviewers and the journal editor for their discerning and valuable input that significantly improved the quality of this article. Any remaining errors or oversights are solely our responsibility.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Caitlin S. Dyckman

References

Agrawal

Arun

. 2003. “Sustainable Governance of Common-pool Resources: Context, Methods, and Politics.” Annual Review of Anthropology 32:243–62.

Anderies

John M.

Janssen

Marco A.

Ostrom

Elinor

. 2004. “A Framework to Analyze the Robustness of Social-ecological Systems from an Institutional Perspective.” Ecology and Society 9 (1). http://www.ecologyandsociety.org/vol9/iss1/art18 (accessed April 4, 2020).

Baer

William C.

1997. “General Plan Evaluation Criteria: An Approach to Making Better Plans.” Journal of the American Planning Association 63 (3): 329–44.

Barnett-Page

Elaine

Thomas

James

. 2009. “Methods for the Synthesis of Qualitative Research: A Critical Review.” BMC Medical Research Methodology 9 (59): 1–11. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-9-59 (accessed April 4, 2020).

Berke

Philip R.

Conroy

Maria Manta

. 2000. “Are We Planning for Sustainable Development?” Journal of the American Planning Association 66 (1): 21–33.

Berke

Philip R.

Cooper

John

Aminto

Meghan

Grabich

Shannon

Horney

Jennifer

. 2014. “Adaptive Planning for Disaster Recovery and Resiliency: An Evaluation of 87 Local Recovery Plans in Eight States.” Journal of the American Planning Association 80 (4): 310–23.

Berke

Philip R.

French

Steven P.

1994. “The Influence of State Planning Mandates on Local Plan Quality.” Journal of Planning Education and Research 13:237–50.

Berke

Philip R.

Godschalk

David

. 2009. “Searching for the Good Plan: A Meta-analysis of Plan Quality Studies.” Journal of Planning Literature 23 (3): 227–40.

Berke

Philip R.

Godschalk

David R.

Kaiser

Edward J.

, with Dan A

Rodriguez

. 2006. Urban Land Use Planning. 5th ed. Urbana: University of Illinois Press.

10.

Britten

Nicky

Campbell

Rona

Pope

Catherine

Donovan

Jenny

Morgan

Myfawny

Pill

Roisin

. 2002. “Using Meta Ethnography to Synthesise Qualitative Research: A Worked Example.” Journal of Health Services Research & Policy 7 (4): 209–13.

11.

Brody

Samuel D.

2003. “Implementing the Principles of Ecosystem Management through Local Land Use Planning.” Population and Environment 24 (6): 511–40.

12.

Bunnell

Gene

Jepson

Edward J.

Jr . 2011. “The Effect of Mandated Planning on Plan Quality.” Journal of the American Planning Association 77 (4): 338–53.

13.

Burby

Raymond J.

May

Peter J.

1997. Making Governments Plan: State Experiments in Managing Land Use. Baltimore, MD: The Johns Hopkins University Press.

14.

Campbell

Pound

Morgan

Daker-White

Britten

Pill

Yardley

Pope

Donovan

2011. “Evaluating Meta-ethnography: Systematic Analysis and Synthesis of Qualitative Research.” Health Technology Assessment 15 (43): 1–164.

15.

Campbell

Rona

Pound

Pandora

Pope

Catherine

Britten

Nicky

Pill

Roisin

Morgan

Myfawny

Donovan

Jenny

. 2003. “Evaluating Meta-ethnography: A Synthesis of Qualitative Research on Lay Experiences of Diabetes and Diabetes Care.” Social Science and Medicine 56:671–84.

16.

Conroy

Maria Manta

Berke

Philip R.

2004. “What Makes a Good Sustainable Development Plan? An Analysis of Factors that Influence Principles of Sustainable Development.” Environment and Planning A 36:1381–96.

17.

Deppisch

Sonja

Hasibovic

Sanin

. 2013. “Social-ecological Resilience Thinking as a Bridging Concept in Transdisciplinary Research on Climate Change Adaptation.” Natural Hazards 67:117–27.

18.

Dyckman

Caitlin S.

2016. “Sustaining the Commons: The Coercive to Cooperative, Resilient, and Adaptive Nature of State Comprehensive Water Planning Legislation.” Journal of the American Planning Association 82 (4): 327–49.

19.

Dyckman

Caitlin S.

Paulsen

Kurt

. 2012. “Not in My Watershed! Will Increased Federal Supervision Really Bring Better Coordination between Land Use and Water Planning?” Journal of Planning Education and Research 32 (1): 91–106.

20.

Feder

Gene S.

Hutson

Madeleine

Ramsay

Jean

Taket

Ann R.

2006. “Women Exposed to Intimate Partner Violence: Expectations and Experience When They Encounter Health Care Professionals: A Meta-analysis of Qualitative Studies.” Archives of Internal Medicine 166:22–37.

21.

Folke

Carl

Carpenter

Stephen R.

Walker

Brian

Scheffer

Marten

Chapin

Terry

Rockstrom

Johan

. 2010. “Resilience Thinking: Integrating Resilience, Adaptability and Transformability.” Ecology and Society 51 (4): 20. http://www.ecologyandsociety.org/vol15/iss4/art20/ (accessed April 4, 2020).

22.

Folke

Carl

Hahn

Thomas

Olssen

Per

Norberg

Jon

. 2005. “Adaptive Governance of Social-ecological Systems.” Annual Review of Environmental Resources 30:441–73.

23.

France

Emma F.

Uny

Isabelle

Ring

Nicola

Turley

Ruth L.

Maxwell

Margaret

Duncan

Edward A. S.

Jepson

Ruth G.

Roberts

Rachel J.

Noyes

Jane

. 2019. “A Methodological Systematic Review of Meta-ethnography Conduct to Articulate the Complex Analytical Phases.” BMC Medical Research Methodology 19 (35): 1–18. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-019-0670-7 (accessed April 4, 2020).

24.

Freeman

Jody

Farber

Daniel A.

2005. “Modular Environmental Regulation.” Duke Law Journal 54 (4): 795–912.

25.

Gaber

John

. 2020. Qualitative Analysis for Planning & Policy: Beyond the Numbers. 2nd ed. New York: Routledge.

26.

Hayes

Andrew F.

Krippendorff

Klaus

. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measures 1:77–89.

27.

Holling

C. S.

1973. “Resilience and Stability of Ecological Systems.” Annual Review of Ecology and Systematics 4:1–23.

28.

Holling

C. S.

Gunderson

Lance H.

2002. “Resilience and Adaptive Cycles.” In Panarchy: Understanding Transformations in Human and Natural Systems, edited by Gunderson

Lance H.

Holling

C. S.

, 25–62. Washington, DC: Island Press.

29.

Kaiser

Edward J.

Godschalk

David R.

Stuart Chapin

, Jr. 1995. Urban Land Use Planning. 4th ed. Urbana: University of Illinois Press.

30.

Krippendorff

Klaus

. 2011. “Agreement and Information in the Reliability of Coding.” Communication Methods and Measures 5 (2): 93–112.

31.

Krippendorff

Klaus

. 2013. Content Analysis. 3rd ed. Thousand Oaks, CA: Sage.

32.

Laurian

Lucie

Day

Maxine

Backhurst

Michael

Berke

Philip

Ericksen

Neil

Crawford

Jan

Dixon

Jenny

Chapman

Sarah

. 2004. “What Drives Plan Implementation? Plans, Planning Agencies and Developers.” Journal of Environmental Planning and Management 47 (4): 555–77.

33.

Lee

Richard Philip

Hart

Ruth Isla

Watson

Rose Mary

Rapley

Tim

. 2015. “Qualitative Synthesis in Practice: Some Pragmatics of Meta-ethnography.” Qualitative Research 15 (3): 334–50.

34.

Noblit

George W.

Dwight Hare

1988. Meta-ethnography: Synthesizing Qualitative Studies. Newbury Park, CA: Sage.

35.

Norton

Richard

. 2005. “Striking the Balance between Environment and Economy in Coastal North Carolina.” Journal of Environmental Planning and Management 48 (2): 177–207.

36.

Norton

Richard

. 2008. “Using Content Analysis to Evaluate Local Master Plans and Zoning Codes.” Land Use Policy 25:432–54.

37.

Ostrom

Elinor

. 1990. Governing the Commons: The Evolution of Institutions for Collective Action. New York: Cambridge University Press.

38.

Ostrom

Elinor

. 2009. “A General Framework for Analyzing the Sustainability of Social-ecological Systems.” Science 325 (5939): 419–22.

39.

Ostrom

Elinor

. 2010. “Polycentric systems for coping with collective action and global environmental change.” Global Environmental Change 20: 550–7.

40.

Ostrom

Elinor

Cox

Michael

. 2010. “Moving beyond Panaceas: A Multi-tiered Diagnostic Approach for Social-ecological Analysis.” Environmental Conservation 37 (4): 451–63.

41.

Pahl-Wostl

Claudia

. 2007. “Transitions towards Adaptive Management of Water Facing Climate and Global Change.” Water Resources Management 21:49–62.

42.

Pahl-Wostl

Claudia

Sendzimir

Jan

Jeffrey

Paul

Aerts

Jeroen

Berkamp

Ger

Cross

Katharine

. 2007. “Managing Change toward Adaptive Water Management through Social Learning.” Ecology and Society 12 (2): 30. https://www.ecologyandsociety.org/vol12/iss2/art30/ (accessed April 4, 2020).

43.

Pendall

Rolf

. 2011. “Municipal Plans, State Mandates, and Property Rights: Lessons from Maine.” Journal of Planning Education and Research 21:154–65.

44.

Saldana

Johnny

. 2016. The Coding Manual for Qualitative Researchers. 3rd ed. Thousand Oaks, CA: Sage.

45.

Spurlock

Danielle

. 2017. “Do Mandates Matter for Plan Quality? Jurisdictional Aggregation for a Watershed Level Comparison.” Journal of Environmental Planning and Management 61 (1): 1–23. doi: 10.1080/09640568.2017.1391070.

46.

Stevens

Mark R.

Lyles

Ward

Berke

Philip R.

2014. “Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research.” Journal of Planning Education and Research 34 (1): 77–93.

47.

Tompkins

Emma

Neil Adger

2004. “Does Adaptive Management of Natural Resources Enhance Resilience to Climate Change?” Ecology and Society 9 (2): 10. http://www.ecologyandsociety.org/vol9/iss2/art10 (accessed April 4, 2020).

48.

Walker

Brian

Holling

C. S.

Carpenter

Stephen R.

Kinzig

Ann

. 2004. “Resilience, Adaptability and Transformability in Social-ecological Systems.” Ecology and Society 9 (2): 5. https://www.ecologyandsociety.org/vol9/iss2/art5/ (accessed April 4, 2020).

49.

Walker

Brian

Salt

David

. 2012. Resilience Practice: Building Capacity to Absorb Disturbance and Maintain Function. Washington, DC: Island Press.

50.

Walsh

Denis

Downe

Soo

. 2005. “Meta-synthesis Method for Qualitative Research: A Literature Review.” Journal of Advanced Nursing 50 (2): 204–11.

Novel Methods,Novel Metrics: Using a Meta-Ethnography to Create a Plan Quality Framework for Sustainable and Resilient Social–Ecological Systems

Abstract

Keywords

Relevant Literature

Plan Quality

SER and SCM

Methods

Primary and Secondary Study Selection Criteria and Coding

Intercoder Reliability Results

Limitations of the Methodology

Findings: The New Framework

The New Resource Plan Quality Assessment Framework

Conclusions and Future Research

Footnotes

Appendix

Acknowledgments

Declaration of Conflicting Interests

Funding

ORCID iD

References