Assessing Collaborative Policymaking Outcomes

Abstract

Administrators and policymakers increasingly rely on collaborative policymaking groups to inform policy development. While this trend is observed in a wide array of policy domains, it is particularly common in the regulation of natural resource-based industries which requires the simultaneous consideration of an interrelated set of economic, technical, and social factors. In this article, we examine outcomes associated with collaborative policymaking groups involved in informing state aquaculture policy, referred to herein as aquaculture partnerships. We define outcomes here as consequences on relevant contextual conditions (social, political, and environmental) that follow from the work or design of collaborative processes. Using data collected through an online survey of partnership participants (n = 123), we examine individual and procedural factors that significantly associate with partnerships’ positive or negative influence on a set of policy and social outcomes, as perceived by their participants. Overall, we find that participants’ ability to mobilize scientific and technical resources to achieve group objectives, perceptions of procedural fairness, and individual-level learning are all positively associated with partnership influence on policy and/or social outcomes. We conclude our article by highlighting the value of this research for both scholars and practitioners interested in better understanding collaborative group dynamics and outcomes relating thereto.

Keywords

outcomes technical expertise process learning

Introduction

Part of the complexity of governance stems from the need to devise context-specific solutions that reflect the unique set of social, political, and environmental considerations tied to individual policy domains. Within each policy context, administrators and policymakers must carefully consider the relevant positions and resources of vested policy stakeholders, public sensitivities, institutional landscapes, and administrative capacities and motivations to further their long-term policy objectives. To help them navigate complex policy contexts, policymakers and administrators increasingly rely on collaborative stakeholder groups comprised of various parties who have a vested interest in a given policy domain (Heikkila & Gerlak, 2014). Engaging diverse stakeholders in collaborative settings is expected to help decision makers craft contextually appropriate policy solutions, harness expert knowledge, reduce the potential for policy-related conflict, increase policy receptivity, and facilitate shared understandings of policy problems and solutions (Beierle & Cayford, 2002; Maggioni, Nelson, & Mazmanian, 2012; Ostrom, 2005; Weible & Sabatier, 2009). A question that naturally emerges in collaborative contexts is how diverse stakeholders with presumably divergent interests, capacity, norms, and policy views can overcome their individual-level differences to achieve mutually desired outcomes (Heikkila & Gerlak, 2014)? And, more importantly, what factors facilitate the attainment of such outcomes given stakeholder differences?

Relative to other dimensions of collaborative arrangements, such as antecedents or processes, assessments of their products have been limited. In part, this represents definitional challenges, whereby products can be variably operationalized in terms of outputs, outcomes, and impacts. Definitions of these three concepts are inconsistent in the literature. Consistent with Koontz and Thomas (2006), we define outputs in the context of collaborative governance as “tangible items generated through collaborative processes” (p. 113). Examples of outputs in the context of this research include specific policy recommendations relating to aquaculture licensing. Our definitions of outcomes and impacts in the collaborative governance context are analogous to Heinrich’s (2002) definitions of these in the context of federal job training programs. We define outcomes as consequences on relevant contextual conditions (social, political, and environmental) that follow from the work or design of collaborative processes and impacts as the consequences on relevant conditions (i.e., outcomes) directly resulting from specific outputs produced through a collaborative process or specific attributes of a collaborative process. An outcome building from the output example above would be increases in aquaculture industry entrants. The corresponding impact would be an increase in the number of industry entrants based on changes in the application process promulgated in the collaborative group’s policy recommendations (i.e., outputs).

Inconsistent definitions of outputs, outcomes, and impacts limit the ability to summarize findings across studies on the products of collaborative governance, but measurement challenges associated with accurately capturing products have also undoubtedly contributed to a relative dearth on literature relating to such. A prominent measurement challenge is the long time scale often required to observe either tangible (e.g., policy recommendations) or nontangible (e.g., stakeholder relations) impacts (Koontz & Thomas, 2006; Leach, Pelkey, & Sabatier, 2002). A second difficulty arises from the inability to identify objective measures of certain types of outcomes and impacts. Scholars can rely on subjective measures in such cases (see Leach et al., 2002), though this approach has notable bias limitations. A third difficulty arises from the prevalence of confounding variables that challenge the linking of certain types of outcomes directly to collaborative processes (Biddle & Koontz, 2014).

The measurement challenges listed above naturally lead scholars to focus on tangible or demonstrable outputs, which is both logical and warranted. This article, however, seeks to build off the work of scholars who simultaneously study social and political outcomes of collaborative arrangements (Leach et al., 2002) and factors relating to such in an increasingly salient policy context. A key motivation guiding this work is the desire to evaluate the outcomes of collaborative groups in such a way that connects to the very logic undergirding the use of this approach more broadly. For example, if collaborative policymaking is valued for its ability to harness relevant expertise, foster social capital among diverse stakeholders, and provide contextually appropriate policy solutions, among other benefits, the commensurate outcome analysis should focus on the extent to which collaborative governance arrangements improve the science or information used in policy decision making, feelings of trust among policy stakeholders engaged in the policy process, and appropriateness of policy solutions. With this in mind, in this article, we contribute to the literature by responding to the following two research questions:

Research Question 1: To what extent do participants in collaborative policymaking groups perceive them as positively or negatively contributing to policy, technical, and social outcomes?

Research Question 2: What procedural and individual-level factors facilitate positive group influence on policy, technical, and social outcomes?

We explore these research questions in the context of 10 collaborative policymaking groups involved in the development of aquaculture policy, referred to herein as aquaculture partnerships. Aquaculture partnerships include some mix of government and nongovernment stakeholders¹ that are convened organically or through an institutional mandate (e.g., legislation) to participate in the aquaculture policymaking process. Similar to other policy settings in which the use of collaborative policymaking techniques are increasingly observed, the aquaculture policy context is one in which policymakers are forced to grapple with an interrelated set of economic, technical, and social factors (Firestone, Kempton, Krueger, & Loper, 2004). Aquaculture partnerships convene diverse sets of stakeholders that variably represent issues relating to each of these factors in an effort to capture their diverse expertise and inform the many facets of aquaculture policy. Data for this research were collected through an online survey (n = 123) of participants in the 10 aquaculture partnerships.

This article proceeds as follows. First, we provide an introduction to the literature on collaborative policymaking as relevant to our research questions. We then provide a more detailed background about aquaculture partnerships and the research methods used therein. We then present findings from our analyses of survey data which highlight the salience of technical expertise, procedural fairness, and individual-level learning as predictors of partnership influence on aquaculture issues. We conclude our article by highlighting the value of this research for both scholars and practitioners interested in better understanding collaborative group design and outcomes.

Conceptual Background

Reflecting its increasing prevalence in the administrative and policymaking process, a rich literature on collaborative governance and its various dimensions (i.e., antecedents, process, and products) has emerged in recent decades. Scholars of public administration and related fields have provided rich insights into factors that drive the creation of collaborative arrangements (Bryson, Crosby, & Stone, 2006) as well as their structural, functional, and procedural characteristics (Heikkila & Gerlak, 2014). A portion of the literature focuses specifically on the outputs, outcomes, or impacts of collaborative processes and factors linking to each (Innes & Booher, 1999; Lubell, 2003), a further subset of which focuses on these concepts specifically in the context of collaborative policymaking.

Key Conceptual Definitions

Collaborative policymaking is a generic term that refers to the engagement of nongovernmental actors in decision making relating to some part of the policy process and is inclusive of different forms of participatory policymaking including stakeholder partnerships, advisory committees, public hearings, and negotiated rulemaking (see Leach et al., 2002, for defining characteristics of each). Outputs are generally commonly defined in recent scholarship as products or services resulting from a particular process (Thomas, 2008); examples provided by Koontz and Thomas (2006) include “plans, projects, and other tangible items” (p. 113). Definitions, and related operationalizations, of outcomes and impacts, however, are more inconsistent in the literature. Provided here are some of the various definitions that are offered (though definitions of outputs, outcomes, and impacts used in this article are based on Heinrich, 2002, as explained in more detail below). Koontz and Thomas (2006) define outcomes as “effects of outputs on environmental or social conditions” (p. 113). Rogers and Weber (2010) define outcomes more broadly and decoupled from outputs. Their conceptualization of outcomes encompasses general improvements in public agency and/or collaborative processes and capacity. Emerson (2009) elevates outcomes to impacts where the former “contribute(s) to more effective problem solving, conflict management, and governance” (p. 222). In a more recent publication, Emerson, Nabatchi, and Balogh (2012) define impacts as “intentional (and unintentional) changes of state within the system context . . . alterations in a preexisting or projected condition that has been deemed undesirable or in need of change” that are “spurred by collaborative dynamics” (p. 18).

Looking beyond studies of collaborative governance to studies of performance measurement more broadly can help offer definitional clarity on the distinctions between outputs, outcomes, and impacts. Heinrich (2002), in particular, offers and advances through an empirical application of performance measurement in the context of federal job training programs, a precise distinction between outputs, outcomes, and impacts. She characterizes outputs as the tangible activities, products, services that commonly are the bases of government appraisal of programmatic performance. These are often assessed in relation to programmatic inputs or costs. She characterizes outcomes in terms of the programmatic products that correspond to the realization of programmatic goal attainment. Finally, Heinrich links outputs and outcomes in her definition of impacts, by characterizing impacts as outcomes resulting from outputs. Because Heinrich’s conceptualizations are clear and empirically robust, we define outputs, outcomes, and impacts in the context of collaborative policymaking in this article as the following:

Outputs are tangible items generated through policy or programmatic efforts.

Outcomes are consequences on relevant contextual conditions (social, political, and environmental) that follow from the work or design of collaborative processes.

Impacts are consequences on relevant conditions (i.e., outcomes) directly resulting from specific outputs produced through a collaborative process or specific attributes of a collaborative process.

Consistent with Heinrich, in our definition, the attribution of outcomes to specific outputs is a critical conceptual dimension of impacts. We also allow in our definition of impacts the attribution of outcomes to specific characteristics of a collaborative process. For example, increased social capital among policy stakeholders in a particular subsystem as a result of a collaborative process would be characterized as an outcome. Increased social capital among policy stakeholders in a particular system linked to products of a collaborative process (e.g., joint research proposals) or deliberative decision-making processes used within a collaborative process would both be examples of impact. In our study, as we explain further in the “Method” section, we do not attribute consequences of interest to particular characteristics of collaborative policymaking groups in U.S. aquaculture and thus typify our study as one of collaborative policymaking outcomes, not impacts.

Linking Collaborative Products to Process and Participants

While public administration and policy scholars vary in their analytical interests and conceptualizations of collaborative group outputs, outcomes, and/or impacts, a common thread in this scholarship is identifying the factors that contribute to such. Factors in two categories—procedural and individual—emerge as being particularly salient in facilitating collective outcomes. Example factors in the procedural category explored in this article include procedural fairness and the presence of a respected mediator and all critical stakeholders in the collaborative process. At the individual level, we consider the following factors based on existing scholarship: individual resources, perception of individuals that the venue in which they are a participant is the only one in which they can advance their goals, duration of participation, and participant learning.

Across a wide array of collective action settings, perceptions that the collective deliberation process treats parties fairly and consistently have been found to be associated with positive outcomes (Herian, Hamm, Tomkins, & Zillig, 2012; Leach & Sabatier, 2005; Lubell, 2003; Tyler, Degoey, & Smith, 1996). Similarly, borrowing from the alternative dispute resolution literature, scholars have shown that perceptions of the existence of a respected mediator within a group will facilitate positive outcomes (Sabatier & Weible, 2007). In addition to being held in high esteem by fellow group members, a respected mediator is someone that actively engages in conflict mitigation. The role of such an actor is expected to be particularly crucial in settings that engage stakeholders with divergent political and policy views. Also relating to stakeholder characteristics and consistent with the foundational assumptions of collaborative governance, scholars have confirmed that the inclusion of all vested parties relating to the work of the collaborative process to be a critical factor influencing its performance (Weber, 1998).

A slightly different facet of the stakeholder environment that can have notable bearing on collaborative group outcomes is stakeholders’ ability to harness financial and scientific/technical resources to further the work of a collaborative (Bidwell & Ryan, 2006). Financial resources are necessary for completing projects, tasks, and other operational functions of the collaborative process. Scientific and technical resources, however, may directly influence the decision-making process by altering the availability of certain types of information (Weible, 2008). Furthermore, insofar as collaborative policymaking groups are considered more structured variants of issue networks (Heclo, 1978), technical expertise is a critical asset, the exchange of which can potentially alter stakeholder dynamics. Finally, Leach and Sabatier (2005) demonstrate the influence of a mutual stalemate in affecting outcomes. A mutual stalemate is the perception by stakeholders participating in a collaborative process that no alternative venue exists in which they can pursue their policy goals. Presumably, stakeholders will be more engaged in a process if it is viewed as being the only venue in which they can address their concerns.

At the individual level, this research also focuses on assessing the salience of time and learning among collaborative group participants as determinants of outcome perceptions. Time, conceptualized here in terms of continuity in participation, is required for maintaining sustained negotiation with other stakeholders toward the attainment of policy objectives as well as observing the impacts of collaborative processes (Sabatier & Weible, 2007). Because incrementalism is favored in the policy process (Baumgartner & Jones, 2010), noticeable policy change is unlikely to be perceived in short time horizons. Similarly, changes in social norms are unlikely observable in short time scales (Ostrom, 1990).

Numerous scholars have examined learning in the context of collaborative policymaking (Leach, Weible, Vince, Siddiki, & Calanni, 2013; Resh, Siddiki, & McConnell, 2014). In their study of learning, Leach et al. (2013) differentiate between simple learning (i.e., knowledge acquisition) and deep learning (i.e., changes in one’s opinions on science or policy issues) and identify factors that contribute to each. Resh et al. (2014) examine the role of government actors in facilitating learning in collaborative policymaking venues. In this article, we are interested in an alternative assessment, wherein learning, operationalized as knowledge acquisition, is treated as a predictor of outcome perceptions. Presumably, individuals participating in collaborative groups who are learning more will perceive greater influence on outcomes thereof because partnerships that are successful at the individual level are also likely to be successful at the collective level or because individual-level learning correlates with more positive perceptions of the group process overall.

Based on the existing scholarship discussed here, it is expected that participants in collaborative policymaking groups will perceive their respective partnerships to have greater positive influence on policy, technical (e.g., science or technology), and social outcomes, when:

Process-related propositions

the group process is perceived as being fair and consistent for all involved parties;

all critical stakeholders are engaged in the process; and

the group has at least one participant who mediates conflict and is held in high esteem;

Individual-related propositions

they perceive that the collaborative group is the only venue in which they can pursue their policy objectives;

they are able to mobilize financial resources to achieve the group’s objectives;

they are able to mobilize scientific and technical resources to achieve the group’s objectives;

they personally experience learning on various dimensions of the issues that are addressed by their group; and

they have been participating in the collaborative process for an extended period of time.

In the following sections, more detail is provided about aquaculture partnerships and the methods used to assess the above propositions therein.

Collaborative Policymaking in U.S. Marine Aquaculture

Aquaculture—also known as fish and shellfish farming—is formally defined by the National Oceanic and Atmospheric Administration (NOAA) as “the breeding, rearing, and harvesting of plants and animals in all types of water environments including ponds, rivers, lakes, and the ocean” (NOAA, 2014, para. 1). Aquaculture development in the United States faces a number of barriers, including an uncertain regulatory landscape (Firestone et al., 2004; Wirth and Luzar, 1999); complex interdependencies among ecological, economic, technical, and social factors (Firestone et al., 2004); resistance from the general public regarding farmed seafood (Amberg & Hall, 2010; Mazur & Curtis, 2006); conflict about aquaculture development (Kaiser & Stead, 2002); and numerous concerns about the industry from disease control to degradation of marine ecosystems (Mazur & Curtis, 2006; Naylor et al., 2000; Treece, 2002). Over the past several years, numerous partnerships have materialized across the United States with the intent of overcoming these barriers by bringing together various, and often diverse, interests associated with the development of the aquaculture industry. Because aquaculture has largely been devolved to the state level, due in part to the geographic and species sensitive nature of aquaculture production, most of the partnerships are housed at the state level.

In most cases, aquaculture partnerships are formally convened, either through a legislative or regulatory mandate. In other cases, partnerships are formed based on the impetus of diverse aquaculture stakeholders who desire a shared venue in which to deliberate and develop recommendations relating to aquaculture industry. Whether formally or organically developed, aquaculture partnerships typically function to advise on industry policies, develop and/or implement research priorities as it relates to aquaculture policy, implement programs, advise on administrative budgetary decision relating to the industry, address the industry’s marketing issues, and/or evaluate the effectiveness of aquaculture policies. Given the salience of the responsibilities assigned to aquaculture partnerships, they provide an appropriate context within which to assess their influence on policy, social, and technical issues. More broadly, they are characteristically similar to diversely composed advisory committees and citizen participation efforts increasingly observed within and outside of natural resource industry–based policy contexts, lending opportunities to generalize theoretically and practically oriented research findings across policy fields.

Method

Aquaculture Partnerships

For research purposes, aquaculture partnerships were defined as local, state, or regional organizations, comprising some mix of aquaculture industry, government agency, and nongovernment organization representatives that collaborate on policy or research, or both, relating to marine aquaculture in the United States. At the time the research was conducted, a total of 10 partnerships were identified nationwide that met our definition and all were included in this study.² Partnerships were identified based on extensive web research, consultation with a project advisory committee³ comprised of expert stakeholders representing different dimensions of aquaculture production, and other aquaculture experts. Coordinators of the 10 identified partnerships were contacted, and each group agreed to participate in this study.

Together, the 10 partnerships included in this research represent each major coastal region of the United States, with 3 located along the Pacific Coast, 5 located along the north-eastern Atlantic Coast, and 2 located on the south-eastern Atlantic Coast.⁴ The 10 partnerships are characteristically similar in terms of structure and function. Reflecting the geographically sensitive nature of aquaculture production, most of the selected partnerships focus on state-level issues, though 1 was created to address aquaculture issues pertaining to an entire region. All of the groups advise on policy issues: 7 as formal advisory committees established through legislative mandates and 3 as collaborative groups initiated or sponsored by government. The partnerships variably focus on variety of aquaculture issues, such as, fish health, public safety, and/or other issues relating to the expansion of the aquaculture industry. All of the partnerships are comprised of vested stakeholders representing diverse aspects of the aquaculture industry. The groups differ in terms of the number of recommendations or policy-related research projects they generate. All of the groups were created within the 25 years, with the oldest being created in the late 1980s and the newest being created in 2007.

Data Collection

Data were collected using an online survey of participants in the sample of 10 aquaculture partnerships. Participants were defined as anyone currently participating in one of the partnerships as well as anyone who closely followed the work of the partnership. Lists of active partnership members at the time of this research and their contact information were obtained from online membership lists or directories or were provided by partnership coordinators. Information regarding participants who were not officially members but closely followed the work of the partnership, and their contact information, was obtained from interviews (described below) and/or partnership coordinators. Across the 10 partnerships, the number of participants ranged from 7 to 43. The construction of the online survey was informed by data obtained through interviews with 43 participants across the 10 partnerships, 32 in person and 11 by telephone.⁵ Interview responses, for example, were used to help researchers identify key dimensions of concepts explored through survey questions from theoretical (as consistent with existing scholarship) and practical perspectives. The interviews and survey explored similar questions, and thus interviews yielded useful information given their intended purpose (i.e., to inform survey measure construction).

The online survey was sent to 198 participants identified across the 10 partnerships, of which 123 or 62% responded.⁶ The online survey contained a variety of questions relating to aquaculture policy generally, the functioning of partnerships, and attitudes and experiences of the participants therein. Furthermore, relevant to the analyses in this article, respondents were provided a list of problems typically associated with marine aquaculture and asked to indicate their perceptions of severity of each problem. They were then provided this same list and asked to indicate whether their respective partnership made the problem better or worse (i.e., partnership influence on issues represented in problem statements). The list of seven aquaculture problems analyzed is provided in Table 1. Also provided in Table 1 are the survey prompts used to elicit responses on problem severity and partnership influence⁷ as related to each problem listed, response scales, and variable characterizations for subsequently performed analyses.

Table 1.

Data Description.

Variable name	Survey item		Response scale
Dependent variables:	Problem Severity: “In your opinion, please indicate the seriousness of the following perceived problems associated with the expansion of the marine aquaculture industry in your region.”	Influence on Problem: “For each of the following issues, please indicate whether the [partnership name] has had any actual impact” so far using a scale ranging from “made the issue much worse” to “made the issue much better.”	Problem Severity: Not a problem at all (1) to very serious problem (5)
Perceived Problem Severity			Influence on Problem: Made the issue much worse (−2) to made the issue much better (2)
Partnership Influence on Problems
Inappropriate regulations governing aquaculture	Inappropriate regulations governing U.S. marine aquaculture
Insufficient policies for facility permits and licenses	Insufficient policies for granting long-term leases, licenses, and/or permits for aquaculture facilities
Negative public perceptions	Negative public perceptions about marine aquaculture
Lack of good science for decision making	Lack of good science to make sound decisions regarding the siting and operation of aquaculture facilities
Distrust among stakeholders	Distrust among stakeholders involved in marine aquaculture
Socioeconomic health of fishers and tribes	Negative impacts from marine aquaculture on the socioeconomic health of local fishing-dependent communities or tribes
Dissemination of misinformation	Dissemination of misinformation about marine aquaculture
Independent variables: Factors Affecting Partnership Influence
Fair process	The [partnership] process treats all parties fairly and consistently		Strongly disagree (−2) to strongly agree (2)
Critical parties absent	Some critical interests are not effectively represented in the [partnership name]		Strongly disagree (−2) to strongly agree (2)
Alternative venue option	If the [partnership] fails to adopt workable solutions, my concerns could probably be satisfied by appealing directly to the legislature, courts, or individual agencies		Strongly disagree (−2) to strongly agree (2)
Respected mediator	There is at least one participant who mediates conflict and is respected by the other [partnership] members		Strongly disagree (−2) to strongly agree (2)
Financial capacity	Capacity your organization has to mobilize financial resources to help achieve [partnership] objectives		No capacity (1) to complete capacity (5)
Scientific and technical capacity	Capacity your organization has to mobilize scientific and technical resources to achieve [partnership] objectives		No capacity (1) to complete capacity (5)
Participant learning	Multi-item measure: The [partnership] has given me a better understanding of (a) other stakeholders’ perspectives; (b) aquaculture science; (c) aquaculture policy, law, or regulations; and (d) aquaculture economics or business. Cronbach’s α = .87		Individual items: Strongly disagree (−2) to strongly agree (2). Multi-item measure range (−8) to (+8)
Extended time in partnership	In what year and month did you first start participating in the [partnership]		1 = 10 years or more; 0 = less than 10 years

Dependent Variable

Perceptions of partnership influence on the following issues (i.e., partnership outcomes), again issues as represented in problem statements, were the main dependent variables in our analyses: “appropriateness” and “sufficiency” of aquaculture regulations, public perceptions of aquaculture, availability of “good” science for aquaculture decision making, level of trust among aquaculture policy stakeholders, socioeconomic health of fishers and tribes, and dissemination of misinformation. In other words, we were interested in determining whether partnerships contributed to more or less appropriate or sufficient aquaculture regulations, improved or worse public perceptions of aquaculture, more or less science deemed “good” for policy decision making, increased or decreased levels of trust among aquaculture policy stakeholders, improved or worsened socioeconomic health of fishers and tribes, and more or less misinformation dissemination.

In the case of the last issue here, it is not partnership participants who are disseminating misinformation. Rather, they are involved in addressing the spread of misinformation relating to the industry by various parties either through efforts of the collaborative process or through the convening of diverse stakeholders with different perspectives, and possibly information, in the same venue. This is not characterized as an impact because influence on this issue, as well as all others in the above list, is not directly associated with specific partnership outputs or process attributes in the data collection methods used in this research. But while we do not link partnership influence on the problems to specific activities, reports, and so on (and thus we characterize our dependent variable in terms of outcomes), it is useful to identify the general types of activities that partnerships engage in that may support their influence on aquaculture-related problems: Inappropriate or insufficient aquaculture regulations and socioeconomic health of fishers are tribes (e.g., development of policy recommendations), negative public perceptions of aquaculture and dissemination of misinformation (e.g., education/outreach), lack of good science in aquaculture decision making (e.g., development of technical reports or research on aquaculture issues), and distrust among stakeholders (e.g., collaborative meetings involving diverse aquaculture stakeholders).

Using perceptions of partnerships’ positive or negative influence on a predefined set of problems as a proxy for actual influence is appropriate given the nature of the issues upon which we seek to determine partnerships’ positive or negative influence. With the exception of the socioeconomic health of fishers and tribes and misinformation dissemination, no clear objective measures exist for analytical purposes and thus subjective measures of such are considered appropriate. Even still, it is important to recognize the limitations of using subjective measures of outcomes and impacts. Common source bias and the halo effect are related phenomena that are relevant to consider in the context of this research (Meier & O’Toole, 2013; Nisbett & Wilson, 1977). Both of these phenomenon point to tendencies of individuals to conflate perceptions of different variables that relate to a shared experience, process, and so on. In the case of this research, the most likely manifestation of either phenomenon is that partnership participants would evaluate group’s influence on the above issues based on their perceptions of other partnership attributes—for example, process or fellow participants.

Independent Variables

In the survey, respondents were also asked specific questions operationalizing variables relating to each of the propositions offered in this article. These variables include the following:

Process variables

fair process

critical parties absent

respected mediator

Individual variables

alternative venue option

financial capacity

scientific and technical capacity

participant learning

time in partnership

These questions and their associated variable characterization and measurement scales are also provided in Table 1. Each of these was treated as an independent variable predicting impact perceptions in our analyses.

Data Analysis

Analyses of survey data were performed in two steps. In the first step, partnership participants’ problem perceptions and perceptions of partnerships’ influence on policy, technical, and social issues were analyzed to (a) descriptively assess the perceived severity of aquaculture problems by participants in the different partnerships and their partnerships’ influence on such (i.e., calculation of group means relating to each problem examined) and (b) assess variance across the partnerships in problem and associated influence perceptions. Significant differences between group means on individual problems were assessed using an independent sample, Kruskal–Wallis test. The Kruskal–Wallis test was determined to capture potentially relevant differences among groups. Aquaculture production and associated policy, technical, and social issues have the potential to differ substantially by state and region, and thus the researchers were interested in ascertaining intergroup heterogeneity.

In the second step of the data analysis, seven multiple regressions were conducted to determine whether individual participants’ perceptions of a fair partnership process, absence of critical parties in the collaborative process, presence of a respected mediator, alternative venue option, financial capacity among partnership members, scientific and technical capacity among partnership members, participant leaning, and extended time in the partnership significantly associate with perceptions of positive or negative partnership influence on the seven problem issues listed in Table 1. This regression method is appropriate given the analytical objective of identifying significant relationships between a set of independent variables and dependent variable of interest. In the regression models, standard errors were clustered on a variable identifying each respondent’s partnership to account for the possibility that residuals are correlated for respondents within a partnership.⁸ One regression model was conducted for each issue; the set of predictor variables (above) were common to all of the models.

Results

Results from the survey data analyses are presented in this section as they relate to each of our posited research questions.

Research Question 1: To what extent do participants in collaborative policymaking groups perceive them as positively or negatively contributing to policy, technical, and social outcomes?

Table 2 reports the partnership means for perceived problems and partnership influence on such. Across all the partnerships, the problems perceived as being most severe were dissemination of misinformation (summed mean⁹: 39.78), negative public perceptions (summed mean: 35.01), inappropriate regulations governing aquaculture (summed mean: 33.4), and insufficient policies for facility permits and licenses (summed mean: 32.4). The problems perceived as being least severe were socioeconomic health of fishers and tribes (summed mean: 19.97) and lack of good science for decision making (summed mean: 24.93).

Table 2.

Mean Perceptions of Partnership Influence on Problems (Inf.) and Problem Severity (Sev.).

Issue	Partnership 1		Partnership 2		Partnership 3		Partnership 4		Partnership 5		Partnership 6		Partnership 7		Partnership 8		Partnership 9		Partnership 10
	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.	Inf.	Sev.
Inappropriate regulations governing aquaculture	0.17	2.81	0.90	3.30	0.10	3.62	0.89	3.43	1.17	3.14	1.00	3.33	0.83	2.92	0.76	3.52	0.50	3.53	1.00	3.80
Insufficient policies for facility permits and licenses	0	3.45	0.70	3.10	0.50	4.25	1.55	3.14	0.67	2.14	1.00	2.50	0.83	3.08	0.52	4.00	0.55	3.94	1.20	2.80
Negative public perceptions	0.33	3.45	0.40	3.10	0.50	3.00	0.89	3.43	0.50	3.86	0	3.83	0.58	2.77	0.90	4.21	0.61	3.76	1.40	3.60
Lack of good science for decision making	0.33	2.73	0.30	2.30	0.50	2.38	0.89	3.29	1.33	2.71	0.80	1.83	0.83	1.92	0.85	2.43	0.55	2.94	1.20	2.40
Distrust among stakeholders	−0.5	3.55	0.70	2.70	0.80	3.00	1.33	2.71	1	3.29	0.60	3.00	1.00	3.00	0.71	3.30	0.78	3.06	1.00	2.60
Socioeconomic health of fishers and tribes	−0.17	2.09	0.20	2.50	0.30	1.38	0.56	2.57	0.17	2.29	0	1.17	0.42	1.92	0.52	2.13	0.33	2.12	0.80	1.80
Dissemination of misinformation	0.08	3.73	0.90	3.30	0.70	3.75	1.22	4.14	1.17	4.14	0.60	4.50	0.33	3.92	0.95	4.48	0.61	3.82	1.00	4.00

Note. Values per partnership represent the mean responses of partnership participants to questions about the influence of respondents’ partnership on the problems listed as well as the perceived severity of these problems. The survey prompt for the problem severity question was as follows: “In your opinion, please indicate the seriousness of the following perceived problems associated with the expansion of the marine aquaculture industry in your region.” The corresponding response scale was 1 = not a problem at all to 5 = very serious problem. The survey prompt for the partnership influence variable was as follows: “For each of the following issues, please indicate whether the [partnership name] has had any actual impact so far using a scale ranging from ‘made the issue much worse’ (−2) to ‘made the issue much better’ (+2).”

Across all partnerships, participants perceived the greatest influence of their respective groups on the availability of good science for decision making (summed mean: 7.58), dissemination of misinformation (summed mean: 7.56), permitting and licensing policy sufficiency (summed mean: 7.52), and level of trust among aquaculture policy stakeholders (summed mean: 7.42). Less influence was perceived on socioeconomic health of fishers and tribes (summed mean: 3.13). Results from this analysis therefore suggest that partnerships are indeed having notable influence on at least some of the problems regarded as the most severe, as perceived by important policy stakeholders in the aquaculture subsystem. They are also having an influence on issues that are not necessarily perceived as being problems for the industry; for example, availability of good science in decision making. This is also important to know as it can signal group capacity, priorities, and/or other important group dynamics.

In terms of problem perceptions, the Kruskal–Wallis test of variance revealed significant differences among partnerships on the following, where p < .01: insufficient policies for facility permits and licenses and negative public perceptions. In terms of influence on problems, significant differences were observed among partnerships on the following: appropriateness of regulations for governing aquaculture, licensing and permitting policy sufficiency, negative public perceptions, and level of trust among stakeholders, where p < .01, and availability of good science for decision making and socioeconomic health of fishers and tribes, where p < .10. For both problem and related influence perceptions, no structural or geographical characteristics were common to partnerships exhibiting similarly high, moderate, or low means.

Research Question 2: What procedural and individual-level factors facilitate positive group influence on policy, technical, and social outcomes?

Descriptive statistics for variables included in the regression models are provided in Table 3. Results from the regressions are provided in Table 4. All models are highly significant and explain a respectable amount of variance (R² = .20-.40) given that the data come from an online survey that often have measurement error (Lubell & Fulton, 2008). Results relating to each of the specific variables explored in the models are provided below.

Table 3.

Descriptive Statistics for Regression Model.

Variable name	Number of observations^a	M	SD	Minimum	Maximum
Fair process	115	0.96	0.85	−2	2
Critical parties absent	113	0.01	0.98	−2	2
Respected mediator	112	0.72	0.73	−1	2
Alternative venue option	113	0	0.88	−2	2
Financial capacity	103	2.14	0.92	1	5
Scientific and technical capacity	110	3.2	1.15	1	5
Participant learning	114	3.38	3.08	−5	8
Extended time in partnership	123	0.25	0.46	0	1

Units of analysis are partnership participants.

Table 4.

Regression Model Results.

	Regulatory appropriateness	Permitting and licensing policy sufficiency	Negative public perceptions	Availability of good science for policy decision making	Level of trust between policy stakeholders	Socioeconomic health of fishers and tribes	Dissemination of misinformation
Fair process	.27***	.24**	.34***	.04	.28***	.16	.14*
Critical parties absent	−.07	−.09	.14	−.04	−.22**	.04	−.09
Respected mediator	.15*	.11	.00	.03	.22	.04	.16
Alternative venue option	−.10	−.09	.01	−.03	.06	−.02	.08
Financial capacity	−.05	−.04	−.10	.02	−.11	−.08	−.12
Scientific and technical capacity	.13**	.15**	.20***	.17***	.11	.11	.18***
Participant learning	.06***	.05***	.04	.12***	.07***	.04***	.05
Extended time in partnership	.27*	.25	.28*	.14	.42**	.28**	.28
Number of observations	96	97	97	97	97	97	97
R ²	.30	.22	.26	.37	.40	.24	.20
Model significance	.00	.00	.00	.00	.00	.00	.00

p < .10. **p < .05. ***p < .01.

Procedural fairness was significant in five of the seven models (p < .10 to p < .01). As predicted, perceptions that the partnership process treats all parties fairly and consistently are associated with greater perceived positive influence on policy and social issues. The only two problems in relation to which procedural fairness was not associated with positive group influence upon were the availability of good science for policy decision making and socioeconomic health of fishers and tribes. In terms of input variables, while financial capacity of partnership participants was not found significant in any of the models, scientific and technical capacity was significant and positive across all but two (p < .05 to p < .01)—level of trust among stakeholders and socioeconomic health of fishers and tribes. Similarly, participant learning was found to be significantly associated (p < .01) with positive group influence on all but two problems—negative public perceptions and dissemination of misinformation. Extended time in a partnership was significant in the models for appropriateness of regulations governing aquaculture (p < .10), negative public perceptions (p < .10), level of trust among policy stakeholders (p < .05), and socioeconomic health of fishers and tribes (p < .05).

The absence of critical parties in the partnership process was found to be significantly associated with group influence on only one issue—level of trust among stakeholders—whereby an increase in agreement that critical parties were absent is associated with less positive influence on this issue (p < .05). Having an alternative venue (e.g., legislature, courts, individual agencies) to which participants can appeal should their partnership fail to adopt workable solutions was not found to be significant in any of the models. The presence of a person in the collaborative process who mediates conflict and is respected by the partnership members was found to be positively associated with group influence on the appropriateness of regulations governing aquaculture.

Discussion

Assessing the effectiveness of collaborative governance arrangements is both logical and necessary. Public administration and policy scholars have variably sought to do this by studying outcomes, outputs, and/or impacts that result from collaborative processes. There has been particular emphasis on studying tangible outputs in light of measurement challenges relating to longtime horizons required for observing noticeable changes in either policy or social norms and difficulty isolating effects of collaborative process amid a vast array of potentially confounding factors (Biddle & Koontz, 2014). Still, given the importance of evaluating less tangible policy, technical, and social outcomes, this article adds to recent scholarship oriented at better understanding what factors contribute to positive collaborative group influence relating to such (Leach & Sabatier, 2005; Lubell, 2003; Rogers & Weber, 2010). We see distinct benefit in analyzing outcomes that relate to the very rationales of collaborative governance. As such, we examine which procedural and individual-level variables are positively associated with perceived partnership influence on the following issues: (a) the appropriateness of aquaculture regulations, (b) sufficiency of permitting and licensing policies, (c) negative public perceptions regarding aquaculture, (d) availability of good science for policy decision making, (e) level of trust among stakeholders in the aquaculture policy arena, (f) socioeconomic health of fishers and tribes, and (g) dissemination of misinformation about aquaculture.

Overall, we find that participants’ ability to mobilize scientific and technical resources to achieve group objectives, perceptions of procedural fairness, and individual-level learning are all positively associated with group influence on the above issues. The positive association between scientific and technical resource capacity of participants and perceptions of group influence is evident in relation to policy-related problems, negative public perceptions, lack of good science for decision making, and dissemination of misinformation. In contrast, capacity to mobilize financial resources was not found to be a significant predictor of group influence in relation to any of the problems. These findings confirm expectations that financial and scientific and technical resources, which are sometimes conceptualized analogously under resources generally, are in fact conceptually distinct and have differing potential in affecting collaborative group dynamics. Whereas financial resources are important for conducting operational tasks of groups and supporting the development and implementation of tangible policy outputs, scientific and technical resources can alter the decision context of the group by bringing new or otherwise critical information to deliberations. Furthermore, when viewed as legitimate and unbiased, scientific and technical resources can introduce neutrality to the decision process that oversteps existing rifts between opposing stakeholders and possibly restructure relations between them. Indeed, as Weible (2008) argues, uncertainty, presumably linked to the absence of scientific and technical information, creates opportunities for stakeholders to rely on their beliefs for decision making and can thus contribute to divisive relations among deliberating parties.

The salience of scientific and technical resources may also be punctuated in the context of aquaculture partnerships as aquaculture production requires a high degree of scientific and technical competence relating to the unique physiological characteristics of farmed species, fish health, natural resource management, and aquaculture marketing and business. However, because aquaculture is akin in many ways to other natural resource-based industries, we argue that the results found in this research are generalizable to other contexts that involve complex interdependencies between social, environmental, policy, and economic realms.

Our findings also suggest a positive association between individual learning and perceptions of partnership influence on policy-related problems, lack of good science for decision making, distrust among aquaculture stakeholders, and the socioeconomic health of fishers and tribes. Our learning measure captures what Leach et al. (2013) refer to as simple learning or knowledge acquisition. The specific measure we use is a multi-item measure reflecting partnership participants’ enhanced understanding of other stakeholders’ perspectives, aquaculture science, aquaculture policy, and aquaculture economics/business. Our findings regarding learning suggest one of two possibilities. One possibility is that partnerships that are successful in fostering individual-level learning on policy, technical, and social dimensions are also effective in producing positive collective outcomes on issues relating to these dimensions more broadly. Another possibility is that individuals who report intragroup learning are more likely to view the collective process as being more effective.

This latter interpretation could reflect a halo effect, or in this case particularly, could lend support to recent scholarship that emphasizes conceptual pathways relating to learning in collaborative policy processes. For example, this finding offers empirical support for Gerlak and Heikkila’s (2011) learning framework that posits a distinction between learning processes and learning products and general relationships between them (Ostrom, 2005), whereby the process of knowledge acquisition in a policy context can translate to policy or institutional changes. They explicitly acknowledge learning products in evaluation terms as encompassing, “immediate outputs, such as the development of a new project, as well as longer term outcomes or impacts, such as the effects of a project on society or the environment” (Gerlak & Heikkila, 2011, p. 622). Although learning is treated as an individual-level outcome rather than process variable in this research per se, the learning measure used is generally consistent with their conceptualization of learning processes. Future research could more actively integrate Gerlak and Heikkila’s learning framework for a more nuanced assessment of how learning processes contribute to the types of learning products analyzed in this article.

No study is without limitations. First, even though we suggest our findings are relevant in the context of recent collaborative governance research (i.e., learning findings), we must consider the possibility that individuals’ perceptions of partnerships impact are biased by their perceptions of other attributes of the groups (Meier & O’Toole, 2013; Nisbett & Wilson, 1977). In addition, this research is constrained by limitations endemic to all survey research. Inevitably it is impossible to completely guard against the possibility of measurement error. However, we argue that the survey approach was appropriate given the types of variables we were interested in exploring in this article. Another limitation of this research is that we capture perceptions of partnerships impact using a cross-sectional study. Indeed, given the nature of our interests, a longitudinal design would be preferable. Incorporating an additional wave of data collection among partnership participants is a logical next step for this research. Despite these limitations, however, we argue that the research presented herein offers valuable insights about the key concepts in the collaborative policymaking literature.

Conclusion

Understanding outcomes relating to different modes of governance is a critical pursuit of scholars of public administration and related fields. As alternative forms of governance continue to develop to accommodate shifting actor, institutional, and public contexts, the need to examine their relative performance (in terms of outputs, outcomes, and impacts) is even more pressing. This research seeks to help illuminate procedural and individual-level factors that relate to collaborative policymaking groups’ influence in the aquaculture policy domain on important policy, technical, and social issues. This research contributes to the literature by highlighting the conceptual differences between different types of resources often studied in the context of collaborative governance arrangements as well as reinforcing recent scholarships that links learning through participation in collaborative settings and learning products such as changes in policy, environmental, or social conditions. For public administrators, this research offers guidance on how to structure or design collaborative processes to achieve desired policy objectives. Whereas their capacity to directly shape individual-level learning may be somewhat constrained, certainly they can encourage inclusivity, particularly by scientific and technical experts, procedural consistency, and low turn-around among group participants through various administrative strategies. A logical next step in this research is to conduct additional waves of data collection among partnership participants to assess changes in perceived group influence on policy, technical, and social problems relevant to their respective groups’ work over time.

Footnotes

Acknowledgements

This research was conducted as part of the Aquaculture Partnerships Project. Members of the research team included Chris Weible, Bill Leach, John Calanni, Scott Vince, and Saba Siddiki. The lead author thanks all members of the research team for their guidance, support, and colleagiality. Also, the authors wish to thank the anonymous reviewers for their constructive feedback on previous versions of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by National Science Foundation Grant 0721067.

Notes

Author Biographies

Saba Siddiki is an Assistant Professor in the School of Public and Environmental Affairs, Indiana University-Purdue University Indianapolis.

Shilpi Goel is a Master of Public Affairs student in the School of Public and Environmental Affairs, Indiana University-Purdue University Indianapolis.

References

Amberg

Hall

T. E.

(2010). Precision and rhetoric in media reporting about contamination in farmed salmon. Science Communication, 32, 489-513.

Baumgartner

F. R.

Jones

B. D.

(2010). Agendas and instability in American politics. Chicago, IL: University of Chicago Press.

Beierle

T. C.

Cayford

(2002). Democracy in practice: Public participation in environmental decisions. Washington, DC: Resources for the Future.

Biddle

J. C.

Koontz

T. M.

(2014). Goal specificity: A proxy measure for improvements in environmental outcomes in collaborative governance. Journal of Environmental Management, 145, 268-276.

Bidwell

R. D.

Ryan

C. M.

(2006). Collaborative partnership design: The implications of organizational affiliation for watershed partnerships. Society & Natural Resources, 19, 827-843.

Bryson

J. M.

Crosby

B. C.

Stone

M. M.

(2006). The design and implementation of cross-sector collaborations: Propositions from the literature. Public Administration Review, 66(Suppl. 1), 44-55.

Emerson

(2009). Synthesizing practice and performance in the field of environmental conflict resolution. In O’Leary

Blomgren Bingham

(Eds.), The collaborative public manager (pp. 215-232). Washington, D.C.: Georgetown University Press.

Emerson

Nabatchi

Balogh

(2012). An integrative framework for collaborative governance. Journal of Public Administration Research and Theory, 22, 1-29.

Firestone

Kempton

Krueger

Loper

C. E.

(2004). Regulating offshore wind power and aquaculture: Messages from land and sea. Cornell Journal of Law and Public Policy, 14, 71-111.

10.

Gerlak

A. K.

Heikkila

(2011). Building a theory of learning in collaboratives: Evidence from the Everglades Restoration Program. Journal of Public Administration Research and Theory, 21, 619-644.

11.

Heclo

(1978). Issue networks and the executive establishment. In King

(Ed.), The new American political system (pp. 87-124). Washington, D.C.: American Enterprise Institute for Public Policy Research.

12.

Heikkila

Gerlak

A. K.

(2014). Investigating collaborative processes over time: A 10-year study of the South Florida ecosystem restoration task force. The American Review of Public Administration. Advance online publication. doi:10.1177/0275074014544196

13.

Heinrich

C. J.

(2002). Outcomes-based performance management in the public sector: Implications for government accountability and effectiveness. Public Administration Review, 62, 712-725.

14.

Herian

M. N.

Hamm

J. A.

Tomkins

A. J.

Zillig

L. M. P.

(2012). Public participation, procedural fairness, and evaluations of local governance: The moderating role of uncertainty. Journal of Public Administration Research and Theory, 22, 815-840.

15.

Innes

J. E.

Booher

D. E.

(1999). Consensus building and complex adaptive systems: A framework for evaluating collaborative planning. Journal of the American Planning Association, 65, 412-423.

16.

Kaiser

Stead

S. M.

(2002). Uncertainties and values in European aquaculture: Communication, management and policy issues in times of “changing public perceptions.” Aquaculture International, 10, 469-490.

17.

Koontz

T. M.

Thomas

C. W.

(2006). What do we know and need to know about the environmental outcomes of collaborative management? Public Administration Review, 66(Suppl. 1), 111-121.

18.

Leach

W. D.

Pelkey

N. W.

Sabatier

P. A.

(2002). Stakeholder partnerships as collaborative policymaking: Evaluation criteria applied to watershed management in California and Washington. Journal of Policy Analysis and Management, 21, 645-670.

19.

Leach

W. D.

Sabatier

P. A.

(2005). To trust an adversary: Integrating rational and psychological models of collaborative policymaking. American Political Science Review, 99, 491-503.

20.

Leach

W. D.

Weible

C. M.

Vince

S. R.

Siddiki

S. N.

Calanni

J. C.

(2013). Fostering learning through collaboration: Knowledge acquisition and belief change in marine aquaculture partnerships. Journal of Public Administration Research and Theory, 24, 591-622.

21.

Lubell

(2003). Collaborative institutions, belief-systems, and perceived policy effectiveness. Political Research Quarterly, 56, 309-323.

22.

Lubell

Fulton

(2008). Local policy networks and agricultural watershed management. Journal of Public Administration Research and Theory, 18, 673-696.

23.

Maggioni

Nelson

Mazmanian

D. A.

(2012). Industry influence in stakeholder-driven state climate change planning efforts. Policy Studies Journal, 40, 234-255.

24.

Mazur

N. A.

Curtis

A. L.

(2006). Risk perceptions, aquaculture, and issues of trust: Lessons from Australia. Society & Natural Resources, 19, 791-808.

25.

Meier

K. J.

O’Toole

L. J.

(2013). Subjective organizational performance and measurement error: Common source bias and spurious relationships. Journal of Public Administration Research and Theory, 23, 429-456.

26.

National Oceanic and Atmospheric Administration. (2014). What is aquaculture? Retrieved from http://www.nmfs.noaa.gov/aquaculture/what_is_aquaculture.html

27.

Naylor

R. L.

Goldburg

R. J.

Primavera

J. H.

Kautsky

Beveridge

M. C.

Clay

. . .Troell

(2000). Effect of aquaculture on world fish supplies. Nature, 405, 1017-1024.

28.

Nisbett

R. E.

Wilson

T. D.

(1977). The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35, 250-256.

29.

Ostrom

(1990). Governing the commons: The evolution of institutions for collective action. Cambridge, UK: Cambridge University Press.

30.

Ostrom

(2005). Understanding institutional diversity. Princeton, NJ: Princeton University Press.

31.

Resh

Siddiki

McConnell

W. R.

(2014). Does the network centrality of government actors matter? Examining the role of government organizations in aquaculture partnerships. Review of Policy Research, 31, 584-609.

32.

Rogers

Weber

E. P.

(2010). Thinking harder about outcomes for collaborative governance arrangements. The American Review of Public Administration, 40, 546-567.

33.

Sabatier

P. A.

Weible

C. M.

(2007). The advocacy coalition framework: Innovations and clarifications. In Theories of policy process. In Sabatier

(Ed.), Theories of the policy process (pp. 189-220). Boulder, CO: Westview.

34.

Thomas

C. W.

(2008, April). Evaluating the performance of collaborative environmental governance. Consortium on Collaborative Governance Mini-Conference, Santa Monica, CA.

35.

Treece

G. D.

(2002). Shrimp farm effluents. In Stickney

McVey

(Eds.), Responsible marine aquaculture (pp. 297-309). New York, NY: CABI Publishing.

36.

Tyler

Degoey

Smith

(1996). Understanding why the justice of group procedures matters: A test of the psychological dynamics of the group-value model. Journal of Personality and Social Psychology, 70, 913-930.

37.

Weber

E. P.

(1998). Pluralism by the rules: Conflict and cooperation in environmental regulation. Washington, D.C.: Georgetown University Press.

38.

Weible

C. M.

(2008). Expert-based information and policy subsystems: A review and synthesis. Policy Studies Journal, 36, 615-635.

39.

Weible

C. M.

Sabatier

P. A.

(2009). Coalitions, science, and belief change: Comparing adversarial and collaborative policy subsystems. Policy Studies Journal, 37, 195-212.

40.

Wirth

F. F.

Luzar

E. J.

(1999). Environmental management of the U.S. aquaculture industry: Insights from a national survey. Society & Natural Resources, 12, 659-672.