Building a new tool to evaluate networks and multi-stakeholder governance systems

Abstract

Networks are increasingly used in public policy to tackle ever more complex societal challenges. Multi-stakeholder governance systems have been set up to strengthen regional innovation systems, to fight the consequences of climate change and to tackle social change. However, multiple challenges mean that evaluating these networks and governance systems is complex and requires specific evaluation approaches. Learning from the strengths and weaknesses of existing approaches, this article presents a new tool (GOCAPASS) which enables evaluators to assess governance systems and networks involving multiple stakeholders. It has already been tested in regional innovation policy and governance of EU macro-regional strategies. Practical testing demonstrated its usefulness as a tool for evaluation and policy learning, though some weaknesses still need to be resolved.

Keywords

evaluation GOCAPASS governance networks tool

Introduction

Nowadays, many societal problems – such as climate change or poverty eradication – are too complex to be solved by one organisation or one government alone. These problems are usually tackled by networks and partnerships of public bodies, private sector entities and sometimes civil society organisations. This encourages new types of collaboration and governance systems. As policy-making and implementation is increasingly built on multi-actor governance networks, evaluation of these networks has become increasingly important.

Evaluation of effectiveness in governance systems and networks is particularly relevant for policy-makers ‘so that scarce public funding can be allocated to … mechanisms that are utilizing resources efficiently while adequately serving public needs’ (Provan and Milward, 2001: 415). Another positive effect is that evaluation:

is not the end point of a collaborative policy-making process, it is an input for continued and deepened cooperation. Evaluation supports learning and facilitates cooperation between stakeholders. (Van der Meer and Edelenbos, 2006: 204)

There are several challenges to evaluating networks and multi-actor governance networks that demand a specific evaluation approach. Such an approach should consider the complexity and interconnectedness of multi-stakeholder governance settings, the different development stages of networks and the diversity of partnerships and networks, including their open and dynamic character. Despite the challenges, there are concepts and approaches that can assist the evaluation of networks and governance systems. Based on existing networks and approaches, this article presents a new tool (GOCAPASS) that enables evaluators to assess the effectiveness of governance systems with multiple stakeholders.

The next section briefly defines key terms used in this article and is followed by a section covering conceptual foundations for the evaluation of governance networks. After that, a section presents the complexities of networks and multi-stakeholder governance systems in public policy, highlighting specific challenges for evaluation. A brief review of practical approaches to network evaluation helps to identify weaknesses in current methods and requirements for a new tool introduced in the next section. Finally, the strengths and weaknesses of the new tool are discussed and general conclusions presented.

Key terms: Networks, governance, systems and capacities

The complexity of societal problems, such as climate change and environmental degradation, has increased to a point where solutions cannot be expected from one organisation alone. These problems are increasingly being tackled by networks and partnerships of public bodies, private entities, and sometimes civil society organisations, giving rise to new types of networks and collaborative governance systems. These new governance arrangements aim to add value when compared with other forms of hierarchal policy-making. They promote an integrated approach to strategic planning which can be cross-border, multi-sector and multi-level. It is also participatory involving the public and business sector, academia and civil society (Uusikylä and Valovirta, 2007). These collaborative systems can be described with certain key terms.

Networks were defined by Agranoff (2003) as interorganisational structures of:

public organizations, involving formal and informal structures, composed of representatives from governmental and non-governmental agencies working interdependently to exchange information and/or jointly formulate and implement policies and programs that are usually designed for action through their respective organizations. (p. 7)

Governance is understood as a large and complex network to address common and shared problems (Heritier and Rhodes, 2011). It is based on ‘deliberative, consensus-based and reciprocal learning forms of policy-making and problem-solving’ (Bellamy et al., 2011: 140). Klijn and Koppenjan (2016) establish that ‘within the network literature, governance refers to actors’ conscious attempts to influence processes in networks’ (p. 36). Governance systems can be found in virtually all areas of social and economic development. More strategic development planning processes involve more inclusive and diverse governance systems for decision-making and implementation. Examples can be found in innovation and competitiveness policies (OECD, 2005), tourism strategies (Duran, 2013), climate change policy (Knieling and Leal Filho, 2013), territorial cooperation (INTERREG, or macro-regional strategies in Europe) and energy policies (Chartier, 2015; OECD/IEA, 2010).

Multi-stakeholder governance systems, according to Jessop (1999), involve ‘interpersonal networking, inter-organizational negotiation, and inter-systemic steering’ (p. 17) and require:

defining new boundary-spanning roles and functions, creating linkage devices, sponsoring new organizations, identifying appropriate lead organizations to coordinate other partners, designing institutions, and generating visions to facilitate self-organization in different fields. It also involves providing mechanisms for collective feedback and learning about the functional linkages and the material interdependencies among different sites and spheres of action, and encouraging a relative coherence among diverse objectives, spatial and temporal horizons, actions, and outcomes of governance arrangements. (p. 23)

This set of governance rules together with steering and control activities to promote the governance network and facilitate its self-organising capacity can be called ‘meta-governance’ (Jessop, 2011) or ‘network management’ (Klijn and Edelenbos, 2007).

In fact, both governance systems and networks can be considered as social systems. In broad terms, a social system is defined by a boundary between itself and its environment. It consists of parts (persons, organisations, roles), the relationships between them (connections, communications) and other elements which determine how the system works and develops (identity, content, functions, structures, procedures etc.).

A distinctive feature of systems is that they are not static but with internal dynamic processes and adaptive to their environment (Luhmann, 1984; Wilke, 2000). Usually systems tend to create new characteristics or functions that ‘emerge’ from the system and which determine its success. Experts highlight that the effectiveness of a system is not the sum of the effectiveness of its individual parts (Wilke, 2000: 195). So analysing and monitoring governance systems needs a review of the individual elements (organisations or people) and especially the functional capacities and systemic features that emerge. Evaluations increasingly use systemic approaches to better deal with complexity (see Caffrey and Munro, 2017; Walton, 2016).

Capacity is recognised by experts as the ability of individuals, institutions and systems to perform functions, solve problems and set and achieve objectives in a sustainable manner (UNDP, 2010: 2). In this paper, ‘capacity’ refers to the ability of multi-actor governance systems to perform policy functions (Wu et al., 2015) and the capability to act, implement, grow and adapt to changes in their environment (Ubels et al., 2010). The fuzzy concept of ‘capacity’ describes elements in collective policy processes such as vision building, strategy development and communication (Ubels et al., 2010), but also different functions of policy processes, e.g. analytical, political or operational capacities (Baser and Morgan, 2008; Wu et al., 2015). In most reflections on multi-level governance, the challenge to develop systemic capacities is well acknowledged.

Conceptual foundations of research on networks and governance systems

The foundations of studies on networks and governance systems can be found in different fields of research (see Figure 1).

Figure 1.

Theory-based research on network and governance evaluation.

An important body of knowledge stems from institutional economics and research on collective action (Coase, 1937; Olson, 1965; Ostrom, 1990; Williamson, 1975). Together with research on social capital (e.g. Woolcock and Narayan, 2000) and social systems (e.g. Luhmann, 1984; Wilke, 2000), these fields of social and economic research define basic assumptions on collective action in networks and governance systems. Research on collaborative public management confirms the value of collective action for enhanced problem-solving capacity (e.g. Agranoff, 1986; Glasbergen, 1995).

Research on governance and metagovernance (Heritier and Rhodes, 2011; Jessop, 1999, 2011) has been vital to better understanding networks in governance systems. Theoretical and applied research on territorial and multi-stakeholder governance models in the EU have contributed important insights (see ESPON and Politecnico di Torino, 2014; Van den Brande, 2014).

Two other conceptual backgrounds have added essential knowledge to in-depth analysis of relationships between (policy) network actors. The first regards network theory using specific methods known as Social Network Analysis (SNA) (Borgatti et al., 2013; Dershem et al., 2011). However, even if SNA allows researchers to compare networks and draw conclusions on their relative effectiveness (Provan and Milward, 1995), it is still a major leap from quantitative description of relationships to meaningful understanding of network effectiveness (Provan and Milward, 2001). The second background concentrates on collaborative and intergovernmental decision-making that has led to ‘actor-centred institutionalism’. This descriptive approach to policy research applies elements of game theory to multi-actor policy frameworks (Scharpf, 1997). It suggests that analysis of networks and partnerships should target not only the actors themselves, but also institutional settings, constellations of actors and their relationships (‘modes of interaction’).

More recently, research on the management of complex governance networks in the public sector (Kickert et al., 1997; Klijn and Koppenjan, 2016; Koppenjan and Klijn, 2004; Sørensen and Torfing, 2007) combined concepts and reflections on the theoretical foundations to analyse in detail the effectiveness and performance of public sector networks in theory and practice. With this, the authors contributed to the emergence of a new research field.

Despite important scientific advances, the evaluation of networks and governance still faces many challenges. Although several research frameworks analyse the effectiveness of networks, practical approaches and methods are still rare.

Is it possible to evaluate complex networks and governance systems?

Despite the valuable research on governance and network evaluation, it is still perceived as a complex and unsolved issue (Hertting and Vedung, 2012). Considering the ‘conception of complex, multi-level, dynamic policy systems’ (Sanderson, 2000: 444), there are implications for the evaluation of collaborative policies and programmes. Researchers have approached the evaluation of networks and governance systems from the perspective of complexity (Koppenjan and Klijn, 2013; Sanderson, 2000) and wicked problems (Ferlie et al., 2013; Noordegraaf et al., 2017). Without doubt, the complex and systemic character of networks needs to be considered.

Challenges

The systemic character leads to different complexities in governance systems. Hertting and Vedung (2012) observe specific substantive and institutional complexities in governance networks. Klijn and Koppenjan (2016) add to this a layer of strategic complexity. Complexities are solved by collaborative action, but can also be increased with more and more governance networks, which affect institutions and organisations. The following characteristics of networks and governance systems make their evaluation challenging:

Network diversity

Governance systems can differ, and usually do, in size, geography, purpose and functional goal. Governance diversity hampers the use of one-size-fits-all evaluation designs and unidimensional evaluation tools.

Fuzzy boundaries

Stakeholders, both organisations and individuals, can usually enter and exit the network for different reasons. The contribution of an organisation can vary considerably especially with a change of person. In addition, networks usually deal with multiple interests and different constituents. All this means that a governance network is dynamic and volatile.

Adaptiveness

A network is an adaptive system which may change its mission and direction as part of a learning process. This leads to specific problems. First, that goals are often not stated in clear and measurable terms, to ensure flexibility to operate as a network. Second, goals are not always fixed but may change (see Klijn and Koppenjan, 2016: 243).

Rules and behaviour

Network and governance system performance depends not only on the quantifiable activities of the network but also on underlying conditions and intangible elements that make collaboration possible, such as trust, motivation, common norms and values etc. So, mere observation of network activities might not identify the reasons for success and failure that are important to interpreting network performance.

Collective results

As mentioned earlier, networks have different levels of outcome and impact that usually go beyond individual objectives for each member organisation or individual. Many outcomes can only be achieved together. Desired impacts or social change may even require the active contribution of wider society. This ‘joint-production problem’ (Provan and Milward, 2001: 415) is usually an attribution challenge if it is not clear whether and to what extent the outcomes and impacts can be attributed to the network and its individual stakeholders. Evaluation designs must take into account the different levels and time horizons of results, outcomes and impacts of joint network and governance action.

Existing approaches

Various approaches have tackled the challenge of network and governance evaluation from different angles. Haarich (2014) discussed several methods to evaluate intangible assets in collaborative policy-making processes, such as organisational assessment methods, social network analysis, approaches to measure social capital, the United Nations Development Programme approach to assess capacity development, or the ‘collective impact’ model (see Figure 2). All these methodological approaches offer valuable input, yet few researchers have tackled the issue of practical models for network evaluation.

Figure 2.

Practical approaches to network and governance evaluation.

Considering the complexities of governance networks, it is no surprise that scholars and practitioners have not agreed on what can or should be evaluated. What seems to be clear is that a different approach is needed, since ‘in the multi-actor context, a different kind of knowledge is required’ (Van der Meer and Edelenbos, 2006: 203). In the past, research focussed mainly on network structure and activities, e.g. a ‘count of contacts and interactions’ (McGuire, 2011: 448). Less attention was paid to analysing network outcomes and impacts. ‘Even in the general network literature … issues of network outcomes and effectiveness are mostly ignored’ (Provan and Milward, 1995: 2). The sentinel work of Provan and Milward was one of the first to analyse different levels of network effectiveness combining community, network and organisation/participant levels. This concluded that ‘the different views of effectiveness at each level need to be considered and resolved, especially in a system that only works effectively through cooperation’ (Provan and Milward, 2001: 422). Kickert et al. (1997) consider that evaluating networks just on goal attainment is not adequate and that the mere observation of network interaction and efficiency might ‘degenerate to instrumentalism’ (p. 172). They propose using ‘ex post satisficing’ of network participants and focusing on network management. In the same context, Uusikylä and Valovirta (2007) propose observing three spheres of multi-actor network performance; 1) intangible factors crucial to effective processes, learning and resources, 2) single-organisational performance targets, and 3) multi-organisational spheres of effectiveness.

Recent developmental approaches and methods analyse and evaluate networks and partnerships that aim to generate social change and involve diverse civil society organisations (e.g. Creech and Ramji, 2004; Network Impact and Center for Evaluation Innovation, 2014a; Wilson-Grau and Nuñez, 2007). Similar studies offer an overview of cases and examples of network and partnership evaluations (e.g. Network Impact and Center for Evaluation Innovation, 2014b; Perkin and Court, 2005). Inspired by these methods, evaluation studies assess the performance of civil society partnerships (e.g. Browne, 2013) or civil society organisation networks (e.g. GEF IEO, 2016). A recent approach to evaluating networks builds on the review of content criteria (cognitive learning), processes (strategic learning) and institutions (institutional learning) (Klijn and Koppenjan, 2016).

In general, few approaches seem applicable to real-world network and governance system evaluations. Some of the existing approaches have been examined for their practical use (see Table 1).

Table 1.

Existing approaches to evaluate governance networks.

Approach	Strengths	Weaknesses
Social Network Analysis – SNA (Borgatti et al., 2013; Dershem et al., 2011)	Focus on the quantity of relationships between elements within a (formal or informal) network (persons, organisations etc.). Important methods to quantify, map and assess relationships between organisations and people in networks. Enables comparison of a network at different times or with other networks.	Does not cover quality elements, goal achievement and impact or satisfaction for network members. Does not answer ‘why’ a network works or not. Requires additional interpretation of results. Requires expert knowledge of the method, statistical analysis and specific SNA software.
Evaluating networks for social change (Network Impact and Center for Evaluation Innovation, 2014a and 2014b)	Based on combining subjective information (i.e. opinion and satisfaction of network members) with objective data (i.e. comparable figures on network development through social network analysis). Considers different dimensions of network effectiveness (connectivity, health and results). Methodology is well defined – non-experts can (partially) use it. Seeks to recommend improvements.	Requires a variety of methods for data collection and analysis. The ‘Result dimension’ is considered but relies only on the satisfaction/opinion of network members. Does not cover results and impacts. There is only a qualitative description of the results, limiting visibility of the potential to improve.
Evaluating governance networks (Klijn and Koppenjan, 2016)	Considers different dimensions of network effectiveness (content, processes and institutions). Considers qualitative and intangible elements that contribute to network effectiveness. Seeks to evaluate the network to recommend improvements.	Requires extensive empirical research (interviews, survey). Some assessment criteria are interdependent, hampering clear assessment. Some assessment criteria can only be evaluated ex-post. There is only a qualitative description of the results, limiting visibility of the potential to improve.

As can be seen, the existing approaches each have important advantages as well as shortcomings when it comes to practical use, thus indicating requirements for a new tool.

Requirements for a new evaluation tool

To increase usability for practitioners, a new tool for evaluating the performance and effectiveness of networks and governance systems should:

use different data sources to triangulate information and to cover quantitative and qualitative aspects of the network,

consider different dimensions of network effectiveness to depict governance system complexity more realistically,

be flexible, adapting to different networks in various policy fields, but solid enough to offer comparable and quantifiable results about network effectiveness,

have well-defined methodology and analytical methods suitable for practitioners,

use measurable assessment criteria in a clear and operational research framework,

allow network stakeholders to learn about their actual network performance,

produce timely evaluation results during implementation of network activities, so practical recommendations can be fed back into the network,

offer quantitative and comparable evaluation results to enable comparison of the network over time, or with other networks,

visualise evaluation results so they can be easily communicated to network members (or other stakeholders) to support discussion on learning and capacity development within the network.

GOCAPASS – proposal for a new tool

Rationale

Traditional designs are not suited to the evaluation of complex governance systems and existing approaches only partially facilitate useful tools for practitioners. Therefore, a new tool to measure and evaluate the effectiveness of networks and governance systems was developed. Governance Capacity Assessment tool or GOCAPASS (Haarich, 2014, 2016a, 2016b, 2016c) objectively measures networking and governance capacities, adapting to different development stages of networks in different policy fields. The new tool is built from a practical perspective, to be easily used during policy implementation processes and to contribute to on-going learning processes.

Design

GOCAPASS can identify, measure and monitor functions and capacities within complex governance systems. The tool enables practitioners to map and monitor multi-governance capacities that cover policy, managerial and coordinative functions as well as the governance environment.

The tool focuses on the different functions of a governance system and 21 functional capacities that are needed for decision-making, management and implementation, communication and cooperation as well as the enabling environment and conditions for development of the governance system (see Table 2).

Table 2.

GOCAPASS – Dimensions and Factors of Governance Performance.

Governance Capacity Dimension	GOCAPASS Functional Capacity Factors
D1 Policy Capacity	1.1 Analysis for decision-making 1.2 Translate needs into action 1.3 Responsibilities 1.4 Vision- and goal-setting 1.5 Priority-setting 1.6 Reflection and feedback
D2 Managerial Capacity	2.1 Capacity to budget 2.2 Adequate implementation support 2.3 Implementation resources 2.4 Monitoring and control 2.5 Stakeholder engagement
D3 Networking and Cooperation Capacity	3.1 Trust as precondition 3.2 Exchange of information 3.3 Communication flows 3.4 Knowledge management 3.5 Active cooperation in policy-making and implementation
D4 Enabling Environment and Governance Levers	4.1 Leadership 4.2 Mutual support (within the system) 4.3 Learning facilities in the system 4.4 Openness and adaptability (learning from others) 4.5 Commitment from the environment

Each functional capacity in the four dimensions is covered by 2–3 research criteria (see Table 3). The dimensions and factors are the same when analysing different networks. The research criteria can be adapted for each evaluation of specific networks or policy fields. This adds flexibility to the tool while ensuring methodological coherence and comparability with other networks. To ensure comparability, a standard measurement scale is used. For each research criterion, the scale ranges from ‘no capacity’ (1) to the ideal, ‘full capacity’ (5). This scale provides a level of detail that is helpful for full analysis and for providing recommendations on how to improve.

Table 3.

Research framework for GOCAPASS (Example of Dimension 3 developed for the assessment of a regional innovation governance network).

GOCAPASS Dimension	GOCAPASS Factor	Research Criteria
D3 – Networking and Cooperation Capacities	3.1 Trust as precondition	• A predisposition for cooperation between and among private (e.g. cluster), and public stakeholders (e.g. departments, regional-national). • Positive previous experiences with cooperation among actors inside and outside the region. • (Even without trust) activities and motivation improve the predisposition for cooperation.
	3.2 Exchange of information	• A comprehensive and effective exchange of information based on mutual trust: - between public administration stakeholders (analytical, planning, monitoring, evaluation), - between private stakeholders (even competitors), - between public administration and the private sector.
	3.3 Communication flows	• Regular informal communication between political, administrative and private stakeholders (networking). • Formal communication mechanisms within public administration (internal communication). • Formal mechanisms to facilitate communication between public and private stakeholders (external communication, dissemination, information management, communication plan).
	3.4 Knowledge Management	• An effective information storage and management system covering regional public measures (and their results), open to the public. • An open and effective information storage and management system for complementary private activities. • An effective foresight and strategic intelligence system to manage knowledge from outside the region and to facilitate learning.
	3.5 Active Cooperation in policy-making and implementation	• Active cooperation between public, private and academic stakeholders in the design of strategy, programmes and measures. • Active cooperation between public, private and academic stakeholders during the implementation of instruments, initiatives and projects. • Active cooperation between public, private and academic stakeholders in decision-making and other strategic regional issues.

Methodology

The GOCAPASS tool foresees different steps (see Figure 3) from data collection and analysis to the visualisation of results and comparison with other networks or baseline/target values.

Figure 3.

GOCAPASS methodology.

Data collection and analysis should have a specific methodology. Data collection is structured around research criteria based on the specific evaluation using: 1) desk review (focus on key documents and written evidence), 2) survey of stakeholder groups (e.g. working group coordinators or network members to get an overview of network structures, content and processes), and 3) in-depth interviews with key stakeholders (for additional qualitative and informal information, including anecdotal evidence). The number of interviews and survey sample can vary with the evaluation scope and size.

Data analysis can be structured on the S-C-P grid with each research criterion reviewed for Structures, Contents and Processes (Table 4), bringing together information from the different sources.

Table 4.

S-C-P Grid for functional capacity factors.

S – Structure	C – Contents	P – Processes
Structures and bodies in charge of …	Documents, reports, anecdotes, images that reproduce or reflect …	Procedures and protocols that facilitate …

Defining research criteria and using the S-C-P grid facilitate ratings based on data from different sources. After processing the data, the evaluator uses a 1–5 scale to rate each factor. For newly established governance systems a 1–3 rating scale (not developed, weak, strong) might be more appropriate, as all capacities could still be at an initial development stage. To establish an adequate value for each factor, research criteria are rated as fully, partially, or not accomplished. Each factor is given an aggregate value based on the research criteria. These factor values are then combined to establish the average value of each GOCAPASS dimension.

Visualisation of results – supporting learning processes

The GOCAPASS tool supports the analysis and improvement of governance systems as part of an on-going learning process. The visualisation and communication of evaluation results to network members and constituencies is useful in raising awareness within the system. Debating the results hopefully leads to ‘deliberative dialogues’ and to strengthening governance capacities, such as trust and learning within the network. GOCAPASS supports learning processes in networks with specific visualisation tools, such as a traffic light dashboard (see Figure 4) that shows the progress for each functional capacity (red: weak, yellow: intermediate, green: strong).

Figure 4.

GOCAPASS visualisation tool.

Application

The GOCAPASS tool is designed to evaluate the effectiveness of networks and multi-stakeholder governance structures in a general and flexible manner. With minor adjustments, the tool can be adapted to the specific characteristics of networks in different policy areas, e.g. innovation and smart specialisation, territorial cooperation, spatial planning, climate change mitigation, etc. The basic GOCAPASS architecture can be adapted to particular policy frameworks with detailed research criteria and questions. The tool guides and assists external evaluators as well as network managers (e.g. in a process of self-evaluation) during systemic assessment of a governance network. This assessment might be carried out for accountability or as part of an internal capacity-building process. GOCAPASS is particularly useful in strategic networks that have clear long-term goals, but which suffer from a lack of defined intermediate results so on-going progress measurement is difficult. In these cases, GOCAPASS helps to visualise progress and areas for improvement so the collaborating network can be more effective and efficient.

Testing: Evaluating regional innovation governance and macro-regional strategy networks

GOCAPASS has been tested in two policy fields. Firstly, to evaluate regional partnerships promoting innovation under the public-private governance of a regional innovation system. GOCAPASS reviewed the emerging governance system for the recently elaborated Aysén Regional Innovation Strategy in Chile, to establish a baseline for future governance capacity evaluations (Haarich, 2016b). Secondly, a specific GOCAPASS application assisted governance capacity monitoring and development in EU transnational and macro-regional strategic frameworks (Haarich, 2016a). So far this has only reached a theoretical stage.

For Aysén, the application started with a thorough examination of the Regional Innovation Strategy and implementation of innovation policies in the region at the beginning of 2015. Detailed research criteria and questions were formulated in this phase.

Data collection involved an intensive desk review of activities and decisions in Aysén regarding public innovation support as well as private innovation and research. There were then nine in-depth interviews with Strategy stakeholders. In addition, a written survey covered 26 regional stakeholders from different areas within the regional innovation system (regional government, national public services, universities and research centres).

During analysis, the development of each regional capacity factor was rated, using the GOCAPASS research framework and a 1–5 scale. Desk review, survey and interview results were combined for an overall mean value for each factor and GOCAPASS dimension. Application of the tool in March 2015 resulted in a baseline for regional innovation governance capacities in Aysén (see Figure 5).

Figure 5.

Baseline measurement with GOCAPASS (Dimension 3) in Aysén, Chile (March 2015).

The baseline showed the relatively low level of all GOCAPASS dimensions, as expected with a newly established governance system. Management Capacity (2.32) was the most developed followed by a relatively favourable Governance Environment (2.14). Networking and Cooperation development was quite low (1.84) and Policy Capacity (1.64) was the least developed of the innovation governance capacities.

Based on this initial assessment, in 2015 a plan to strengthen regional capacities was designed together with regional stakeholders and implemented. A first step was to present the evaluation to stakeholders, raising awareness of networking and collaboration for innovation support and how it can be made more effective. This awareness-raising was seen as important within the capacity development process. Overall, the network coordinators considered the GOCAPASS tool useful as it reported detailed answers on the effectiveness and learning needs of the governance system easily and quickly. Other tools would have required more time and research effort and/or produce mostly qualitative and descriptive results, less able to raise awareness among the regional stakeholders.

A second pilot application of GOCAPASS was prepared to assess the effectiveness of governance systems in EU macro-regional strategies. Currently, there a macro-regional strategies in the Baltic Sea, Danube, Adriatic-Ionian and Alpine regions. Even without a standard definition for a macro-region or for such a strategy, common challenges for EU Member States and third countries in a geographical area are better faced with strengthened cooperation. Obviously, governance systems for macro-regional strategies are highly complex, as they embrace not only different countries, but also various sector policy fields and stakeholders from several administrative levels (EU, national, sub-national). To date, GOCAPASS was prepared and presented as a tool to measure effectiveness and capacity within macro-regional governance networks (Haarich, 2016a), though no practical case has been tested so far. This is partially because of the complexity of macro-regional governance with many governance layers and thematic networks, but also due to increased interest in evaluating substantive policy results, leaving less time and space for learning about processes in macro-regional networks.

Discussion

GOCAPASS has demonstrated its usefulness in practical and theoretical applications. The overall design with its dimensions, factors and methodology, proved valid for both cases when evaluating networks.

The main strength of GOCAPASS is that it starts from a pre-designed research framework, so it can be quickly set up and adapted to any network or governance system. As the test cases have shown, it can be easily adjusted to different contexts by adapting the research criteria and the resulting survey and interview questions. The adjustment does not require complex research efforts meaning network managers can use this self-evaluation tool with only minor external support. Other strengths are that data collection does not require extensive time and research resources and even if it uses different data sources and considers quantitative and qualitative information, results visualisation helps to raise the awareness of network members and to support capacity development and learning processes within networks. In addition, quantified measurement helps to compare the effectiveness of one network over time or with other networks.

While GOCAPASS supports evaluation of processes and learning in networks, it does not evaluate policy field results that may (or not) have been achieved due to the network. The focus on processes and lack of an add-on to evaluate the achievement of objectives and substantive policy changes, can be seen as a weakness of GOCAPASS. Many governance systems want to know first and foremost about the effectiveness of achieving results and then about their internal processes and the effectiveness of their collaboration. GOCAPASS could add result and impact evaluation using traditional methods such as impact evaluation, outcome mapping or contribution analysis. Furthermore, measurement is not totally objective. Even if measuring with quantitative values, the rating process depends on the evaluator. Judgements may vary from one expert to another, reducing objectivity and limiting comparability. This weakness can be solved by clear instructions to evaluators and an even more solid research framework. Another weakness of GOCAPASS is the need for a baseline measurement or for comparable, unified standards covering the dimensions and factors, to ensure meaningful conclusions from the assessment. However, this need for standards can also be seen as an opportunity. In this sense, GOCAPASS could benefit from establishing certified quality standards for effective networking and collaboration. Then GOCAPASS would contribute – as an audit tool – to quality management for networks and governance systems.

Conclusions

Evaluations of networks and complex governance systems are more and more important, as today many policies are decided and implemented through collaborative approaches. However, the development of approaches and methods to assess the performance of networks at their diverse levels is hampered by multiple challenges. Based on existing studies and practical approaches a new concept for evaluating governance networks has been developed: GOCAPASS. This analytical tool helps to open the black box of governance performance. GOCAPASS has been already tested, demonstrating advantages that make it a valuable evaluation and policy learning tool for networks. Further research is needed to test its usability in other policy fields and to solve perceived weaknesses. Despite its infant flaws, GOCAPASS may even serve as a starting point to establish quality standards for effective networking and collaboration in governance systems.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

Dr Silke Haarich is an evaluator of regional development policies and Senior Consultant at Spatial Foresight GmbH, a Luxembourg-based company which provides territorial policy support and research.

References

Agranoff

(1986) Intergovernmental Management: Human Services Problem-solving in Six Metropolitan Areas. Albany, NY: State University of New York Press.

Agranoff

(2003) Leveraging Networks: A Guide to Public Managers Working Across Organizations. Arlington, VA: IBM Endowment for the Business of Government.

Baser

Morgan

(2008) Capacity, change and performance. Study report. Discussion Paper No. 59B. Maastricht, The Netherlands: European Centre for Development Policy Management. Available at: http://ecdpm.org/wp-content/uploads/2013/11/DP-59B-Capacity-Change-Performance-Study-Report-2008.pdf (accessed 23 May 2017).

Bellamy

Castiglione

Follesdal

et al . (2011) Evaluating trustworthiness, representation and political accountability in new modes of governance. In: Heritier

Rhodes

(eds) New Modes of Governance in Europe. Governing in the Shadow of Hierarchy. London: Palgrave, 135–62.

Borgatti

Everett

Johnson

(2013) Analyzing Social Networks. London: SAGE.

Browne

(2013) Monitoring and evaluating civil society partnerships (GSDRC Helpdesk Research Report 1024). Birmingham, UK: GSDRC, University of Birmingham.

Caffrey

Munro

(2017) A systems approach to policy evaluation. Evaluation 23(4): 463–78.

Chartier

(2015) Energy democracy. A case study of energy generation in Växjö. Master thesis in Sustainable Development at Uppsala University, No. 271, 44pp, 30ECTS/hp. Available at: http://www.diva-portal.se/smash/get/diva2:848378/FULLTEXT01.pdf (accessed 23 May 2017).

Coase

(1937) The nature of the firm. Economica 4(16): 386–405.

10.

Creech

Ramji

(2004) Knowledge networks: Guidelines for assessment. Working paper prepared for the International Institute for Sustainable Development. Winnipeg, Manitoba: IISD.

11.

Dershem

Dagargulia

Saganelidze

et al . (2011) NGO Network Analysis Handbook: How to Measure and Map Linkages between NGOs. Tbilisi, Georgia: Save the Children.

12.

Duran

(2013) Governance for the tourism sector and its measurement. UNWTO Statistics and TSA Issue Paper Series STSA/IP/2013/01 (Online). Available at: http://statistics.unwto.org/en/content/papers (accessed 23 May 2017).

13.

ESPON and Politecnico di Torino (2014) Towards better territorial governance in Europe: A guide for practitioners, policy and decision makers based on contributions from the ESPON TANGO Project. Luxembourg: ESPON.

14.

Ferlie

Fitzgerald

McGivern

et al . (2013) Making Wicked Problems Governable? The Case of Managed Networks in Health Care. Oxford: Oxford University Press.

15.

GEF IEO – Global Environment Facility Independent Evaluation Office (2016) Evaluation of the GEF–Civil Society Organization Network. Evaluation Report No. 108, Washington, DC: GEF IEO.

16.

Glasbergen

(ed.) (1995) Managing Environmental Disputes: Network Management as an Alternative. Dordrecht: Kluwer Academic Publishers.

17.

Haarich

(2014) How to measure capacity development? A new tool for the evaluation of intangible assets in regional development. Paper presented at the 11th Biennial EES Conference ‘Evaluation for an Equitable Society’, 1–3 October, Dublin.

18.

Haarich

(2016a) How to know if multi-level governance works? A new approach to map and monitor governance systems of macro-regional strategies. Paper presented at the RSA Annual Conference ‘Building Bridges: Cities and Regions in a Transnational World’, 3–6 April, Graz.

19.

Haarich

(2016b) Innovation governance in Chile – a methodology to measure cultural elements in innovation systems. Paper presented at the Third ISA Forum of Sociology, RC23. Sociology of Innovation, 10–14 July, Vienna.

20.

Haarich

(2016c) Evaluation of capacities in multi-level governance networks: The GOCAPASS tool. In: Dotti

(ed.) Learning from implementation and evaluation of the EU Cohesion Policy: Lessons from a research-policy dialogue. RSA Research Network on Cohesion Policy. Brussels.

21.

Heritier

Rhodes

(eds) (2011) New Modes of Governance in Europe: Governing in the Shadow of Hierarchy. Palgrave Studies in European Union Politics Series. London: Palgrave.

22.

Hertting

Vedung

(2012) Purposes and criteria in network governance evaluation: How far does standard evaluation vocabulary takes us? Evaluation 18(1): 27–46.

23.

Jessop

(1999) Governance failure. In: Stoker

(ed.) The New Politics of British Local Governance. London: Macmillan, 11–32.

24.

Jessop

(2011) Metagovernance. In: Bevir

(ed.) Handbook of Governance. London: SAGE, 106–23.

25.

Kickert

WJM

Klijn

E-H

Koppenjan

JFM

(1997) Managing networks in the public sector: Findings and reflections. In: Kickert

WJM

Klijn

E-H

Koppenjan

JFM

(eds) Managing Complex Networks: Strategies for the Public Sector. Thousand Oaks, CA: SAGE.

26.

Klijn

E-H

Edelenbos

(2007) Meta-governance as network management. In: Sørensen

Torfing

(eds) Theories of Democratic Network Governance. London: Palgrave-Macmillan.

27.

Klijn

E-H

Koppenjan

JFM

(2016) Governance Networks in the Public Sector. London: Routledge.

28.

Knieling

Leal Filho

(eds) (2013) Climate Change Governance. Berlin Heidelberg: Springer-Verlag.

29.

Koppenjan

JFM

Klijn

(2004) Managing Uncertainties in Networks: A Network Approach to Problem Solving and Decision Making. London: Routledge.

30.

Koppenjan

JFM

Klijn

(2013) What can governance theory learn from complexity theory? Mirroring two perspectives. In: Keast

Mandell

Agranoff

(eds) Network Theory in the Public Sector: Building New Theoretical Frameworks. New York: Routledge, 187–206.

31.

Luhmann

(1984) Soziale Systeme. Frankfurt aM: Suhrkamp

32.

McGuire

(2011) Network management. In: Bevir

(ed.) Handbook of Governance. London: SAGE, 436–53.

33.

Network Impact and Center for Evaluation Innovation (2014a) Part 1 of a guide to network evaluation. Framing paper: The state of network evaluation. Working Paper.

34.

Network Impact and Center for Evaluation Innovation (2014b) Part 2 of a guide to network evaluation. Evaluating networks for social change: A casebook. Working Paper.

35.

Noordegraaf

Douglas

Bos

et al . (2017) How to evaluate the governance of transboundary problems? Assessing a national counterterrorism strategy. Evaluation 23(4): 389–406.

36.

OECD (2005) Governance of innovation systems. Volume 1: Synthesis Report. Paris: OECD.

37.

OECD/IEA (2010) Energy Efficiency Governance: Handbook. International Energy Agency. Paris: OECD.

38.

Olson

(1965) The Logic of Collective Action: Public Goods and the Theory of Groups. Cambridge, MA: Harvard University Press.

39.

Ostrom

(1990) Governing the Commons: The Evolution of Institutions of Collective Action. Cambridge: Cambridge University Press.

40.

Perkin

Court

(2005) Networks and policy processes in international development: A literature review. Working Paper 252. Overseas Development Institute, London.

41.

Provan

Milward

(1995) A preliminary theory of network effectiveness: A comparative study of four community mental health systems. Administrative Science Quarterly 40(1): 1–33.

42.

Provan

Milward

(2001) Do networks really work? A framework for evaluating public-sector organizational networks. Public Administration Review 61(4): 414–23.

43.

Sanderson

(2000) Evaluation in Complex policy systems. Evaluation 6(4): 433–54.

44.

Scharpf

(1997) Games Real Actors Play: Actor-Centered Institutionalism in Policy Research. Boulder, CO: Westview Press.

45.

Sørensen

Torfing

(2007) Theoretical approaches to metagovernance. In: Sørensen

Torfing

(eds) Theories of Democratic Network Governance. London: Palgrave-Macmillan.

46.

Ubels

Acquaye-Baddoo

Fowler

(eds) (2010) Capacity Development in Practice. London: Earthscan Ltd.

47.

UNDP (2010) Measuring capacity. Available at: http://content-ext.undp.org/aplaws_publications/2679640/UNDP_Measuring_Capacity_July_2010.pdf (accessed 23 May 2017)

48.

Uusikylä

Valovirta

(2007) Three spheres of performance governance: Spanning the boundaries from single-organization focus towards a partnership network. Evaluation 13(4): 399–419.

49.

Van den Brande

(2014) Report on multilevel governance and partnership. Prepared at the request of the Commissioner for Regional and Urban Policy Johannes Hahn. October.

50.

Van der Meer

F-B

Edelenbos

(2006) Evaluation in multi-actor policy processes. Evaluation 12(2): 201–18.

51.

Walton

(2016) Expert views on applying complexity theory in evaluation: Opportunities and barriers. Evaluation 22(4): 410–23.

52.

Wilke

(2000) Systemtheorie I: Grundlagen. 6th Edition. Stuttgart: Lucius.

53.

Williamson

(1975) Markets and Hierarchies. New York: Free Press.

54.

Wilson-Grau

Nuñez

(2007) Evaluating international social-change networks: A conceptual framework for a participatory approach. Development in Practice 17(2): 258–71.

55.

Woolcock

Narayan

(2000) Social capital: Implications for development, theory, research, and policy. World Bank Research Observer 15(2): 225–49.

56.

Ramesh

Howlett

(2015) Policy capacity: A conceptual framework for understanding policy competences and capabilities. Policy and Society 34: 165–171.