An Application of a Multicriteria Model to Assess the Quality of Local Governance

Local Governance in Portugal

Local government is democratic in Portugal since 1974 (first elections in 1976). From then on, municipalities became the major players in the country’s territorial development and one of the most important pressure groups in Portuguese politics (Tavares and Camões 2010). Currently, there are 308 municipalities responsible for delivering essential infrastructure services (mainly water, wastewater, urban transport, and waste services). Local governments also play an important role in other areas such as culture, tourism, and, increasingly, social welfare and basic education. Portugal is a suitable context for testing the development of an MGI through MCDA modeling because all local governments operate under the same rules and have a similar institutional architecture (some features vary as a function of population size but the powers and institutions remain the same across the country).

Despite its contributions to social cohesion and proximity, the recent history of local government in Portugal has also been bounded by institutional failure and wrongful governance practices. The great authority and discretion given to mayors and the fact that local governments are major employers, regulators, and service providers in many municipalities has often led to corruption and clientelism (De Sousa 2008). Furthermore, new modes of delivery of urban services and/or infrastructures (e.g., the creation of municipal companies) and new types of interaction with the private sector (e.g., the development of local public–private partnership [PPP] arrangements) raised important governance issues. In the current context, there are few incentives for achieving good municipal governance (de Sousa et al. 2015). Local governments have been known to deal with these issues and with reporting and accountability procedures quite differently from one municipality to another (da Cruz et al. 2016).

Methodology and Ownership of the Problem

MCDA literature and scholarship studies theoretically sound and meaningful ways of transforming “impacts” into “scores” (i.e., associate a number in a scale to a real-world performance) and transforming “partial” scores (i.e., scores in a particular criterion) into “overall” scores (i.e., aggregating the scores of the various criteria to come about with a single overall score). Using an additive model (sum of weighted scores) to aggregate the scores of each criterion and calculate the overall governance level has several advantages (Mateus, Ferreira, and Carreira 2008). More than just being able to rank municipalities—for example, according to their overall score—MCDA modeling allows for evaluating outcomes against each criterion individually (according to their partial scores) or for each dimension of governance (sum of weighted scores of the criteria contained in each particular dimension of governance) (da Cruz and Marques 2013). Nevertheless, “in a multi-criteria framework, what really matters is the process since the problem structuring will determine the result” (Munda 2004, p. 673). This is why it is essential to design a participatory process to structure the model and take into account the values and opinions of the problem owner or legitimate decision maker(s). Contrary to most multiple criteria problems, the purpose or process of assessing the quality of local governance does not have a single, easily identifiable, and legitimate decision maker. The MGI for Portugal was modeled with the input of practitioners and stakeholders with responsibilities over (or affected by) local governance.

After resolving the decision-maker issue, it is necessary to collect his or her or their input to (in brief) (1) validate the assessment framework and the criteria of (good) local governance, (2) select suitable quantitative or qualitative descriptors to measure performance in each criterion, and (3) define the reference levels of the criteria so that the weighting coefficients may be obtained (Bana e Costa, Carnero, and Oliveira 2012). Because “the process” is the main concern, each MCDA model is tailored to fit a particular problem. And although the structuring process might be troublesome, the additive hierarchical model that aggregates the scores of the various criteria is quite simple. A hierarchical model is a composition of simple additive models, adapted to a hierarchical criteria structure (Mateus, Ferreira, and Carreira 2008)—for example, “good governance” at the top of the hierarchy, followed by several “dimensions of governance,” followed by individual underlying criteria. An additive model can be represented through equation (1):

G ( m i ) = ∑ j = 1 n c j × G j ( m i ) with { G j ( good j ) = 100 G j ( neutral j ) = 0 ,

where G(m_i) is the overall governance level of municipality m_i, G_j(m_i) is the score of the municipality in the criterion j, good _j and neutral _j are the reference levels of performance on criterion j, and c_j is the weighting coefficient of criterion j, such that ∑ j = 1 n c j = 1 and c j > 0 .

As can be seen in equation (1), the scores of 0 and 100 were arbitrarily assigned to the “Neutral” and the “Good” reference levels in each criterion. Whereas establishing these anchors is not a requirement (e.g., the minimum and maximum values could have been selected to construct the interval scale), experience shows that selecting the “Neutral” (below which performance would be considered to be negative—governance worst practices) and the “Good” (above which performance would be considered to be extremely positive—governance best practices) performance levels has some cognitive advantages (Bana e Costa and Oliveira 2012). Structured in this manner, the scores will have intrinsic meaning to the user (and also to the decision maker while eliciting qualitative judgments to compute the weights in the decision conferences—see the “Decision Conferencing” section).

Consultation with Key Stakeholders

In the scoping phase of this study, virtually all the major entities whose missions concerned (even if only marginally) local governance in Portugal were contacted. The purpose was to present the objectives of the MGI, gather feedback on what should be measured and why, and learn what data they possess (to feed the MCDA model). The name and scope of these entities are the following:

From this list, six entities immediately showed interest in the research and scheduled meetings with the authors (AMA, IGF, InCI, DGPJ, DCIAP, and TIAC). In these meetings, the MGI framework (definition of governance and the several dimensions), the possible criteria, and the data available were the main topics discussed.

Populating the Value Tree

To operationalize the concept of municipal governance, an adaptation of the definition proposed in the WGI project was assumed (Kaufmann, Kraay, and Mastruzzi 2011). Governance was defined as the “traditions and institutions by which authority in a country, region or municipality is exercised.” This includes (1) the process by which governments are selected, monitored, and replaced (“Voice and accountability” and “Political stability”); (2) the capacity of the government to effectively formulate and implement sound policies (“Government effectiveness” and “Regulatory quality”); and (3) the respect of citizens and the state for the institutions that govern economic and social interactions among them (“Rule of law” and “Control of corruption”).

Although Kaufmann, Kraay, and Mastruzzi (2011) defined six dimensions of governance, considering the Portuguese local administration, the “Rule of law” and the “Control of corruption” can be treated as one dimension (municipalities abide by the same rules and the judicial system operates at the national level). Thus, the assessment framework was structured as follows:

Note that any other conceptual framework could have been implemented. These definitions and dimensions of governance were adopted because they are widely recognized and used by practitioners and scholars. In any case, the purpose was simply to provide a starting point for the discussions carried out with the decision-making group (DMG; see the “Decision Conferencing” section). After the consultation phase with key stakeholders, the research team was able to suggest the value tree represented in Figure 1 (to be completely accurate, the criteria A3 and B3 were added during the decision conferences). Still, it is fair to wonder whether the value tree would be very different if the WGI framework were not suggested to expedite the process, and it was left open for the stakeholders to complete (e.g., through an additional decision conference just to conceptualize the problem). This may be a limitation. However, the extra time commitment could also have jeopardized the feasibility of the modeling process.

OPEN IN VIEWER

Figure 1.

Dimensions and criteria of the Portuguese MGI (using M-MACBETH software).

To transform the many aspects of the problem into evaluation criteria, all the aspects considered to be relevant (by the decision maker) should be considered. Nevertheless, some constraints have to be respected, for instance: Criteria must be nonredundant and preferentially independent (Siskos, Askounis, and Psarras 2014) and the data should be up-to-date and retrievable for all municipalities (da Cruz and Marques 2013). Criteria must also have theoretical grounding (Andrews, Hay, and Myers 2010). If good municipal governance is interpreted as the way the local government–general society interactions should occur, then governance assessments rely on a set of criteria that are unavoidably normative (Bouckaert and Van de Walle 2003). Table 1 presents the normative principles behind the MGI’s criteria.

OPEN IN VIEWER

Table 1.

Normative Assumptions Behind the Governance Criteria.

Governance Dimensions	Governance Criteria	Normative Belief
Voice and accountability	A1	An active participation of local civic groups and individual citizens in the meetings of municipal parliaments makes local governments more accountable.
	A2	Allowing citizens to decide on how to allocate a significant part of the municipal budget (i.e., formulating and selecting local investments) is an effective way of giving voice to citizens’ concerns.
	A3	Citizens should be able to know the career path of the elected officials and to ascertain whether or not they are prone to be influenced by certain lobbies or have motivations other than protecting the public interest at all times; the opposition should have the means to effectively carry out this monitoring even more closely.
	A4	Despite the strict national accounting rules for all local governments, the quality of this reporting differs substantially from one municipality to another (especially regarding off-budget spending and off-balance sheet debt).
	A5 and A6	High levels of transparency allow citizens to know what the processes, structures, and products of government are; publicly disclosing some crucial items increases public trust and provides incentives to accountable public management.
Political stability	B1	High electoral participation is a symptom of a healthy democracy; efforts to involve citizens and raise awareness by local parties and/or candidates along with public trust on local institutions should result in higher voter turnouts.
	B2	Holding the majority of the seats is crucial for empowering local governments; certain local reforms might only be feasible if the winning list has political strength.
	B3	The ability to generate consensus by taking into account the views and opinions of others (namely, the nonelected councilmen) is beneficial for society and promotes political stability.
	B4	The public interest should be safeguarded against political maneuvering, and a change in the executive should not prevent good policies from being pursued; continuous, credible, and long-term planning effectively involving local stakeholders contributes to achieving this.
Government effectiveness	C1	Very high debt levels may hinder the economic sustainability of municipalities; effective local governments should be able to meet their responsibilities in terms of public-service delivery without jeopardizing future borrowing capacity.
	C2	The credibility of local policies and planning is strongly associated with the credibility of the budget.
	C3, C4, and C5	The availability and quality of local infrastructure services (considered to be services of general economic interest) is essential for the well-being of citizens.
	C6	Although often financially unsustainable (noneconomic services of general interest), social, cultural, education, and recreational services are crucial to promote development and social cohesion.
Market access and regulation	D1	Favoritism (local governments) and/or collusion (suppliers) make prices detach from costs; with effective competition for the market, few suppliers should not be able to win several public contracts.
	D2	The use of price signals (e.g., inclining blocks or seasonal rates) and ensuring the financial sustainability of utility services are good regulatory practices.
	D3	Higher local taxes hinder consumption and private-sector development.
Rule of law and prevention of corruption	E1	Suppliers and contractors often have to cope with late payments from local governments, and some municipalities build a (bad) reputation due to the time taken to settle invoices.
	E2	Public procurement is a key area for risks of corruption at the local level, and the major problems arise in direct awards.
	E3	Good public contracts (well-drafted, legal) protect the public interest.
	E4	Individuals or businesses are more prone to submit a complaint to the ombudsman when they feel that the local government or its entities (e.g., municipal companies) fail to respect the rule of law.

Introduction

The validation and fine-tuning of the set of criteria and respective descriptors as well as the determination of weights of the additive model were carried out in two decision conferences (Phillips 2007). Several entities were consulted and invited to establish a wide-ranging group of decision makers (the DMG), representing the citizens, the local administration, the central government, and the audit/monitoring institutions. All entities representing the local administration refused to participate (namely, the ANMP, the National Association of Civil Parishes, and the National Association of Local Civil Servants).

In the end, the DMG was composed of the Director-General of the TC and Secretary-General of the Council for the Prevention of Corruption (CPC), the President of TIAC, a representative from the Department of Innovation and Knowledge Management of AMA, and the Inspector of Finance (Director) from IGF responsible for local administration issues. The decision conferences took place in Lisbon on March 1 and 12, 2013, and the two authors of this article acted as facilitators. The MGI is a model of good local governance that is aligned with the values of the participants in the decision conferences who fine-tuned and validated the criteria set. A different DMG would likely render a different MGI (i.e., the values and/or priorities of the participants could be different and that would reflect on the criteria, descriptors, and relative weighting coefficients of the model).

Fine-Tuning the Criteria and Performance Descriptors

During the decision conferences, several adjustments were made to the MGI (the DMG was allowed to change everything about the model). Some of the modifications were conceptual. For instance, to be more in line with the Portuguese local administration reality, the DMG decided to change the name of dimension “D” to “Market access and regulation” and of dimension “E” to “Rule of law and prevention of corruption.” The criteria “A3—Political accountability” and “B3—Pluralism in decision-making” and respective descriptors (see the online appendix) were added during the first decision conference. The extensive expertise of the elements of the DMG on local administration matters was crucial in this process.

The criteria are operationalized by quantitative or qualitative descriptors (ordered sets of plausible impact levels). The natural, proxy, or constructed descriptors must preserve the independence in terms of preference of the criteria (Mateus, Ferreira, and Carreira 2008). The final performance descriptors adopted for the Portuguese MGI during the decision conferences are presented in the online appendix. By operationalizing broad and complex concepts, these innovative descriptors may be a helpful resource for local governance assessment frameworks in other international jurisdictions.

Data availability (up to date and systematically obtainable for all Portuguese municipalities) was a major constraint to the selection of performance descriptors. For instance, as pointed out by the DMG, for criterion D1 “Market access,” the average number of bidders in public tenders could be a preferable performance descriptor (a higher number would indicate lower levels of favoritism and collusion), but this information is not available for all municipalities. Instead, we had to consider the average number of contracts obtained by each supplier for contracts more than €150,000 (threshold above which a public tender is mandatory by law) in the last four years (local governments have a four-year term) as a proxy descriptor. Several data sources feed the performance descriptors presented in the online appendix, for example, the National Elections Commission, the TC, the Ombudsman, DGAL, local governments’ websites, InCI’s public procurement online database, minutes of the meetings of municipal parliaments and local executives, annual reports of local governments and municipal companies (including financial statements), the National Statistics Institute, the annual reports of the Water and Waste Services Regulation Authority, and the Portuguese Environmental Agency.

Finally, it is important to note that the criteria and descriptors underlying this model to measure the quality of local governance include both institutional aspects (e.g., the use of participatory budgeting) and output/outcome aspects (e.g., the quality of services). Methodologically, this does not represent a problem; the only rules the criteria/descriptors need to follow concern their comprehensiveness, nonredundancy, and preferential independence (Keeney 1992). Conceptually, however, this could be problematic if one sees (the quality of) governance strictly as a product of institutional features and processes. This is obviously not the case of the model presented in this article (see the definition in the “Populating the Value Tree” section).

Including the dimension “Government effectiveness” can be subject to criticism. Still, the key tenet of the approach proposed here is that the resulting model should ultimately measure what the DMG wants it to measure. Therefore, if this group decides to look at the institutional features and outcomes of governance (and has recognized legitimacy to do so), then the developed MGI model should comply with this conceptual preference, taking into account the local context. Obviously, any individual that is not a member of this DMG can disagree with the definitions, contents, and preferences embedded in this model.

Despite the fact that all entities representing the Portuguese local administration refused to participate in the decision conferences—which represents a problem to this approach—the DMG included leading representatives from three crucial interest or stakeholder groups vis-à-vis the quality of local governance in Portugal (independent auditing institutions, central government agencies, and citizens). Therefore, although it does not include the preferences of the entities being assessed (which is not so rare in evaluation frameworks), the composition of this DMG should grant a considerable amount of legitimacy to the model developed here.

Scoring Functions

Several numerical (e.g., direct rating or the bisection method, von Winterfeldt and Edwards 1986) and nonnumerical (e.g., the Measuring Attractiveness by a Categorical Based Evaluation Technique—MACBETH; Bana e Costa and Oliveira 2012) methodologies have been used in the literature to construct scoring functions. These functions convert performance impacts into scores in an interval scale. However, given the absence of a univocal problem owner, the very time-consuming process of modeling nonlinear scoring functions could originate “decision fatigue” among the representatives that volunteered to be part of the DMG. Therefore, to develop a model that is feasible and able to estimate the governance level satisfactorily, linear scoring functions (or preference scales with equal distances between consecutive levels for qualitative descriptors) were assumed and validated by the DMG for all criteria.

Certainly, being a simplification, using linear scoring functions has its limitations. For instance, regarding the criterion B2 “Political strength of decisions,” the scoring function might be a concave down increasing curve (“too many” seats for the winning list in a given municipality may even be a sign of democratic deficit); however, because in Portugal, winning lists rarely are above the 60% share (seats are attributed using the D’Hondt method), using a linear relationship is not so problematic. Moreover, the DMG was aware of this and allowed to change it during decision conferencing (and, for example, to establish minimum and maximum scores, below 0.0 and above 100.0). It was concluded that in the few criteria where the nonlinearity could be more important (e.g., voter turnout), the actual performances were clustered in a small range between the established reference levels (in these few cases, the linearity was regarded as perfectly reasonable by the DMG).

Computing the Weights

After the validation of the MGI value tree (with 23 criteria), the DMG was asked to set the “Neutral” and the “Good” performance levels in each criterion (the selected levels for all criteria are presented in Table 2). As for modeling scoring functions, the literature provides many numerical techniques to compute weighting coefficients (e.g., swing weighting or the trade-off procedure; see Greco, Ehrgott, and Figueira 2010). Nevertheless, it would be counterproductive to ask the members of a nontechnical, heterogeneous DMG to express their preference judgments numerically (Bana e Costa, Carnero, and Oliveira 2012). We adopted the MACBETH approach to avoid this cognitive uneasiness (Bana e Costa and Oliveira 2012). With this technique, it is possible to determine the weights by asking the DMG to make pairwise comparisons through qualitative judgments of the differences in preference of certain reference profiles.

OPEN IN VIEWER

Table 2.

“Neutral” and “Good” Reference Levels and the Performances and Scores Attained by Lisbon.

Criteria	Reference Levels		Performances for Lisbon	Scores for Lisbon
	Good	Neutral
A1	Level II	Level IV	Level II	100.00
A2	5.0%	1.0%	3.5%	62.50
A3	Level II	Level III	Level V	−200.00
A4	Level II	Level III	Level IV	−100.00
A5	Level II	Level III	Level IV	−100.00
A6	5.9	4.7	5.5	66.67
B1	75%	50%	53%	12.00
B2	75%	50%	53%	12.00
B3	80%	60%	60%	0.00
B4	Level II	Level IV	Level III	50.00
C1	30%	60%	154%	−313.33
C2	80%	70%	64%	−60.00
C3	Level II	Level IV	Level V	−50.00
C4	Level II	Level IV	Level I	150.00
C5	Level II	Level IV	Level II	100.00
C6	€89 per capita	€55 per capita	€58 per capita	8.82
D1	1.1 contracts/supplier	1.5 contracts/supplier	1.92 contracts/supplier	−105.00
D2	Level II	Level IV	Level II	100.00
D3	€164 per capita	€285 per capita	€592 per capita	−253.72
E1	20 days	90 days	95 days	−7.14
E2	90%	66%	48%	−75.00
E3	99%	95%	99%	100.00
E4	3.0 complaints/104 inh.	6.0 complaints/104 inh.	6.4 complaints/104 inh.	−13.33

Note. inh. = inhabitants.

The procedure to compute the MGI weights was as follows. For each dimension (with n criteria), the DMG was asked to consider a set of n + 1 hypothetical municipalities, where n municipalities have a “Good” performance in one criterion and a “Neutral” performance in the remainder (each municipality has “Good” performance in a different criterion), and one municipality is “Neutral” all over (Figure 2 was shown to the DMG to explain this).

OPEN IN VIEWER

Figure 2.

Example presented to the decision-making group to explain the weighting protocol (MACBETH judgments between reference profiles).

The DMG was then asked to place the n municipalities in order of preference (evidently, the “Neutral” all over is the least preferred). After this assortment, the participants had to compare these municipalities in terms of preference by providing qualitative judgments using seven possible categories: “no,” “very weak,” “weak,” “moderate,” “strong,” “very strong,” or “extreme” difference. To assist us in this process, we used the M-MACBETH software, which allows the DMG to fill in a matrix of categorical judgments on the spot and then derives a compatible scale (if the judgments are consistent). Bana e Costa, Carnero, and Oliveira (2012) detailed the linear programming algorithm that determines the weights according to the qualitative judgments. To be able to compute all the weights in the two sessions, the DMG only had to elicit judgments between two consecutive reference profiles (corresponding to the first diagonal of the MACBETH matrix, as shown in Figure 3 for the “Rule of law and prevention of corruption” dimension).

OPEN IN VIEWER

Figure 3.

Matrix of judgments for the dimension “Rule of law and prevention of corruption” (using M-MACBETH software).

The steps described above had to be repeated for each of the five dimensions of the MGI (i.e., one matrix such as the one shown in Figure 3 for each dimension). This allowed to compute the (intra) weights of the criteria in each dimension of municipal governance. To obtain the overall governance score, the (inter or global) weights of the MGI also had to be calculated. To achieve this, the DMG compared one criterion from each dimension in a new matrix of judgments (see Figure 4, the criteria with higher weights in each dimension were arbitrarily chosen to carry out this comparison). With this final set of judgments, it is possible to normalize all weights through linear transformations. This hierarchical approach presented a clear advantage: Trying to compute the global weights at once (instead of one dimension at a time) would result in a 24 × 24 matrix, and ordering the hypothetical municipalities (i.e., the reference profiles) would have been very difficult for the DMG.

OPEN IN VIEWER

Figure 4.

Matrix of judgments for the hierarchical model (using M-MACBETH software).

The main results of the two decision conferences, that is, the global weights of the MGI model, are presented in Figure 5. As can easily be seen, a swing from “Neutral” to “Good” (or vice versa) in criterion “C1—Debt management” has the greatest impact in the overall score, followed by the criteria measuring the quality of essential services. Because the weight of each dimension of municipal governance is equal to the sum of the global weights of the criteria contained in it, the “Government effectiveness” dimension is the one with the highest weight (next to “Voice and accountability,” “Rule of law and prevention of corruption,” “Political stability,” and, finally, “Market access and regulation”). With these weighting coefficients, the reference values presented in Table 2, and the detailed explanation of the descriptors presented in the online appendix, one can assess the quality of governance of virtually any Portuguese municipality (all feeding data are publicly accessible).

OPEN IN VIEWER

Figure 5.

Weights of the MGI.