Design of a Basic System of Indicators for Monitoring and Evaluating Spanish Cooperation’s Culture and Development Strategy

Abstract

Objective:

This article describes the process implemented in order to define and build up a Basic Monitoring and Evaluation System for Spanish Cooperation’s Culture and Development Strategy (CD-S).

Research Design:

Delphi techniques were used to assess a wide catalogue of indicators for each of the strategic areas included in the CD-S. Afterward, those indicators considered most suitable for monitoring and evaluating purposes were selected based on the assessments provided by a panel of experts.

Results:

As a consequence, the resulting system, made up by a total of 80 indicators, was designed to be a practical, manageable, and targeted tool for its potential users, who include managers and technical officers who map out strategy together with stakeholders involved in its implementation, and specifically with staff at cultural centers abroad and in technical aid offices.

Keywords

basic system of indicators monitoring and evaluation system Delphi technique culture and development strategy

Introduction

Spanish Cooperation’s Master Plan¹ forms the framework within which the national vision of development aid is established, public policy in this area is stated, and the goals to be achieved through sectorial and geographical aid strategies are set. The Master Plan defines the mission of cultural aid and the role of stakeholders involved in achieving these goals and also examines the consistency of sectorial policies, harmonization between partner countries, and alignment with the public policy that has been agreed in development processes. Among other aspects, the Master Plan seeks to reconcile, unite, and harness the potential of the synergies of culture and aid to development goals. Implementing this public policy involves developing a strategy to structure the mechanisms for action in this area. As a result, the Culture and Development Strategy (CD-S) paper (Sectorial Strategy Paper—Culture and Development, SSP-CD; AECID 2007) was drawn up, a framework document for Spanish Cooperation’s cultural action that seeks to bring together the efforts made in the field of cultural aid for development. The CD-S is structured into seven strategic areas (Table 1), which in turn are divided into priority actions designed to achieve the goals set out in these areas (Appendix A).

Table 1.

Culture and Development Strategy: Strategic Areas and Their Goals

Strategic Area	Main Goal	Specific Goals
Training	Training human capital for cultural management with an emphasis on culture and development projects	Drive processes that develop human resources, enhancing people’s autonomy in managing the aspects of cultural life that have an impact on development
Politics	Political dimension of culture in its contribution to development	Shape the contributions that cultural policies can make to improve governance, promoting the institutional development of culture and encouraging citizen participation to help create active citizenship in cultural life
Economics	Economic dimension of culture in its contribution to development	Enhance culture’s contribution to economic development through the promotion, creation, and production of cultural and creative companies, industries, and institutions and driving cultural employment and links with other economic sectors
Education	Relationship and complementary nature of education and culture	Strengthen cultural structures and content in formal and nonformal education, seeking greater understanding and participation by students in contemporary cultural and artistic phenomena
Heritage	Sustainable management of cultural heritage for development	Support and work with the numerous existing processes in conserving, restoring, preserving, and enhancing tangible and intangible heritage, identifying the contribution that they can make towards sustainable development
Communication	Relations between communication and culture with an impact on development	Sponsor communication processes in communities to help toward the integration of the sectors that make up the community. Contribute to the development of audiences, generate new cultural industries, and encourage coproduction
Cultural rights	Promoting the recognition of cultural rights	Enhance awareness in society about issues related to cultural diversity and the development of human rights in the field of culture, supporting studies into the different aspects of cultural legislation

The Agencia Española de Cooperación Internacional para el Desarrollo (AECID—Spanish Agency for International Cooperation) is aware of the importance of measuring the impact of cultural activities on human development and hence of the need for evaluation as a key factor in the cultural policy cycle. Consequently, in 2007, its Dirección de Relaciones Culturales y Científicas (DRCC—Directorate of Cultural and Scientific Relations) sponsored the creation of indicators to assess the implementation of its CD-S. This initial process resulted in the publication of the book Cómo evaluar proyectos de cultura para el desarrollo (How to evaluate cultural projects for development; AECID 2009). Over the following 2 years (2009–2010), the AECID continued to drive the design and implementation of tools for analyzing, measuring, and evaluating culture and development policy.

In this article, we show the process developed (Figure 1) in order to obtain a Basic System of indicators for monitoring and evaluating Spanish Cooperation’s CD-S (SSP-CD). This system is finally made up of a total of 80 indicators, and it covers all the strategic areas and priority actions of the SSP-CD.

Figure 1.

Research process.

Implementation of the Delphi Technique

The Delphi method is closely tied to the RAND Corporation.² In 1953, Dalkey and Helmer began their research using iterative questioning of a panel of experts, a technique which subsequently came to be called Delphi (Dalkey and Helmer 1963). This method was developed in the early 1960s at a time when quantitative modeling using computers was still a long way off. In 1964, Olaf Helmer and Theodore J. Gordon published a study based on questioning a panel of 82 experts (Gordon and Helmer 1964) about future scientific and technological discoveries beyond the 20th century. A little later, in June 1969, Norman C. Dalkey (1969) published The Delphi Method: An Experimental Study of Group Opinion. Linston and Turoff (1975) defined the Delphi technique “as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem.” The main features of the Delphi method include (Dalkey 1969; Linston and Turoff 1975; Landeta 2002; Loo 2002):

Anonymity of the experts. No member of the panel should know who the other members are. This avoids any bias arising from contact between them (Landeta 2002) by eliminating the negative influences on individual responses due to the personality (Kresten et al. 2007) and/or reputation of the rest of the experts.

Iteration and controlled feedback. Iteration is achieved by submitting the same questionnaire in successive rounds. In addition, at the beginning of each round, the experts are given a summary of the results of the previous round including any significant comments made by the participants. In the light of this information, the experts can then maintain or change their opinions.

Statistical response of the panel. The opinion of each of the experts should be reflected in the final answer and not just the view of the majority.

As noted by Powell (2003, 376), based on Linston and Turoff (1975), “the Delphi is in essence a series of sequential questionnaires or ‘rounds’, interspersed by controlled feedback, that seek to gain the most reliable consensus of opinion of a group of experts.”

In our research, the Delphi process was structured as usual into three stages: initial, exploratory, and final. In the initial stage, the Delphi coordination and supervision team was set up, the context and purpose of the research were clearly defined, the experts were selected, and the catalogue of indicators to be evaluated was drawn up. The exploratory stage involved designing the questionnaires to be used in the successive rounds, monitoring their completion by the experts and analysis of the results obtained in each round. The final stage consisted of the statistical processing and analysis of the information gathered and drawing up a final report.

Of course, the Delphi process was carried out not without difficulties. The main drawback that we had to face was directly related to the catalogue of indicators to be evaluated. First, we were aware of the fact that the number of indicators to be evaluated by each expert was high (especially if we take into account that an indicator is evaluated on three different criteria). This could increase the drop-out rate. Second, we had to use a tool that (i) allows us to efficiently manage the high amount of data generated by the experts as a result of the evaluations and (ii) facilitates us the subsequent treatment of data. Related to the latter point, and after considering different possibilities, we made the decision to design online questionnaires for gathering the information using LimeSurvey, an open source survey application. At the same time, this software provided the tools needed to do an individualized tracking of the filling in of the questionnaires. Thus, when inactivity was detected we sent either an e-mail to all experts inviting to participate, a customized e-mail reminding the aim of the project, or even we made several phone calls to know the reasons for nonparticipation (which appeared to be mainly the nonavailability of time due to overload, and the extension of the questionnaire required to pay a lot of attention and much time to be answered). With these actions, we achieved a high response rate.

Some of the key aspects of the Delphi process that was used are shown below.

Catalogue of Cultural Indicators

The AECID is aware of the need to evaluate public policy on development aid (Paris Declaration 2005; Accra Agenda for Action 2008), and so in 2007 it began discussing and mapping out evaluation indicators which could be used by officers tasked with implementing the strategy and also by stakeholders and cultural cooperation project managers looking for reference points for the design, monitoring, and evaluation of their schemes. The outcome was the design of a working method for the creation of evaluation indicators and the definition of a total of 312 cultural indicators for evaluating the cultural and development activities implemented as part of the CD-S (AECID 2007). This first list of indicators was supplemented by (i) indicators from statistics published by the Statistics Offices and Ministries of Culture in Spanish aid partner countries and (ii) indicators compiled from relevant variables contained in the DGCC’s activity report. The final catalogue of cultural indicators assessed by the experts consisted of a total of 508 indicators. Appendix B shows the distribution of indicators by strategic area, and within the latter by priority action.

Panel of Experts

Without doubt one of the keys to success in studies using the Delphi method is the appropriate selection of experts or evaluators (Gordon 1994; Powell 2003). The usual process is to identify people who have the knowledge required about the study topic by means of an exhaustive search of the literature in the field and to seek out potential users who as stakeholders are fully familiar with the situation being analyzed and/or through recommendations made by the institutions involved in the project. Since the ultimate goal of the research was to determine a set of cultural indicators that would enable the CD-S to be monitored and evaluated, we felt that the Delphi panel should be made up of experts from groups involved in programs to be implemented by Centros Culturales de España en el Exterior (CCE—Spanish Cultural Centers Abroad), that is to say managers and management and technical specialists at the Centers, who are potential end users of the final product (Linstone and Turoff 1975; Jones and Hunter 1995). Experts on the CD-S not connected with the institutions involved were also contacted. Thus, bearing in mind (i) the distribution of Cultural Centers abroad, (ii) their specialization in terms of programs and actions they have planned, and (iii) the diversity of evaluators according to their technical grade, origin, and the specific features of the group they represented, the expert panel was made up of project managers, CEE managers and officers, DRCC cultural managers and officers, officers from the Dirección General de Planificación y Evaluación de Políticas de Desarrollo (DGPOLDE—Directorate General of Development Policy Planning and Evaluation), experts taking part in the first stage of the project, independent consultants, and universities.

The Delphi panel consisted of a total of 41 experts and each one was given two to three strategic areas to evaluate. While there is no consensus about the optimal number of experts for a Delphi study, the theoretical minimum³ number is roughly around seven (Dalkey, Brown, and Cochran 1969; Wilhelm 2001; Landeta 2002), although adding more experts significantly reduces the error rate (Dalkey, Brown, and Cochran 1969). As can be seen in Table 2, the number of experts assigned to the various strategic areas of the CD-S is higher than this theoretical minimum. We selected a sufficient number of experts at the start of the Delphi to ensure a qualified panel at the end of the study (Hallowell and Gambatese 2010), as it was predictable that some participants drop out during the process. In relative terms, the number of experts involved in the Delphi process for each strategic area might be seen as high, given that in absolute terms the number of experts associated with the study area was small.

Table 2.

Distribution of Experts by Strategic Area

Strategic Area	Number of Experts
Training	18
Politics	17
Economics	17
Education	19
Heritage	16
Communication	17
Cultural rights	12

Delphi Process: Rounds

Once the catalogue of indicators to be evaluated had been specified and the group of experts selected, data were gathered by filling in some online questionnaires with a Likert-type scale of 7 points, drawn up using LimeSurvey. There were a number of reasons for choosing this software, including (i) appropriateness for electronic evaluation, (ii) ease of online access, (iii) practical and user-friendly interface for the experts, and (iv) its capacity for information storage and retrieval between rounds.

There was one questionnaire per strategic area and this enabled each cultural indicator to be assessed using three criteria (Figure 2):

Figure 2.

Questionnaire (Round 2). Evaluation of an indicator.

Relevance: the degree of importance and usefulness of the indicator when potentially measuring the phenomenon being observed, enabling decision making by the user.

Specificity: to what extent change in the indicator is actually the result of changes in the phenomenon to be monitored and how the effect of exogenous factors on this change is minimized.

Feasibility: the ability to set a desired value for the indicator, in other words to what extent a quantifiable target can be assigned to the indicator being evaluated.

In the field of development aid, and specifically in the area of Monitoring and evaluation systems, the acronym SMART (Specific, Measurable, Achievable, Relevant, Time-Bound) has been coined to describe “good” indicators (Drucker 1954; Smith 1999; Jones 2007). The three criteria used for evaluating each indicator are based on SMART and, in the evaluation process, the experts were asked to calculate the relative weight of them (Yong and Wenhao 2011).

During the first half of December 2009, a pretest was run to check that the website to be used to manage the information was working properly and that the questionnaires met the requirements of the study. Two rounds were used, as the bulk of the changes occur in the transition from the first to the second round—van Zolingen and Klaasen (2003) cited by Hanafin et al. (2005)—and because evaluating each one of a set of indicators based on three criteria is a tedious task. Hence, it was held that conducting a larger number of rounds would lead to the experts losing interest and/or dropping out of the project (Loo 2002).

The first round of evaluations began in mid-January 2010 and lasted 3 weeks. The number of questionnaires that had been correctly filled in at the end of the first round was 82, which corresponds to an overall response rate of 70.69%.

In the second round, the questionnaires, which were customized for each expert, had to (i) keep the evaluation given in the previous round, (ii) show the main statistical measures (median and interquartile range)⁴ of the overall results, and (iii) allow the expert to maintain or modify their initial assessment by comparing it with statistical information about the response of all the experts. Only the experts who had completed the evaluation of indicators in the previous round could take part in this one.

Gathering information from the second round lasted from the second half of February to March 2010. In total, 74 questionnaires were filled in correctly, giving a response rate in the second round of 86% and of 63.79% with respect to the total. Table 3 shows the distribution by strategic area. The politics and cultural rights areas were completed in their entirety while economics and education had a lower, though still nonetheless high, response rate.

Table 3.

Response Rate in the Second Round and With Respect to the Total

Strategic area	Questionnaires Filled In	Response Rate (Second Round; %)	Total Response Rate (%)
Training	11	90	61.11
Politics	12	100	70.59
Economics	9	75	52.94
Education	10	77	52.63
Heritage	8	80	50
Communication	13	81	76.47
Cultural rights	11	100	91.67
Total	74	86	63.79

Effectiveness of the Method

According to Landeta (2002), the key element in the completion of the Delphi process is to achieve a consensus between the experts’ opinions. However, paradoxically there is no consensus on how this consensus should be understood and interpreted. In a review of selected Delphi studies, Powell (2003) concluded that consensus may be defined or achieved in various ways (Weatherman and Swenson 1974; McCallister 1992; Beech 1997; Gibson 1998). Landeta (2002:90) defined consensus as “the degree of convergence of individual estimates, which is reached when the opinions have an acceptable degree of proximity (small interquartile range).” He argues that the existence of a real risk of loss of information and building fictitious consensuses means that a criterion other than consensus should be used to end the process, such as the stability of responses between rounds (Duffield 1993; Landeta 2002).

In general, consensus/stability is determined using an empirical approach by obtaining measures of dispersion in responses between rounds (Hanafin et al. 2007). In this study, the relative interquartile range⁵ (RIR) was used as a measure of dispersion of the responses of the experts to determine the stability of the estimates. High RIR values indicate a weak consensus among experts. The use of this stability criterion should be qualified, given the nature of the research. In each of the seven strategic areas, a set of indicators (i = 1, . . . s) based on three criteria (c = 1, 2, 3)⁶ for each expert (e = 1, . . .E) were formally evaluated, yielding scores that can be denoted by $X_{i, e}^{c (1)}$ , where the superscript (1) refers to the first round. Similarly, $X_{i, e}^{c (2)}$ scores were obtained in the second round. Thus, at the indicator criterion level (c = 1, 2, 3), the second round will be graded as effective if ${R I R}_{i}^{c (1)} > {R I R}_{i}^{c (2)}$ and neutral if ${R I R}_{i}^{c (1)} = {R I R}_{i}^{c (2)}$ .

For example, Table 4 shows descriptive statistics of the scores given by the experts in each of the three criteria (Relevance, Specificity, and Feasibility) for the indicator “Annual rate of change in the number of countries, partners in priority action 1 in strategic area 7, whose authorities officially attend international forums about cultural rights.”

Table 4.

Calculating the Effectiveness of an Indicator

	First Round						Second Round
Criterion	Max	Min	Q1	Q3	Median	RIR⁽¹⁾	Max	Min	Q1	Q3	Median	RIR⁽²⁾
Relevance	7.0	3.0	4.0	7.0	5.0	0.6	7.0	3.0	4.0	6.5	5.0	0.5
Specificity	7.0	3.0	4.0	6.0	4.0	0.5	7.0	3.0	4.0	6.0	4.0	0.5
Feasibility	7.0	3.0	5.0	7.0	5.0	0.4	7.0	3.0	5.0	6.5	5.0	0.3

Note. RIR = relative interquartile range.

¹First Round

²Second Round

Thus, for this indicator, RIR⁽¹⁾ is higher than RIR⁽²⁾ for the criteria Relevance and Feasibility (which implies the second round of the Delphi was effective), and RIR⁽¹⁾ is equal to RIR⁽²⁾for the criterion Specificity (in this case, the second round was neutral).

Table 5 provides a summary of the results for the effectiveness of the second round of the Delphi method by criterion evaluated.

Table 5.

Effectiveness of the Second Delphi Round by Criterion

Criterion	Effective	Neutral	Not Effective
Relevance	244 (48%)	205 (40%)	61 (12%)
Specificity	277 (54%)	178 (35%)	55 (11%)
Feasibility	279 (55%)	156 (31%)	75 (15%)

Some points should be highlighted from the results shown in Table 5. The greatest consensus was observed when assessing relevance and specificity. These findings are consistent with the fact that the experts are for the most part users of the data, and these aspects correspond precisely to the usefulness of an indicator for its users. There is slightly less consensus on the feasibility assessment, which can be explained by the fact that the experts are users of the indicators but their tasks do not include obtaining them, that is, their production. In general, the percentage of criteria that were not effective is low.

At the indicator level (i = 1, 2, . . .s), the second round was considered to have been effective if at least two of the criteria on which it was evaluated are effective or neutral. Table 6 shows the effectiveness results for the assessment of the indicators by strategic area.

Table 6.

Indicators: Effectiveness of the Second Delphi Round by Strategic Area

Strategic Area	Effective and Neutral (%)	Not Effective (%)
Training	83.67	16.33
Politics	98.46	1.54
Economics	91.07	8.93
Education	70.24	29.76
Heritage	77.36	22.64
Communication	98.20	1.80
Cultural rights	86.11	13.89

From this standpoint, and in the following order, the politics, communication, and economics areas had the highest percentages of effective indicators. The education and heritage areas were at the other end of the scale. However, the effectiveness percentages of the indicators obtained were high, ranging from 98.46% (politics) to 70.24% (education).

Consequently, the results obtained by the Delphi method are consistent and therefore it is possible to accept the effectiveness of the second round as a method for obtaining a consensus among the experts concerning their opinions (stability of responses) about the adequacy of the indicators evaluated in terms of relevance, specificity, and feasibility.

The next section seeks to set out reasonable operational criteria for selecting those indicators deemed to be best for inclusion in the basic system of monitoring and evaluation indicators.

Selection of Indicators

The process of selecting indicators should seek to use the information provided by the panel of experts in the Delphi procedure as efficiently as possible. In addition, it must pay special attention to the particular structure of the data set generated. Data from the Delphi method define a set of trivariate distributions for each indicator in the catalogue. The variables that make up the trivariate distribution for each indicator refer to the scores obtained by the indicator for the three criteria (relevance, specificity, and feasibility) that have been evaluated. Furthermore, the structure of the data obtained through the Delphi method is special because the trivariate distributions are grouped or nested into priority actions and these in turn into strategic areas (see Appendix B).

The methodological approach to analyzing the Delphi results for the selection of indicators that are to be included in the Basic System was based on the type of indicators and the definition of a Composite Relative Operability Index (ROI).

Type of Indicators

The methodology for developing an indicator system is based on two related aspects; a Logical Framework Approach (LFA) and the CIPP (Context, Input, Process, Product) evaluation model (Figure 3).

Figure 3.

Interrelation between the logical framework approach (LFA) and Context, Input, Process, Product (CIPP) models and the Indicator System. Adapted from Carrasco (2006).

Used by the AECID, LFA dates from the late 1960s as an attempt to order the main contents of project design and was developed by the United States Agency for International Development. In the early 1980s, the German Technical Development Agency (GTZ) provided a methodological structure for the technique under the ZOPP concept, a German acronym for “Goal Oriented Project Planning.” Finally, in 1993, the European Community adopted it under the name LFA as the standard methodology for development aid projects (European Commission 2004).

The LFA (Sartorius 1991; Gasper 2000; Bakewell and Garbutt 2005) is a tool that makes it easier for an evaluator to examine the performance of a program during all its stages. It means that a program’s goals and their causal relationships can be systematically and logically presented (Camacho et al. 2001). It also assesses whether these goals have been attained and maps out factors external to the program that may influence their achievement. It is the main programming tool used in planning by objectives and has been progressively tailored to the management of aid projects. The logical framework matrix, which is made to monitor and evaluate projects, should reflect what the plan, program, or project (PPP) is seeking to achieve.

The CIPP evaluation model (Stufflebeam 1983; Worthen and Sanders 1987; Eseryel 2002) is drawn from education analysis. In this model, evaluation is viewed as a process (King 2007) designed to define, obtain, and provide information which is useful for evaluating, judging, and weighing up alternatives, that is to say, it is a process that generates viable and therefore useful information that will enable us to take the appropriate decisions for our purpose from the diagnosis and planning stage (project) through structuring the implementation of processes and actions and up to the final results stage.

As shown in Figure 3, the system of indicators, which in turn can be subdivided into a Basic System and a Strategic System, is structured by the interrelationship between the logical framework and CIPP models (Bonefoy and Armijo 2005; citation deleted for blind review). Specifically, the Basic System is essential for short-term planning; it makes it easier to manage, make decisions, and set a baseline that will make it possible to establish the medium- and long-term goals of the strategies. The strategic system is specifically orientated to evaluate the effects (medium-term results) and the impacts (long-term results) of the Spanish Cooperation strategic planning (See Figure 3).

In this article, we have focused on the design of the Basic System which consists of simple, operational indicators that will specify the activities, beneficiaries, and outcomes of cultural activity based on the planning carried out. In general, the Basic System involves short-term indicators for resources (or inputs), processes, and products, defined by Bonefoy and Armijo (2005:27) and AECID (2007:49) as:

Resource indicators: show the nature and number of the factors that are directly or indirectly used to carry out an intervention in order to achieve the purpose specified in the project. Example: “Budget allocated to actions to support the creation of instruments to enhance capacity-building and employment related to the rehabilitation and revaluation of heritage” (Strategic area 5: Sustainable management of Cultural Heritage at the service of development).

Process indicators: measure the performance of actions, processes, and activities scheduled using available resources. Example: “Number of specific training programs which use the capabilities of the local cultural agents” (Strategic area 1: Training of human capital for cultural management).

Product indicators: provide information about the quantity and quality of the product, goods, or services generated by scheduled activities, processes, and actions. It is the short-term result, the direct purpose of the planning process. Example: “Number of beneficiaries of actions to promote access to information technology in education, reading, and cultural services” (Strategic area 4: Relationship and complementarity between Education and Culture).

Definition of a Composite ROI

Given the nested structure of the data referred to above, it seems reasonable to define and build a composite operability index from the relevance, effectiveness, and feasibility scores awarded by the experts to the indicators in the catalogue, which are resource, process, and product indicators. The indicators that demonstrate a better overall performance in each priority action by achieving a higher ROI will be selected for the basic system of monitoring and evaluation indicators.

Weighting coefficients have to be assigned to the three variables in order to draw up a composite ROI for the indicators for the various priority actions by strategic area. This will provide a single measure of relative operability per indicator, which in turn will speed up the selection process and ensure that the composite index obtained contains information about the three criteria evaluated by the experts. Thus, the general formula of the ROI(I_j ) can be written as follows for any system indicator, denoted I_j ,:

R O I (I_{j}) = (w_{R} R_{j} + w_{S} S_{j} + w_{F} F_{j}),

where R_j , S_j , and F_j represent the median scores for relevance, specificity, and feasibility obtained by the indicator j. w_R, w_S , and w_F denote the weighting or weights of each criterion in the index (Grandzol and Gershon 1994; Tague 2004).

The w_R, w_S , and w_F weightings could in principle be determined using multivariate analysis techniques for dimension reduction based on the panel of experts’ assessments. However, the particular structure of the data obtained from the Delphi process suggested that this procedure would not be entirely satisfactory for the purpose of selecting indicators for the Basic System. The main reason for the inadequacy of multivariate techniques in this case is that, by definition, the extraction of common factors in the variability of the scores for the criteria would lead to the calculation of not very intuitive interpretation constructs of questionable practical relevance in the selection of indicators for the Basic System. Furthermore, obtaining the principal components to condense the bulk of the variability in the scores for the three criteria had to be done on at least two levels, that is, it had to be replicated for each priority action and for each strategic area, a procedure that has a high probability of generating inconsistent results. In any case, the determination of weightings using multivariate techniques for dimension reduction would mean that the weightings assigned to the three criteria would be different for the indicators for the priority actions and strategic areas, thus hindering a comparison of their respective ROIs.

Consequently, as an alternative to multivariate analysis, it was decided to set fixed weightings for each criterion based on the modal value given by the panel of experts. Thus, the weightings used in the ROI for each indicator were initially preset to the following values: w_R = 0.5, w_S = 0.25, and w_F = 0.25. Since the goal is to select indicators for inclusion in an information system that is both operational and useful, we think it seems reasonable to give more relative weight to the relevance criterion of the indicator since it expresses the degree to which the indicator meets the needs of stakeholders who make decisions. In any case, the system is flexible and its administrator (AECID staff) can adjust these weightings, so that the ROIs are automatically reevaluated. The ROIs for the cultural rights strategic area indicators are set out in Appendix C.⁷

Results: A Basic Monitoring and Evaluation System

In order for the selection of indicators to be representative of the numerous activities included in cultural and development policies, it seems appropriate to impose the restriction that all priority actions in the various strategic areas should have indicators in this selection, thus ensuring that they are represented in the basic system of monitoring and evaluation indicators. Consequently, the selection procedure looks at the indicators grouped by the priority action to which they refer. Given these premises, it would appear advisable to select indicators based not only on the ROI value but also on the feasibility scores assigned by the experts. Although the three criteria evaluated are important because they define an indicator’s overall operability, feasibility, defined as the ability to specify objective numerical values for the indicator, it appears to be a logical prerequisite for the inclusion of a specific indicator in the basic system of monitoring and evaluation indicators. If, for instance, an indicator is defined as highly relevant and specific but the experts think it is not very feasible, it seems inappropriate to select it as there are misgivings about the possibility of actually observing or calculating its numerical values, and selecting it would have been pointless. On the strength of this reasoning, the indicators were chosen by setting a different minimum or threshold feasibility value per priority action that specifies which indicators in the opinion of the experts are the most feasible for each priority action. If the experts have overall rated an indicator as not very feasible, this would suggest they have their doubts about whether it would be possible to assign objective numerical values to it, which in turn would mean that this indicator would be ineligible to be part of the basic system. The feasibility threshold, which is specific to each priority action, is defined so that at least two indicators for each priority action can be retained. Hence, the indicators retained⁸ were scaled in descending order as a function of their ROI value on the condition that they reached the feasibility threshold that was preset for the priority action concerned. As an example, Table 7 shows the feasibility thresholds used to retain cultural rights indicators.

Table 7.

Feasibility Thresholds in the Cultural Rights Strategic Area

Priority Action	Average Feasibility		Feasibility Threshold	Number of indicators
Priority Action	Maximum	Minimum	Feasibility Threshold	Number of indicators
PA-1	5.45	3.90	5.2	2
PA-2	5.45	3.90	5.2	2
PA-3	5.27	3.90	5.1	2
PA-4	5.09	4.27	4.9	2

This evaluation procedure, under which the two indicators with the highest ROI for each priority action are selected from among those which score above the feasibility threshold calculated for the priority action concerned, ended with the final selection of 80 indicators (Table 8).

Table 8.

Distribution of Cultural Indicators: Original Catalogue versus Basic System

Strategic Area	Indicators in the Original Catalogue	Indicators in the Basic System
Training	49	12
Politics	65	11
Economics	112	12
Education	84	13
Heritage	52	12
Communication	110	12
Cultural rights	36	8
Total	508	80

Table 9 lists the cultural rights strategic area indicators, together with their respective ROIs, selected to form part of the basic monitoring and evaluation system.

Table 9.

Basic System of Indicators: Cultural Rights Indicators

Priority Action	Code	Indicator	ROI
PA-1	B73	Annual rate of change in the number of studies about cultural legislation that promotes respect for cultural rights	5.295
PA-1	B74	Annual rate of change in the number of partner countries whose authorities officially attend international forums about cultural rights	5.114
PA-2	B75	Annual rate of change in the number of third sector organizations involved in the planning, management, and dissemination of culture as a means to promote sustainable development	5.682
PA-2	B76	Annual rate of change in the number of towns with fewer than 50,000 inhabitants that are beneficiaries of cultural facility plans in partner countries	5.250
PA-3	B77	Annual rate of change in the number of international trade agreements signed by the Spanish authorities that respect the cultural diversity convention	4.995
PA-3	B78	Annual rate of change in the number of trade agreements and treaties signed by the authorities of the countries in which action has been taken which explicitly include the cultural diversity convention	5.341
PA-4	B79	Annual rate of change in the number of actions to guarantee equal opportunities in participation in cultural life and ensure freedom of expression and communication, cultural activities, and creative expression	5.091
PA-4	B80	Annual rate of change related to the presence in Cultural Centers of unconventional forms and expressions of everyday culture connected with minority cultures and groups traditionally excluded from society	5.205

So far, the DGPOLDE is in charge of the evaluation of programs funded by the Spanish Cooperation.⁹ This evaluation is aimed to determine the levels of effectiveness, efficiency, relevance, and viability achieved by the programs as well as the estimation of the impacts of its implementation (OECD 2010). To carry out this evaluation, the documentation of the program is analyzed and interviews in depth to a sample of representative stakeholders (staff, beneficiaries, etc.) are made. The DGPOLDE has overseen the execution of 30 evaluations,¹⁰ which have been focused on the analysis of specific actions at the project level.

However, the basic system of indicators aims to measure, in each of the 16 Spanish Cultural Centers Abroad, the performance of the Spanish Cultural Cooperation in the seven strategic lines of the CD-S. To make operative the Basic System, data necessary to calculate the 80 indicators have to be gathered. Then, the system will support both planning and monitoring of Spanish Cultural Action Abroad. Obviously, to the extent that the system is maintained over time, accuracy and quality of information obtained will be higher.

Summary

Spanish Cooperation Planning involves first drawing up a Master Plan as the common core for sectorial priorities which are then developed through specific strategies set out in SSP. These include the CD-S (SSP-CD).

The CD-S is built on a series of strategic areas whose main purpose is to drive social and economic development in countries to which Spain provides aid through cultural mechanisms. Each strategic area includes a number of priority actions that (i) lead to the performance of a series of activities, (ii) require a level of funding which in turn makes it possible to mobilize human and capital resources involved in the action, and (iii) are required in order to carry out activities designed to make tangible improvements in terms of greater social and economic development.

In this article, we have described the main actions taken to obtain a basic monitoring and evaluation system for Spanish Cooperation’s CD-S (SSP-CD). First, it has been described how the Delphi technique was used to assess a catalogue of indicators for each of the strategic areas and how the indicators considered most suitable to constitute the Basic System were selected based on the assessments provided by a panel of experts. Second, given the nested structure of the SSP-CD (strategic areas and priority actions) and taking into account the interrelation between the LFA and the CIPP model, the Basic System was configured according to (i) the type of indicators and (ii) the definition of a composite ROI.

The final result achieved is a system made up of a total of 80 indicators, which covers all the strategic areas and priority actions in the SSP-CD. The system has been designed to be a practical, manageable, and targeted tool for its potential users, that is, among others managers and technical officers who map out strategy, together with stakeholders involved in its implementation, and specifically staff at Cultural Centers Abroad and in Technical Aid Offices.

Finally, and prior to the implementation of the Basic System, it is necessary to conduct a test to check that the data to calculate the selected indicators can be easily gathered at the Spanish Cultural Centers Abroad. This is the next challenge ahead of AECID and us to obtain an effective system.

Footnotes

Appendix A

Appendix B

Table B1.

Distribution of Indicators by Strategic Area and Priority Action

Strategic area	Initial	Indicators by Priority Action (PA)		Total Indicators
Training	46	PA-1	8	49
		PA-2	6
		PA-3	10
		PA-4	7
		PA-5	11
		PA-6	7
Politics	40	PA-1	13	65
		PA-2	10
		PA-3	8
		PA-4	8
		PA-5	26
Economics	48	PA-1	22	112
		PA-2	8
		PA-3	18
		PA-4	11
		PA-5	29
		PA-6	24
Education	48	PA-1	24	84
		PA-2	15
		PA-3	8
		PA-4	8
		PA-5	7
		PA-6	22
Heritage	45	PA-1	16	52
		PA-2	7
		PA-3	7
		PA-4	7
		PA-5	7
		PA-6	8
Communication	49	PA-1	28	110
		PA-2	10
		PA-3	8
		PA-4	21
		PA-5	17
		PA-6	26
Cultural rights	36	PA-1	10	36
		PA-2	8
		PA-3	9
		PA-4	9
Total	312			508

Appendix C

Table C1.

Relative Operability Index (ROI) of the Indicators for Priority Action 1 (Cultural Rights Strategic Area)

Operability Index	Indicator
4.068	Annual rate of change in the number of stakeholders involved in efforts to promote the joint participation of the international community based on cultural rights to regulate and harmonize community life and opportunities for dialogue
4.250	Annual rate of change in the budget allocated to actions to promote the joint participation of the international community based on cultural rights to regulate and harmonize community life and opportunities for dialogue
4.841	Annual rate of change in the number of actions to promote the joint participation of the international community based on cultural rights to regulate and harmonize community life and opportunities for dialogue, carried out in one year compared to the previous year by country
4.568	Annual rate of change in the number of beneficiaries of actions to promote the joint participation of the international community based on cultural rights to regulate and harmonize community life and opportunities for dialogue
5.205	Annual rate of change in the number of international meetings attended by countries where activities have been carried out under priority action 1 in strategic area 7 which reflect the principles of cultural rights
5.136	Annual rate of change in the number of bilateral or multilateral projects run in countries where activities have been carried out under priority action 1 in strategic area 7 concerning the defense of cultural rights
5.364	Annual rate of change in Spain’s financial contribution to international funds for the defense of cultural diversity
5.295	Annual rate of change in the number of studies about cultural legislation that promote respect for cultural rights (at national, regional and international level) sponsored by supranational or multilateral organizations
5.114	Annual rate of change in the number of countries, partners in priority action 1 in strategic area 7, whose authorities officially attend international forums about cultural rights
4.500	Annual rate of change in the number of cities in partner countries for priority action 1 in strategic area 7 which have hosted international forums about cultural rights

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Agencia Española de Cooperación Internacional para el Desarrollo, AECID (project 08-CAP-0795).

Notes

References

AECID (Agencia Española de Cooperación Internacional para el Desarrollo). 2007. Spanish Strategy for Culture and Development Cooperation. Madrid, Spain: AECID.

AECID (Agencia Española de Cooperación Internacional para el Desarrollo). 2009. Cómo Evaluar Proyectos de Cultura Para el Desarrollo: Una Aproximación Metodológica a la Construcción De Indicadores. Madrid, Spain: AECID.

Bakewell

Garbutt

. 2005. The Use and Abuse of the Logical Framework Approach. Accessed February 14, 2012, www.intrac.org/data/files/resources/518/The-Use-and-Abuse-of-the-Logical-Framework-Approach.pdf.

Beech

1997. “Studying the Future: A Delphi Study of How Multidisciplinary Clinical Staff View the Likely Development of Two Community Mental Health Centres over the Course of the Next 2 years.” Journal of Advanced Nursing 25:331–38.

Bonefoy

J. C.

Armijo

. 2005. Indicadores de Desempeño En El Sector Público. Series Manuales, nº 45. Santiago de Chile: Instituto Latinoamericano y del Caribe de Planificación Económica y Social.

Camacho

Cámara

Cascante

Sainz

. 2001. El enfoque del Marco Lógico: 10 Casos Prácticos. Cuaderno Para la Identificación y Diseño de Proyectos de Desarrollo. Madrid, Spain: Fundación CIDEAL y Acciones de Desarrollo y Cooperación.

Carrasco

. 2006. “Medir la Cultura: Una tarea inacabada.” Periférica 7:140–168.

Dalkey

1969. The Delphi Method: An Experimental Study of Group Opinion. Research Memorandum, RM-5888-PR. Santa Monica, CA: The Rand Corporation.

Dalkey

Brown

Cochan

. 1969. The Delphi Method III: Use of Self Rating to Improve Group Estimates. Research Memorandum, RM-6115-PR. Santa Monica, CA: The Rand Corporation.

10.

Dalkey

N. C.

Helmer

. 1963. “An Experimental Application of the Delphi Method to the Use of Experts.” Management Science 9:458–67.

11.

Drucker

1954. The Practice of Management. New York: Harper and Row.

12.

Duffield

. 1993. “The Delphi Technique: a comparison of results obtained using two expert panels.” International Journal of Nursing Studies 30:227–237.

13.

Eseryel

2002. “Approaches to Evaluation of Training: Theory & Practice.” Journal of Educational Technology and Society 5:93–98.

14.

European Commission. 2004. Aid Delivery Methods. Vol. 1. Project Cycle Management Guidelines. Brussels, Belgium: EuropeAid Cooperation Office.

15.

Gasper

2000. “Evaluating the ‘Logical Framework Approach' towards Learning-oriented Development Evaluation.” Public Administration and Development 20:17–28.

16.

Gibson

J. M. E.

1998. “Using the Delphi Technique to Identify the Content and Context of Nurses’ Continuing Professional Development Needs.” Journal of Clinical Nursing 7:451–59.

17.

Gordon

T. J.

1994. The Delphi Method. AC/UNU Millennium Project. Accessed February 14, 2012, http://www.gerenciamento.ufba.br/Downloads/delphi%20(1).pdf.

18.

Gordon

T. J.

Helmer

. 1964. Report on a Long Range Forecasting Study. Research Memorandum, RM-2982. Santa Monica, CA: The Rand Corporation.

19.

Grandzol

Gershon

. 1994. “Multiple Criteria Decision Making.” Quality Progress 27:69–73.

20.

Hallowell

M. R.

Gambatese

J. A.

. 2010. “Qualitative Research: Application of the Delphi Method to CEM Research.” Journal of Construction Engineering and Management 136:99–107.

21.

Hanafin

Brooks

A.-M.

Carroll

Fitzgerald

Gabhainn

S. N.

Sixsmith

. 2007. “Achieving Consensus in Developing a National Set of Child Well-being Indicators.” Social Indicators Research 80:79–104.

22.

High Level Forum on Aid Effectiveness. 2008. Accra Agenda for Action. Accessed February 14, 2012, http://siteresources.worldbank.org/ACCRAEXT/Resources/4700790-1217425866038/AAA-4-SEPTEMBER-FINAL-16h00.pdf.

23.

Jones

Hunter

. 1995. “Consensus Methods for Medical and Health Services Research.” Journal of British Medical 311:379–80.

24.

Jones

2007. Project Management Survival: A Practical Guide to Leading, Managing & Delivering Challenging Projects. London, UK: Kogan Page.

25.

King

J. A.

2007. “Developing Evaluation Capacity through Process Use.” New Directions for Evaluation 116:45–59.

26.

Kresten

Green

Armstrong

Graefe

. 2007. “Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared.” Foresight: The International Journal of Applied Forecasting 8:17–20.

27.

Landeta

2002. El Método Delphi: Una Técnica De Previsión Del Futuro. Barcelona, Spain: Ariel.

28.

Linston

Turoff

. 1975. The Delphi Method. Techniques and Applications. Reading, MA: Addison-Wesley.

29.

Loo

2002. “The Delphi Method: A Powerful Tool for Strategic Management.” Policing: An International Journal of Police Strategies and Management 25:762–69.

30.

McCallister

A. D.

1992. Determining Characteristics of the Forest Industry in the 21st Century: A Delphi Study with Implications for Agriculture Education Programs. Unpublished doctoral dissertation, University of Georgia, Athens, GA.

31.

OECD (Organization for Economic Co-operation and Development). 2005. Paris Declaration on Aid Effectiveness. Paris, France: OECD. Accessed August 13, 2012, http://www.oecd.org/development/aideffectiveness/34428351.pdf.

32.

OECD (Organization for Economic Co-operation and Development). DAC (Development Assistance Committee). 2010. Evaluating Development Co-Operation. Summary of Key Norms and Standards. París, France. Accessed February 14, 2012, http://www.maec.es/es/MenuPpal/CooperacionInternacional/Evaluacion/Documents/2010Principios%20y%20Estandares%20CAD.pdf.

33.

Powell

2003. “The Delphi Technique: Myths and Realities.” Journal of Advanced Nursing 41:376–82.

34.

Sartorius

R. H.

1991. “The Logical Framework Approach to Project Design and Management.” American Journal of Evaluation 12:139–47.

35.

Smith

D. K.

1999. Make Success Measurable! A Mindbook-Workbook for Setting Goals and Taking Action. New York: John Wiley.

36.

Stufflebeam

D. L.

1983. “The CIPP Model for Program Evaluation.” In Evaluation Models: Viewpoints on Educational and Human Services Evaluation (pp. 117–141), edited by Madaus

G. F.

Scriven

Stufflebeam.Boston, MA

D. L.

: Kluwer Nijhof.

37.

Tague

N. R.

2004. The Quality Toolbox. Milwauke, WI: ASQ Quality Press.

38.

Van Zolingen

S. J.

Klaasen

C. A.

. 2003. “Selection Process in a Delphi Study about Key Qualifications in Senior Secondary Vocational Education.” Technological Forecasting and Social Change 70:317–40.

39.

Weatherman

Swenson

. 1974. “Delphi Technique”. In Futurism in Education: Methodologies (pp. 97–114), edited by Hencley

S. P.

Yates

J. R.

. Berkeley, CA: McCutchan.

40.

Wilhelm

W. J.

2001. “Alchemy of the Oracle: The Delphi Technique.” The Delta Pi Epsilon Journal 43:6–26.

41.

Worthen

B. R.

Sanders

J. R.

. 1987. Educational Evaluation. New York: Longman.

42.

Yong

Wenhao

. 2011. “Developing a City Governance Index: Based on Surveys in Five Major Chinese Cities.” Social Indicators Research. Published online first, August 2, 2011.