Improvements and risks of the proposed evaluation of Cohesion Policy in the 2021–27 period: A personal reflection to open a debate

Abstract

This review offers an articulated argument and commentary on the evaluation section of the proposal for a new regulation on EU Cohesion Policy. The main arguments are based on the comparison of the new proposal with past experience and unresolved problems in previous programming periods. The new proposal includes some improvements which may reinforce future evaluation, but also some shortcomings that may weaken evaluations and reproduce past failures. There is time enough to discuss and modify the proposal of the regulation to avoid these risks. My ideas and suggestions focus on the necessity to continue and strengthen the commitment to improve the evaluation process and quality but do not imply a drastic change of the proposed framework.

Keywords

Cohesion Policy evaluation evaluation approach evaluation capacity evaluation rules

Introduction

On 29 May 2018 the European Commission (EC) issued proposals for new regulations for the European Structural and Investment (ESI) funds in the 2021–27 period;¹ in articles 39 and 40 the proposal presents the new rules for the evaluation. In Europe the evaluation of the ESI funds, or Cohesion Policy (CP), has a large influence: it involves all the 28 (27 in the future) EU Member States (MSs) and implements annual public investments for around €40 billion. This makes the CP the largest and most evaluated policy in the world – the scale of the evaluation market is the most important in Europe for both private and public organisations. In the 2007–13 period, when on-going evaluation was not compulsory, around 250 ex-ante evaluation reports and more than 1000 on-going and ex-post evaluations were produced by the MSs and the EU-wide ex-post evaluation of each fund was carried out by the EC.

This review briefly analyses the proposed rules for evaluation in the hope of promoting a wider debate. For reason of space the analysis concerns only the rules for evaluation and does not discuss monitoring, performance framework and controls, which are to varying extents interrelated with evaluation. After a summary of the evolution of the ESI funds evaluation since 1989, which is necessary to contextualise the analysis, the contribution presents the proposal of the EC and compares it to the existing rules of the current programming period (2014–20). Finally, risks and opportunities that derive from the proposal are considered and some improvements suggested.

A complex and evolving evaluation approach

The evaluation of CP is highly ‘institutionalised’: rules established in the EU Regulation require the compliance of all the MSs and all the funds. It is also a very complex evaluation: its governance includes the EC, the MSs and the Regions responsible for interventions; the programmes to evaluate are multi-annual, integrate many different measures and target different social groups or economic sectors. Institutional arrangements and methodological issues are interrelated: rules identify the types of evaluation and push for specific methodological approaches. In addition, since the pioneering phase in the 1990s the EC has played a leadership role in making the regulation operational and identifying the methodological approaches to use in the requested evaluations. This happened in a context where for the majority of MSs the evaluation of the ESI funds has represented the first systematic and significant experience of evaluation of their public policies.

It is useful to briefly review the evolution of the evaluation of ESI funds. Table 1 summarises the main features of this evaluation during the five programming periods started in 1989. Below, the principal characteristics of ESI funds evaluation over these 30 years are briefly examined:

the first characteristic is the type of evaluations required. Only in the first two periods were the ex-ante and the ex-post evaluations required and only in the 2000–06 period did the mid-term evaluation (and subsequently the in-itinere evaluations) become a stable component of ESI funds evaluation. From that moment evaluation has been a continuing source of support for the different main Structural Fund actors – EC, MSs, Managing Authorities (MAs) – and has had to respond to a plurality of needs (planning, accountability, learning) for each actor involved in the ESI funds implementation;

the second characteristic is intended coverage, and the capacity to evaluate all programmes and typologies of intervention. Full coverage of all types of evaluations only became a reality in the 2000–06 period and the obligation to carry out mid-term evaluations was a determining factor in promoting it. In fact, in the following 2007–13 period, when the obligation was abolished, the coverage of the in-itinere evaluations significantly decreased, while in the current 2014–20 period obligation has been reintroduced and full coverage seems to have been achieved;

the third characteristic concerns the evaluation approach and methodology. In the initial periods the ‘programme’ approach² – a mix of methods which evaluates the entire programme composed of different policy areas and instruments in a unique study – prevailed. However, the difficulty of this approach in providing an in-depth analysis of the different policies and their synergies became evident at the end of the 2000–06 period. Hence, a ‘thematic’ approach, that focused separately on each policy area and instrument, was preferred in the ex-post evaluation. This change also modified the scope of the evaluation; in the initial periods evaluation was more oriented toward assessing the effects of the programmes on the whole economic and social development, subsequently it focused more on the effectiveness of the single interventions or policy areas and econometric models or successive meta-analyses were delegated to assess the global effects. Since the end of the 2007–13 period and after numerous criticisms,³ an improvement in the methodological design has been promoted by the EC. Impact evaluations and related methodologies (in jargon, ‘counterfactual’ or ‘theory-based’) began to be disseminated and implemented. The EC has also started to scrutinise the quality of evaluation reports and evaluation processes managed by the MAs and has implemented various forms of training and technical assistance to improve them.

The limited quality of the evaluation is the result of several factors, often at work simultaneously. The main factors are: a formal compliance with the EU rules and a bureaucratic approach to evaluation; a poor methodological design often the consequence of inadequate MA evaluation capacity and lack of control over evaluators’ approach and goals; insufficient independence of the evaluators; and the difficulty, or reluctance, in debating and using the findings of the evaluations. More generally, the poor quality of the evaluation often follows from inadequate administrative processes to manage evaluation implementation rather than the lack of technical knowledge in the MS; and

a fourth, and final, characteristic is the complexity of the ESI evaluation process and the consequent compromises in its institutional formalisation. ESI funds are implemented according to a ‘shared-management’ method in which EC and MSs pursue different tasks but are both responsible for the interventions. Consequently, in the regulations evaluation has to respond to different competences and interests: EC focuses more on the ex-post evaluation which it is totally responsible for, while MSs pay more attention to ex-ante and in-itinere evaluations falling under their responsibility. Furthermore, the MAs are responsible for national interventions and evaluations which introduce a potential ‘conflict of interest’ and do not guarantee an effective specialisation of the body responsible for the evaluation. However, this solution is considered more realistic in comparison with the introduction of an additional authority and represents the ‘lesser of two evils’. There is also the relation between the timing of the policies and the timing of the evaluations. The different policies included in ESI programme funds produce their results at different times and need to be evaluated accordingly. However, the timing for evaluations does not take into consideration the timing of the implemented polices (see, for instance, the case of the mid-term evaluation in 2000–06).

Table 1.

Evolution of the evaluation of the ESI funds in the different programming periods.

Programming Period	Main rules	European Commission	Member States	Notes
1989–93	Only ex-ante and ex-post evaluation are instituted.	The EC carries out the ex-post evaluation. A ‘programme’ approach is experimented in the ex-post evaluation.	No evaluation carried out by the MSs.	This is the pioneering phase of the evaluation and the ex-post evaluation is its most important result in methodological and knowledge terms.
1994–99	The regulation provides the implementation of ex-ante evaluation, in-itinere ‘surveillance’, and ex-post evaluation. The in-itinere ‘surveillance’ can include on-going evaluations.	The EC implements the ex-ante and the ex-post evaluation. The Court of Auditors criticises the results of the ex-post evaluation because it was considered generic.	MSs implement their first on-going evaluations, but these were very few and not systematic. Evaluation in the MSs are under the responsibility of the Managing Authorities (MAs). This remains unchanged until now.	The EC guidelines are inspired by the programme evaluation approach. Several criticisms on the efficacy of the CP are raised and a debate on evaluation demands and methodologies begins.
2000–06	The ex-ante, mid-term and ex-post evaluations are compulsory and are clearly identified in the regulations. MSs are explicitly responsible for the ex-ante and mid-term evaluations. The mid-term evaluation has to be completed and discussed by 2005.	The EC controls compliance with the regulation and enforces the evaluation obligation. The EC guidelines are based on a programme approach, but in the ex-post evaluation national evaluations result are weak and only few are usable. The EC prefers a thematic approach to manage the complexity of CP.	The MSs implement the ex-ante evaluations and evaluated all programmes by 2005 (mid-term evaluation). Many mid-term evaluations were poor because many interventions were often in the initial phase and were only formally compliant to the rules.	A big effort is made and evaluation covers all the programmes. The mid-term deadline permits a general discussion of the evaluation findings. However, the limits of the evaluation quality are evident (formal or not fully independent, based on indicators and without original analyses, no impact evaluations, etc.). The EC starts a systematic reflection on this.
2007–13	The mid-term evaluation is substituted by in-itinere evaluations, which are no longer compulsory. Ex-ante and ex-post evaluations continue to be compulsory. The evaluation plan and the evaluation steering group are proposed by the EC but are not obligatory.	The EC carries out the ex-post-evaluation in which it confirms the preference for a thematic approach. The EC promotes evaluation quality and design, particularly in impact evaluations (counterfactual as well as also theory-based).	In absence of an obligation, MSs launched on-going evaluations with delay and did not evaluate many programmes and thematic priorities. New MSs are more active. Limited but significant improvements in the methodologies are evident in the MSs.	Evaluation enters a more ‘mature’ stage, but is still lacking in quality. In other policies the role of evaluation is increasing within the EC (impact assessment and better law-making) and CP has to pay increasing attention to ‘result-orientation’.
2014–20	Ex-ante, in-itinere and ex-post evaluations are compulsory. In the in-itinere evaluations MSs have to produce an impact evaluation for each priority axis of each programme. Evaluation plans are compulsory. MSs are requested to ensure the needed evaluation capacity.	The EC verifies the evaluation plans in the Steering Committee and issues guidelines for impact evaluations (counterfactual or theory-based) and the evaluation process (terms of references, follow up, etc.). The EC put in force also a specialised technical assistance to support MSs.	Evaluation plans are of mixed quality and often evaluation implementation is delayed, but in general evaluation planning is in force. Impact evaluations are limited in number at the moment, but many of them are planned for the next years.	It is too early for a judgment. In many cases the delays in implementing evaluation are signals of a persisting formalistic and disinterested approach. A wider range of methodological approaches are planned and awareness of methodological issues seems to have increased.

Source: authors own elaboration on Naldini (2013).

The current proposals

This evolution shows that changes in the evaluation of ESI funds might both progress and regress from one period to another; and an increased awareness of methodological and organisational issues does not automatically lead to a better quality evaluation. So what of the current proposals?

In its preface the EC proposal for a new regulation affirms that evaluation rules have been defined according to the inter-institutional agreement between the European Parliament, the Council of the EU and the EC of July 2016 on ‘Better law-making’. This agreement recognises evaluation as a fundamental instrument for preparing new legislation and indicates that it has to focus on basic criteria of efficiency, effectiveness, relevance, coherence and value added.

Article 39 states that the managing authority (MA) of an ESI funds programme ‘shall carry out evaluations of the programme. Each evaluation shall assess the programme’s effectiveness, efficiency, relevance, coherence and EU added value with the aim to improve the quality of the design and implementation of programmes.’ One year after the approval of the programme the MA has to submit an evaluation plan to the Steering Committee of the programme, where it identifies the main evaluations to be carried out. In 2029 an impact evaluation of each programme is to be implemented. MAs have to ensure the independence of the evaluation and the procedures to make available data necessary for evaluations. All the completed evaluations have to be published on a website.

According to article 40, the EC ‘shall carry out a mid-term evaluation to examine the effectiveness, efficiency, relevance, coherence and EU added value of each Fund by the end of 2024’. Then, by 31 December 2031, the EC ‘shall carry out a retrospective evaluation to examine the effectiveness, efficiency, relevance, coherence and EU added value of each Fund’.

This proposed evaluation is in line with past experiences, but when compared to current rules⁴ introduces some significant changes:

in the regulation evaluation rules are more succinct than before and, consequently, more generic;

ex-ante evaluation is abolished, while it was compulsory in current and previous programming periods;

the focus of the evaluation passes from the individual or homogenous components (‘priority axes’ or ‘specific objectives’) of the programme to the whole programme;

impact evaluation is postponed until 2029 and covers the entire programme; on the contrary at the moment impact evaluations are carried out on each single priority axes during the implementation of the interventions and in line with their progress;

the EC is required to implement a mid-term evaluation in 2024; this is a new evaluation in comparison to requirements in the current period; and

references to the independence of the evaluators and cooperation of the MA in providing necessary data are maintained, but other references, such as the responsibility of MSs for evaluation capacity or the need of operational guideline by the EC, are absent.

Each of these points deserves specific consideration:

the simplification of the rules in general is welcome, but runs the risk of weakening the obligation to carry out evaluations; as seen in the past, this very often implies a reduced coverage of the evaluation. In current Regulation 1303/2013, comma 3 of article 56 was quite precise with regard to the type and the minimum number of compulsory evaluations (‘At least once during the programming period, an evaluation shall assess how support from the ESI Funds has contributed to the objectives for each priority’) and the evaluation plans have been defined according to that requirement. Now, the statement in article 39 says only that ‘evaluations’ have to be carried out;

the decision to abolish the ex-ante can be understood in relation to the formal and ineffective approach to ex-ante evaluations demonstrated by many MAs at the beginning of this period. However, it could be seen as a setback for the evaluation method promoted by the EC,⁵ which intends the evaluation as a continuous support to the policy cycle, from the preparation to the implementation and assessment of the policy initiative.⁶ Moreover, ex-ante evaluations are also the basis of the Inter-institutional Agreement mentioned at the beginning of the proposed regulation. The ex-ante assessment remains in force only for the financial instruments (article 52 of the proposal), but the effective contribution of these evaluations in this current period may be debatable as much as that of the programme ex-ante evaluations;

the return to the programme approach, after having promoted the thematic approach for more than 10 years, may send a confusing message and re-institutes the methodological problems encountered, and not overcome, in the past (see above). Obviously, the interpretation of article 39 can be flexible and thematic evaluations may be used to assess the programme, but the message of the proposed regulation does not suggest this kind of solution. In addition, the reference to the classic evaluation criteria (effectiveness, efficiency, relevance, coherence and EU added value) and the lack of any specifications of the evaluation questions to address may induce MAs to carry out evaluations based only on the relations between programme indicators. This would bring evaluation of ESI funds back to its old vices and formalistic ways;

the impact evaluation which refers to the entire programme, is under the responsibility of the MSs and has to be implemented by 2029. A final evaluation of each programme is essential to improve accountability and provide stakeholders and policy makers with a synthetic view of what has been achieved. However, the deadline raises some doubts because 2029 may be too late for using the results of the evaluation in the preparation of the following programming period and too early to reveal specific impacts if interventions start late or require a long time to show their effects;

the re-introduction of the mid-term evaluation conducted by the Commission is a positive novelty. It can be useful to take stock of the progress of all the programmes at the same time and is also needed to support the mid-term review of the programmes in 2025. In this regard, it is worth highlighting that this evaluation is the responsibility of the EC and not of the MAs. The contrary happened in the 2000–06 period when many resulting evaluations were weak and of little use. Probably this decision intends to avoid that risk and puts the responsibility of the mid-term evaluation under an authority not directly involved in the daily implementation.⁷ Its deadline is consistent with the policy cycle (the mid-term review in 2025), but, considering the fact that programmes on average start to spend resources two years after the beginning of the period, in 2024 the evaluation may have little to analyse; and

finally, the lack of emphasis on evaluation capacity and the role of the EC in providing methodological support and guidance is unexpected. In recent years the EC has lavished much energy on the improvement of evaluation capacity and quality; not reaffirming the importance of these aspects could undermine previous efforts and release the MSs of their responsibilities for quality assurance.

More generally, these changes in evaluation seem strongly affected by a major objective of the proposed reform, that is: ‘reduce unnecessary administrative burden for beneficiaries and managing bodies’ (p. 7) even though there is little evidence that evaluation currently suffers from an excessive administrative burden. Present limitations derive more from capacity problems or lack of interest in some MAs and a consequent low quality of many evaluation reports. The necessary improvements require time but also a persistent and shared commitment of the EC and the MS to strengthen the evaluation process. I would argue that this should be clearly stated in the regulation and a few, but unambiguous, rules should more clearly specify how evaluations should be carried out.

Possible ways to strengthen evaluations in the 2021–27 period

In the view of the author, and following on from the analysis outlined above, there is scope to further improve the proposed rules for evaluation in the next programming period. This is where I hope to provoke further debate at both policy and practitioner levels.

I would suggest that the rules need to:

specify better the minimum number and types of in-itinere evaluations required from the MSs, which would also strengthen the EC’s own mid-term evaluations;

clarify the possibility of integrating a thematic approach alongside a programme approach, especially when this is justified by the complexity of the programme concerned;

introduce an explanation of the main types of evaluation questions to address (as, for instance, ‘are the right targets reached?’, ‘what are the main operational bottlenecks?’, ‘what works?’, ‘why it works and under what circumstances?’, etc.). This would give greater meaning to the criteria mentioned such as effectiveness, etc. This would also orientate evaluations to building real policy-relevant knowledge; and

affirm the objective of reinforcing the evaluation capacity and quality with operational guidelines, and supported by the mutual efforts of both the EC and the MSs.

The specification of operational guidelines for the MAs could take various forms. I would advocate the need for some methodological guidance. This might include, for example:

the need to adopt a basic and robust methodological design for the programme approach capable of managing high levels of programme complexity;

to suggest that evaluation criteria are translated into practical and useful evaluation questions; and

to identify minimum standards in the evaluation process, from the moment of the definition of the Evaluation Plan to the dissemination of the findings, to ensure quality, credibility and the usability of evaluation results.

I do not underestimate the challenges of my suggestions, but from past experience could be implemented relatively easily. Some other issues, mentioned previously, are challenging because they intersect with the division of competences between the EC and the MSs in the co-management system of the ESI funds. Their solution may therefore imply modifications in this co-management system supported by a broad consensus among key actors. In any case, I would argue these matters should not be ignored.

Thus the first issue, reintroducing the obligation for ex-ante evaluation, would require a role for the evaluator separate from the MA. This would give him/her the needed time, resources and independence to carry out the ex-ante evaluation and restore its utility during the negotiation between the EC and the MS. The second issue is which body is responsible for the evaluation when the MA does not demonstrate sufficient commitment and capacity. One solution would be the identification of a new authority responsible for evaluation in the MS concerned, or alternatively granting the EC or a national authority substitution power when a single MA is not up to the job. The third issue is the problematic coexistence of institutional deadlines and methodological constraints in different policies concurrently supported by the CP. Resolving this issue would require a more flexible timing of evaluations while also paying more attention to building cumulative knowledge on the implemented policies rather than relying on the production of a single evaluation report.

Conclusions

The ideas put forward above focus on the necessity to continue to strengthen and improve the evaluation process and quality. This necessarily requires balancing rules (and obligations) with capacity building, in line with the institutionalised nature and current ambitions for the evaluation of ESI funds. However, this does not imply a drastic change of the framework proposed in draft regulations. Rather it requires clarification together with enhanced integration with positive experiences from past evaluation of Structural Fund programmes.

I am aware that these arguments need extensive debate to include different points of view informed by the kind of technical analysis that I have only begun in this review. The fundamental objective is simple: to enrich the role and the contribution of the evaluation of CP, which annually spends around 0.3% of the EU GDP and involves all the territories and all the main institutions of the EU.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

Andrea Naldini is Director of Policy Analysis and Evaluation at ISMERI EUROPA and was Secretary of the Italian Evaluation Association. His main fields of expertise are regional policy and public policy monitoring and evaluation.

References

Dunlop

Radaelli

(2017) If evaluation is the solution, what’s the problem? In: Buonanno

Zahariadis

(eds) The Routledge Handbook of European Public Policy. London, UK: Taylor & Francis.

European Commission (2013) Regulation (EU) n. 1303/2013.

European Commission (2015) The better regulation guidelines in the Commission. Available at: https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-how/better-regulation-guidelines-and-toolbox_en

European Commission (2018) Communication 375/2018. Final Proposal for a Regulation of the European Parliament and of the Council laying down common provisions on the European Regional Development Fund, the European Social Fund Plus, the Cohesion Fund, and the European Maritime and Fisheries Fund and financial rules for those and for the Asylum and Migration Fund, the Internal Security Fund and the Border Management and Visa Instrument. Available at: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2018%3A375%3AFIN

Martini

(2008) How counterfactuals got lost on the way to Brussels. In: Fouquet

(ed.) Evaluation des politiques publiques en Europe: cultures et futures. Paris: Harmattan.

Naldini

(2013) Lo stato della valutazione della politica di coesione in Italia. In: Vergani

(ed.) Il libro bianco della valutazione in Italia. Milano: Franco Angeli.

Rossi

Lipsey

Freeman

(2003) Evaluation: A Systematic Approach. Thousand Oaks, CA: SAGE.