The growing role of evaluation in parliaments: holding governments accountable?

Abstract

To date, the role of evaluation in parliaments has hardly been analyzed. This is a surprise as members of parliament are stakeholders par excellence, who could have an interest in evaluation. But do they? Through a systematic analysis of written questions and interrogations in Germany and Flanders we investigate MPs’ attention to evaluations. The following aspects are analyzed, from a comparative perspective: the content of questions on evaluation, the political profile of the MPs posing the questions, the share of questions on evaluation, and the distribution of questions over political parties, policy areas and over time.

Points for practitioners

The role and use of evaluation in the democratic process might be enhanced. On the one side members of parliament might wish to be better informed on evaluation methods as well as on possibilities for using evaluation results for accountability and learning purposes. On the other side evaluators might strive for strengthening the demand for evaluation in the policy-making process and not only public administration.

Keywords

accountability evaluation parliamentary question policy information written question

Introduction

Both ex ante and ex post, the supply of policy information constitutes a crucial instrument for parliament to fulfill its role. Besides a chain of delegation – from voter to administrator – feedback mechanisms are built in for the purposes of accountability and democratic control (Strøm, 2000). Although the predominance of political parties and, in many countries, coalition agreements heavily influence the way the parliamentary task is put into practice and the extent to which parliament can or does influence policy-making, the operation of institutional checks and procedures to gather information is crucial to the legitimacy of the democratic system.

A key instrument used to gain insights into the conception, progress and results of policy is policy evaluation. Evaluation can be viewed as a structured process that creates and synthesizes information intended to reduce the level of uncertainty for stakeholders about a given program or policy. It is intended to answer questions or test hypotheses, the results of which are then incorporated into the information bases used by those who have a stake in the program or policy (McDavid and Hawthorn, 2006). With this definition of evaluation, it is evident that members of parliament, as representatives of people who pay for and constitute target groups of policies, are stakeholders par excellence and could have an interest in evaluation. But do they? This is a central question in this article. Specifically, we address the questions to what extent and how policy evaluation receives attention in parliament.

To answer these questions we need to realize, in line with the above-mentioned definition of evaluation, that evaluation will not be the sole source of policy information for members of parliament. Politics relies continuously on various forms of expertise, which enter the political process through administrations, special committees, and commissions as well as informal networks. Evaluative information is available to parliamentarians in many different forms, including reports on managerial performance and policy context monitoring, policy research and evaluations. Chelimsky (2009: 52) calls the cross-branch politics and overall ‘checks-and-balances’ architecture of government a ‘lofty arena in which the executive and legislative branches vie for power, engage in political partisanship, and direct ramified but well-aimed strikes on evaluative processes, outcomes, and reporting – especially reporting’.

The weak institutionalization of evaluation in parliaments is striking. In the International Atlas of Evaluation (Furubo and Sandahl, 2002), parliament shows the weakest degree of institutionalization of evaluation across all countries compared to, for example, the administration. Many parliaments only have modest resources and expert personal staff, making it less feasible to conduct evaluations themselves.

However, in many countries evaluation has become increasingly practiced within the executive branch, and parliaments have indirectly become more intense consumers of evaluative information, propelled by a discourse on evidence-based policy-making. This means that parliaments can be very active users of evaluation and still have no ‘institutional arrangements for conducting evaluations and disseminating them to decision-makers’ (Furubo and Sandahl, 2002: 8). However, there are indications that the general interest in evaluation rose within the parliaments. And parliaments do have the opportunity to put evaluation on the political agenda. For example, an imperative to carry out evaluations might be included in laws through sunset legislation or via the introduction of evaluation clauses. In fact, the evaluation of legislation is often triggered by the parliament (Bussmann, 2008, 2010), and the parliament can demand evaluations by means of a parliamentary procedural request. In these cases, the executive branch needs to deliver evaluations after a predefined period of time. Another way for parliament to trigger evaluation is by asking the supreme court of audit to conduct analyses that contain evaluation perspectives. More indirectly, members of parliament (MPs) have the opportunity to ask questions to ministers on policy evaluation, on its planning, content, progress, results, use, etc. In fact, this distinct interest in evaluation is part of the key role of parliament. In addition, answers to MPs’ questions can influence their decisions within their legislative role. They can also refer to evaluations to underpin their legislative work.

The role and use of evaluation in bureaucracies has been studied intensively, whereas the role of evaluation in parliaments has merely been analyzed. Also, evidence on how members of parliament use evaluative information is rather limited. Pollitt (2005: 52) summarizes this in a literature review, stating that ‘little research has been done on whether and how elected representatives use the performance information which is contained in evaluation reports, performance audits, quality assessments and inspections’. Horber-Papazian (2011) analyzed the introduction of performance contracts in the Swiss canton Valais in strengthening the role at the political level. She illustrates the asymmetry in the flow of information between parliament and the executive, including the administration’s strategy to withhold information: ‘The process has a tendency to shine a spotlight on results obtained and areas of information that highlight success, convey a positive impression and maximize their room to manoeuver. In this context, parliament is clearly at a disadvantage and in an asymmetric position in terms of access to information and, hence also, in terms of power at its disposal’ (2011: 142). Also, Johnson and Talbot (2007: 113) write in regard to the UK: ‘we find that parliament itself has been more challenged by performance reporting than challenging of the executive, despite attempts by parliament itself to institutionalize performance scrutiny’.

The use of evaluations within parliament is as difficult to trace as the use of evaluation in general. Sometimes MPs refer explicitly to evaluations conducted elsewhere. Caspari et al. (2003) documented the long path from a first interpellation in the German Bundestag concerning evaluating sustainability in development aid to a parliamentary decision nine years later. Hereafter, more interpellations followed and the committee on development aid continued to work on this issue.

In this article, the instrument of parliamentary questions and interpellations constitutes the unit of analysis for two cases: the Federal Parliament in Germany (Bundestag) and the Flemish Parliament in Belgium (Vlaams Parlement). The selection of the two cases follows the ‘most similar case’ design, as both countries are so-called consensual democracies and their party systems belong to the Benelux type of party constellation (Laver and Hunt, 1992). Both parliaments also cover a wide array of policy domains for which they are responsible. The only study on the institutionalization of evaluation in different countries, including comparative results for Belgium and Germany, shows a relatively low degree of institutionalization for both countries (Varone and Jacob, 2004). Yet, in both countries the evaluation culture is currently quite mature (see below).

Our research is based on the analysis of parliamentary documents, which allows for an insight into MPs’ interest in and use of evaluations. For the underlying research, evaluations in various policy areas are included. Due to varying systems of documentation, the research had to follow different steps of identification and analysis (see below). Also, depending on their importance and use within the case, predominantly interpellations (Germany) or written questions (Flanders) have been analyzed. In general, the background for parliamentary questions is either ‘information-seeking’, ‘prelocutionary’ or both (Russo and Wiberg, 2010). In both cases, the chosen time period (2004–09) covers one coalition period.

Elaborating on the research question, the following aspects will be analyzed: the content of questions on evaluation, the political profile of the MPs posing the questions, the share of questions on evaluation in the total amount of questions, and the distribution of questions over political parties, policy areas and over time within the same legislature.

Evaluation in the German Parliament (Bundestag)

In Germany, where evaluation is embedded more procedurally than institutionally, the national influence is mainly limited to laws and the discretionary decisions of ministries. There is not an overall national institutionalization of evaluation at the whole-of-government level. The influence of the Supreme Audit Institution (Bundesrechnungshof) has a rather indirect influence on evaluation because evaluation is more embedded in particular sectors, where reforms and policy shifts have led to a rise in the demand for evaluation and additional funding for evaluation research.

The role of evaluation in parliament and government has risen, which can, for example, be illustrated by its inclusion in coalition agreements. In 2002, evaluation played a very minor role in the German federal coalition agreement (Koalitionsvertrag, 2002). Thereafter, evaluation became much more integrated into reform projects as well as in pivotal policy areas. In 2005, evaluation played a role in 17 policy areas of the coalition agreement, in which the parties of the government-elect clearly stated which programs or laws they intended to evaluate, particularly regarding how far they intended to wait for evaluation results prior to further decision-making (Koalitionsvertrag, 2005). Within these coalition agreements, a clear timeframe for the evaluation was often indicated, e.g. after three years or in the middle of the electoral period. The government can thereafter be judged on its accomplished intentions to evaluate and some of the parliamentary questions refer to these promises.

Institutional mechanisms in Parliament

In the German Bundestag (German Parliament), evaluations are addressed in the form of motions, reports, major and minor interpellations, as well as oral and written questions. Questions addressed to the Federal Government enable a dialogue within the Bundestag. The following types of written questions are possible:

– Written questions (‘schriftliche Fragen’): every MP may put up to four written questions per month and the government should answer the questions within a week. Questions are to be published weekly.

– Minor interpellations (‘kleine Anfragen’): the government has to answer written interpellations within a fortnight.

– Major interpellations (‘große Anfragen’): Major interpellations usually consist of long lists of questions, sometimes more than 200, relating to a specific area or topic. These are handed in by a faction of the opposition or government parties, but alternatively can also be asked by a minimum of 5 percent of the total MPs. It often takes a few months for the government to answer these questions. The major interpellation is stronger in its effect because the government’s answer can be discussed in parliament.

In the following text, ‘written questions’ refers to this first type, as explained above, although the questions in minor and major interpellations are also written, of course.

Empirical scope

For this analysis, a search for the central keyword ‘evaluation’¹ was conducted within the Parliamentary Material Information System.² This system includes all forms of written questions as well as their answers. In some answers, references to evaluations can be found even if not requested, but in this research only evaluation-related questions were analyzed. The timeframe for this research was the complete 16th legislative term from October 2005 until October 2009. Germany is a federal republic and according to the subsidiarity principle, some policy fields are more dealt with at the Länder level (regions). In this research, all questions and policy fields dominantly assigned to the federal level have been taken into account. The coding of policy fields followed the organization of federal ministries at that time.

Analysis

An examination of the distribution of policy sectors shows that many evaluation-related questions occurred around interior policy, labor market and social policy, as well as education and research policy (Table 1). This was followed by the three policy sectors ‘transport, building and urban development’, foreign policy, and ‘family, elderly people, women and youth’. Within interior policy, many questions were about programs related to migration and integration issues. Since the large, so-called Hartz reform, labor market policy and social policy are constantly undergoing further smaller reforms and adaptations. The evaluation of the Hartz reform was the largest evaluation of that decade and drove evaluation in the labor market sector (Speer, 2012). Also, high political interest has been witnessed in recent years in the fields of education and research, and evaluation became a widespread practice here. These three policy fields have seen (political) novelties and evaluation become increasingly part of governance. Similarly, the attention of MPs increased for evaluations within these policy sectors. This is also the case for the adjacent field of ‘family, elderly people, women and youth’. In those four policy fields the proportion of evaluation-related questions is relatively higher compared to the proportion of all questions per policy field. Another field with a relatively high number of evaluation questions is economic cooperation and development, which has the longest tradition and highest institutionalization of evaluation in Germany. So, the evaluation-related questions may on the one side reflect evaluation activity in the fields, but at the same time also reflect the current political interest and reform activities. Both are often going hand in hand.

Table 1.

Evaluation-related questions by policy sector (German Bundestag, 16th legislature 2005–09)

	Major interpellations		Minor interpellations		Written questions		Sum of all three types of questions		Evaluation-related questions overall
Policy field	Total no.	No. including evaluation- related questions	Total no.	No. including evaluation- related questions	No. of written, oral and urgent questions	No. of written questions on evaluation	Total no.	Percent	Total no.	Percent
Foreign policy	15	5	205	11	1114	0	1334	7.1	16	7.4
Labor and social affairs	5	4	311	17	1280	8	1596	8.4	29	13.4
Education and research	0	0	174	19	588	4	762	4.0	23	10.6
Food, agriculture and consumer protection	1	0	139	6	773	1	913	4.8	7	3.2
Finance	1	1	325	5	2049	1	2375	12.6	7	3.2
Family, elderly people, women and youth	6	1	128	12	492	2	626	3.3	15	6.9
Health	0	0	141	10	716	1	857	4.5	11	5.1
Interior	11	2	590	28	1671	6	2272	12.0	36	16.7
Justice	5	3	124	4	562	2	691	3.7	9	4.2
Environment, nature conservation and nuclear safety	3	1	167	3	987	3	1157	6.1	7	3.2
Transport, building and urban development	3	1	437	16	2552	3	2992	15.8	20	9.3
Defense	2	1	202	3	930	1	1134	6.0	5	2.3
Economics and technology	6	1	238	7	1213	4	1457	7.7	12	5.6
Economic cooperation and development	3	0	46	11	175	1	224	1.2	12	5.6
Federal chancellor, federal press office	2	1	72	5	431	1	504	2.4	7	3.2
	63	21	3299	157	15,553	38	18,895	100	216	100

Source: Authors’ calculations.

In the 16th German Bundestag, there were five parliamentary groups: the Christian Democratic Union/Christian Social Union (CDU/CSU), the Social Democratic Party (SPD), the Free Democratic Party (FDP), the Left Party (DIE LINKE), and the Alliance 90/the Greens (BÜNDNIS 90/DIE GRÜNEN). CDU/CSU and SPD formed a grand coalition. The initiative to ask questions related to evaluations came predominantly from the opposition parties (FDP, DIE LINKE, BÜNDNIS 90/DIE GRÜNEN).

What can we learn from the content analysis of the evaluation-related questions? The largest share of questions is on monitoring the evaluation activity of the government. The queries were often formulated as: ‘Is an evaluation of the new regulation (e.g.) planned? Who will conduct the evaluation? When can the evaluation results be expected?’ Accordingly, the answers generally briefly indicate an intention to evaluate. If the evaluation has already been commissioned, the name of the institute is given, and the expected timeframe is presented. These types of questions and answers focus on holding the government accountable for carrying out evaluations, but as long as the evaluations are not further utilized it remains a purely symbolic and rhetorical ritual. For these questions and answers no parliamentarian even needs to read a single evaluation report. However, some questions go more into detail about monitoring evaluation activity by including questions on the evaluation design, methods and evaluation criteria to be used. For example, details on how to measure labor market effects of agro-genetic engineering or details on measurements of the target group of asylum seekers within Equal partnerships was requested. In summary, it also shows that the expectation to evaluate has risen and the government needs to justify non-evaluation activity by answering questions such as ‘why has this program or this question not been evaluated?’

A second category of questions focuses solely on requesting information mainly on the level of evaluation results. Questions are typically: ‘Is the government aware of an evaluation in the field x and what are the results?’ or ‘Are the results available and are they published?’ These types of question are, on the one hand, also monitoring evaluation activity so that there might be an overlap between the first two categories, but, on the other hand, they request concrete information and disclosure.

A third category of questions was asking for the consequences of evaluation results, often in the sense of evidence-based decision-making: ‘If there are first positive evaluation results, why is the government not acting accordingly?’, ‘What are the consequences of the evaluation results?’, ‘To what degree will the government follow the recommendations in the evaluation report?’

Finally, a few questions contested evaluation results, often combined with the question how far the government agrees with the evaluation results. Those questions delved deeply into the evaluation design and results. For example, in one case it was assumed that the evaluators did not understand the concept, in another the stakeholder representation was put into question and also other stakeholder groups contesting the results were quoted.

Most questions on evaluation were included in minor interpellations. In 157 minor interpellations, generally between one and five questions were on evaluations. A few minor interpellations focused exclusively on evaluation. Thus, 5 percent of the minor interpellations included individual questions on evaluation, whereas 33 percent of the 63 major interpellations included an interest in evaluation. When taking just the written questions into account the fraction of interest in evaluation within the major interpellation is much lower because they include an overall higher number of questions. That said, only 0.3 percent of the total number of questions within the 16th German Bundestag (12,789) touches on evaluation-related topics. So evaluation-related questions are still not dominant.

As the major and minor interpellations are handed in by groups of MPs, specifically by political party fractions, it is difficult to draw any conclusions. For example, the names of the chief whips are usually included. Only written questions were handed in by individual MPs and, between 2005 and 2009, 28 MPs addressed questions on evaluation. Some asked two questions during this period and only one MP (Ina Lenke from the FDP) asked five, all of which have a very different focus. No personal preferences or specializations could be observed here.

In absolute numbers, most of the questions on evaluation within major interpellations came from BÜNDNIS 90/DIE GRÜNEN, whereas within minor interpellations DIE LINKE was the most active and the FDP issued the highest number of written questions. This pattern is following the overall varying use of the three different forms of activity across political parties. Therefore, it is not typical for evaluation-related questions, but rather follows general trends of opposition party activities. However, the relative amount of evaluation-related questions shows a higher proportion for the FDP in major interpellations, and for the BÜNDNIS 90/DIE GRÜNEN in minor interpellations. These data should not be overinterpreted, as one interpellation can contain different individual questions (evaluation related or not). However, MPs of the government parties handed in more than 2800 written questions in total, but only three were related to evaluations. This is consistent with the generally lower interest in evaluative questions from the government side and the interest of the opposition parties to control the government.

All three opposition parties are smaller fractions compared to the grand coalition government, and they do not vary much in their numbers of MPs. Accordingly, the opposition parties that control the government are responsible for between 29 percent and 38 percent per party of overall evaluation-related questions. DIE LINKE has the highest number of questions per parliamentary seat, which is confirmed by the fact that the number of minor interpellations, in which category they have the highest number of questions, has a multiplier effect by including several questions in one single interpellation.

Across the parties, DIE LINKE showed the highest interest in evidence-based decision-making, and questioned evaluation results (contesting) more in comparison to the other parties. However, it remains an open question whether this is mainly driven by the control of opposition parties or perhaps by ideological preferences. To answer this question, research over multiple legislative terms needs to be conducted.

Evaluation in the Flemish Parliament

The Flemish Parliament constitutes the legislative power in Flanders, and is responsible for voting decrees, controlling their implementation and approving the budget of the Flemish regional government (25 billion euro). Since the 1970s, several state reforms have made the Flemish Parliament exclusively or partly responsible for a growing array of policy issues relating to both persons and territory: language, culture, education, health, youth and family, spatial planning, environment, agriculture, housing, energy, local government, work, mobility, scientific research and even international affairs.

Institutional mechanisms in Parliament

The Flemish Parliament has several instruments at its disposal to fulfill its surveying power vis-à-vis the government. Apart from a plenary weekly hour for ‘questions of the day’, MPs can ask for explanations from ministers within the parliamentary committees. Furthermore, written questions can be submitted that should receive an answer from the minister within 20 days. To address a larger issue or problem, an MP can interrogate the minister. If the matter is general in scope, it may be referred to the plenary session. If the minister’s answer is deemed unsatisfactory, the MP can submit a motion of distrust, disapproving the minister’s or government’s policy. When such a motion is approved by a majority of MPs, the government has to be replaced (without elections). MPs can also formulate a recommendation, known as a motivated motion. If adopted, it constitutes an important political signal to the government, which is then obliged to take it into account. The Flemish Parliament also has the right of inquiry: it can install a parliamentary committee of inquiry, an instrument that is, however, rarely used.

Another institutional instrument at the disposal of the Flemish Parliament is asking the (federal) Court of Audit to conduct an audit or inquiry on one or more aspects of policy. Since 1998 the Court of Audit has been entitled to assess policies on the three E’s: economy, efficiency, and effectiveness with so-called performance audits. However, the Flemish Parliament has only seldom made such a request (five times in 10 years) (Vlaams Parlement, 2008).

For the 2004–09 legislature, a commission for ‘decree evaluation’ was established (Vlaams Parlement, 2010). However, its proceedings uncovered that ‘decree evaluation’ is in the first place legalistically oriented towards better regulation rather than evaluation. Formal procedures of ‘regulatory impact assessment’ are operational but appear not to fulfill the role of real ex ante evaluation (SERV, 2006).

The new president of the Flemish Parliament developed an action plan aimed at a better performing Parliament in its surveillance role (Peumans, 2010). Besides goals relating to a more structured debate, some new tools were also announced that could contribute to the evaluating role of Parliament. In so-called topical debates, a specific issue can be evaluated more thoroughly by holding hearings with several stakeholders, after which a societal policy note is drafted.

Empirical scope

In this article the empirical analysis is based on the written questions (and answers) during the legislature of 2004–09.³ This legislature was characterized by the large-scale NPM reform of the administration, which was expected to, and was already demonstrated to, have an impact on interest in evaluation within the executive. Further (long-term) research should reveal whether MPs will also join in this trend.

In the Flemish Parliament, written questions are the controlling instrument par excellence; they are what is most frequently used and are used more than all other parliamentary initiatives combined. The written character of the procedure allows for detailed questions and well-documented answers from the minister. Being a primary tool, they often result in other parliamentary initiatives. Questions do not necessarily need to concern matters of general interest and they can also discuss issues that cannot be treated in committees or plenary sessions. There are few procedural rules, and questions can be submitted in the entire parliamentary year, even within sabbatical periods (Vlaams Parlement, 2009). All these characteristics guarantee a comprehensive approach to our research subject.

For reasons of feasibility, we selected a number of policy sectors, varying widely in budget and nature.⁴ The following eight sectors constitute our area of analysis:⁵ education, agriculture, spatial planning, civic integration (including equal opportunities; anti-poverty policy, etc.), mobility, public works, and culture as well as finance and budget.

To master the enormous amount of data, we predetermined key terms. Given the inconsistent use of concepts characterizing the evaluation field (De Peuter and Pattyn, 2009), we deliberately did not restrict the analysis to questions in which the term ‘evaluation’ was explicitly used, but also included related terms that could actually refer to evaluations.⁶ A content analysis was subsequently conducted to filter the questions that relate to evaluation following the definition mentioned above.

Analysis

The number of evaluation-related MP questions per policy sectors are presented in Figure 1.

Figure 1.

Overview of evaluation-related MPs’ questions per policy sector.

In general, our analysis yielded a total of 180 relevant evaluation-related questions. The total number of (admissible) written questions during the 2004–09 legislature was 11,822 (Vlaams Parlement, 2009). Roughly, the eight policy sectors studied count for 43 percent of the total number of written questions.⁷ This means that only 3.5 percent of the written questions, for this group of these policy sectors, focused on evaluation.

For most areas (mobility, public works, education, finance and budget) the number of questions focused on evaluation reflects the total number of questions per area. More evaluation-related questions might have been expected within the fields of spatial planning, civic integration and agriculture based on their total number of written questions.

The number of parliamentary questions on evaluation varies clearly across sectors. Within some high-ranked sectors, e.g. education, policy evaluation practice is already more developed. This can be confirmed in the case of the Flemish Parliament. The inverse reasoning also counts to some extent: younger policy domains, such as ‘civic integration’, are lagging behind in terms of attention for policy evaluation. Yet, ‘tradition’ or ‘age of the policy sector’ can definitely not explain the full picture. The area of ‘finance and budget’ has traditionally had strong links with policy evaluation, but there are hardly any MPs’ questions with regard to policy evaluations in this domain. A possible explanation lies in its crosscutting links with other policy sectors. Based on international trends, one could also expect a high figure for the sector ‘agriculture’. After all, the European Union has played a major triggering role in terms of diffusion of policy evaluation in this sector, in the framework of the structural funds.

Mobility and public works instead excel in terms of the number of evaluation-related questions posed by MPs. The largely regional character of these sectors likely explains this high number in a regional parliament such as Flanders. Politicians tend to be more interested in programs, initiatives and actions that take place within their local electoral habitats, which causes them to ask a substantial number of ad hoc questions on these topics (see below).

What is the focus of parliamentary questions? Evaluation practices can be classified along various axes. A first classic typology distinguishes evaluation practices according to their timing in the policy cycle. We can traditionally discern between ex ante evaluations (taking place prior to the actual policy decision), ad interim evaluations (taking place during the implementation stage of the policy) and ex post evaluations (taking place after the finalization of the policy), with the latter two both taking a retrospective perspective. Are MPs equally interested in both prospective and retrospective evaluation types? For the Flemish case about 70 percent of the questions could be classified in this way, and from this set a predominance of questions on retrospective evaluations is observed. However, the share of questions on either ex ante evaluation or retrospective evaluations varies with policy sector, possibly relating to differences in practice and types of evaluations conducted, types of problems and policy issues, etc.

A clearer focus on written questions on evaluation can be determined in terms of the stage of the evaluation process to which a particular question relates. We can roughly distinguish questions about the initiation of an evaluation, the implementation of the evaluation, the results of the evaluation and the use of the evaluation. In the Flemish Parliament, the initiation, the results and the use of evaluations are of most interest to parliamentarians, and represent an almost equal share of the total number of questions. The process of the evaluation itself is not really questioned. Again, one can notice variety among the different sectors studied. Four sectors in particular show a deviant pattern in terms of a bias for a specific kind of question. MPs asking questions on evaluation in the field of education are mostly interested in the use of evaluation (‘How are evaluation findings being translated into policy adaptations?’). On the other hand, in domains such as public works, agriculture and mobility, questions about the initiation of an evaluation (‘Has an evaluation been planned or started?’) are found to to be more dominant.

Apart from these four categories, a fifth frequently noticed type of question concerns whether an evaluation has been carried out or not: ‘Has this already been evaluated?’ With that kind of question MPs are often referring to evaluation clauses incorporated in legislation.

Reference to specific evaluation techniques is only seldom made in written questions. In the 180 questions examined, we found one reference to cost–benefit analysis and one to multi-criteria analysis. The same observation holds for evaluation criteria. In the few instances where criteria are mentioned, reference is made to ‘efficiency’ and ‘(cost-)effectiveness’. Other criteria have not been named.

Does party membership and status matter when it comes to parliamentary questions on evaluation? Flemish (and Belgian) politics excels in terms of ‘partitocracy’, meaning that political parties dominantly influence the policy-making process and play a key role in both the legislative and the executive function. Some consider partitocracy to be a major explanatory factor of why evaluation practice generally has difficulties in taking ground in the Belgian politico-administrative arena. Varone et al. (2005), for example, argue that, since political decisions are often the result of difficult compromises between various coalition parties, it is generally not in the interest of the majority to question these compromises on objective grounds via evaluations.

Table 5 shows the breakdown of our analyzed set of questions according to the respective parties, clustered in majority and opposition parties. Because larger parties have more MPs who could ask questions, we calculated the relative number of questions per parliamentary seat.

Table 2.

Focus of evaluation-related questions within the 16th legislature of the German Bundestag

	Major interpellation	Minor interpellations	Written questions	Total	Percent
Monitoring evaluation activity	18	116	19	153	62
Requesting information	6	30	5	41	17
Evidence-based decision-making/asking for consequences from evaluation results	4	28	14	46	19
Contesting evaluation results	1	7	0	8	3
	29	181	38	248

Source: Authors’ calculations.

Table 3.

Evaluation-related questions by political party ‘Evaluation’ in the 16th Legislative Term (German Bundestag 2005–09)

	Government (Gov.)			Opposition (Opp.)
	CDU/CSU and SPD	CDU/CSU	SPD	FDP	Bündnis 90/ Die Grünen	DIE LINKE	Others	Total Gov.	Total Opp.
Motion	7	1	0	16	13	6	0	8	35
Report, expertise, program	5	0	0	0	0	0	1	5	1
Major interpellation	1	0	0	1	1	0	0	0	2
Minor interpellation	0	0	0	9	11	24	0	0	44
Oral question	0	0	0	0	1	1	0	0	2
Written question	0	0	0	3	4	3	0	0	10
Total	13	1	0	29	30	34	1	13	94

Source: Authors’ calculations.

Table 4.

Number of seats and questions on evaluation per political party

	No. of seats	Share of seats per party	No. of evaluation-related questions	Share of evaluation-related questions	No. of evaluation-related questions per seat
CDU/CSU	226	37%	2	1%	0.01
SPD	222	36%	1	0%	0.00
FDP	61	10%	62	29%	1.02
DIE LINKE	54	9%	83	38%	1.54
Bündnis 90/ Die Grünen	51	8%	68	31%	1.33
Total	614		216

Source: Authors’ calculations and Feldkamp (2011).

Table 5.

Number of seats and questions on evaluation per political party

	Political parties	Number of parliamentary seats (d.d. 13 June 2004)	Absolute number of evaluation- related questions	Share per party in evaluation related questions	Relative number of evaluation- related questions per parliamentary seat	Share per party in total number of questions
Majority	VLD-Vivant (Liberals)	25	20	11%	0.80	12%
	CD&V (Christian Democrats)	29	54	30%	1.86	35%
	SP.A-Spirit (Socialists)	25	10	6%	0.40	9%
	N-VA (Flemish Nationalists)	7	45	25%	6.40	13%
Opposition	Vlaams Belang (Extreme Rightwing)	32	30	17%	0.94	24%
Opposition	GROEN! (Greens)	6	13	7%	2.17	5%
Other	Independent MP members	1	8	4%	8.00	1%

Source: Authors’ calculations.

During the legislature, the number of seats per party altered slightly several times, with the transfer of some MPs to other parties. Also some party cartels disintegrated during the legislature.

The figures in the last column refer to all questions, including also the policy fields which were not studied in this research.

From the data, the assumption that questions on evaluation would mostly come from opposition parties is not supported. In fact, one of the majority parties has the highest number of questions per parliamentary seat while the two highest absolute numbers of questions also come from coalition parties. The number of evaluation-related questions of opposition versus majority is also in line with the total number of questions asked by the opposition and majority. Neither can a clear-cut division between left-wing and right-wing parties be made; left-wing parties (SP.A-Spirit and GROEN!) vary from 0.40 to 2.17 questions per seat, while right-wing parties (VLD-Vivant, NV-A and Vlaams Belang) vary from 0.8 to 6.4 questions per seat.

When comparing each party’s share in the evaluation-related questions with its share in the total number of questions, the figures for NV-A are remarkable. As we will discuss below, the high figure for NV-A can be largely explained by the keen interest of one particular MP. The comparison should also be made with care as there might be a bias stemming from the policy fields selection. The shares regarding the total number of questions encompass all policy fields, unlike the shares on evaluation-related questions. In other words, some parties could have a specific interest in the selected policy areas.

Another explanation might be that MPs from coalition parties deliberately ask questions to ‘their’ minister. The minister could then put his or her policy and initiatives in the spotlight. However, the data on written questions about evaluation do not support this rationale. Table 6 shows that in the majority of cases studied only a small percentage of questions come from MPs of the same political party as the responsible minister. The percentage is higher for the Christian-Democrats (CD&V).

Table 6.

Questions on evaluation from Minister’s own party MPs

Policy sector	Party of minister	Number of questions on evaluation from MPs of same party	Share in total number of questions on evaluation per policy field
Finance	VLD-Vivant	1	12.5
Spatial planning	VLD-Vivant	1	6.25
Education	SP.A	0	0
Public works	CD&V	10	22.7
Agriculture	CD&V	2	28.6
Culture	SP.A	2	10.5
Integration	VLD-Vivant	0	0
Mobility	SP.A	2	4.4

Source: Authors’ calculations.

Is interest in evaluation a dedicated affair of a select group of MPs? Table 7 displays for each political party the number of evaluation-related questions and the ratio of MPs asking these questions compared to the total number of MPs. Within most parties, asking questions on evaluation is done by a majority of the parliamentarians, and there are many examples of where the same MP addresses evaluation in more than one policy domain. Remarkably, 30 out of the 45 questions from NVA stem from the same MP.⁸ Interestingly, this MP now chairs the Flemish Parliament. The action plan for improved functioning of parliament reflects his particular interest in evaluation. Time will tell whether the plan will have an impact on the diffusion of interest in evaluation among other MPs.

Table 7.

Spread of questions on evaluation among MPs per political party

Political party	Number of questions	Share of MPs asking the questions
VLD-Vivant (Liberals)	20	12/25 = 48%
CD&V (Christian Democrats)	54	19/29 = 66%
SP.A-Spirit (Socialists)	10	7/25 = 28%
N-VA (Flemish Nationalists)	45	7/7 = 100%
Vlaams Belang (Extreme Rightwing)	30	11/32 = 34%
GROEN! (Greens)	13	4/6 = 67%

Source: Authors’ calculations.

The final question that we touch upon in the analysis is when MPs are primarily interested in evaluation. One indicator is the distribution of parliamentary questions in time during the period of the legislature examined. For Flanders we observe a peak in the second year of the legislature and a slightly greater number of questions during the first half of the legislature. One possible explanation is that MPs have more questions on evaluation when new or adapted policies are being prepared. This observation is not necessarily out of line with the fact that most questions relate to retrospective evaluations, since the latter are perfectly legitimate from a learning perspective when preparing new or adapted policies.

Summary and outlook

Research on parliamentary questions itself is quite young (Martin, 2011) and the role of evaluation within it is even younger. Although the research on the cases varied due to different documentary systems for screening written questions, we can conclude with some observations and reflections from a comparative perspective.

The analysis revealed that MPs ask evaluation-related questions in a wide array of policy sectors. The type of questions asked in Flanders concentrates on verifying whether an evaluation is initiated, its results, or its use. In Germany, many questions were also on holding the government accountable to undertake evaluation and not on the quality of evaluations. German MPs often do not ask for evaluation results, but more for process information, such as when the evaluation has started, who has been selected as evaluator/evaluation institute, and when the evaluation results will be expected. Questions on that level make evaluation quite a new discursive element in parliamentary discussions and monitoring of government activity rather than leading to an increased use of evaluative information, shaping evaluations or being a stimulus for methodological advancements. The spread of interest in evaluation across policy sectors can be explained by the evaluation activity in these sectors, and the importance of the topics on the national/regional political agenda.

The distribution of questions on evaluation among MPs from government vs opposition parties clearly varies between the two cases. In Germany, it is a matter of the opposition asking for information on evaluation activities, whereas in Flanders, MPs from governing parties are highly active. Also, ministers receive questions on evaluation more from other majority parties than from their own party.

Conclusions on interparty differences are difficult to draw since it remains unclear whether interest is directly related to the varying roles in the parliament (government vs opposition) or more influenced by other factors, such as personal interest in evaluation or party culture and attitude toward ‘evidence-based policy-making’. On the basis of interviews with 16 members of the Norwegian parliament, Nyborg (1998) identified a correlation between attitudes to cost–benefit analysis (CBA) and political orientation. The left-wing parliamentarians interviewed were most skeptical and the conservatives the most positive vis-à-vis CBA. This was explained by the fact that the CBA was not perceived as ‘a political neutral tool for project analysis’. So, more qualitative research might shed more light on the interparty preferences vis-à-vis evaluation in general and specific approaches.

Looking at the data from a political party perspective, we can conclude that parliamentary attention – measured by the number of questions on evaluation – is mostly in line with general intra-parliamentary dynamics. Deviant shares for specific parties seem to be largely the result of the activity and personal dedication of individual MPs. Both in Germany and Flanders some MPs became ‘political entrepreneurs’ for evaluation.

The distribution of questions over the course of the legislative period showed dominance for the first half in Flanders, while in Germany interest peaked in the middle of the term. On the one hand, one could argue that at the beginning of the legislature the preparation of new or adapted policies can trigger MPs’ interest in evaluation, while on the other hand interest in evaluation could also rise only after certain policies have become more mature following their introduction.

The latter point also relates to the focus of the questions on evaluation. In Flanders more questions relate to retrospective evaluations compared to ex ante evaluations. In Germany very little attention is given by MPs to ex ante evaluation. Approaching the focus of questions in terms of the evaluation process, in both countries MPs ask questions at different stages: the initiation of an evaluation, the progress, the findings and on the use of results, but with variation among policy sectors.

From this observation we can also conclude that attention to evaluation in parliaments does not directly mean that MPs are receptive to using evaluation to learn about the impact of policies and programs. Questions on evaluation are part of a broader controlling and monitoring role of parliament vis-à-vis government. Parliamentarians seem to push for evaluation, but in fact largely deal with it at the monitoring level.

Our research sheds light on just parts of the parliamentary work and how it is related to evaluation. The use of evaluative knowledge for policy development and budget decisions remains patchy and the use of evaluative findings for decision-making remains opaque. The underlying role of incentives and disincentives for the use of evaluative information as well as secrecy in decision-making in parliaments needs further research (Stiglitz, 1998). The relationship between the use of evaluative knowledge and evidence-based policy should also be analyzed across countries (Nutley et al., 2010). More research is needed to analyze the role of evaluation in the work of committees, in budget decisions and in other decision-making processes. Future research could also be conducted across various countries and parliamentary systems – for example, on the Westminster system prototype of the United Kingdom and the semi-presidential political system of France – to cover political systems with other forms of institutional designs and to be able to analyze their respective influence on the role of evaluation in parliament. In addition, comparative research across regions should also be undertaken to understand the dynamics within the same nations and political systems.

Footnotes

Notes

Sandra Speer is an independent evaluator, based in Wiesbaden, Germany; the focus of her work is on national and international evaluation research for public agencies as well as evaluations in various fields.

Valérie Pattyn is a postdoctoral researcher at KU Leuven Public Governance Institute, and coordinator of the policy research center Governmental Organization and Decisive Governance Flanders. Her current research interests include evidence-based policy, policy capacity, evaluation capacity-building, policy evaluation, policy advisors, and comparative methods. Her recent work has been published in journals such as Evaluation, Public Management Review, International Journal of Public Administration, and The Canadian Journal of Program Evaluation.

Bart De Peuter is a doctoral researcher at KU Leuven Public Governance. His research interests include policy evaluation, local government and performance monitoring. He is co-founder and member of the coordination committee of the Flemish Evaluation Platform. He coordinates evaluation studies and combines this with academic research on evaluation.

References

Bussmann

(2008) The emergence of evaluation in Switzerland. Evaluation 14(4): 499–506.

Bussmann

(2010) Evaluation of legislation: Skating on thin ice. Evaluation 16(3): 279–293.

Caspari

Kevenhörster

Stockmann

(2003) Das Schweigen des Parlaments. Aus Politik und Zeitgeschichte B13–14: 7–14.

Chelimsky

(2009) Integrating evaluation units into the political environment of government: The role of evaluation policy. New Directions for Evaluation 123: 51–66.

De Peuter B and Pattyn V (2009) Evaluation capacity: enabler or exponent of evaluation culture. In: Fouquet A and Méasson L (eds) L’évaluation des politiques publiques en Europe: cultures et futurs. Paris, L’Harmattan, pp. 133--142.

Feldkamp

(2011) Datenhandbuch zur Geschichte des Deutschen Bundestages 1990 bis 2010, Eine Veröffentlichung des Archivs des Deutschen Bundestages, Baden-Baden: Nomos-Verlagsgesellschaft.

Furubo

J-E

Sandahl

(2002) A diffusion perspective on global developments in evaluation. In: Furubo

J-E

Rist

Sandahl

(eds) International Atlas of Evaluation, New Brunswick, NJ: Transaction Publishers, pp. 1–23.

Horber-Papazian

(2011) The power of illusion: Evaluative information and political steering in Valais. In: Eliadis

Furubo

J-E

Jacob

(eds) Evaluation: Seeking Truth or Power?, New Brunswick, NJ: Transaction Publishers, pp. 131–147.

Johnson

Talbot

(2007) The UK Parliament and performance: Challenging or challenged? International Review of Administrative Sciences 73(1): 113–131.

10.

Koalitionsvertrag (2002) 2002–2006: Erneuerung – Gerechtigkeit–Nachhaltigkeit, für ein wirtschaftlich starkes, soziales und ökologisches Deutschland, 16. Oktober, Berlin. Available at: http://www.boell.de/downloads/stiftung/2002_Koalitionsvertrag.pdf.

11.

Koalitionsvertrag zwischen CDU, CSU und SPD (2005) Gemeinsam für Deutschland – Mit Mut und Menschlichkeit. 11 November. Available at: http://www.cdu.de/doc/pdf/05_11_11_Koalitionsvertrag.pdf.

12.

Laver

Hunt

(1992) Policy and Party Competition, London: Routledge.

13.

McDavid

Hawthorn

(2006) Program Evaluation and Performance Measurement: An Introduction to Practice, Thousand Oaks, CA: Sage.

14.

Martin

(2011) Parliamentary questions, the behavior of legislators, and the function of legislatures: An introduction. Journal of Legislative Studies 17(3): 259–270.

15.

Nutley

Morton

Jung

Boaz

(2010) Evidence and policy in six European countries: Diverse approaches and common challenges. Evidence & Policy 6(2): 131–144.

16.

Nyborg

(1998) Some Norwegian politicians’ use of cost–benefit analysis. Public Choice 95: 381–401.

17.

Peeters

(2009) Een vergelijkend onderzoek naar de rol van het Vlaams Parlement bij beleidsevaluaties. Meesterproef ingediend tot het verkrijgen van de graad van Master in het Overheidsmanagement en Beleid Leuven: KU Leuven.

18.

Peumans

(2010) Plenum 2009–2014. Ons Vlaams Parlement in actie. Plenum 2009–2014, Brussel: Vlaams Parlement.

19.

Pollitt

(2005) Performance information for democracy: The missing link? Evaluation 12(1): 38–55.

20.

Put

(2000) Doelmatigheidsonderzoek in het Rekenhof. Vlaams Tijdschrift voor Overheidsmanagement 5(2–3): 19–24.

21.

Russo

Wiberg

(2010) Parliamentary questioning in 17 European parliaments: Some steps towards comparison. Journal of Legislative Studies 16(2): 215–232.

22.

SERV (2006) Advies reguleringsimpactanalyse RIA, Brussel: Sociale Economische Raad Vlaanderen.

23.

Speer S (2012) Sectoral evaluation cultures: A comparison of the education and labor market sectors in Germany. In: Barbier J-C and Hawkins P (eds) Evaluation Cultures. Transaction Publishers, pp. 65--88.

24.

Stiglitz

(1998) Distinguished lecture on economics in government, the private uses of public interests: Incentives and institutions. Journal of Economic Perspectives 12(2): 3–22.

25.

Strøm

(2000) Delegation and accountability in parliamentary democracies. European Journal of Political Research 37: 261–289.

26.

Varone

Jacob

(2004) Institutionnalisation de l'Évaluation et Nouvelle Gestion Publique: Un État de Lieux Comparatif. Revue Internationale de Politique Comparée 11(2): 271–292.

27.

Varone

Jacob

de Winter

(2005) Polity, politics and policy evaluation in Belgium. Evaluation 11(3): 253–273.

28.

Vlaams Parlement (2008) Gedachtewisseling over de nota van de Juridische Dienst van het Vlaams Parlement, over de mogelijke rol van het Vlaams Parlement inzake Decreetsevaluatie, Brussel: Vlaams Parlement, Commissie voor Binnenlandse Aangelegenheden, Bestuurszaken, Institutionele en Bestuurlijke Hervorming en Decreetsevaluatie.

29.

Vlaams Parlement (2005) Decreet Gemeentedecreet, Brussel: Vlaams Parlement.

30.

Vlaams Parlement (2009) Legislatuurverslag 2004–2009, Brussel: Vlaams Parlement.