Abstract
There is a lack of published evidence which demonstrates the impacts of public engagement (PE) in science and technology policy. This might represent the failure of PE to achieve policy impacts or indicate a lack of effective procedures for discerning the uptake by policy makers of PE-derived outputs. While efforts have been made to identify and categorize different types of policy impact, research has rarely attempted to link policy impact with PE procedures, political procedures, or the connections between them. In this article, we propose a simple conceptual model to capture this information, based on semistructured interviews with both policy makers and PE practitioners. A range of criteria are identified to increase the policy impact of PE. The role of PE practitioners in realizing impacts through their interactions with policy makers in the informal “in-between” spaces of public engagement is emphasized. However, the potential contradictions between the pursuit of policy impacts and the more traditional conceptualizations of PE effectiveness are discussed. The main barrier to the identification of policy impacts from PE may lie within policy processes themselves. Political institutions have responsibility to establish formalized procedures for monitoring the uptake and use of evidence from PE in their decision-making processes.
Introduction
There is widespread recognition in the literature of a lack of credible evidence to measure and demonstrate the policy impacts of public engagement (PE) in science and technology (Abels 2007; Kurath and Gisler 2009; Pidgeon and Rogers-Hayden 2007; Powell and Colin 2009; PytlikZillig and Tomkins 2011; Rowe and Frewer 2000; Wathen et al. 2011; Wilsdon, Wynne, and Stilgoe 2005). There are various reasons as to why demand for increased societal inclusion into policy processes has arisen, including, inter alia, institutional perceptions of a general decline in societal trust in the motives of institutional actors (in particular in industrial and regulatory sectors) regarding policy and policy implementation (Houghton et al. 2008; Petts 2008; Wagner and Armstrong 2010). There is simultaneously increased societal demand for transparency and inclusivity in decision-making processes regarding policy development. In addition, the consideration of a broader range of expertise in assessing different policy options might lead to better outcomes, as more evidence (lay knowledge, perceptions, and preferences) is considered as part of the decision-making process (Reed 2008; Renn 2006). Without substantiated evidence of policy impact, it is unclear whether the deficiency represents the failings of PE to actualize policy impacts or whether it simply indicates that the means for discerning policy impact are poorly developed. There is certainly anecdotal evidence of policy impacts arising from PE, which suggests that PE does have the potential to influence policy making. What remains limited, however, is an understanding of the causal relationships between PE and policy, as well as the potential contradictions between them in their alternative quests for legitimacy. It is important for both engagement practitioners and policy makers to demonstrate policy impacts in order to better evaluate the effectiveness of PE, allow monitoring and continuous improvement of engagement practices and their policy connections, demonstrate the worth of PE for policy, and enhance its reputation and credibility in the eyes of policy makers and funders.
We review the literature on the relationship between PE and policy impacts to inform a conceptual model, which is refined and substantiated using empirical evidence derived from interviews with PE practitioners and policy makers. Bringing together a range of criteria, and relating them to the likelihood of impacts being realized, allows us to offer recommendations that consider the issues, contexts, and potential contradictions between the characteristics of PE mechanisms, policy processes, and the features linking them together. We conducted semistructured interviews with PE practitioners from across Europe, as well as policy makers, primarily from the European Commission. This approach, which transcends the interface between PE practice and policy, highlights interactions and relationships across these areas, which have been given insufficient attention in the evaluation literature. It also allows for a critical reflection on the outstanding barriers to further integration between PE and policy.
The Problem of Policy Impact
The problem of identifying policy impacts arises because of the difficulty for PE practitioners and evaluators to track PE outputs once they have entered the policy realm; the time lag between engagement (and evaluation) activities and potential policy outcomes; the numerous direct and indirect ways to realize policy impact; and because PE impacts are not easily differentiated from a plethora of other potential influences on political decision making. This has meant that the most significant focus of attention in the PE evaluation literature has been on the evaluation of PE procedures and mechanisms, which may be used as surrogates for evaluating effectiveness in terms of outputs (Abels 2007; Rowe and Frewer 2004).
The concept of policy “resonance” (rather than “impact”) has been proposed to account for difficulties in recognizing impacts and to avoid implying a linear model of engagement (Joly and Kaufmann 2008). The idea of resonance recognizes a propensity to influence, as opposed to the achievement of a tangible and measurable outcome. Resonance, therefore, might be a useful term for anticipating the likely future effect of PE mechanisms in the policy realm when there is no direct means of measuring that impact. However, if the remit of evaluation is to be extended into the sphere of political process, if impact is employed in a way that does not imply finality or closure of an engagement process, and if we are able to evaluate retrospectively—given enough time for changes to be discerned—then impact remains a useful term for evaluating PE in terms familiar to a range of different audiences.
Researchers in certain fields (e.g., technology assessment) have tried to typologize policy impacts arising from societal engagement (Decker and Ladikas 2004; Hennen and Ladikas 2009). They divide impacts into three principal headings: raising knowledge, forming attitudes and opinions, and initiating actions. The first two are “conceptual impacts” whereas the last encompasses “instrumental impacts” (Phillipson et al. 2012). The typologizing of policy impacts is a useful first step toward monitoring and understanding them. However, without linking those impacts to the features and characteristics of the realms of PE, it is difficult to offer recommendations to either policy makers or practitioners for how to improve PE and maximize its policy impact.
A Simple Conceptual Model for Examining and Evaluating the Policy Impact of PE
Notwithstanding the lack of information about the relationship between PE and policy, and the difficulties of studying behind closed door policy-making processes, the literature suggests some important lessons and reflections on this relationship. To our knowledge, there has been no previous attempt to draw together insights embedded in the wider literature and categorize them according to their spatiotemporal and causal forebears: that is, to examine the “wheres,” the “whens,” and the “whys” of PE-derived policy impacts. This lack of comparative or systematic attempts to examine the relationship between procedures and impacts was also identified by Hansen and Allansdottir (2011), who conducted a comparative analysis of participatory technology assessment in relation to policy impact. They acknowledged, however, that they were only able to comment on the presence/absence of policy impact in different contexts, rather than on the differences attributable to procedural design.
The insights elaborated in this section are derived from the PE literature relating to policy impacts or outcomes. We also acknowledge that there is a much wider literature on relevant topics beyond the strict limits of PE. The purpose of retaining a focus on PE is to ensure that the criteria developed remain relevant to informing the debate on the links specifically between PE and policy, as well as developing a framework for examining these links. Based on our interpretation of the literature, we divide the relationship between PE and policy impact into three interlinked areas: (i) the features and nature of the PE mechanism, (ii) the features and nature of the policy-making process, and (iii) the features linking the PE mechanism with policy making (Figure 1). We now examine the three realms in more detail, according to the barriers and opportunities for PE-derived policy impacts.
The three realms of public engagement-derived policy impact.
Features of the PE Mechanism
In Rowe and Frewer’s (2000) framework for evaluation, the most directly relevant criteria associated with PE mechanisms’ policy impact are the criterion of influence and the criterion of task definition. These criteria stipulate that realizing policy impact is more likely where there is an up-front agreement on how the outputs of PE will be used and how they will be integrated into policy making (Chilvers 2008). Pidgeon and Rogers-Hayden (2007, 205), meanwhile, stress the importance of the comprehensibility of the recommendations arising from PE, and, in particular, their utility to policy makers.
The classic evaluation literature also stresses the importance of the fairness, legitimacy, and accountability of PE mechanisms. Such criteria relate to the central tenet of deliberative democratic processes, namely that their application should lead to better policy outcomes. The problem for evaluators is that fairness and legitimacy are not automatically imbued on a process incorporating PE, but are correlated with the intents of the persons involved (Rowe and Frewer 2005). Moreover, even the fairest of processes in terms of representativeness and accountability may fail to translate into discernible policy impacts. Typically, legitimacy is considered from the perspective of people participating in the PE process. Less attention has been paid to PE legitimacy in the eyes of policy makers, whose perceptions of legitimacy might be more likely to have a bearing on the uptake of PE outputs into policy making (see also Features Linking Policy Making with PE subsection). This is interesting because what policy makers and publics consider as legitimate may differ or be incongruent. For instance, a PE mechanism could fail to meet generally agreed upon evaluation criteria, but still have a policy impact, if it was perceived as legitimate in the eyes of policy makers. Conversely, a well-executed and fair mechanism might not result in discernible policy impacts, if the policy makers themselves do not perceive it as legitimate.
The appropriateness of the scale, topic, and timing of engagement are potentially relevant to policy making. At any particular moment, there will be topics of more interest to policy makers than others. This links to the timing of engagement, and it is normally seen as most useful when undertaken as early as possible in the policy-making process, although it could also be applied to the selection of policy alternatives once these have been identified (König et al. 2010). It is feasible, therefore, that an otherwise well-executed engagement exercise might not realize policy impacts if it is mistimed (Abels 2007). For this reason, Joly and Kaufmann (2008) argue that policy resonance is more likely where policy engages with people in the creation and maintenance of an ongoing dialogic space, as opposed to discreet, stand-alone engagement events. These authors also raise the issue of scale. The outputs from PE may have a greater impact at the level of policy making appropriate to the issue at hand, which has consequences for the nature and scale of the PE mechanism itself (Joly and Kaufmann 2008). For example, a local-level PE might be more likely to lead to locally relevant policy outputs, yet it could equally give rise to regionally or nationally important outputs. The focus, therefore, should be on the appropriate use of PE outputs, in terms of where their impact will be most relevant.
Two final features of the PE mechanisms relate to the practices and capacities of PE practitioners. Fung (2003) argues that policy impacts are more likely if PE practitioners are actively involved in monitoring those impacts. By doing so, practitioners are more likely to consider policy impact in the process of designing, implementing, and communicating the PE process. Powell and Colin (2009, 335), meanwhile, emphasize the political capacities of PE practitioners in terms of (i) their understanding of the policy context, (ii) their understanding of the political processes, (iii) their knowledge of which political actors/institutions to engage with, and (iv) their ability to communicate effectively.
Features of the Policy-making Processes and Institutions
Little is known about the influence of political procedures for the uptake and use of PE-derived outputs in policy making. This may be due to the variety of procedures in place in different contexts, or to the lack of access by practitioners and researchers to the inner workings of bureaucratic institutions. This tends to result in much of the focus on policy -making’s role in the uptake and legitimacy of PE outputs’ relations to policy makers’ motives, perceptions, and attitudes. Powell and Colin (2009), for instance, argue that engagement that is motivated by the political desire to gain public acceptance of an issue does not buy into the ideals of PE and cannot, therefore, lead to legitimate policy impacts.
Since the legitimacy of PE and its outputs in the eyes of policy makers has not been studied as a potential determinant of PE impact, it merits further discussion. Legitimacy might relate to the validity of the mechanism employed, to the societal groups represented (together with their perceived interests), and to the nature of the outputs produced. This recognizes that the outputs are policy appropriate and that policy makers might consider it an infringement of their responsibility if the recommendations arising are too prescriptive regarding subsequent actions (Hennen and Ladikas 2009). Political legitimacy of PE is sometimes inferred from the impact of engagement on the knowledge and attitudes of the wider public (Krabbenborg 2012).
Finally, there is an expectation that policy makers will, at least to some extent, take PE outputs into account in their decision making (Fung 2003). Researchers have less frequently considered a methodology for determining if, to what extent, how, and why this does, or does not, happen. The political or “organizational” capacity of institutions engaging with PE has been highlighted as an important but overlooked element in the evaluation literature (Jabbar and Abelson 2011).
Features Linking Policy Making with PE
Where PE is formally attached to the institutionalized political agenda, it is more likely that its outputs will be assimilated into decision making (for instance, where public consultation/engagement is formally incorporated into policy making through legislation and/or adherence to agreed standards). This links to arguments about PE’s need to be seen as part of a process of ongoing engagement to allow genuinely deliberative interaction between the public and the policy (Abels 2007; Joly and Kaufmann 2008; Wilsdon, Wynne, and Stilgoe 2005).
Krabbenborg (2012) showed the importance of treating the features linking PE with policy making separately. In the case of the Dutch national dialogue on nanotechnologies, the dialogue was seen as legitimate by policy makers since it was government initiated, aimed at informing policy, and sponsored with 4.5 million euros. The PE activities themselves were very diverse, organized bottom-up, and received sufficient funding. However, outcomes, that is, ethical and societal issues raised by participants, were not communicated to policy makers. The organizing committee instead focused on traditional outreach factors to demonstrate legitimacy (number of people reached, increased knowledge and awareness). Thus, the means by which links were formed between PE and policy makers was the primary barrier to achieving policy impact.
A very direct way for policy makers to be involved in knowledge production and public interaction is through face-to-face participation in PE mechanisms (Rowe and Frewer 2005). This may improve communication and facilitate the development of interpersonal relationships (and trust) that lead to mutual learning. Such interaction would have to be sensitively handled, however, since the presence of policy makers at PE events could be seen as imposing, preventing open discussion and leading to potentially biased outputs, and increasing resource requirements.
Findings: Perspectives of PE Practitioners and Policy Makers
We used the framework presented in Figure 1 to direct semistructured interviews among both policy makers and PE practitioners. We conducted fourteen telephone interviews, lasting between forty and eighty minutes and involving seven PE “practitioners” and seven “policy makers.” The sample of practitioners came from members of the Public Engagement in Research and Research Engagement in Society (PERARES) project and comprised practitioners affiliated with academic institutions and/or nongovernmental organizations (NGOs). Interviews took place between June and August 2012. “Policy maker” refers broadly to civil servants with direct, inside experience of the policy-making process. Our sample comprises predominantly civil servants from the European Commission Directorate General (DG) for Research and Innovation and one former European Union (EU) and UK civil servant. This sample draws heavily on experience in PE in the process of research, which must be emphasized when considering the wider relevance of the findings. While respondents drew on their own experiences, they were asked during interviews to comment on the relationship between PE and policy making broadly, rather than in relation only to research policy. It is also worth noting that the majority of the policy maker respondents had prior experience of working within/with other policy DGs in the European Commission.
The interviews were coded using thematic analysis, according to both the framework presented in the third section and the themes emerging inductively from the data itself (Boyatzis 1998). The themes were developed through listening to recorded interviews, with relevant sections transcribed according to their use. In the following overview of findings from the interviews, we denote practitioner perspectives by the prefix “PR” and policy makers by the prefix “PO.”
Features of PE Mechanism and Approach
Scale, topic, and timing
Participants expressed the view that the relevant scale at which PE took place should depend on the particular topic under consideration and, in particular, at the scale of decision making at which that topic can be best addressed (e.g., a local topic addressed at the local level and a topic of international importance at the international level). In terms of scale, both PE practitioners and policy makers emphasized that local-level engagement exercises were more likely to have (local) policy impacts. There was also a suggestion that the benefits would be greater since there was greater contextual sensitivity and cognizance of the interaction of multiple policy objectives at the implementation coalface: To build capacity like that locally, or regionally is really where you can make a difference. And it’s there where policies are implemented … and … are [often] implemented together with other policies, that may even contradict each other at that particular level. So engaging citizens to look at these different policies together for their region, for their area, in the context in which they live, I think that’s where … public engagement could have a much, much greater impact. (PO3)
They did not suggest, however, that engagement at other scales could not deliver policy impacts. Indeed, policy makers from the European Commission stressed that for engagement to directly affect policy making at the EU level, it needed to be pan-European in nature. This suggests that the appropriateness of the scale of engagement for the issue at hand, rather than the scale per se, is the decisive factor regarding impact. Policy makers also stressed that it was easier to discern policy impacts at the local level but that this did not mean that Europe-wide engagement activities did not have impact.
Several participants emphasized the importance of conducting engagement across different scales in order to maximize the quality and impact of the process. For that reason, a number of the interviewees praised the PERARES project for addressing scalar problems by integrating Science Shop style engagement at the local level with online debates at the international level. 1 The participants highlighted that engagement that dealt with controversial topics already under public scrutiny are more likely to influence policy makers. It was also argued that this type of issue should probably have been addressed by PE earlier in the policy process to prevent the escalation of controversy in the first place. The interviewees stressed that how a topic is framed is important for ensuring policy impact (PR7 and PO2). PE practitioners therefore have the responsibility to communicate the relevance of their PE work in a manner that is relevant to the policy context.
Respondents also pointed out that the timing of engagement in relation to the cycle of policy development is critical (PR2 and PO3). They highlighted that longer term engagement activities are more likely to lead to policy impacts through a slower process, as “numerous small interventions” lead to a critical mass of PE-derived evidence. The short-term nature of project-style PE was recognized, particularly by policy makers, as an impediment to this (PO2 and PO5)—an idea that supports the need for engagement to be seen as part of a continuous process.
There are various ways in which the topic, scale, and timing of engagement interact that can have a bearing on policy impact. Ultimately, policy impact will be heightened when the topic, scale, and timing of engagement are optimized based on the policy-contextual awareness of those commissioning and undertaking the PE.
Monitoring and evaluation
Both policy makers and PE practitioners suggested that there need to be better tools (quantitative and qualitative) for PE practitioners to monitor the policy impacts of their activities (PR2, PR7, and PO4). One PE practitioner pointed out that although they had seen a discernible policy change following a recent PE, they had absolutely no way of knowing to what extent their activities had led to this policy change (PR4). This concern gets to the heart of the evaluation problem. Practitioners need to trace their outputs for policy impact, and monitor and evaluate the use of information derived from PE once it has entered the policy realm. Access to this realm on the part of PE practitioners remains a considerable problem and highlights the need for greater transparency and monitoring within policy-making institutions themselves (see Features of Policy Making subsection).
Approach to PE and perceived legitimacy
The interviewees did not agree that one approach or mechanism adopted to undertake PE was any more likely to have a policy impact than another. They stressed, however, that the perceived credibility of the approach in the eyes of policy makers had an important bearing on the uptake of PE-derived evidence (PO2). Furthermore, there was a sense from the policy makers that the limitations of PE outputs are not sufficiently communicated to allow them to make a judgment on its credibility as a source of information (PO1 and PO7): There is [sic] so many uncertainties in the way it is done, in the methodologies, in the who is doing it, and the how, that it is very hard to have real legitimacy of such processes. (PO7)
Policy makers recognize a need to be able to assess the reliability of opinion-based evidence alongside the other sources of information that they use to make policy decisions. PE practitioners need to communicate the limitations of their work better. Policy makers will then be able to judge the reliability and representativeness of PE as sources of evidence. One way in which the credibility of PE can be enhanced in the eyes of policy makers is through its integration into a research program. The PE outputs will then be research outputs. If we as researchers say something [like] ‘this is the result of our research, which is an EU FP7 research’, then the local decision makers cannot just simply say you are stupid, you are dumb and value-driven, and so on, which they like to say to activists, so yeah, research has, in this sense … social power, or policy-forming power. (PR4)
The PERARES project explicitly sought to increase the engagement of the public and civil society organisations (CSOs) in the setting of research agendas, and so this finding may not be surprising. Nevertheless, those interviewed had varied backgrounds and experiences with different forms of PE. There was also wide support for PE through research among the policy makers, who viewed it as a potentially more legitimate—and hence policy-appropriate—form of engagement (PO1, PO3, PO6, and PO7).
Practitioner skills and attributes
Policy maker PO3 highlighted the enormous diversity in the approaches to PE, the outputs produced and the resources committed to engagement activities commissioned by the EU. She suggested that this appeared to be largely dictated by who was responsible for undertaking the PE. Furthermore, there appeared to be no standards for consistently undertaking PE or for generating outputs from it. This was supported by policy maker PO1, who argued that PE practitioners need to be trained experts to make the process more efficient and credible in the eyes of policy makers (see also Approach to PE and perceived legitimacy subsection). The issue of perceived legitimacy was also raised by a practitioner who pointed out that because his organization was associated with an oppositional political party in his country, the outputs of his engagement were overlooked by the ruling party (PR4). There was some acknowledgment of the importance of skilled practitioners from the PE practitioner community, though it was also said that effective PE can potentially involve a vast array of different skills that any one person might not possess (PR2). Another practitioner expressed caution at the idea of professionalizing PE practice through regulation, standards, and qualifications, since she felt this would lead to a loss of the more innate personal qualities and genuineness of motive of PE practitioners that are also important qualities for successful PE (PR7).
Features Linking PE with Policy Making
Integration of policy makers and policy making into PE
Both practitioners and policy makers emphasized the need for explicit integration between PE and policy making if the policy impact of PE was to be maximized (PR1, PR2, PR3, PR6, PO1, PO2, PO4, and PO6). Policy-commissioned or policy-driven PE exercises were identified as the most likely to lead to discernible policy impacts: policy makers would have a known and direct interest in the outcome of the research, and the engagement would have been framed according to the policy context. Practitioners tended to suggest that it was important for PE to be policy driven (PR2 and PR6), whereas policy makers stressed the importance of PE being policy initiated. In other words, policy makers appeared to be more likely to take heed of the outputs of an engagement exercise that had been undertaken at their request and within the appropriate time required to fit it into the policy-making cycle associated with a specific issue. One policy maker, for instance, suggested that even if PE had been already undertaken with respect to a particular issue it would be likely that the EU would want to commission its own PE, according to its own terms, if it was deemed necessary for the policy process (PO1).
The direct involvement of policy makers within the PE activity itself was identified as a way for policy and PE to be better integrated. This could be achieved through direct face-to-face involvement with the public during an event (depending on the PE approach taken), or through involvement in an advisory or steering group to ensure the relevance of the PE to the policy process. Both practitioners and policy makers were generally supportive of this idea in principle, though the practical limitations (e.g., in terms of time commitment or other resources) were identified by policy makers as potentially problematic. One practitioner, who conducts Science Shops
2
that involve municipal policy makers on a support committee, emphasized that it was important to involve policy makers who are positive about the benefits of PE: You mainly get the ones who are interested in public engagement, and the ones who are not interested—I’d rather leave them out. You need people who are interested or enthusiastic about this and then they can try to make the others in their own organization interested; it’s [easier] for them, than for me to do that. (PR3)
Policy maker PO4 provided an example of an engagement process that directly involved policy makers in order to have a meaningful impact on policy. The project involved face-to-face interaction among a range of stakeholders and policy makers, influenced policy, created common interest among disparate stakeholders, and established relationships between stakeholders and policy makers that outlasted the project. This is important, given the problems identified with the finiteness of the “project” approach to PE (see Scale, Topic, and Timing subsection).
The informal interaction of PE practitioners with policy makers
Respondents suggested that informal interaction between policy makers and PE is potentially more influential than formal interaction. They identified as important the ability of PE practitioners to engage with policy makers, forge relationships, and communicate with them in an appropriate fashion.
Practitioner PR7 drew attention to what she called “the soft end of public engagement,” which, as a practitioner, “you can’t necessarily put in your annual targets … and be evaluated against.” For her, this entailed interacting with policy makers in informal settings where it is possible to “capture hearts and minds” rather than putting something on their agenda. She emphasized the importance of the practitioner acknowledging such activities and making time to engage in them: The [PE] initiatives that were more successful often were where they placed slightly more of a priority on doing that informal policy-work, and … I think it’s one of those things too, that if you don’t … properly build it in and believe in your own mind that it is genuinely a part of the work, … you get caught up with something else and you … don’t go along to that conference or you don’t go along to that meet and greet, or … you miss the spaces where you could be doing that work. (PR7, emphasis added)
She also said that such practices might be more important than having a formal interaction that was not on the terms of the policy maker, or might not be the best moment to influence them: You could fire a policy maker into those kinds of things [a PE event] and if they were the right person that would be grand and it could be somebody else who felt that they just had to be there, and … you know, has three things on their desk that they’re trying to finish and they’ve been deputized and sent there by somebody and … they’re not in the right frame of mind. Whereas actually if you went along to an event that they were running and just happened to have a conversation with them for two minutes you might actually get more out of that than looking at something for a full day that you were running. (PR7)
Policy maker PO2 stressed that personal relationships with policy makers can be built, which also facilitates the building of trust and likelihood of PE practitioners “being listened to”: I’ve always said … really what it needs is the [practitioners] to actually find a way of talking to the policy-people in the departments, … so getting to know them, talk to them, find out how they do things and then … if there’s somebody you know and you’ve got a question: you for example, if I’m working in a government department and you know me … you can say to me ‘look, I’ve got this idea, have you ever thought about this’, and while they’re doing this consultation I might think ‘oh that’s a good idea’—but if I didn’t know you I probably wouldn’t—so its contacts. (PO2)
Moreover, by building personal relationships with policy makers, practitioners can understand better and more quickly how the outcomes of PE need to be translated and communicated in an appropriate fashion for their assimilation into policy making. This is because by being in their “midst … you can pick these things up much more easily”: I mean for policy things it’s got to be pretty brief normally … if you can summarize something—almost like an abstract but a bit more punchy, then that’s a good way of presenting it, and that’s the way it tends to get across but it always works better if you’ve got personal contacts to do it with, I think because otherwise you send a report to a government department and it goes straight on a shelf. (PO2, emphasis added)
The informal interactions between practitioners and policy makers need to be given greater attention by PE practitioners. The need for PE practitioners to be more politically aware and networked in order to facilitate the communication of PE outputs into a policy useable and trusted format is also emphasized by these observations.
Features of Policy Making
The nature of policy making and the political procedures dealing with PE
Many of the interviewees argued that the nature of decision making, the nature of political institutions, and the nature of political procedures represent a barrier both to realizing PE impacts and to being able to delineate and monitor those impacts (PR3, PR4, PR7, PO1, PO2, PO3, and PO6). For example, decision making is based on so many different factors that it is difficult to know and monitor the extent to which a PE process has influenced decisions: Policy-makers at the end of the day, will make a political decision, of which scientific evidence is one factor, public opinion is another factor, economics are another factor, pure politics is another factor … and so on. And so there’s all those things and you can see it in different situations there comes a judgment as to which is most important. (PO2)
Furthermore, policy makers often operate under incredible pressure with insufficient resources to utilize all of the information that is available to them. Even if the political will to undertake PE exists, insufficient resources could limit the actual influence of PE outputs (PR7 and PO3). The temporal nature of much policy making, and the turnover of policy staff were also identified as potential barriers to the realization of policy impacts. Respondents argued that much policy making is still reactive and conducted in a short time frame, which may prohibit application of PE in the time available before policy decisions are needed (PO1). This also means that PE that is not policy-commissioned needs to be well timed in order to coincide with a relatively short window in which information is assessed in advance of a policy decision (see Scale, Topic, and Timing subsection). In contrast, where policy making is a longer iterative process, the relatively rapid turnover of staff in policy-making institutions may lead to a lack of continuity in the relationship between PE and policy makers, and make it difficult to track the impact of PE outputs when various policy makers have been responsible for policy development (PO3). This also relates to the involvement of many different people with different roles and perspectives in the decision-making process (see Auditing and Monitoring of PE-derived Evidence in Political Institutions subsection).
Policy-maker attitudes and motives
Several interviewees highlighted that realizing policy impacts from PE can be inhibited by the motives and attitudes of individual policy makers. In view of the diversity of evidence that they must consider, in many cases policy makers simply pick and choose what they want to take from the available evidence (PO2); an individual, therefore, can be very influential in terms of the evidence that gets used in policy (PR6). Within the European Commission one policy maker argued that: The mainstream thinking is that public engagement can hamper scientific excellence … or could hamper innovation. (PO4)
She went on to argue that such thinking is changing and, in particular, the work of the Science in Society work program in DG Research and Innovation argues vehemently for a view of PE as a process that “enriches excellence” and “promotes innovation.” In other situations, however, several practitioners and policy makers argued that public opinion often outweighs the evidence provided by science.
Although differences in individual attitudes toward PE are important, the interviewees suggested that their significance in light of other issues is not overriding. One policy maker from the EU Commission argued that the attitudes of policy makers is less important now, since the requirement for PE is built into European legislation (PO1). Both practitioners and policy makers in the Science in Society work program of DG Research argued that they were seeing a genuine and positive shift in attitudes toward PE. This suggests that the practical constraints placed on policy makers (see The Nature of Policy Making and the Political Procedures Dealing with PE subsection), and the need to engage with them in the right way and at the right time (see The Informal Interaction of PE Practitioners with Policy Makers subsection), are more important than policy makers’ attitudes per se. There is still a lack of consistency across different policy areas in the triggers for (and methods and means of) assimilating evidence from PE. This suggests that greater consistency and procedural standardization within policy processes might be more important than differences in attitude between individual policy makers.
Auditing and monitoring of PE-derived evidence in political institutions
The lack of institutional procedures for monitoring and reporting on the use of evidence in the decision-making process may be important (PO1, PO2, PO3, PO5, PO6, and PO7). One policy maker argued that this is partly because there is no record of how evidence gets used and reinterpreted as it passes from individual to individual and between different policy realms and institutions: The Commission is only part of the story … the minute that a policy document enters the inter-institutional context—with the Parliament and with the Council—then … it’s nothing but a black box … it’s very difficult to trace then, why were certain words changed or why were certain sentences dropped or replaced with others and there’s very little traceability and that’s, the traceability of the evidence is something that I think we really need to work on. (PO3)
Policy maker PO3 also suggested there is a lack of up-front accountability on the use of evidence in the assessment of policy. She pointed out that the policy impact assessment process used by the EU is the place where the evidence used should be clearly on display, but a review she undertook found that, in most cases, no reference was made to the evidence used to make decisions. Another European policy maker suggested that the impact assessment itself was not the problem for evaluating the use of evidence, but rather its timing. He argued that the impact assessment is usually required so early in the policy-making process that it does not account for the changes that take place during subsequent policy development (PO7). Moreover, there was no system in place for retrospectively looking at past policies, and how evidence was, or was not, integrated into them. However, even when information from PE is considered, but not taken up by policy makers, PE is not precluded from having a policy impact. In the interests of fairness and legitimacy, it is essential that what might be justifiable reasons for not using the evidence from PE in policy making are made transparent.
These findings suggest that there is a need for better mechanisms to identify the use of evidence in policy formation in the first place (e.g., at the impact assessment stage); better traceability of how evidence is used/dropped as it moves through the policy-making process; and greater attention to retrospective analysis of the use of PE in previous policy developments.
Discussion and Conclusions
Figure 2 summarizes the key features of PE mechanisms, policy process, and the links between them. These links will be important if the impacts of PE on policy making are to be maximized. We recognize that this does not present a comprehensive set of criteria for evaluating engagement exercises and, indeed, there will be different types of engagement and different reasons for undertaking engagement that will place a greater or lesser degree of significance on the attainment of policy impacts. The features we present in Figure 2 principally include those features that have been identified as increasing the likelihood of impacts arising. In practice, policy impact will depend on a combination of these factors, with some factors being more important in some situations than others. Equally, there may well be a range of other issues over which PE practitioners, and even policy makers, might not have control over. A focus on “likelihood” recognizes this uncertainty and suggests that making PE “policy resonant”—inasmuch as it attempts to preempt likely future outcomes—may be a more realistic objective. However, included within the list of features in Figure 2 is a range of measures that would facilitate the capture of information on discernible policy impacts, which are absent or lacking from current methods and procedures. While emphasis has been placed on the attitudes and motives of policy makers in realizing impacts from PE, we suggest that this detracts attention from the lack of appropriate measures in place to monitor and evaluate the use and uptake of PE-derived evidence within the policy realm. Attitudes toward PE among policy makers may be less significant now than they were ten to twenty years ago, and, in the first instance, there is a need to pay greater attention to the means of tracking and evaluating impacts in the policy realm.
Features to maximize and/or monitor the impact of public engagement on policy making.
The lack of attention to political procedures has focused attention on individual attitudes among policy makers. In contrast, the emphasis on procedures within the evaluation of approaches to PE has downplayed the significance of the role of individual practitioners in facilitating the relevance of PE to policy making. We have highlighted the importance of the informal work of PE practitioners in the “in-between spaces” of engagement: that is, the efforts of practitioners to establish relationships and build trust and open communication channels with policy makers. This has several interrelated benefits, including enhanced trust and perceived legitimacy between the parties, enhanced awareness of the policy processes and constraints on the part of the practitioner, ways of tracking outputs from PE within the policy realm, and development of PE as part of a long-term engagement with policy. In building lasting relationships with and between policy makers and the public, it is clear that the skills and abilities of practitioners to engage and interact informally are as important as their skills in organizing formal PE procedures. These more informal skills have tended to be overlooked. The findings from this research point to a need for greater recognition of the role of practitioners and for further research on their work in the more hidden, in-between zones of PE.
As mentioned, the factors presented in Figure 2 cannot be taken as a formulaic set of ingredients that will automatically give rise to greater policy impact. There is a fundamental question raised regarding the extent to which greater policy resonance equates with “better” PE. Possible incompatibilities between traditional measures of PE effectiveness (Rowe and Frewer 2004) and the imperative to seek policy impacts need to be considered. Legitimacy in the eyes of policy makers is likely to enhance PE’s impact on policy making. However, it is important to note that striving for policy legitimacy is not at odds with the aims of the PE itself. PE practitioners have a duty to ensure the legitimacy of PE in terms of its representativeness and political neutrality. Deliberative democracy has not taken the politics out of policy, but it is important that its proponents ensure that PE does not become a political tool. Assessment of successful policy impact is not a criterion that can simply be tacked-on to existing evaluation approaches. Instead, policy impact needs to be judged in its own right and weighed up against validated criteria for assessing the effectiveness of PE. This also means that the capacity of PE practitioners to affect change in the policy realm has to be recognized as limited. It is conceivable, for example, that a practitioner could operationalize all of the factors in Figure 2 that are within his or her remit, and have no bearing on policy outcomes.
We suggest that the onus of responsibility for maximizing the policy impact of PE rests with political institutions. This requires the implementation of the necessary procedures within policy making to increase the transparency of decisions.
For a long time, PE mechanisms have incorporated well-established and formalized evaluation criteria and methods, but political processes have not. There is a gap between policy impact procedures that typically take place early in the policy formulation process and policy evaluations that typically occur after a policy has been implemented. What is missing is an audit of the final decision-making process—a process which often involves last-minute modifications and compromises that are not reported. Such an audit would shed light on how decisions are made, including under what circumstances evidence is or is not taken up and used. This offers greater potential for identifying more concrete relationships between PE practices and policy. While this may be associated with politically sensitivity ground, and will require additional resources, the benefits include greater accountability in the eyes of the public and the provision of a means to evaluate and improve PE processes to maximize their benefits. PE will then lead to better political decisions.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors acknowledged that this study was undertaken as part of the PERARES project (Public Engagement with Research and Research Engagement with Society), which received funding from the European Community’s Seventh Framework Programme, contract number FP7-SiS-244264. The views and opinions expressed in this publication are the sole responsibility of the authors and do not necessarily reflect the views of the European Commission.