Formalization and separation: A systematic basis for interpreting approaches to summarizing science for climate policy

Formalization and separation: the canonical view, the STS critique and beyond

It is important to be clear at the outset that there are prima facie arguments for and against formalization and separation. Starting with separation, an initial impulse might be to argue that scientific expertise ought to be shielded from contact with the policy world until the best scientific view is established; only then should the two sets of actors be brought together. On this view, separation promotes scientific objectivity and disinterestedness. However, it has long been recognized that policy actors are more likely to pay attention to those whose work they know and trust; moreover, scientists who have close contact with the world of policy may be expected to know better how to communicate effectively with policy audiences (Macgill, 1993). These considerations imply that separation may be self-defeating, cutting scientists off from their policymaking peers. Hence, there appear to be good reasons for promoting as well as for opposing separation.

The situation is similar for formalization. Formalistic approaches have the apparent benefit of being more uniform, reliable and universalistic. The use of measuring instruments and formal tools of other kinds is widely taken as a warrant of objectivity across the spectrum of academic disciplines, as well as in many fields outside academia. On the other hand, it is recognized that formalization has limits: not all knowledge can be formalized to the same degree, especially not in novel or unanticipated situations. Indeed, it might be more expedient for policymakers faced with non-routine phenomena to turn to the expert judgement of an experienced scientist than to a regular scientific professional following a codified procedure.

It is into this complicated terrain, where good arguments are offered for as well as against high and low levels of formalization and separation alike, that we tread when exploring these balancing acts. Analyses of science for policy often oscillate between strong and competing contrasts, disregarding, even concealing, the vast area of possibilities that opens up once the two dimensions are combined. Even though individually the dimensions are well known to every scholar interested in science for policy and to every practitioner involved in synthesizing science for policy purposes, we propose a way to bring them together in an explicit fashion. However, we first elaborate the two dimensions, giving examples from important theoretical positions evincing a strong attraction to the endpoints of each axis.

Starting with the dimension of separation, the established science policy literature often defends the integrity of science, rehearsing arguments of disinterestedness and impartiality (Yearley, 2005: 161). Over the last two decades, drawing mainly on studies of science and environmental policymaking, political scientist Peter M. Haas (2007) has made strong claims of this kind. Lately, Haas has turned to new forms of empirical evidence to advance his view that the scientific standpoint should be kept apart from policy deliberations until it has reached a certain stage (Haas, 2007; Haas and Stevens, 2011). Prior to that stage, on this view, the institutional framing of the science–policy interplay should be characterized by separation. Haas argues that, in order to be effective and influential, science needs to be detached from policy when setting its research agenda and when establishing consensus. Communication with policymakers should not start until consensus is reached. Consensus established among experts separated from political demands is what allows science to ‘speak truth to power’ (Price, 1965). What commonly happens, in Haas’s view, is that science connects to the policy process at too early a stage, before scientific consensus is reached, with the result that science loses its autonomy. According to Haas and Stevens (2011), the work of the IPCC is a clear example of this. When separation between science and policy is lost, scientific controversies are fuelled and science loses much of its value. However, Haas has little to say about formalization within the scientific process. He contents himself with suggesting that consensus be built through a deliberative and autonomous peer review process carried out by recognized experts (Haas and Stevens, 2011).

The canonical image of science also involves methodological rigour, which is widely regarded as the hallmark of high-quality science (Bohlin, in progress). This applies equally to the practice of individual scientists and to large-scale scientific assessments such as those conducted by the IPCC. Whatever the scope and scale of the work, the reliability of the outcomes is assumed to depend on the degree to which formalized procedures are adopted. The broad and highly influential ‘evidence-based’ movement is a case in point. In medicine, where it originated, systematic reviews are the chief vehicle used to synthesize primary research findings. The methodology of systematic reviews is highly formalized, specifying procedures for identifying publications relevant to a given issue, assessing the quality of the studies thus identified, extracting data from the material selected and so forth (for the history of this methodology, see Bohlin, 2012; Chalmers et al., 2002). Most organizations and agencies conducting systematic reviews rely on teams of reviewers representing academic expertise on the methodology employed and the subject matter addressed. This practice instantiates separation in the sense recommended by Haas. Although steps are sometimes taken to integrate potential users, this is a subordinate theme within the evidence movement, which implies an attraction to the higher endpoint of the dimension of formalization.

It is hardly an exaggeration to suggest that STS scholars focus on the dimensions of formalization and separation as a constitutive part of their field. This is mostly done in a highly critical manner. Formalization and separation are portrayed as simplifications concealing the complex nature of scientific practices. For decades, Harry Collins (1975, 1992, 2010) has questioned what he refers to as the algorithmic model of science, claiming that explicit rules cannot remove the need for skilled judgement. The locus of knowledge is not formal rules but a ‘community of expert practitioners’ (Collins, 1992: 159). The meaning of rules is culturally contingent, and hence learning them means acquiring a cultural competence. This line of research clearly manifests the STS agenda of deconstruction, defined by Sheila Jasanoff (1999) as ‘the analytic process by which scientific claims are disentangled into their material, social and rhetorical components’ (p. 65). Behind its self-serving appearance of rigorous rule-following, for those who look carefully, scientific practice is characterized by informal, enculturated expertise.

There is, however, STS research that focuses on strictly formalized procedures as an important and productive way of doing science. The works of Daston (1992) and Daston and Galison (1992, 2007) are exemplary in this regard. Historicizing notions of objectivity, they demonstrate that ‘mechanical’ objectivity differs markedly from the ideal of producing ‘true’ representations of nature. Mechanical objectivity is pursued through strict protocols designed to eliminate the distortion supposedly produced by subjective judgements. Although the ideal cannot ever be fully realized, the effects of efforts to ensure that knowledge production conforms to it are tangible. A similar pattern is demonstrated by Porter (1992, 1995), who also employs the concept of mechanical objectivity. His work is primarily focused on historical cases of bureaucracies subjected to external pressure, whose response was to legitimate decisions by referencing formal rules and procedures (Porter, 1992, 1995). Acknowledging a higher degree of formalization than that represented in STS analyses of expert judgements, this view of the significance of procedures aligns with our focus on formalization. We also find support for this way of thinking in Actor–Network Theory (ANT), which focuses on delegation to material entities such as inscriptions, standards and other formal tools that fulfil a crucial function in any production of representations of natural or social phenomena (for instance, Latour, 1999; Timmermans and Berg, 2003). The contrasting positions discussed here, one stressing the indispensability of informal judgement, the other exploring the basis and effects of formalization, are explicated by Sismondo (2010: 136–147), who dubs them the ‘humanist’ and the ‘formal’ model.

The STS ambition of presenting a more accurate and complex picture challenges taken-for-granted notions of scientific autonomy and the separation of science and policy commonly lauded by scientists and actors outside the scientific community. STS analysts tend to argue that science is always shaped by political and policy considerations. The credibility of scientific propositions is invariably the outcome of activities of interested parties, and behind ‘pure’ scientific results we find links between a multitude of actors (Latour, 1987). Such an approach, too, is part of the deconstructivist agenda. Furthermore, the concept of co-production foregrounds the close connections between scientific knowledge and policy (Jasanoff, 2004). Separation is commonly taken to be emblematic of an over-simplified way of describing a complex relationship, and in addition refers to technocratic arrangements devised to solve social problems by the use of expert knowledge alone (Jasanoff, 2003; Wynne, 2001).

Co-production is a complex concept, which sometimes connotes little more than an interplay between knowledge and social order. However, an area that might be termed ‘boundary studies’, which often prioritizes the idiom of co-production, is highly relevant to our concerns. Mobilizing science for policy outcomes, boundary organizations are characterized by their proximity to both scientific institutions (for credibility) and to political institutions (for legitimacy) (Guston, 1999, 2000). Drawing on Gieryn’s (1983, 1999) seminal notion of boundary work, the performance of boundary organizations is analysed for ‘what works and what does not work’ (Miller, 2001: 496). Rather than maintain prior commitments to concepts of how science and policy should be organized, this approach focuses on how actors use resources to protect the authority of their work. In efforts aimed at synthesizing research for policy purposes, for instance, what is regarded as belonging to the scientific and the policy realm is contingent; the boundary can be drawn in different ways. In studies of international organizations, such as the IPCC, Miller (2001) extends the notion of boundary organization into a more general understanding of hybrid management. This is motivated by the more complex situations in which international organizations navigate while not being able to lean on well-defined established institutions of science and politics that often exist in national contexts. ‘[B]oundary organizations need to be able to manage hybrids – that is, to put scientific elements together, take them apart, establish and maintain boundaries between different forms of life, and coordinate activities taking place in multiple domains’ (Miller, 2001: 487). Studies of boundary organizations performing balancing acts between science and policy are congruent with our way of understanding the dimension of separation.

So far, we have explicated the two dimensions of formalization and separation, arguing that they are important for the public image of scientific practice, for informing many practical arrangements, as well as for understandings in STS of scientific practice and the science–policy interface. In addition, we have argued that the two dimensions are usually addressed separately; the above exposition is no exception. However, an important aim of this article is to analyse the dimensions together. STS provides a good starting point for this aim. In this field, as we have seen, issues relevant to formalization and separation are frequently analysed empirically and in conceptual terms, often in a highly critical manner, though, as presented above, there are alternative lines of research. We now turn to studies in which connections are explored between the dimensions. While there are interesting examples of this kind, this work can be improved by making explicit use of the two dimensions of formalization and separation and systematically analysing ways in which they are correlated. For example, in an insightful discussion of balancing acts performed by engaged experts between technocratic and democratic ideals in US regulation (a typical study of separation vs integration), Jasanoff (1990) concludes that formalized knowledge is suited to satisfying separated technocratic ideals. Jasanoff demonstrates, in other words, a possible connection between a high degree of formalization and a high degree of separation. Numerous studies by Collins (1992, 1998) on gravitational research exemplify a different kind of connection. He demonstrates that, in a small and well-defined group of highly qualified scientists, key decisions rely entirely on informal judgements. In our terms, Collins, contrary to Jasanoff, links a high degree of separation to a low degree of formalization. The connections Porter (1995) makes go both ways: he claims that quantification, as an example of formalized procedures, can secure authority and protect autonomy (separation), but also that autonomy can allow informal judgements. These examples suggest that not only are the two dimensions important topics, but that positions along one dimension may be connected with positions along the other.

Jasanoff, Porter and Collins all make roughly causal claims on connections between what we call formalization and separation. Although the practices addressed in their studies are very different, it should be clear that such connections can be found in well-known STS work. ¹ However, the connections are not well developed, and STS research relating to climate change is no exception. Miller (2001), for instance, has investigated how expert advisory committees in the international climate regime are established as organizations (separation), as well as how these organizations develop standards and procedures, including methods for measuring greenhouse gas emissions (formalization). While discussing both as examples of hybrid management, he does not clearly distinguish between the two dimensions. Here and elsewhere, we believe that the analysis can be taken several steps further by considering the interaction between formalization and separation.

This introduction to formalistic and separated initiatives and to the idea of combining the dimensions illustrates what Latour (1993) calls ‘processes of purification’, that is, an attraction to the endpoints of the two dimensions (pp. 10–11). STS scholars often challenge processes of purification, as exemplified in the canonical image of science. To date, however, focus has been on deconstructing separated and formalistic initiatives rather than on recognizing the possible productiveness of such efforts. Hence, the endpoints of the dimensions are lent support while the possibilities afforded by the space between them largely remain unexplored. We demonstrate the need for a more elaborate understanding of science for policy, taking advantage of the possibility of studying formalization in conjunction with separation. Although frequently adumbrated in STS work, this theme has not been developed in a systematic manner. As mentioned above, there may be good reasons for separation and formalization as well as for integration and allowing space for informal judgements. Crucially, there is far more than the purified endpoints for science-based policies to lean on.

Science for policy: three climate examples

A key theme of current discussions on climate change concerns ways in which scientific knowledge can be put to use; without references to climate research, climate policies would lack all legitimacy. Yet there is no consensus concerning appropriate mechanisms for determining what relevant evidence is, let alone for implementing it. In this section, we discuss three examples of how climate research is synthesized for policy purposes. ² One example is the IPCC; the others are drawn from the European Union (EU) and the United Kingdom. Together, they illustrate the myriad initiatives taken recently, as well as the strong agreement on the importance of devising smarter ways of making use of science in the policy arena. The IPCC is widely recognized as the biggest and most important initiative for summarizing climate research for policy. The second example, of a research repository designed for and made available to policymakers in the EU, is part of the long-standing tradition of secondary literature in science that has exploded in the digital era. The third example illustrates the classical manner of using expertise in policymaking, namely government nomination of a committee of experts. The most important consideration in our selection of examples is that they bring theoretically salient differences into relief, as they exemplify distinct positions along the two dimensions of formalization and separation. Other comparisons might be relevant in other contexts; ours is restricted to their respective positions along the two dimensions.

IPCC

The most prominent example of expert evidence for climate policy is the IPCC, which was established in 1988 by the World Meteorological Organization (WMO) and the United Nations (UN) Environment Program ‘to provide the governments of the world with a clear scientific view of what is happening to the world’s climate’. ³ Since its first Assessment Report in 1990, the IPCC has continued to release reports (AR5 was published during 2013 and 2014) supporting the work and negotiation processes under the UNFCCC.

The IPCC is organized as a network of scientists, whose core activity is to assess and synthesize published, peer-reviewed research. None of the scientists are hired by the IPCC, and the IPCC does not conduct research of its own. A small secretariat coordinates the effort. The highest decision-making organ is the Panel, mostly consisting of government representatives from the member states of the UN and WMO. The Panel makes major decisions on, for instance, the scope and outline of IPCC reports, the structure and mandate of IPCC working groups, IPCC principles and procedures, election of the IPCC Chair, as well as approval, adoption and acceptance of reports (IPCC, 2013; see also Agrawala, 1998; Bolin, 2007; Skodvin, 2000).

Concerning separation, it is important to note that the IPCC and the UNFCCC had to work out a division of responsibilities, roughly speaking between knowledge about climate change (IPCC) and how to negotiate international agreements (UNFCCC) (Miller, 2004). The ostensible purpose of the IPCC is to provide clear, neutral and unbiased knowledge: ‘The work of the organization is therefore policy-relevant and yet policy-neutral, never policy-prescriptive’. ⁴ It employs a mixed-mode approach in which government representatives (together with some scientists) decide the scope of the assessments and review the results before publication, while scientists independently prepare the first and second draft reports. The second draft is then reviewed by both scientists and government representatives before scientists prepare a final draft.

The dimension of separation is clearly illustrated in Figure 1, indicating that the IPCC framework may be placed somewhere in the middle of this axis. The space allowed for interventions by scientists and government representatives, respectively, in successive stages of the process, is specified in detail. The autonomy of scientific experts is balanced against the political applicability of the reports. Some scholars have suggested that the IPCC assessment process ‘is unique in the intergovernmental arena in its combination of expert scientific analysis with government review and negotiation’ (Yamin and Depledge, 2004: 475).

OPEN IN VIEWER

Figure 1.

Illustration of the IPCC writing and review process. The green (lighter shading) boxes in the upper part of the figure signify that the process is dominated by government representatives, while blue (darker) areas signify that scientists are in control of the process. The diagonally striped box represents a more even balance between the two.

An important characteristic of this model is that government representatives are involved in the work of producing science for policy from the earliest stages of the process. The motivation for this arrangement is to increase policymakers’ ownership of the reports and to secure relevance and legitimacy, as well as to establish a common scientific ground for the international climate negotiations under the UNFCCC. Some scholars argue that this hybrid model needs more separation (Haas and Stevens, 2011), while others argue that it evinces too much (Jasanoff, 2010).

Concerning formalization, the assessments carried out by the IPCC are guided by a comprehensive set of procedures (e.g. IPCC, 2013). Over the years, the IPCC has developed a sophisticated process-cycle for producing the reports. One important issue concerns the selection of authors. After the Panel approves the overall outline, governments and organizations nominate candidates to work on the reports. Most nominees have a strong academic record, but it is possible to nominate experts from outside academia. The final decisions on authors are generally made by the relevant Working Group/Task Force Bureau. The members of the Bureau, again, ‘should have appropriate scientific and technical qualifications and experience relevant to the work of the Bureau, as defined by the Panel’ (IPCC, 2011: 3). A substantial number of authors are replaced after the publication of each Assessment Report. Altogether, several thousand scientists are involved in preparing each IPCC Assessment Report. In AR5, for the Working Group 1 report alone, published in 2013, some 800 scientists are credited as coordinating lead authors, lead authors, review editors and contributing authors, with a further 1100 as expert reviewers. It goes without saying that coordinating this exercise requires a considerable effort.

The formalization involved, however, is of a specific kind. The rules and guidelines are intended to regulate the selection of actors and specify the responsibilities of review editors, lead authors and other personnel involved in the process of producing the reports. Concerning the actual writing of draft chapters, on the other hand, official IPCC documents offer little guidance. This part of the process is at the discretion of the scientists involved. No instruction for the processes of identifying, evaluating and bringing together data from separate studies is specified in IPCC documents. The quality of this work relies, apparently, on the kind of informal judgement that is a key component of enculturation in scientific communities.

The nature of the formalization of the IPCC peer review process may be defined in terms of a well-worn STS distinction, that of social versus cognitive aspects of knowledge production (e.g. Collins, 1992: 132). Social aspects of the working routines adopted are highly formalized by rules set out in official documents, whereas the degree to which cognitive aspects are formalized is far lower. However, in addition to guidance notes addressing the selection of authors, the responsibilities of review editors, and so on, a framework for specifying and communicating uncertainty has been developed. Recommendations on ways of describing uncertainty, an issue which clearly concerns the substance of the knowledge claims made, were first proposed in 2000 (Moss and Schneider, 2000). The treatment of uncertainty has subsequently been addressed in separate series of guidance notes, and the IAC (2010) report devoted a chapter to this issue. The most recent IPCC guidance note on uncertainty provides formal tools, including an instrument requiring author teams to specify the type, amount and consistency of the evidence available, as well as the extent to which experts agree on interpretations of this information (Mastrandrea et al., 2010: 3). Determining the degree to which IPCC reports are formalized is not, therefore, a straightforward matter. Social aspects of the procedures involved are formalized to a considerable degree, and increasingly so. The cognitive components of the assessment process, on the other hand, rely on a mix of formal techniques and informal routines embedded in the cultures of specific research communities. Just as with separation, the IPCC framework may be placed near the middle along the axis of formalization. But an increasing degree of formalization should be noticed, the documents specifying the procedures by which the assessment is carried out having gradually become ever more detailed over the years. The gist of the IAC (2010) report was, and the IPCC response to its recommendations has been, increased formalization.

UK – the Committee on Climate Change

Turning to the national level, the United Kingdom offers an informative case. In 2008, the Gordon Brown–led UK government passed the Climate Change Act, committing the United Kingdom to cut CO₂ emissions by 80 percent by 2050, in accordance with the most ambitious recommendations from the IPCC and in line with EU policy. In addition, the Act established a permanent, independent advisory body of experts, the Committee on Climate Change (CCC). According to the CCC corporate plan, its overall strategic objective is to give advice to the government on emission targets and carbon budgets, and to report on progress towards these targets. In addition, the CCC is able to conduct independent research and analysis upon request from the authorities, and engage with society to share research and invite input to their analyses (CCC, 2012). It is clearly stated in the Climate Change Act that the members need to have extensive competences in a variety of fields, including climate science, climate change policy, technology development and diffusion, and economics. ¹⁰

The Climate Change Act is specific in giving the CCC a key role in relation to policy. Thus, before ‘laying before Parliament a draft of a statutory instrument’ amending the 2050 target or the baseline year, the ‘Secretary of State must obtain, and take into account, the advice of the Committee on Climate Change’. ¹¹ The paragraph continues, ‘If the order makes provision different from that recommended by the Committee, the Secretary of State must also publish a statement setting out the reasons for that decision’. ¹² The CCC is thus a key means by which the government ties itself (and demonstrates that it ties itself) to taking this future target seriously.

In this case, therefore, expert advice is given a very strong position in policymaking. It is quite common that expert advice informs policy discussion, but less common, at least in climate policy, that the government must take expert advice into account to comply with legislation. This requirement should be understood against the background of the effort of the

UK central government, under incoming Prime Minister Blair from 1997, […] to develop a more coherent approach to policy development, championing evidence-based policy as a major element in developing fresh thinking and increased policy capability as required by a reformist government. (Head, 2010: 79)

It is, however, worth noting that the focus in this example is not on ways of summarizing research, on formal guidelines or research repositories. In this case, the function served by such resources is delegated, instead, to a small committee, representing science (but also related to policy and business), and their un-formalized judgements. A small group of eight experts, the CCC is expected to solve the problem of summarizing research and give advice on how to use it. No formalization is proposed for either of the two steps. This appears to contradict the administration’s aspiration for evidence-based policy, the backbone of which is the formalization of syntheses of scientific evidence for policymaking. Regarding separation, the CCC seems to interact closely with policymakers. The members of the Committee are experienced in policymaking and seem to be chosen from that angle, but no criteria for their selection are given. The work of the CCC, setting goals for emissions and proposing carbon budgets, is in fact more closely related to policy than to science. The CCC is free to choose how to summarize research as well as how to propose policy recommendations and assess the political realism of its proposals. To the best of our knowledge, no requirements to make this work explicit exist.

There has long been wide agreement that the informal character of the expert committee model causes serious problems, such as the tendency that consensus established in committees unduly reflects the views of strong personalities. These limitations have provided the impetus for the development of formal consensus methods ever since the Delphi technique was pioneered in the 1950s (Linstone and Turoff, 2011). Consensus methods are linked to what we call social formalization, which is of great importance to the procedures adopted by the IPCC. Being at odds with the very notion of social formalization, the institutional model on which the work of the CCC is based also evinces a very low degree of cognitive formalization. The CCC’s location on our axis of formalization is at the low end.

The degree of informality characterizing the committee model is likely to be conducive to intimate exchanges with policymakers. In the case of CCC, such contacts are facilitated by the selection of its members from among scientists with well-established ties to policy circles; its second chairman, from September 2012, John Gummer (Lord Deben), is a former environment minister. The requirement of the Climate Change Act of 2008, moreover, that government decisions that override CCC recommendations be carefully justified, suggests that policymakers have to pay close attention to the advice offered. However, paying close attention to advice need not mean that it is embraced or heeded. Proposed carbon budgets and the technologies used for electricity generation are fiercely debated in the United Kingdom. ¹³ Because the work of the CCC is closely linked to policy, and its members are experienced in working closely with decision-makers in policy and business, the level of science–policy separation must be deemed low in this case.

The analytical potential of the two-dimensional model

The three examples presented in the previous section illustrate different ways actors synthesize science for policy purposes. So far, we have argued simply that the three initiatives can all be located in a two-dimensional space defined by degrees of formalization and separation. We concluded that the analytical discussion about science-for-policy tends to oscillate between very high and very low expectations of what can be achieved by formalization and separation; this is a kind of flip-flop discussion (cf. Collins and Pinch, 1993), exhibiting a strong attraction towards the purified endpoints while neglecting most of the space opened up by our two-dimensional diagram. The fact that many climate initiatives are taking place at different locations within this vast space, as well as their movement across the space over time, needs to be better understood. Analysts’ preference for endpoints poses an obstacle to such an understanding.

In the following section, we first map the three examples onto the diagram and then focus on the analytical potential of the model. Our aim is to make a contribution that goes beyond the routine efforts at deconstructing the false pretensions that formalization and separation frequently involve. In doing so, we draw attention to some studies that have a similar or comparable ambition.

Our three examples occupy different spots in the diagram (see Figure 2). They are close to the endpoints of the axis of separation (CCC low, EU-SfEP relatively high), as well as in the middle (IPCC). Along the axis of formalization, our examples are similarly located close to the endpoints (CCC low, IPCC relatively high) as well as in the middle (EU-SfEP).

OPEN IN VIEWER

Figure 2.

The three examples mapped onto the diagram combining the dimensions of separation and formalization.

None of our examples is located at the top of both axes. The IPCC has ambitions – increasingly so – of formalizing its procedures, but is still quite some distance away from the top end of that dimension. The IPCC is not taking advantage of available techniques representing a high degree of formalization, such as the methodology of systematic reviews and meta-analysis (why this is the case is an interesting question that lies beyond the scope of this article). Concerning separation, the IPCC is positioned in the middle, and is criticized by science policy analysts representing the ‘speaking truth to power’ model (Haas and Stevens, 2011) for not being separated enough and by STS scholars for being too separated (Jasanoff, 2010). The Climategate debate is another example of this kind of criticism of climate scientists being too intimately involved in policy matters. SfEP is located high on separation, but not at the top, since the policy relevance of the material is always considered and policy people are consulted, even though this work is overseen by scientists. On the axis of formalization, SfEP is located in the middle. In addition, there are climate policy initiatives which merit being placed low on both dimensions, as illustrated by the CCC.

Having stated that the diagram opens up an area that is often neglected or even rendered invisible by flip-flop discussions structured by an attraction to the purified endpoints of the two dimensions, we contend that there is no single ‘best’ position in the space outlined. There is no ideal Archimedean position that suits all kinds of questions or all varieties of policy users, no position to which all organizations and initiatives should gravitate. Rather, the identification of the ‘right’ position is itself a matter of judgement, experimentation and learning, and various trade-offs need to be negotiated in every case. Honouring these two basic premises, we present two claims that develop the analytical potential of the diagram as an STS-inspired framework for understanding and critically analysing initiatives positioned at different locations in the space.

Our first analytic claim is that the study of organizations’ trajectories within the two-dimensional space is likely to be of key importance. Analysts should be interested not just in a snapshot of where bodies and initiatives are located but also in the movements that they make and whether different organizations move in similar directions. The mixed-mode process of the IPCC, for example, means that the organization’s assessment process oscillates between high and low degrees of separation during its different phases (see Figure 1); accordingly, the middle position assigned to the IPCC in the diagram simplifies matters (see Figure 2). Moreover, with regard to formalization, it is important to note that the IPCC is climbing higher on this dimension over time. Analyses should be sensitive to the dynamism involved in relation to the two dimensions and an organization’s way of dealing with them over time (for a similar observation, see Miller, 2001: 484–485).

Our second claim is that formalization should not be rejected in principle, but instead criticized in examples where it is not handled well, where there seems to be a significant mismatch between actors’ ambitions and the position they have adopted within the space of our diagram. As we argued in a previous section, good reasons may be adduced in favour of separation and of formalization, as well as for integration and informality. Analysts should ask what the actors’ aims are and with what external constraints they have to deal, and assess their achievements from the circumstances of the case, which, more often than not, change over time.

Taking the IPCC as an example, the increasing degree of formalization should be analysed against the backdrop of external pressures and threats. For example, the recommendations from the 2010 evaluation of the IAC, which was a response to the Climategate crisis, have helped push the IPCC to elaborate its rules of procedure, implying a move higher on the axis of formalization. In this case, the IPCC’s move to higher formalization may not have been the IPCC’s own ambition but an accommodation to external demands (for similar indications of formalization as a response to external distrust, see Porter, 1995; Sismondo, 2010: 143).

SfEP has the laudable aim of making novel, reliable scientific information available to policy communities. Policy actors clearly cannot keep abreast of all potentially relevant scientific literature, so having a digest of the latest information is doubtless of value. However, the SfEP initiative is run without a rigorous methodology for deciding which forms of information to summarize and present. The moderate level of formalization means that it is difficult for SfEP to identify a systematic basis for selecting the literature to be summarized, even if this is exactly what is aimed for. CCC wishes to benefit from the objectivity of scientific procedures, but the scientific basis of its recommendations is not open to public scrutiny. In this sense, the organization’s self-understanding and the practical compatibility between the organization’s publically espoused view of science and its day-to-day activities are a significant issue.

This claim about mismatches, such as failing to handle the two dimensions well in practice, can help support and improve STS analyses of science-based policy initiatives. We therefore elaborate this potential by outlining three possible paths for future analyses of mismatches. Here the importance of trajectories will also be evident.

First, we need to ask whether actors are honest when they describe their position in the space. Do they switch between different representational repertoires when offering accounts of what they are doing, alternating between what may be called front-stage and backstage ways of reasoning, depending on their audience? The Climategate incident is an example of scientists using one kind of repertoire when talking to each other over e-mails and another when addressing the world through the IPCC reports. What can often be noticed in such mismatches is that actors, during front-stage performances, present their activities as being more formalized and separated than they actually are (Wynne, 2010). One important reason for this is that the canonical view of science is a strong public image that supports expectations that scientists try to, but cannot, live up to. Scholars in STS convincingly demonstrate the importance of informal judgements and integration in backstage activities (actual practice), whereas separation and formalization are often used when these activities are presented in front-stage performances (Collins, 1992; Jasanoff, 1990; Latour, 1993). Studies of such mismatches and their effects, long addressed in the field of STS (Hilgartner, 2000; Mulkay et al., 1983), may now be pursued in conjunction with our two-dimensional model (for a discussion of scientists as ‘honest brokers’ in relation to climate issues – who are ‘expanding the choice for policy makers’ without taking a personal stand – see Hulme, 2009: 99; Pielke, 2007).

Second, are actors betraying their ideals? Would they prefer being located at one place but choose to do things differently because it is more practical, convenient or opportune? The ideal of not using non-peer-reviewed literature when summarizing research for policy purposes provides an example. This mismatch of using a source of data that one ‘really should not’ illustrates an adaptation of preferences to what it is possible to achieve. This betrayal can afterwards be reconciled by establishing new procedures that adapt the ideals to reality. For example, in some cases, IPCC chapters have drawn – quite reasonably, in the absence of other forms of information – on grey literature and reports from non-governmental organizations (NGOs). But, though such sources can be highly informative, the problem remains that they are not supported by academic peer review. The IAC review (2010: 17) recommended clearer guidelines for dealing with such information, which later were adopted by the IPCC. STS scholars have suggested betrayal, that is, adaptation to external constraints, to be a factor for success (Callon, 1986; Latour, 1986). Betrayals of ideals may be studied over time and represented as trajectories in our two-dimensional diagram. These trajectories may also draw on studies of organizational change and reflexive learning (e.g. Pallett and Chilvers, 2015).

Third, it is clear that organizations are not autonomous when locating themselves within the formalization–separation space. In particular, it is apparent that in contexts of public controversy, there are pressures towards formalization and separation as a response to accusations of partisan judgement and covert ‘activism’. A move away from expert judgement to transparent, formalized procedures seems an appealing way to convince the public that one’s policy conclusions are not a matter of opinion but the result of impersonal, ‘scientific’ methods (Porter, 1995). In a similar way, a move from integration towards separation is attractive in order to avoid being ‘illegitimately’ influenced by non-credible actors, and thereby presenting one’s findings and conclusions as autonomously generated; this is named ‘the separatist principle’ (Jasanoff, 1990: 231). This, again, is related to external pressure – generated from a strong public ideal of formalization and separation on which scientists and experts base their results, advice and conclusions – prompting organizations to move within the diagram. Examples of this come from studies of the way scientists frame their research as alarmist or not (Brysse et al., 2013), and from studies of the way uncertainty is given meaning (e.g. Lahsen, 2010).

Thus, a key finding, and a way to summarize the three observations of mismatches, is that organizations may be subject to pressure to adopt a public image of their position in the space that is contrary to the one they would ideally adopt. And – of course – there is no guarantee that the highly rationalized images of science that circulate in public contexts are realizable; they might more accurately be regarded as the product of demands for increased rule-based ‘objectivity’ in the Audit Society (Power, 1997) – a public ‘performance’ of an over-positivistic conception of what ‘real science’ is like. This is not to say that organizations are being ‘pushed around’ in this space without any control over their own fates. Organizations may aim to move ahead of criticisms or simply to increase the perceived reliability of their procedures. For example, there are indications that the IPCC has moved towards greater formalization in response to external reviews, most notably that of the IAC. But the IPCC also has been conducting internal reviews of the way that uncertainty is handled, recognized and expressed. The first text offering guidance to IPCC authors on how to deal with uncertainties was published ten years before the IAC evaluation, as an input to the third assessment cycle (Moss and Schneider, 2000). Subsequently, an increasingly formal set of procedures was developed under which members of the Technical Support Units for each Working Group oversee the adoption of evidentiary standards in each chapter.

STS scholars have long stressed the argument that we all, academics as well as publics, would gain from lowering our expectations of what science can deliver with regard to certain (formalized and separated) knowledge (Collins and Pinch, 1993: 154; cf. Sundqvist, 2011). This would help resolve the mismatches we have discussed in this section. We subscribe to this view, yet insist that this does not imply that the only task for STS scholars is to deconstruct activities aiming for formalization and separation. Encouraging careful, nuanced investigation of what actors are trying to achieve, the forms of practice they adopt and the constraints they face, the diagram offers an analytical framework for studies of unavoidable mismatches and their development over time.

Conclusion

Formalization and separation are two fundamental dimensions of any attempt to summarize science for policy purposes. In this article, these dimensions, which are usually discussed separately, have been combined in a single, comprehensive framework. This has allowed us to outline a space of possibilities within which all current science-for-policy initiatives, not least in the climate field, can be located. The area in the middle of our diagram is routinely neglected and made invisible by confrontational but sterile disputes over the relative merits of polarized and purified endpoints – high and low positions – along the two dimensions of formalization and separation.

Our three climate examples exhibit considerable variation with regard to the two dimensions. Covering the informal and integrated classic expert committee work, the semi-formal and separated research repository that has mushroomed in the digital era, and the most prominent international body performing scientific assessments, our examples evince a range of organizational designs balancing, on the one hand, formalization and judgements and, on the other, separation and integration. Drawing on these three examples in order to make a series of theoretical claims, we have suggested some preliminary ideas on possible ways of elaborating a conceptual STS framework that acknowledges different locations in the area opened up by the two-dimensional diagram. We have argued that formalization and separation, as well as their counterparts, judgements and integration, are fair positions that may be supported by an analysis of what should be formalized and separated. From there, critical analysis may be carried out concerning trajectories and mismatches.

Advancing two analytical claims, we encourage STS scholars to refrain from routinely criticizing formalization and separation, and to address, instead, specific initiatives in which they are not well handled, critically studying trajectories and mismatches in science-for-policy initiatives. The STS agenda should be one of demonstrating the complexity of such activities and, in a constructive way, analysing and assisting initiatives performing balancing acts beyond the simple attraction of purified endpoints. This implies embracing a normative programme, being prepared to offer recommendations concerning what to separate and what to formalize, and how far to go in these ambitions. Our two-dimensional diagram provides a framework for STS research addressing trajectories and mismatches – scrutinizing the aims of the actors dealing with specific objects – and improving nuanced discussions and critical reviews of a range of practical initiatives concerning science for policy, not least in the climate field. There is no lack of important and interesting initiatives being launched within the space of summarizing science for policy which is opened up by the dimensions of formalization and separation.