Impact,theory of change,and the horizons of scientific practice

Abstract

Demands for research to generate impact, along with proliferating institutional regimes for evaluating impact, are a ubiquitous aspect of contemporary scientific practice. Based on participant observation at an agro-environmental research institute in southwest China, this article explores three iterations of a tool for planning and evaluating impactful science called ‘theory of change’ (TOC). Despite their ostensible common grounding in TOC, I show how an impact scientist’s framework, a donor’s monitoring and evaluation regime, and a communication consultant’s branding strategy each suggest very different normative structures for scientific practice. These structures entail: particular horizons towards which scientific research is to be practiced, precise points in time at which the future effects of research are to be anticipated, and specific assumptions about how scientists’ agency should play out across time. Taking the peculiar sensibilities of TOC as a comparative framework, I illuminate IFF scientists’ implicit imaginations of how contemporary science does and should generate effects in the world.

Keywords

agroforestry audit cultures impact monitoring and evaluation planning research for development

Demands for scientists to generate impactful research are ubiquitous. The US National Science Foundation (NSF) requires funding applications to describe a research project’s potential to generate ‘broader impacts’ (NSF, 2017). Similarly, the UK’s Research Excellence Framework (REF) expects universities to generate ‘impact’ defined as ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ (HEFCE, 2017). In this article, I examine the models for scientific practice that such demands imply through ethnography of a planning and evaluation tool – theory of change (TOC) – that promises scientists the ability to bring impacts into being. TOC originated in development NGOs but in recent years it has become prevalent in diverse fields from philanthropy and international aid to universities and funding organizations. It is, for example, a central component in the ‘impact framework’ used by Australia’s federal research agency, the Commonwealth Scientific and Industrial Research Organisation (CSIRO, 2018). Examining three iterations of TOC encountered by a group of agro-environmental researchers at the Institute for Farms and Forests (IFF) (a pseudonym¹) in southwest China, I explore how emerging regimes for planning and evaluating impact attempt to transform scientific practice, and I consider scientists’ alternative frameworks for making research do good in the world.

Agricultural and environmental sciences constitute fields that have long been oriented toward real-world problems. Indeed, before TOC and intense institutional demands to demonstrate their impact arrived at IFF, the Institute’s scientists were already motivated to produce research that contributes to positive social and environmental change. The transformations at stake are, therefore, perhaps more subtle, though not necessarily less important, than they would be for other fields. In the context of university reforms, numerous commentators have highlighted how demands for impactful and marketable research are undermining institutional support for, among other things, feminist, critical and humanistic teaching and research (Ginsberg, 2011; Newfield, 2008; Thornton, 2010; Wright and Shore, 2017). For fields such as agricultural and environmental sciences that are already more directly and practically orientated towards real-world problems, we might not expect emerging demands for impact to generate much of a disturbance. Exploring iterations of TOC in the context of agro-environmental research, however, reveals how impact agendas are about more than simply whether research is orientated towards some form of practical or concrete effect on the world. The key, rather, is the structure according to which effects in the world are anticipated and the process by which scientists are to manage the pathway from research to impact. Three aspects are especially significant to understanding the divergence between the various orientations to impactful research that I encountered at IFF: the horizons that motivate research, the moments at which future effects are anticipated, and how scientists imagine their agency playing out across time.

Here my analysis builds upon attention in STS scholarship to imaginations of the future and to scientists’ aspirations to orchestrate the future lives of knowledge and technology (Callon, 1984; de Laet and Mol, 2000; Jasanoff and Kim, 2015; Latour, 1988). Krzywoszynska et al.’s (2018) recent study of participatory scenario workshops provides a particularly provocative set of contrasts in this regard. The goal of the workshops that Krzywoszynska et al. describe was to open up ‘techno-scientific future imaginaries’ of energy systems to dialogue with members of the public (p. 788). This was to involve lay participants ‘assum[ing] a role of responsibility for the future … [and] engag[ing] with this future in a creative manner’ (p. 793). This role, however, turned out to be incongruous with participants’ assumptions about the future and their abilities to shape it. Far from the ‘open future model’ that guided the design of the scenario workshops, participants ‘saw technological progress as both unavoidable and unpredictable’ (p. 793). This contrast suggests a different set of alternatives than those evinced by the various models of TOC I examine in this article. Nevertheless, like my ethnography of TOC, Krzywoszynska et al.’s analysis illuminates the significance and diversity of ways in which people engage futures, as well as of the ‘assumptions about agency – about who or what has the power to shape the future and how’ – that these engagements express (Krzywoszynska et al., 2018: 794, citing Adam and Groves, 2007). Echoing this analysis of failed scenario workshops, my ethnography highlights the diverse structures according to which staff at research institutions and funding organizations anticipate the ability of science to transform the world.

This focus departs from critical studies of audit cultures that interrogate the forms of visibility and opacity that evaluation regimes generate – studies that focus on ‘what audits and rankings bring into focus and what they render invisible or unsayable’ (Shore and Wright, 2015: 422; also Calvillo, 2018; Giri, 2000; Goodwin, 2018; Rottenburg, 2009). My interest, instead, is in how planning and evaluation technologies structure, sequence and anticipate the effects of scientists’ actions upon the future. ANT-inspired ethnographies describe development projects as ‘systems of representation’ (Mosse, 2005) and project evaluations as ‘immutable mobiles’ (Rottenburg, 2009). Such analysis implies that an evaluation regime is, like a scientific laboratory, an institution for constructing representations and stabilizing facts (also Davis et al., 2012; Merry, 2016; cf. Pickering, 1995). The relationship I create between science and audit cultures works in the opposite direction: Taking TOC as my analytic framework renders science like a planning and evaluation regime.

I elaborate three divergent models for scientific practice that draw explicitly on TOC: the outcomes thinking approach of an impact scientist, a donor’s monitoring and evaluation (M&E) regime, and a communication consultant’s brand-building strategy. Having outlined these models, as well as some of the disjunctures between them, I develop an IFF colleague’s suggestion that IFF scientists already have their own implicit TOCs, by borrowing the peculiar sensibilities of TOC as my own comparative framework. I use this framework to explore two further models of scientific practice at IFF: research as it is conventionally imagined by IFF colleagues, and a call for scientists to proactively interest broader publics in their research.

Comparing these five models for scientific practice, I develop three points: First, I highlight the normative stakes in new institutional agendas for impact. These agendas cannot be reduced to a straightforward shift towards research that is more oriented to practical, real-world effects. Rather, I demonstrate how the three TOC-based models differ from IFF scientists’ conventional practices (as well as from each other) in terms of the horizons that motivate research and the structures according to which scientists anticipate their role in bringing about real-world change. Second, by exploring the diverse ways that TOC is applied – even within a single organization – I draw attention to the multiplicity of forms that institutional agendas for promoting impactful research can take. Third, taking my lead from the peculiar sensibilities of TOC, I explicate what I refer to as IFF scientists’ ‘implicit TOCs’, or their alternative imaginations of how contemporary science does and should generate effects in the world.

Research site and methods

This article is based on 25 months of participant observation at IFF’s China office conducted primarily between 2014 and 2016. IFF is a global research for development organization with its headquarters in East Africa. Made up of around twenty research staff, ten administrative staff and forty affiliated graduate students, IFF-China is one of six regional offices. Approximately half of staff and students are Chinese, with large contingents from elsewhere in Asia, as well as Africa, Europe and North America. A major focus of IFF’s global research is agroforestry – the use of trees within agricultural landscapes. The China office’s research is nevertheless quite varied, including fungal diversity studies, social studies of natural resource management, ethnobotanical surveys, global climate modelling, on-farm agroforestry trials and phenological research. Alongside this research, a huge amount of IFF staff’s professional work entails attending workshops and navigating the evaluation regimes described in this article. Broadly speaking, IFF’s institutional mission is to generate knowledge and advance policies in a manner that benefits the poor and the environment. Imagining what this mission implies for practice, however, is central to the work of TOC described below. Indeed, the question of what IFF does, and why it does it, is a large part of what is at stake in the divergent ideas of science and impact that I explore in this article.

During my field research, I worked for IFF as a research assistant. In addition to numerous interviews and informal conversations, I was a direct participant in several IFF projects, and I participated in the production of the TOC flowcharts that IFF-China produced at the behest of its international headquarters and its donor the Food Security Fund. This mode of participant observation allowed me to witness first-hand the frustrations and unspoken incongruities that IFF staff experienced as they came to terms with demands to use TOC. My hands-on learning process often involved colleagues correcting my misinterpretations and making fun of my out-of-place comments. As I highlight at certain points in this article, these moments often illuminate differences between the logics of TOC and the assumptions according to which IFF scientists would otherwise go about their work.

When I embarked on my field research, one of my primary goals was to investigate the challenges that Chinese state bureaucracies pose to scientific practice in China. I quickly came to realize, however, that IFF staff are frequently more concerned and more frustrated by the headaches of negotiating demands from their own headquarters and from international donors. TOC provides a common thread to these two sets of demands. IFF’s largest international donor, the Food Security Fund, ties its funding to IFF’s adherence to a strict M&E regime that revolves around TOC. Representatives of IFF headquarters likewise champion TOC and, in various ways, urge IFF-China colleagues to orientate their work around TOC. IFF headquarters do not impose TOC in quite as regimented a manner as the Food Security Fund. Nevertheless, IFF-China relies significantly on financial resources that are controlled by headquarters, and staff are acutely aware that headquarters mandates cannot simply be ignored. IFF staff are, in this respect, compelled to take seriously each of the three iterations of TOC that I describe below.

TOC 1: The aspirational future of outcomes thinking

As I arrived at IFF in 2014, several senior scientists were returning from the Institution’s annual global research staff meeting in East Africa. Though IFF’s China office outperforms other regional offices in terms of publications, the global leadership had rebuked senior scientists in China for failing to generate ‘real-world impacts’ from their research. To help remedy this, the China office’s director invited Lesley, a colleague from IFF’s international headquarters, to China to host a workshop on ‘outcomes thinking’. Lesley’s job title – ‘participation and impact scientist’ – reflects her position amongst a growing body of professional experts entrusted with optimizing scientific research for impact. During her workshop, Lesley introduced a version of TOC that makes an aspirational future the new driving force for IFF’s collective work, and that demands a concrete, albeit provisional, plan for who will take up IFF’s research and to what use.

Central to Lesley’s outcomes thinking workshop was the introduction of a new set of categories with which she asked scientists to conceptualize an impact pathway and TOC flowchart for their research. Lesley put significant energy into explaining the differences between ‘outputs’, ‘outcomes’ and ‘impacts’, as well as between ‘next-users’ and ‘end-users’. In this scheme, an output refers to a deliverable such as a policy paper or a workshop. According to Lesley, agricultural researchers have too often targeted outputs – especially in the form of scientific publications and policy papers – and then thought no further. Lesley, however, encourages her colleagues in China to imagine pathways for their work that extend beyond the horizon of knowledge production and onto an impact for a specified end-user group. This involves IFF activities targeting changes in the ‘knowledge, attitudes and skills’ of next-users such as policy-makers or NGOs.² These changes will precipitate outcomes such as changes in policy or the adoption of new development projects that will benefit an ultimate beneficiary – the end-users. Thus, an impact pathway might map a causal chain from IFF research on rubber cultivation to a specified change in policy (an outcome) by a local government (a next-user), to an increase in crop yields (an impact) for rubber farmers (an end-user).

One aspect of outcomes thinking that Lesley emphasizes is that scientists cannot undertake the process of generating outcomes after the fact of scientific knowledge creation. According to outcomes thinking, shaping the practices of next-users is most effective when scientists engage next-users from the beginning of a project. This implies a collaborative agenda and is closely connected to broader trends for multi-stakeholder meetings and platforms (Brown and Green, 2017; Hall and Sanders, 2015; Rabeharisoa and Callon, 2004; Sanyang et al., 2016). Multi-stakeholder platforms bring together people from diverse backgrounds to work to address a common set of problems. IFF was an initiating partner in the establishment of the Lengshan Multi-Stakeholder Platform – a collaborative project that aimed to address social and ecological challenges in the tropical Chinese prefecture Lengshan. Rather than decide on their research focus autonomously, platform facilitators ask IFF scientists to attend platform meetings along with stakeholders from business, government, and other research organizations – all of whom are potential next-users. From an outcomes thinking perspective, this platform offers an opportunity for scientists to identify key next-users and to shape their research approach to the demands and needs of these next-users. By continuing this collaborative engagement throughout the research, scientists will, so the platform organizers tell them, enhance their ability to mobilize the interests of next-users, and thereby their prospects for effecting desirable changes in next-users’ knowledge, attitudes and skills.³

Having explained this new outcomes philosophy and vocabulary, Lesley introduced a tool for planning a specific pathway for impactful research: TOC. Organized as a flow chart (see Figure 1), activities are placed at the bottom. Arrows connect these activities to outputs and then to the outcomes that the outputs will generate. At the very top of the TOC flowchart is a ‘vision’. This vision is an image of the world that one wishes to bring into being – the aspirational future one imagines as the cumulative effect of the impacts one will generate. Building a TOC flowchart, one begins at the top – with the vision – and then works backwards towards the activities.

Figure 1.

TOC flowchart.

The vision that IFF scientists crafted during Lesley’s workshop is:

Rural landscapes sustain healthy and culturally diverse ecosystems that ensure food security and provide health, wealth, education, cohesion and equity. Public, private and non-profit institutions supporting policies, investments and interactions that integrate sustainable land management with healthy urban-rural food and ecosystem linkages.

The utopianism of this vision did not escape IFF staff: One workshop participant joked that once we have reached this point we could all go home. Though utopianism appears inappropriate to this scientist’s conventional notions of research design, the novelty of TOC compared to conventional research planning does not lie in convergent degrees of realism. By beginning with this endpoint, Lesley did not suppose that our vision represented a point that we would ever actually reach. The power of this vision lies in the horizon that it generates. Miyazaki (2003: 261) has highlighted how the gap between reality and ideal that characterizes utopianism generates a sense of incompleteness that ‘gives the present moment a future orientation’, and sustains a ‘prospective momentum’ (see also Watanabe, 2019). Similarly, by asking workshop participants to begin by crafting a vision for the world, Lesley gave the subsequent work of designing research activities a new aspirational future horizon to work towards, and with it a new kind of prospective momentum.

With the aspirational future of our vision in mind, Lesley asked us to decide what changes in next-user knowledge, attitudes, and skills would be required to achieve these changes – the outcomes we wish to generate. One such outcome is:

Public policy actors develop sustainable and equitable policies (targeted towards food security, wealth, health, education, cohesion, equity) and direct the necessary human resources and capital towards implementing the policies.

Finally, Lesley asked the workshop participants to discuss activities and outputs that could bring about these outcomes. By this point, the process of manufacturing a TOC flowchart seems quite detached from the utopianism of the office’s vision. Lesley intended that discussion at this moment be pragmatic and grounded in what was feasible: ‘How could we get local government to adopt policies promoting sustainable rubber cultivation?’; ‘What kind of research would be necessary for stakeholders to know which agricultural systems are most sustainable?’; ‘Could we set up a multi-stakeholder platform?’; ‘Who should we invite to participate?’ Working backwards from our aspirational future, nevertheless, meant that workshop participants sustained a prospective momentum: Plans for research activities were made with a vision already in mind for the future these activities would bring closer to being.

At the time of the workshop, most of IFF’s ongoing projects in the region had been initiated with no thought to outcomes or impact as these concepts are understood in outcomes thinking. Lesley asked that we redesign or retro-fit existing projects so that they could be brought in line with IFF’s new TOC flowchart. To fit ongoing projects into this flowchart IFF scientists would have to generate a kind of prospective momentum that their projects had thus far lacked. This involved IFF researchers hastily conceiving social science research to complement existing biophysical research, in the hope that this might highlight the relevance of their research to the achievement of ‘sustainable and equitable policies’. Other IFF staff proposed workshops with stakeholder groups to share research findings in a more proactive manner than the traditional policy paper. Here, scientists were imagining a new kind of value to their work: their contribution to bringing into being the aspirational future atop IFF’s TOC flowchart.

TOC 2: M&E in the future perfect

Lesley’s workshop focused primarily on TOC as a planning tool, but TOC can also facilitate the measurement of outcomes and impacts. Scientists at IFF were already familiar with quantitative assessment of research through the quantification of peer-reviewed publications – either by their volume, or by their quality via proxies such as impact factor or h-index. As with the increased emphasis on impact elsewhere, however, TOC implies a shift from concern with outputs such as publications to concern with real-world outcomes and impacts. In this regard, TOC can provide a novel basis for auditing science, or for what the development community call ‘monitoring and evaluation’ or ‘M&E’.

The Food Security Fund, which funds an IFF project that investigates the efficacy of new agroforestry systems in upland Myanmar, demands a particularly stringent M&E framework. This framework, ‘Monitoring and Evaluation for Learning and Accountability (MELA)’, consists of numerous components. Though they share a common grounding in TOC, the contrast between Lesley’s outcomes thinking and the MELA framework reveals a difference between provisionally anticipated effects compared to targets that are predetermined and inflexible. Subsumed within this M&E regime, a TOC’s horizon is transformed: the aspirational future of a world to be built gives way to the future perfect of quantitative targets.

A core component of MELA is a TOC flowchart similar in style to the one participants had developed during Lesley’s workshop. A key outcome in the new TOC IFF produced for the MELA framework is: ‘Relevant stakeholders understand better how agroforestry improves livelihoods, resilience and ecological health in the Uplands’. Working towards this outcome means that, alongside the kind of on-farm agroforestry trials that IFF scientists would conventionally have planned, IFF scientists are having to pursue numerous knowledge dissemination activities. These include plans for hosting multi-stakeholder meetings and for producing an agroforestry curriculum for use by a local university.

A second component is a measurement template that demands evidence IFF is producing outcomes in the manner set out in the TOC flowchart. This entails an Excel spreadsheet with the outcomes from the flowchart listed in the first column, and subsequent columns then detailing indicators and targets (see Figure 2). So, for example, to measure the outcome ‘Relevant stakeholders understand better how agroforestry improves livelihoods, resilience and ecological health in the Uplands’, the IFF project team devised the indicator ‘% of knowledge platform members with improved agroforestry knowledge’ and set a target of a 20% improvement in agroforestry knowledge by 2018.

Figure 2.

Extract from the MELA spreadsheet.

What makes the MELA framework especially troublesome for IFF scientists is not merely the intensity of the Food Security Fund’s scrutiny, but that MELA entails IFF scientists developing precise quantitative performance indicators in advance of activities commencing. This advance determination gives detailed structure and form to a decisive retrospective moment of evaluation that awaits the project’s conclusion. The MELA framework thus invites IFF staff to think in the future perfect tense: ‘By project completion, we will have caused X to have happened.’

The MELA framework has been a source of great frustration for IFF colleagues, who have expended significant energy developing and revising it. One headache for the project team has been the incongruity between the short-term scope of the monitoring framework – which covers only the four years of Food Security Fund financed work from 2014 to 2018 – compared with goals that will take many years to achieve. To address this, the project team’s initial draft of the MELA framework included the measurement of indicators – including ‘% of knowledge platform members with improved agroforestry knowledge’ – beyond the 2018 conclusion of the project. The Food Security Fund, however, rejected these proposed long-term targets, insisting that IFF base its measurement framework exclusively on what is achievable within the 2014–2018 funding period. Food Security Fund project managers conceded that this timeframe would make achieving end-user impacts such as increased local incomes unlikely, but suggested that IFF could nevertheless demonstrate next-user outcomes such as improvements in agroforestry knowledge within four years. In effect, this means truncating the horizon of the project’s TOC. With impacts unmeasured, end-users became irrelevant to project evaluation: what counts are the next-user outcomes. Here, the distant aspirational future that characterized Lesley’s version of TOC gives way to focus on the date-punctuated near future (Guyer, 2007: 417) of the donor’s project evaluation.

As described above, IFF scientists experience an incongruity between the short-term horizon of the MELA framework and the project’s long-term goals. IFF colleagues, however, also highlight a more profound absurdity in M&E. Jacob, a veteran of numerous agro-environmental research for development projects, told me that he finds not only the MELA framework, but monitoring frameworks in general to be an ‘utter pain in the backside’. At the root of his frustration is the demand to determine the precise course a project will take in advance of its inception (see Hales, 2016). Jacob pointed out that this is an utterly unrealistic demand: One cannot know what to expect of a project before it has even begun and should rather adapt to the situation as it evolves. Then, Jacob told me, once your project is complete, you must waste time justifying why you have not met goals that you only set because you were forced to do so. From Jacob’s perspective, the development of M&E metrics in advance of the project demands an inappropriate predetermination of project goals, making a farce of the ultimate retrospective evaluation these goals are used to conduct. More appropriate from Jacob’s point of view would be provisional goals that can be adapted and revised as a project progresses.⁴

Though erased right from the start by the MELA framework, a provisional mode of a project plan is not foreign to TOC. Indeed, during Lesley’s workshop, participants were told to treat their TOC as a ‘working hypothesis’ – something to regularly revisit and revise. Even the MELA framework asks project implementers such as IFF to ‘facilitate the use of M&E results to improve the project’, and in discussions of MELA, Food Security Fund project managers would often reassure IFF that they are ‘flexible’ and willing to consider revisions to the project’s MELA framework. Indeed, many versions of TOC and M&E foreground precisely this kind of provisional thinking: this is what Jensen and Winthereik (2013: 148–149) observe when they argue that ‘perpetual revisability [is] an important dimension of the monitoring movement’. IFF’s experience of drawn out interactions with Food Security Fund project managers to approve the initial version of the MELA framework, however, mean that IFF staff have little confidence in assertions of flexibility and revisability. As such, IFF staff have approached the MELA framework not as a provisional position, but as a fixed, future perfect image of the project’s endpoint.

The rendering of MELA as offering a predetermined future perfect endpoint also reflects a tension inherent in the MELA framework’s marriage of TOC with a particular form of auditable metrics. As Sam – a London-based outcomes thinking consultant unconnected to IFF – put it in response to my description of IFF’s experience with the MELA framework, the demand for quantifiable evidence to prove impact inevitably leads to the ‘bastardization’ of TOC as ‘a thought process’. In Sam’s view, the use of TOC as a way of demonstrating impact or providing definitive assessments of success is incompatible with the conceptualization of TOC as a reflexive planning process. M&E, he told me, should be used as a learning tool, as something that allows project managers to learn what is and isn’t working, allowing them to revise activities and objectives as they go. This is a perspective echoed in comments Lesley offered having read an earlier draft of this article. As she put it, ‘monitoring and evaluation of a TOC is longitudinal, it is continuous and happens in real-time. The purpose is to identify when the whole effort may be drifting away from [the TOC’s] visionary end state, rather than moving towards it.’ For Sam and Lesley, this kind of learning is inevitably frustrated when focus on quantifying success turns M&E into a box-ticking exercise. The predetermined, future perfect endpoint that MELA demands erases the potential for a TOC to operate as a provisional plan for real-time evaluation.

This erasure is at the heart of Jacob’s frustration with M&E, but it is also something which frustrates people such as Sam and Lesley for whom this erasure undermines fundamental rationales for TOC and the adaptive learning goals of M&E. Despite its global reach, TOC is in this respect not a singular and immutable form (Winthereik and Jensen, 2017). The aspirational future horizon of Lesley’s outcomes thinking is very different from the future perfect horizon generated by the Food Security Fund’s MELA framework.

TOC 3: The eternal impetus of brand-building

Lesley was not the only member of IFF’s global staff to visit China with the hopes of remedying the office’s failure to generate impacts from its research. Shortly following the same rebuke that brought about Lesley’s invitation to China, the leadership at IFF-China hired a new communications officer and invited IFF’s global communications consultant to visit China to help the office build a new communications strategy. During a presentation he gave to his colleagues in China, Alistair, the communications consultant, expressed his delight that IFF headquarters had recently recruited an M&E team. Alistair celebrated how this team would allow IFF to show their return on investment and to thereby make a better case to policy makers and donors for the efficacy of agroforestry, as well as for the work that IFF does in researching and promoting agroforestry. For Alistair, the TOC imperative to change the knowledge, attitudes, and skills of next-users is analogous to the work of a business selling a product. According to Alistair, ‘TOC’ is simply another way of saying ‘business plan’. Alistair wants, moreover, to use M&E metrics that can evince the efficacy of agroforestry and the impact of agroforestry research as a marketing tool. Here, Alistair takes M&E’s decisive retrospective judgement and turns it into a tool for a brand-building strategy – a strategy which like Lesley’s outcomes thinking seeks to empower IFF to bring about its target outcomes and impacts. Compared to the relatively specific target-users and target uses implied by outcomes thinking, however, this branding strategy implies a more generalized initiative to exert an influence on the world. In this strategy, the motivating horizon once again shifts: IFF’s activities are to be driven not by an aspirational future or by future perfect goals but by the eternal impetus for brand growth.

Whereas the IFF team’s concerns within the MELA M&E framework focuses almost exclusively on the individual project at hand, Alistair’s idea of branding implies a much broader understanding of IFF’s relationship to next-users. In Alistair’s conception, the brand of agroforestry stands on a different level than the reputation of any individual project. The reputation of a well-developed brand is more than simply the sum of its products and will sustain itself despite individual instances of project failure – just as, in Alistair’s example, the brand Coca Cola survived the notorious failure of its product New Coke. Individual projects can serve as success stories with which next-users can be persuaded of the benefits of agroforestry, but this must be understood as part of a broader brand management strategy. Projects like Agroforestry for Myanmar are thus subsumed as means to a grander project. Alistair told IFF’s China office that IFF is already reaping the rewards of brand management. It was because of agroforestry’s growing brand recognition, he explained, that he could get IFF’s director-general into an important inter-governmental forum on global security; and colleagues at another agroforestry organization have succeeded in persuading the EU to integrate agroforestry practices into the common agricultural policy (CAP) (cf. Mosquera-Losada et al., 2018). Thanks to the brand of agroforestry, Alistair boasted, the concept is now ‘percolating policy documents’. Proactive brand management has, in Alistair’s view, transformed agroforestry science from an obscure body of knowledge to a vital component of multinational agricultural and environmental policy.

The way in which Alistair imagines and celebrates branding as a strategy for knowledge circulation and policy transformation shares a certain affinity with aspects of early actor-network theory (ANT) scholarship.⁵ In his ground-breaking essay on the scallops of St Brieuc Bay, Callon (1984) describes the efforts of three marine biologists to enroll scallops, fishermen and scientific colleagues to the service of their conservation strategy. At a fundamental level, Alistair’s take on scientific practice looks at lot like that of Callon’s marine biologists: The goal, in its crudest terms, is for ‘researchers to impose themselves and their definition of the situation on others’ (Callon, 1984: 196). Likewise, Alistair’s call for IFF scientists to market their work could almost be a selective re-working of Latour’s sociological framework into a program for action: an ‘actant manages to persuade others to fall into line, it thereby increases its strength and becomes stronger than those it aligned and convinced’ (Latour, 1988: 172). This somewhat Machiavellian reading of science, as de Laet and Mol (2000: 227) put it, ‘says that technologies depend on a power-seeking strategist who, given a laboratory, plots to change the world’.

Aside from his more unabashedly world-making ambitions, there is also a more subtle distinction between Alistair’s rendering of TOC compared with Lesley’s or the Food Security Fund’s. Whereas MELA’s future perfect horizon demands that IFF evinces its ability to effect changes within the limited timeframe of a project, branding implies a collective agency that projects itself into an endless future. This endless quality superficially echoes the prospective momentum generated by the aspiration future of Lesley’s TOC. Branding, however, implies a distinctive form of impetus. Alistair did not imagine an ideal state for agroforestry’s brand: There was no generative gap between the reality of agroforestry’s brand and an ideal vision of it to which Alistair aspired. Rather, an entailment of branding as a commercial strategy is the infinite imperative for growth: ‘Brand management over time is the permanent pursuit of growth’ (Kapferer, 2012: 219). This imperative to pursue permanent growth gives the work of brand-building an eternal quality. Whereas Lesley’s outcomes thinking generates momentum towards an aspirational future, Alistair’s branding strategy implies an eternal impetus to the task of enlarging IFF’s world-making power.

TOC as comparative framework

One of the new impact-focused activities that IFF scientists devised in response to Lesley’s workshop was an experiment, conducted in collaboration with a county-level forestry bureau and local mushroom harvesters, to compare a range of indigenous mushroom management practices. The project team – of which I was a member – conceived this project as a means to build upon IFF’s existing fungal diversity research and its staff’s extensive mycological expertise in a manner that would satisfy the imperative to generate outcomes and impacts. The project’s target impact was for mushroom harvesters to adopt more productive, and therefore more profitable, mushroom harvesting practices. Working backwards from this end-user impact, we formulated a next-user outcome of forestry extension workers gaining improved knowledge of mushroom management. We therefore decided to begin by devising an experiment that would produce new knowledge of mushroom management that we could pass on to forestry extension offices.

Not only did TOC provoke IFF scientists to conceive of this new research project, it also gave them a new means for evaluating the project’s success – or in this case failure. Having completed the mushroom management experiment, data analysis showed no significant difference between the efficacies of the management strategies compared in the experiment. Reporting these results to Matt, who led the office’s mushroom research, I suggested that although our research could not say anything of practical use to mushroom harvesters, our data might have the potential to generate a decent scientific paper. Matt laughed at this. By rendering a peer-reviewed paper as the project’s only output, I had forgotten the point of the research. We had originally anticipated that our comparative analysis would allow us to propose a best practice that we could subsequently disseminate to harvesters (the end-users) via forestry bureau extension officers (the next-users). From this perspective, our project seemed somewhat of a failure.

There is also a sense, however, in which by arriving at this conclusion both Matt’s team and I had adopted a distorted version of Lesley’s TOC. In our impatience to satisfy the imperative to generate impacts, the project team had hastily imagined a TOC flowchart that did not consider a role for researchers as next-users. Had we done so, we might have considered how our research (even with null results) might have helped us to persuade scientific colleagues of which further avenues for research would and would not be worth pursuing. This would have involved considering how shaping the knowledge and attitudes of scientific colleagues might serve as a causal link in a more complex pathway to our target impact of enhancing sustainable mushroom yields. As Lesley highlighted in response to an earlier draft of this article, moreover, rather than seeing these null results as a failure of the project, we might have taken the deviation from our intended impact pathway as an opportunity for ‘real-time evaluation’. Seeing the project drift away from our TOC’s causal pathway, we might have taken the opportunity to re-think the pathway to our aspirational future. Though we had taken on board many of its lessons, we had perhaps also transformed TOC into something that was narrower and more rigid than what Lesley described in her workshop.

When he laughed at my appeal to publication as the project’s endpoint, Matt had nevertheless taken on board a fundamental lesson from Lesley’s workshop. Our goal had been to generate outcomes and impacts, and publishing research results could not, as I had implied, stand in for this goal. The short shrift Matt gave to my proposal for a publication as the project’s endpoint highlights the novelty of the structure that outcomes thinking imposes upon IFF research. The near future endpoint of a peer-reviewed article is no longer an acceptable horizon to work towards. Driven by the gap between reality and the aspirational future of IFF’s vision for the world, Lesley’s TOC forces scientists to focus instead on a protracted causal chain in which their activities will produce next-user outcomes that will in turn produce end-user impacts.

Most of the participants in Lesley’s workshop were, like Matt and me, encountering TOC for the first time. Bob, a China-based social scientist, was an exception. Not unlike Lesley, Bob saw his role at IFF as one of optimizing the connection between scientific research and development impacts. Reflecting on Lesley’s workshop, Bob suggested to me that everyone at IFF already works according to implicit versions of TOC. What the workshop was doing, in Bob’s analysis, is simply forcing scientists to make their tacit TOCs explicit and to reflect upon them. In a narrow sense, Bob is wrong: His colleagues conceptualize and practice science in a manner that is incommensurable with Lesley’s TOC. Notwithstanding the disjuncture between Lesley’s conceptualization of TOC and IFF researcher’s translation of it into practice, Matt and many of Bob’s colleagues recognize the radical transformation that TOC implies for scientific practice.

While IFF scientists’ usual practices are very different from those prescribed by TOC, there is nevertheless also a sense in which, following Bob, we might say IFF scientists operate according to implicit TOCs. Though in a different structure from the one Lesley espoused, IFF scientists likewise imagine future effects for their work – effects that are understood and pursued according to a tacit structure that, as Bob pointed out, is rarely made explicit.

Implicit TOC 1: Endpoints and unanticipated futures

As familiar as Alistair’s brand-building strategy may be since Callon’s account of marine biology in St Brieuc, it is incongruous with how IFF staff ordinarily conceive of scientific practice. The world-making ambitions of Alistair’s brand-building jars with the more modest ambitions of most IFF research staff. Equally, IFF scientists do not usually imagine themselves working towards the kind of aspirational future that Lesley had them formulate in her workshop. More vaguely, though no less earnestly, IFF scientists aspire to contribute to endeavors such as ‘addressing climate change’, ‘safeguarding biodiversity’, and ‘improving rural livelihoods’. Scientists’ imaginations of the connection between research outputs and the actualization of such endeavors are, moreover, generally free of any pretension to a plan. Instead, IFF scientists more often focus their attention on the endpoint of a publication, while fostering hopes that the knowledge such publications contain might be somehow put to an as-yet-undefined use in the hands of an as-yet-unidentified other.

This orientation is evident in a paper presenting an overview of edible mushrooms in Asia, to which several IFF colleagues contributed. The paper concludes with a discussion section highlighting, among other things, how little is currently known about mushrooms of the region. As Matt, one of the co-authors, described it to me, the paper’s goal was to ‘provoke activity’ in Asian mycology. In this paper, there is closure of a kind: a desire to establish Asian mushrooms as a ‘matter of concern’ (Latour, 2004). Control over how this concern plays out is nevertheless ceded to others (de Laet and Mol, 2000). Establishing mushrooms as a problem for future scientists is, of course, no neutral act: No less than pursuing outcomes thinking or brand-building, provoking activity is an act that (if successful) will compel others to action. The structure and sequencing of these anticipated effects is, however, fundamentally different. Whereas a TOC flowchart begins with a vision of the future IFF will compel others to build, Matt and his colleagues’ mushroom paper does not preconceive the future its work would bring into being.

This incongruity was apparent in my misunderstanding of another colleague’s desire to conduct applied research. Responding to my interest in her work, an IFF postdoc called Ruyue described to me how she is writing a proposal to compare the carbon storage capacity of different mycorrhizal fungi. One of the reasons this research interests her, Ruyue explained, is that she wants to use her expertise in soil ecology to do applied research. Describing the potential application, Ruyue told me that IFF already has databases capable of aiding a forester’s tree selection, but she suggested that with her proposed research IFF could generate a complementary database to guide selection of mycorrhizal species for inoculation into forestry and agroforestry systems. The idea of applied research brought to my mind Lesley’s workshop, so I responded to Ruyue by asking exactly how she intends to turn her research into changes in policy and on-the-ground practices. I was mistaken, however, to assume that Ruyue’s interest in applied research implies anything resembling outcomes thinking. Ruyue’s straightforward response to me was that the first thing is to do the research: Worrying about the specifics of how someone might engage policy-makers should wait until after the research is completed.

Lesley had described TOC as beginning with an aspirational future. Working backwards from this vision, Lesley’s TOC entails working through a causal chain of impacts, outcomes, and outputs. Only once this protracted chain of future effects has been mapped out, does Lesley think it appropriate to decide on research activities to pursue in the present. Encouraging IFF scientists to engage multi-stakeholder platforms, moreover, Lesley compels scientists to identify and cultivate relationships with next-users prior to beginning research. Ruyue, by contrast, thinks it inappropriate to make concrete plans beyond the near-future horizon of conducting research. Ruyue designs research that she hopes might be useful – that will have an application – but she has no designs for who will make use of her research. When Ruyue imagines the future of her research, she, like Matt, defers the problem of planning its concrete application and ultimate impact on the world to the future.

As described above, the inflexible determination of a TOC flowchart and associated target outcomes that is found in MELA is not inevitable to TOC. TOC flowcharts are, like a research proposal, often imagined as merely provisional. The distinction between Ruyue’s applied research and TOC is therefore not merely that one is open-ended and the other is not: both are potentially open-ended, but they are different forms of open-endedness (Miyazaki, 2014). Though both a TOC flowchart and a research proposal can operate as a provisional plan, a key difference is that a research plan has an actual endpoint: In a conventional research proposal, the research will be finished, and the plan will reach its conclusion. As described above, the vision that sits atop Lesley’s TOC does not represent an actual destination, but is an aspirational future that imbues work towards it with prospective momentum. Alistair’s branding strategy, meanwhile, derives its momentum from the eternal impetus for a brand’s growth. By contrast, Ruyue and Matt imagine an actual endpoint to their research: Matt’s research was already completed, and Ruyue’s will be. This is an endpoint, however, that these scientists hope will generate further as-yet-unknowable effects: It is at once an endpoint and the genesis of unanticipated futures.

Implicit TOC 2: Public interest as endpoint

TOC first arrived at IFF’s China office at the behest of senior staff at IFF’s global headquarters keen to generate greater impact from IFF’s work, but many scientists at IFF’s China office are themselves also deeply dissatisfied with what they see as their failure to conduct research that contributes to social and ecological change. Matt, for example, lamented to me that in his five years at IFF, ‘We haven’t really planted any trees.’ As he sees it, one of the organization’s core objectives is to research and promote agroforestry – the integration of trees into agricultural landscapes. But, he told me, the only trees he has seen planted since his arrival at IFF’s southwest China office are a few alder trees for a small experiment his team had conducted, and more recently ‘a few badly planted trees by the roadside’ as part of a landscape restoration project. Another colleague, Jiaolong, often jokes that the inevitable conclusion from my field research will be that ‘IFF is useless’. For Jiaolong, the failure of her and her colleagues to have any positive effect on the world is a frequent pre-occupation and is a problem that requires scientists to re-think how they engage the world. As she put it in one of her blogs, ‘[scientists] often complain that the public don’t understand science and don’t think that popularizing science is important, but the question is, have we helped to foster their interest in science?’

Jiaolong does not pretend to have an easy solution to the problem of making science generate positive change, but she does demonstrate the possibility for making this a very different kind of problem to that posed by Lesley, the Food Security Fund or Alistair. In her blog, Jiaolong asks, ‘How many people have read your research?’ She then points out that not only do scientists make little effort to interest the public in their research, their research is not even accessible: Most journals are pay-to-access, and many Chinese scientists publish only in English. Jiaolong’s response to this is to identify an imperative for scientists to ask themselves: ‘Can we use different methods to more quickly transmit our research results to our primary and middle school classmates, to our classmates’ classmates, to our kin’s kin?’ With this in mind, we might ask, what did Matt and the co-authors of his mushroom paper do to foster interest amongst publics beyond professional scientists? This question would draw attention to the fact that when Matt told me he wanted to ‘provoke activity’, what he meant was activity amongst other scientists and not among any broader publics beyond the sciences. Though Jiaolong’s question highlights a failing in Matt’s work, it is a very different failing than the failings that Lesley’s outcomes thinking, the Food Security Fund’s M&E framework or Alistair’s branding strategy would highlight. Rather than asking what Matt did to foster public interest, evaluating his research within these frameworks might lead us to ask: How is Matt’s paper bringing us closer to the vision of the world we want to bring into being? What changes in the knowledge attitudes and skills of specified next-users was his research paper meant to effect? What measurable outcomes has his paper generated? Or, what does his research do to enhance agroforestry’s brand? Each of these questions implies a distinctive imagination of how science generates effects in the world.

Jiaolong’s proposal is to shift the endpoint of scientific practice from the point of presentation to scientific colleagues – that is, publication in peer-reviewed journals – to the point of presentation to a broader audience. Like Lesley’s outcomes thinking, Jiaolong draws attention to the limits of peer-reviewed (and peer-read) journals as the primary horizon for scientific practice. Compared with Lesley’s approach of identifying specific next-users, however, Jiaolong’s call to interest our classmates’ classmates and our kin’s kin evokes a somewhat amorphous public – a notion that echoes the similarly amorphous scientific community amongst whom Matt wished to provoke activity. Whatever the audience, or the medium, for Jiaolong’s public-oriented endpoint, it is a very different kind of challenge than the one implied by Lesley. In Jiaolong’s implicit TOC, the impetus for scientific practice to transform the world does not depend upon the prior elaboration (provisional or otherwise) of an aspirational vision for the future. Jiaolong (or for that matter Ruyue and Matt) is no less concerned than Lesley with the desire for IFF’s research to be useful. The difference, however, is in the sequencing and structure of expectations for utility: when, how and by who research is put to use, as well as when, how and by who this use is anticipated and planned for.

Conclusion

The TOC that Lesley asks her colleagues to embrace implies a transformation in the structure of scientific practice. Rather than allowing scientists to cede the problem of how research is used, Lesley asks that scientists plan for how, when and by whom their research findings are put to use. The impetus for these designs is a new kind of horizon, an aspirational future world that research strives to bring into being. This is a particular way of imbuing the present with a future orientation: A gap between a reality and a projected ideal sustains the prospective momentum of Lesley’s TOC (Miyazaki, 2003; Watanabe, 2019). Diverse iterations of TOC, however, demonstrate the multiplicity of future orientations that can structure knowledge production. In this regard, the aspirational future that orients Lesley’s TOC implies a very different orientation than does the future perfect deliverables of MELA. This is again different to the futurity implied by the eternal impetus of brand growth contained in Alistair’s rendering of TOC as a business strategy. The diversity of ways in which this one planning and evaluation technology is operationalized within the same institution suggests that demands for ‘impact’ are ubiquitous but by no means uniform.

Table 1 summarizes key characteristics of the three iterations of TOC described in this article, as well as of IFF scientists’ implicit TOCs. At issue in the divergence between each of these models is not merely whether science is expected to generate practical or concrete effects in the world, but a multitude of distinct, and often mutually incompatible, ways of structuring and motivating scientific practice. Despite their ostensible common grounding in TOC, Lesley’s outcomes thinking, the Food Security Fund’s M&E, and Alistair’s branding strategy each suggest very different normative models for how scientists engage stakeholders beyond the academy, and for how we imagine the role of scientists in shaping the future. Jiaolong’s call for her colleagues to foster public interest in the sciences, moreover, demonstrates that highlighting a lack of vision, the absence of measurable targets, or the inadequacy of a scientific brand are not the only extant challenges to contemporary science, nor the only starting points for re-imagining its future.

Table 1.

TOCs and implicit TOCs at IFF.

	Motivating horizon	Scientists anticipation of and role in bringing about real-world effects
TOC 1: Lesley’s outcomes thinking	An aspirational future world that scientists work to bring into being.	Scientists envisage provisional pathways from research activities to impacts. Scientists take responsibility for shaping ‘next-users’ (e.g. policy-makers) in ways that will bring about target impacts.
TOC 2: a donor M&E framework	Future perfect M&E targets.	Scientists devise a truncated pathway from research to impact and then focus on effecting predetermined near-future changes in next-users.
TOC 3: Alistair’s brand-building	The eternal impetus to grow agroforestry science’s brand.	Scientists pursue research that will help to bolster the brand of IFF and of agroforestry science more broadly. A strong brand is vital because it enhances scientists’ power to shape and influence the world.
Implicit TOC 1: conventional research at IFF	Research published for a scientific readership. Hopes that, once published, research will be put to practical use.	Scientists conduct research that is relevant to and has potential applications in the context of real-world problems (e.g. climate change). Scientists make no pretentions to plan or control who applies their findings or to what ends.
Implicit TOC 2: Jiaolong’s call for scientists to interest broader publics	Broader publics taking an interest in research. Shares outcomes thinking’s critique of publication as scientists’ motivating horizon.	Scientists conduct and communicate research in ways that proactively interest non-scientific communities. Unlike outcomes thinking, scientists do not attempt to plan or control specific effects that research should bring about amongst the target audience.

In this regard, as much as the heterogeneity of models for impactful science that animate the audit cultures of IFF and its donors, I want to underline the provocation that TOC provides for us to reflect upon the structure of scientific practice as it is and as it might become. Again, this is not simply a question of a straightforward difference between a theoretical and an applied orientation: Even before Lesley’s workshop, many IFF colleagues, like many of us in STS and anthropology, already aspired for their research to make a positive difference in the world. The value that Bob located in TOC, however, is in its capacity to render explicit scientists’ existing aspirations and to thereby create an opportunity for critical reflection upon them. As I have shown, Lesley introduced Bob and his colleagues to a TOC that entails distinctive imagination of: the horizons towards which scientific research is to be practiced, the points in time at which the future effects of research are to be anticipated, and how scientists’ agency is to play out across time. Following through on Bob’s suggestion that IFF scientists already have implicit TOCs has meant elaborating IFF scientists’ aspirations and practices in terms of these same dimensions. Doing so not only allows an appreciation of how diverse iterations of TOC might differ from prevalent models of scientific practice, but can also help us to conceptualize latent alternatives to the peculiar structures of ‘impact’ demanded by contemporary audit cultures. In this regard, we can do more than simply critique the logics and sensibilities of planning and evaluation tools like TOC, we can also borrow and re-purpose them. We can take the contours of TOC as the basis for conceptualizing and re-imagining the kinds of agency we want and expect scientific practice to exert in the world.

Footnotes

Acknowledgements

For comments on various drafts of this article, I thank Ben Eyre, Vinnie Ialenti, Sheila Jasanoff, Yan Long, Mary Mitchell, Paul Nadasdy, Annelise Riles, Scott Sorrel, Emiko Stock, and the editor and reviewers at SSS. I extend the same gratitude to several IFF colleagues who commented on a draft of this article, but who I do not name in order to protect anonymity of my ethnographic interlocutors. This paper has also benefited greatly from feedback from participants in Manchester Anthropology’s PGR Seminar, The IGLP Asian Regional Workshop, January 2017, and the ‘Impact and its Discontents’ conference hosted by the University of Manchester in May 2017.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article is based upon work supported by the National Science Foundation under Grant No. 1357194, Cornell East Asia Program, Cornell Mario Einaudi Center for International Studies, and Cornell Society for the Humanities.

ORCID iD

Timothy McLellan

Notes

Author biography

Timothy McLellan is a Postdoctoral Fellow at Northwestern University’s Roberta Buffett Institute for Global Affairs. His research examines how bureaucracies interrupt research practices and explores aspirations to democratize the sciences. He is working on a book based on two years of ethnographic fieldwork at an agroforestry research institute in southwest China.

References

Adam

Groves

(2007) Future Matters: Action, Knowledge, Ethics. Leiden: Brill.

Brown

Green

(2017) Demonstrating development: Meetings as management in Kenya’s health sector. Journal of the Royal Anthropological Institute 23(S1): 46–63.

Callon

(1984) Some elements of a sociology of translation: Domestication of the scallops and the fishermen of St Brieuc Bay. The Sociological Review 32(1): 196–233.

Calvillo

(2018) Political airs: From monitoring to attuned sensing air pollution. Social Studies of Science 48(3): 372–388.

Cash

Borck

Patt

(2006) Countering the loading-dock approach to linking science and decision making comparative analysis of El Niño/Southern Oscillation (ENSO) forecasting systems. Science, Technology & Human Values 31(4): 465–494.

CSIRO (2018) Ensuring we deliver impact. Available at: https://www.csiro.au/en/About/Our-impact/Our-impact-model/Ensuring-we-deliver-impact (accessed 6 August 2018).

Davis

Kingsbury

Merry

(2012) Indicators as a technology of global governance. Law & Society Review 46(1): 71–104.

De Laet

Mol

(2000) The Zimbabwe bush pump: Mechanics of a fluid technology. Social Studies of Science 30(2): 225–263.

Escobar

(1995) Encountering Development: The Making and Unmaking of the Third World. Princeton: Princeton University Press.

10.

Ferguson

(1994) The Anti-Politics Machine: ‘Development,’ Depoliticization, and Bureaucratic Power in Lesotho. Minneapolis: University of Minnesota Press.

11.

Ginsberg

(2011) The Fall of the Faculty: The Rise of the All-Administrative University and Why It Matters. Oxford: Oxford University Press.

12.

Giri

(2000) Audited accountability and the imperative of responsibility: Beyond the primacy of the political. In: Strathern

(ed.) Audit Cultures: Anthropological Studies in Accountability, Ethics, and the Academy. London: Routledge, 173–195.

13.

Goodwin

(2018) Cultures of caring: Healthcare ‘scandals’, inquiries, and the remaking of accountabilities. Social Studies of Science 48(1): 101–124.

14.

Guyer

(2007) Prophecy and the near future: Thoughts on macroeconomic, evangelical, and punctuated time. American Ethnologist 34(3): 409–421.

15.

Hales

(2016) Metrics and market logics in global health. In: Adams

(ed.) Metrics: What Counts in Global Health. Durham: Duke University Press, 125–143.

16.

Hall

Sanders

(2015) Accountability and the academy: Producing knowledge about the human dimensions of climate change. Journal of the Royal Anthropological Institute 21(2): 438–461.

17.

HEFCE (2017) REF impact. Available at: http://www.hefce.ac.uk/rsrch/REFimpact/ (accessed 10 April 2017).

18.

Hilgartner

(2015) Capturing the imaginary: Vanguards, visions, and the synthetic biology revolution. In: Hilgartner

Miller

Hagendijk

(eds) Science & Democracy: Knowledge as Wealth and Power in the Biosciences and Beyond. New York: Routledge, 33–55.

19.

Jasanoff

(2004) Afterword. In: Jasanoff

(ed.) States of Knowledge: The Co-Production of Science and Social Order. London: Routledge, 274–282.

20.

Jasanoff

Kim

(eds) (2015) Dreamscapes of Modernity: Sociotechnical Imaginaries and the Fabrication of Power. Chicago: University of Chicago Press.

21.

Jensen

Winthereik

(2013) Monitoring Movements in Development Aid: Recursive Partnerships and Infrastructures. Cambridge: MIT Press.

22.

Kapferer

J-N

(2012) The New Strategic Brand Management: Advanced Insights and Strategic Thinking. London: Kogan Page Publishers.

23.

Krzywoszynska

Matt

Buckley

, et al. (2018) Opening up the participation laboratory: The cocreation of publics and futures in upstream participation. Science, Technology, & Human Values 43(5): 785–809.

24.

Latour

(1987) Science in Action: How to Follow Scientists and Engineers Through Society. Cambridge: Harvard University Press.

25.

Latour

(1988) The Pasteurization of France. Cambridge: Harvard University Press.

26.

Latour

(2004) Why has critique run out of steam? From matters of fact to matters of concern. Critical Inquiry 30(2): 225–248.

27.

Merry

(2016) The Seductions of Quantification: Measuring Human Rights, Gender Violence, and Sex Trafficking. Chicago: University of Chicago Press.

28.

Miyazaki

(2003) The temporalities of the market. American Anthropologist 105(2): 255–265.

29.

Miyazaki

(2014) Insistence and response: On ethnographic replication. Common Knowledge 20(3): 518–526.

30.

Mosquera-Losada

Santiago-Freijanes

Pisanelli

, et al. (2018) Agroforestry in the European common agricultural policy. Agroforestry Systems 92(4): 1117–1127.

31.

Mosse

(2005) Cultivating Development: An Ethnography of Aid Policy and Practice. London: Pluto Press.

32.

Newfield

(2008) Unmaking the Public University: The Forty-Year Assault on the Middle Class. Cambridge: Harvard University Press.

33.

NSF (2017) Broader impacts. Available at: https://www.nsf.gov/od/oia/special/broaderimpacts/ (accessed 10 April 2017).

34.

Pickering

(1995) The Mangle of Practice: Time, Agency, and Science. Chicago: University of Chicago Press.

35.

Rabeharisoa

Callon

(2004) Patients and scientists in French muscular dystrophy research. In: Jasanoff

(ed.) States of Knowledge: The Co-Production of Science and Social Order. London: Routledge, 142–160.

36.

Rottenburg

(2009) Far-Fetched Facts: A Parable of Development Aid. Cambridge: MIT Press.

37.

Sanyang

Taonda

SJ-B

Kuiseu

, et al. (2016) A paradigm shift in African agricultural research for development: The role of innovation platforms. International Journal of Agricultural Sustainability 14(2): 187–213.

38.

Shore

Wright

(2015) Audit culture revisited: Rankings, ratings, and the reassembling of society. Current Anthropology 56(3): 421–444.

39.

Thornton

(2010) Universities upside down: The impact of the new knowledge economy. Canadian Journal of Women and the Law 21(2): 375–393.

40.

Watanabe

(2019) Becoming One: Religion, Development, and Environmentalism in a Japanese NGO in Myanmar. Honolulu: University of Hawai’i Press.

41.

Winthereik

Jensen

(2017) Learning from experiments in optimization: Post-critical perspectives on monitoring and evaluation. Journal of Cultural Economy 10(3): 251–264.

42.

Wright

Shore

(eds) (2017) Death of the Public University?: Uncertain Futures for Higher Education in the Knowledge Economy. New York. Berghahn Books.