Abstract
Ex ante evaluations have received little attention in the literature, resulting in a scattered set of contributions, despite their potential impact on society at large and on the academic careers of the members of research teams proposing interdisciplinary projects. Firstly, a review of the existing literature on the evaluation of interdisciplinary research proposals is presented. Then, two alternatives to the traditional peer-review process are described in the context of interdisciplinary research. The conclusions are of potential use both to academics involved in interdisciplinary research projects and to those responsible for funding such projects.
Introduction
Mainstream scientific research at universities and public research institutes has been developed within closed communities of researchers sharing the same specialist backgrounds. However, many social problems can no longer be appropriately addressed in the context of only one scientific discipline and new forms of knowledge production and research practice are required. New forces have helped to shape this recent approach to research: first, large-scale competitive research funding systems, which allocate funds to large-scale research teams involving multiple research institutions in collaboration with industry; and, second, science/technology policies that emphasize and promote interdisciplinary research (Anzai et al., 2012).
Recognition of different forms of knowledge and their value, a wider spectrum of voices calling for further innovative solutions and a need to assure the relevance of research topics is reflected in increasing demands for publicly funded research. Research activities are justified by demonstrating economic and social impacts, which results in more focused research programs and broader activities to enhance the sharing of knowledge (Fazey et al., 2014).
In a research environment that is expected to cross disciplinary boundaries, as a means of gaining greater relevance for society, interdisciplinary research emerges as the lighthouse that contributes to scientific breakthroughs and fosters innovation (Rafols et al., 2012), addressing societal challenges and anticipating the problems that may arise. As a consequence, the research community has, over past decades, witnessed a surge in the interdisciplinarity of science policy discourse as well as the explicit promotion of interdisciplinary research through graduate scholarship and funding programs (Yegros et al., 2013) linked to the belief that interdisciplinarity is now crucial for the progress of fundamental science.
However, the complexity of interdisciplinary research defies a single definition (Huutoniemi et al., 2010). As a result, there are additional difficulties related to the ex ante evaluation of interdisciplinary research projects. Indeed, ‘there is no consensus on how to measure interdisciplinarity in practice’ (Huutoniemi et al., 2010: 79; Huutoniemi, 2012). Even so, when broadly considered, there is a common understanding at a conceptual level of what actually constitutes interdisciplinary research (Academy of Finland, Committee on Facilitating Interdisciplinary Research and Committee on Science, EURAB, OECD and RCUK).
Decision-makers in funding agencies and on program committees, and researchers themselves, need to know how interdisciplinary research can be assessed, as public support crucially reinforces and promotes integration and research collaboration (Anzai et al., 2012). Moreover, research has shown that peer review and the so-called objective assessment methods, in the form of quantitative journal rankings (Rafols et al., 2012), tend to be biased against interdisciplinary research (Langfeldt, 2006; Laudel & Origgi, 2006).
The previous literature on grant proposal assessments has focused mainly on geographical comparative studies and on peer-review reliability (analyzing bias, effectiveness/efficiency and validity), while research on interdisciplinary peer review has been scarce (Holbrook & Hrotic, 2013). The advantages and pitfalls of peer-review evaluation will be discussed through a narrative review of the relevant literature on the ex ante evaluation of interdisciplinary research projects. Then, two alternatives to the traditional peer-review process, adapted to the particular case of interdisciplinary research, will be described.
A narrative literature review that summarizes the emergent literature on this matter is a relevant and solid starting point for the academic community (Paré et al., 2014) with an interest in interdisciplinary research. This review provides a bridge from that starting point to related areas, summarizing prior knowledge on ex ante evaluation of interdisciplinary research projects.
Interdisciplinary research conceptualization
The first step in this review involves a conceptual and empirical definition of what a discipline actually is, in order to clarify the definition of interdisciplinary research. A discipline can be defined as a central problem with items that are considered of relevance to it, as well as related explanations, goals and theories (Wagner et al., 2011). This intellectual structure denotes a set of codified rules, beliefs, perceptions and procedures with regard to the production and evaluation of knowledge (Huutoniemi, 2012) in such a way that ‘disciplines provide crucial knowledge, methodologies, and tools for interdisciplinary and transdisciplinary work’ (Klein, 2006: 121).
An academic scientific discipline has the following features (Brekke, 2014: 520): (1) a particular research object; (2) a corpus of accumulated specialist knowledge referring to the research object; (3) theories and concepts that effectively organize the accumulated specialist knowledge; (4) specific terminologies adjusted to the research object; (5) research methods in line with the specific research requirements of each discipline; and (6) an institutional presence in subjects taught at universities and colleges, their respective academic departments and associated professional organizations.
Much of the effort to define interdisciplinary research examines the features that distinguish it from multidisciplinary and transdisciplinary research so as to prevent imprecise and ultimately interchangeable use of these terms (Lyall et al., 2015).
Multidisciplinary research represents a juxtaposition of disciplines without any of the interchanges or the integration (Lyall et al., 2015) that characterize interdisciplinary research.
Transdisciplinary research, a step forward from interdisciplinary research, denotes a new mode of knowledge production that draws on expertise from a wider range of organizations. It implies collaborative partnerships for sustainability that integrate research from different disciplines with the knowledge of stakeholders in society (Wagner et al., 2011). Transdisciplinary research represents a different form of knowledge production (concepts and/or methodological frameworks), which draws on expertise from a wider range of organizations, and collaborative partnerships (Lyall et al., 2015; Stokols et al., 2003; Wagner et al. 2011).
Transdisciplinary research therefore involves a broad range of expertise, which might include the end users of such research (Lyall et al., 2015), in part to address the ‘applicability gap’, together with practitioners and academics. In this regard, no universally shared definition of transdisciplinary research may be said to exist, despite the existence of various proposals (Jahn & Keil, 2015).
Interdisciplinary research, in between multi- and transdisciplinary research, is characterized by the collaboration and the integration of concepts and methods (and in turn may lead to the generation of new concepts and knowledge) (Lyall et al., 2015). According to Huutoniemi et al. (2010), interdisciplinarity represents interaction between different corpora of knowledge and research practice.
Wagner et al. (2011) state that interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts and theories, which form a holistic view or common understanding of a complex issue, question or problem. Evidence that it is an integrative synthesis, greater than the sum of its parts, is shown by the following indicators: (1) micro-combinations of models or global schemes that unify disparate approaches; (2) consultative and partnering modes, not multidisciplinary contracting of services; (3) coordinated and collaborative inputs and organizational frameworks; (4) formation of a new community of knowledge-holders with a hybrid interlanguage; (5) generation of new insights and disciplinary relationships; (6) a larger, more holistic understanding of the core problem or question; and (7) altered perspectives and a revised hypothesis.
The key features of these research types are summed up in Table 1.
Research types. Key features.
Integration, a key concept found in all of the proposed definitions, which reflects a diversity of knowledge sources in research teams or individuals (Porter & Rafols, 2009), reveals two aspects of knowledge systems: (1) diversity, or the number, balance and degrees of difference between each specific corpus of knowledge; and (2) coherence, or the extent of the relations between specific topics, concepts, tools and data used in a research process (Rafols & Meyer, 2010).
However, interdisciplinary research has in many cases been turned into a broad-brush term that refers to a wide variety of research strategies and practices, which calls for greater rigor in its definition. The precise definition of interdisciplinary research is directly related to the question of how it can be assessed (Pohl et al., 2011), as it shapes competing assumptions with regard to quality (Huutoniemi, 2010). As Pohl et al. (2011: 4) state, the way in which interdisciplinary research is defined is crucial, as ‘to evaluate means to compare a project proposal to a – more or less explicitly defined – ideal interdisciplinary … project’.
In the evaluation of interdisciplinary research, in search of an ideal project, the different types of interdisciplinary research should also be specifically considered, which is not a trivial issue. Tang et al. (2014: 20) and Molas-Gallart et al. (2014: 14–15) recently proposed two modalities in interdisciplinary research, as a consequence of a study commissioned by the Economic and Social Research Council, with the aim of developing an understanding of the way research-users appreciate interdisciplinary research:
A long-range interdisciplinary research modality (also called unifying modality) in which a set of cognitively distant disciplines are integrated through the joint work of interdisciplinary teams to address specific clearly defined problems. To do so successfully, the initiative draws on the active collaboration of stakeholders and generates solutions that can be directly applied to the problematic situation, often as part of the research initiative itself.
A short-range interdisciplinary (or orchestrated) research modality. In this case, the contributions from different, cognitively close perspectives are harnessed to address a broadly defined social or economic problem; in other words, a societal challenge. The researchers work within their areas of expertise and provide a diversity of perspectives. Coherence is maintained by dealing only with cognitively close disciplines (in our case only in the social sciences and humanities). The research results provide novel multidisciplinary perspectives that can then be conveyed to the broad stakeholder communities interested in the societal challenge that is addressed. The way such communities use this information can be difficult to trace, but they are likely to use it in combination with information provided from other sources.
This inductive classification of the types of interdisciplinary research is similar to the deductive classification of Pohl et al. (2011), which identified ‘fundamental understanding’ and ‘problem solving’, respectively. However, Pohl et al. (2011) considered an intermediary category, which they called ‘reflection-in-action’ and which implies critical reviews and changes to current practice and developments in the making, typically involving heterogeneous disciplines.
The classifications advanced by Pohl et al. (2011), Tang et al. (2014) and Molas-Gallart et al. (2014) focus on the issue of interdisciplinary research and how it is approached, to distinguish the different types, while the proposal from Huutoniemi (2010) is based on how new interdisciplinary knowledge is generated, distinguishing between: (a) ‘mastering multiple disciplines’, meaning to enrich disciplinary knowledge production through the cross-fertilization of disciplinary research; (b) ‘emphasizing integration and synergy’ as an alternative, integrative model of knowledge production from different disciplines for the integration of knowledge; and (c) ‘criticizing disciplinarity’, which requires moving beyond disciplines to establish knowledge production with changing boundaries.
A summary of crucial interdisciplinary research indicators and types is presented in Table 2.
Interdisciplinary research (IDR).
These different interdisciplinary research modalities require different approaches for ex ante (as well as ex post) evaluation, which have to be guided by clear criteria that will depend on the interdisciplinary research modality. In any evaluation the form of collaboration, the variety of disciplines to bridge, the type of problems to be addressed and the extent and form of stakeholder involvement should all be considered. These criteria should guide the selection of reviewers for the panels (Molas-Gallart et al., 2014; Tang et al., 2014).
Thus if the types of interdisciplinary research are considered as those identified by Huutoniemi (2010), the evaluation in the first case should be based on simultaneous fulfillment of the respective disciplinary standards, while the evaluation in the second approach should combine existing disciplinary standards with new standards. Finally, evaluation in the context of critical attitudes toward disciplinarity requires the transformation of existing standards.
Ex ante evaluation of interdisciplinary research
According to Fazey et al. (2014), evaluation involves determining value, significance, worth or condition through an appraisal process that is used in social science research to answer policy-oriented questions and to assist with decision-making. In the particular case of interdisciplinary research, the assessment task is particularly challenging as, by definition, this type of research is a hybrid compound of more than one discipline (Huutoniemi, 2012), each one with different (and even conflicting) notions of quality. This situation leads to criteria at different levels differing in terms of both knowledge domain and institutional location, as well as the purpose of the integration and the forms it can take (Klein, 2006).
Peer-review evaluation
A well-known and widespread practice for evaluating research work is peer review. In peer review, scientific peers (commonly colleagues working in the same field on similar topics) with demonstrated competence assess research work (Huutoniemi, 2012). A summary of peer-review processes in project evaluation appears in Figure 1. The evaluation of projects and their output is subject to peer review, as are the academic performance of individuals and, even, the evaluation of programs or complete policy strategies (Molas-Gallart, 2012).
The peer-review process: grant applications.
Implicitly, the peer-review evaluation process means that qualified experts certify proposed or completed work using the so-called ‘yardstick’ of rational and ‘objective’ decision making (Klein, 2006). The quality control system has therefore been upheld in science by the peer-review system, which, first, selects the highest-quality work (by using qualified peers in the selection process) and, second, supports the assessment of legitimacy (by communicating relative transparency in the assessment) (Hansson & Mønsted, 2012).
The peer-review system is considered to be the most effective tool for promoting the critical selection of research, essential to the evolution of scientific knowledge, in the stream of critical rationalism that the scientific community supports (Bornmann, 2011). Inter-rater reliability, the most valuable collective product of panel deliberation, is theoretically guaranteed by means of peer consensus, which provides clear signals for funding decisions (Huutoniemi, 2012).
However, given that interdisciplinary research is always a new synthesis of expertise, it is at the very least difficult to find peers, because few or very few of them could be competent in all aspects of the project. This imbalance may, importantly, cause a bias in grant decision making against interdisciplinary grant proposals, because peers show an almost unconscious bias toward research in their own field (Laudel, 2006). Previous research has shown poor quality in peer review both for scientific research and for journal publication in biomedical areas (Graves et al., 2011), which is partially due to real and legitimate differences in opinions over what constitutes good science (Fogelhom et al., 2012).
Laudel and Origgi (2006) summed up the standard outputs of the interdisciplinary research assessment process as follows: (a) contradictory assessments are common; (b) the intellectual breadth of a proposal is typically mentioned as a positive quality and missing disciplinary grounding is typically mentioned as a negative one; (c) applications joining disciplines with a regular pattern of collaboration are more likely to be accepted than those joining disciplines with less interaction; (d) the use of indirect measures (such as patents, publications, reputation, awards) and reference to them, as well as trust in other experts is common when reviewers have to evaluate projects that are not within their field of expertise.
Although scientists have shown a high level of satisfaction with the peer-review system in some previous research on grant proposals and journal manuscripts (Bornmann, 2011), Frey (2003) claims that the system implies a kind of intellectual prostitution, since the authors are slavishly compelled to incorporate the amendments demanded by the reviewers if they wish to see their manuscripts published. The peer-review system is called into question in the case of grant proposals in the Cole et al. (1981) study, as they found that, in the assessment of 150 grant proposals submitted to the National Science Foundation, the decision to fund depended to a great extent ‘upon which reviewers happen to be selected’ (Cole et al., 1981: 881).
Huutoniemi (2012) highlighted two cognitive tendencies in peer review as particularly damaging to interdisciplinary research. First, ‘cognitive homophily’, or the evaluator’s tendency to perceive excellence when a proposal that has been evaluated resonates with the work they themselves are conducting. Second, ‘gate keeping’, when evaluators, trying to protect their own epistemic territory from new perspectives and approaches, tend to be conservative. Both tendencies can mean that the prospects for interdisciplinary research are considered unorthodox examples of disciplinary research (Huutoniemi, 2012) and therefore obtain low scores in the peer-review process.
A high quality peer-review process should be reliable, fair and valid. Reliable means a high level of agreement between individual judgments; fair refers to the absence of bias, which implies any feature of a reviewer’s ‘cognitive or attitudinal mind-set that could interfere with an objective evaluation’ (Shatz, 2004: 36) and, finally, valid is related to the ‘best’ proposal selection (although there is no generally accepted definition of a proposal that is ‘worthy of funding’).
Reliability in the assessment of scientific work is often impossible to achieve (Cole, 2000) when considering cutting-edge interdisciplinary research proposals at the frontiers of research. It is severely restricted by the range of different criteria that the representation of such diverse perspectives entails (Marsh & Ball, 1991): too much agreement because of a lack of diversity in reviewer selection would be symptomatic of a non-appropriate review process (Bailar, 1991).
Simple procedures may be used to improve the reliability of peer-review evaluations, such as providing reviewers with detailed instructions (or evaluation forms) that list the essential elements of the evaluation. Another alternative is training, given that research has demonstrated that the degree of inter-reviewer agreement improves when reviewers evaluate more proposals and when there are more reviewers for each proposal (Lee et al., 2013).
A more complicated way of achieving reliability is through the composition of the panel of reviewers, searching for an optimum balance of overlapping disciplinary competences among panel members. In this way the members of the panel can debate ‘the strengths and weaknesses of different research approaches in relation to the proposal’ (Huutoniemi, 2012: 905).
If the objective is to promote interdisciplinary research, Langfeldt (2006) recommends more of a risk-taking approach to the review system, minimizing peer review. However, other proposals, such as open peer review (Pöschl, 2013), are not applicable in the case of grant proposals, due to the confidential information they contain.
With regard to bias, Lee et al. (2013) question the assumption that disagreement between reviewers is undesirable or non-appropriate. ‘Research on bias as a function of author characteristics adopts the untested assumption that authors belonging to different social categories submit manuscripts and grant proposals of comparable quality’ (Lee et al., 2013: 13). Previous empirical research on bias that considered such variables as the gender of authors (Bornmann, 2008, 2011), their origin in non-English speaking countries and interdisciplinary research (Lee et al., 2013) have yielded mixed results.
Finally, predictive validity is frequently inferred from the number of citations in publications, which is meant to serve as an indicator of the impact of international research. However, in the case of grant proposals the task of peer reviewers is to evaluate the potential of the proposed research (whether the proposed content is significant and feasible), as opposed to peer reviews of submissions to journals, in which the results are assessed. A grant proposal assessment is therefore perceived ‘as entailing a heightened risk for judgments and decisions with low predictive validity’ (Bornmann, 2011: 231).
Nonetheless, in spite of the above viewpoints, it is generally accepted in interdisciplinary research evaluation that a panel of experts from different fields is better equipped than any individual expert. This viewpoint is due to the fact that collective deliberation among the members of a panel means a more comprehensive and balanced evaluation, rather than evaluation by means of a composite review of several independent evaluators (Boix Mansilla, 2006; Langfeldt, 2006).
In-depth ethnographic research on multidisciplinary peer-review panels confirmed that a group context may indeed dissuade reviewers from institutional bias and encourage methodological pluralism, as the credibility of panel members was damaged whenever they promoted their own fields too strongly (Lamont, 2009; Lamont et al., 2006). Studies on the decision-making processes of evaluation panels suggest that collective evaluation processes often involve a clear division of scholarly tasks, little interaction and tacit compromises (Huutoniemi, 2012). Lamont (2009) terms those compromises ‘horse-trading’; in other words, enabling other panelists’ objectives to be achieved in return for reciprocal behavior from those same panelists later on (Lamont & Huutoniemi, 2011).
Improvements to the peer-review process of research project applications
Laudel (2006) emphasizes the importance of designing a process that enables a proper assessment of interdisciplinary research. Some alternatives to peer review have been proposed. Holste et al. (2012) proposed and tested a scientometric-statistical model for the analysis of Start-up Grants from the European Research Council. It was not done with the aim of substituting the peer-review process, but rather to stimulate the generation of methods for monitoring the effectiveness of the peer-review process from a scientometric perspective that in no way relies exclusively on the statistics of publications and citations. This scientometric-statistical model requires the generation of indicators, and metrics for those indicators, in accordance with the specific characteristics of the interdisciplinary research the funding agency wishes to encourage.
Other recent proposals for interdisciplinary evaluation, such as those of the Interdisciplinary Adjudication Committee of the Canada Research Chairs (Canada Research Chairs, 2014) and the Institute of Advanced Study of Durham University (Strang & McLeish, 2015), are also based on peer review.
Although the best known instrument of the Swiss National Science Foundation for the evaluation of inter- and transdisciplinary research (Defila & Di Giulio, 1999) suggests that the evaluation of interdisciplinary research projects has aspects that can be evaluated by non-academics (such as formal requirements by the secretary of the funding institution or project management by an expert in organizational development), this aspect does not mean that other components of the proposals should not be evaluated by expert scientists.
The collaborative research network proposed by Laudel and the peer-review proposal of Rand Europe (Guthrie et al., 2013) are two interesting ways of improving peer review, through the stimulation of ideas for an improved assessment system in the evaluation of interdisciplinary research proposals. However, the Td-net Network for Trandisciplinary Research Proposals was preferred because its question-based structure makes it a highly practical instrument that provides guidelines for transdisciplinary research project assessment. The Collaborative Research Network (Laudel, 2006; Laudel & Origgi, 2006) and Td-net (Pohl et al., 2011) are explained below in greater detail.
Collaborative Research Network
According to Laudel (2006) there are two possible answers to the key question of measuring success in the peer review of interdisciplinary research:
Success implies selecting the best proposals for funding, although this approach requires parameters to specify how they should best be understood.
Success is the agreement of all parties (i.e. both successful and unsuccessful applicants as well as reviewers, officers of the funding agency and funders) that the best proposals were funded in a fair procedure with competent assessments. Unfortunately, even this agreement may raise doubts due to idiosyncrasies, power imbalances, personal agendas, etc.
Laudel (2006) proposed what in her opinion was the most realistic and empirically verifiable criterion of success: whether the procedure could in some way compensate the additional strain placed on the peer-review process by the interdisciplinarity of the proposals and the multidisciplinarity of the panel. Hence, Laudel (2006) highlighted the role played by the specific institutional conditions (the rules of the game) and by other conditions of action, to compensate for the impact of specific epistemic conditions (knowledge heterogeneity). Through the empirical analysis of two collaborative research networks, Laudel (2006) set up three conditions for interdisciplinary research review:
The analysis of all aspects of an interdisciplinary proposal can be guaranteed if applicants participate in the selection of reviewers, while the role of the funding agency is to assure the minimization of conflict of interests and cognitive particularities.
The assessment process is based on group discussion, both between reviewers and applicants and between reviewers (a time-consuming and demanding process, as reviewers have to establish interdisciplinary communication that requires a good command of the language of other disciplines).
The process is embedded in the everyday scientific communication of applicants and reviewers, who conduct scientific discussions at workshops and conferences on the Collaborative Research Network.
Such a process involves three important consequences. First, there is a relative empowerment of the applicants (in return, relative constraints are placed on that empowerment). Reviewer performance is assessable due to the non-anonymous procedure in which the applicants have the right to propose and to veto reviewers before the peer-review procedure begins. Moreover the review-process procedure requires scientific argumentation in support of all objections (against projects and reviewers). Second, the procedure focuses heavily on reviewer competence because reviewers have to learn about projects and cannot insist on their own narrow perspectives. Cognitive particularism is reduced thanks to remote reviewers and neutral members of the grants committee. Group discussions together with the participation of reviewers in scientific meetings generate the common background knowledge necessary for successful interdisciplinary communication. Finally, this process also involves costs for the funding agency.
A combination of all those elements ensures that the interdisciplinary peer-review process functions, according to Laudel (2006), in spite of its weaknesses: expensive and time-consuming processes. Moderate interdisciplinarity is required because of the importance of a common basis for communication and consensus.
Table 3 provides a summary of the main features of the Collaborative Research Network.
Collaborative Research Network: Description.
Source: Laudel (2006) and Laudel & Origgi (2006).
In a different context, the research results of Huutoniemi (2012) lend weight to the conclusions of Laudel (2006). Huutoniemi (2012) researched peer-review deliberations (in terms of scope and disciplinary heterogeneity) in four different evaluation panels, revealing communication across disciplinary boundaries and the formation of interdisciplinary judgments. The common practice of communication over epistemological differences in the form of deliberation is the remedy for most disagreements, overcoming individual understanding of a proposal in the search for agreement.
Agreement, Huutoniemi specifies, in this case means finding the conventions to tolerate, more or less explicitly, divergent views, adjusting initial perspectives and resolving different viewpoints, even though their precise understanding of the merits of a particular proposal may not be held in common.
Also in line with the work of Laudel (2006), the results of Huutoniemi (2012: 917) suggest that the mix of disciplinary specialities in a review panel may explain its consensual practices, which support the importance attached to panel member selection, as ‘depending on other disciplinary standards and other panelists’ behavior, a panel member may acknowledge the strengths and weaknesses of a proposal differently when negotiating with colleagues from her or his own field than when doing so with colleagues from other fields’. The inevitable variance in reviewers’ judgments, Huutoniemi (2012) states, can hardly mean that the outcome of the evaluation is merely dependent on chance.
Td-net Network for Transdisciplinary Research proposal
In 2011 the Swiss Academy of Arts and Sciences proposed a practical set of questions to assess interdisciplinary research, in accordance with its major purpose, aiming to highlight the quality and the synthesis of integration (Pohl et al., 2011).
With regard to the panel reviewers for interdisciplinary research proposals, Td-net identified four elements to be changed and adopted.
First, the composition of the panel should include evaluators with different disciplinary expertise, mixing specialists and generalists, and incorporating interpreters as intermediaries in the panel. Second, the assortment of external reviewers should include both specialist and generalist reviewers with different areas of expertise, selecting the reviewers in accordance with the purpose. Third, the design of the review process should select reviewers and evaluation criteria in a joint process decided on by the reviewers, the panel and the applicants, in such a way that the process is adapted to the purpose.
Finally, the fourth point involved the questions that the reviewers have to answer. Td-net proposed a set of questions for reviewers, who were asked to evaluate a proposal on the basis of specific interdisciplinarity qualities, highlighting its integration and synergy, and considering the following aspects as crucial:
Breadth (the number and the diversity of disciplines, methods and analytical scales – molecular, cellular, individual, social and global – that are combined in the project).
Integration (the reasoning for an integrative approach, whether the applicants have the concrete methods or tools to make integration happen and the assessment of the proposed integration in terms of suitability and a balanced interplay of perspectives).
Reflection and learning (whether there is time for mutual understanding and learning considering the different disciplines in the project and the possibility of reframing or adjusting the planned integration and collaboration).
Problem solving (whether an interdisciplinary approach is really required for the issue in the proposal and whether the project–participants tandem can make a change).
Management, social and leadership skills (referring to a capability of the management structure to match the project aims and to support them, and whether the team members are able to revise their scientific perspective and to manage stress).
As the assessment of interdisciplinary research implies an evaluation of the expertise involved in it, as well as the way its specific purpose is integrated, a clear identification of its purpose is necessary so as to verify its appropriateness and relevance. In this regard, three types of interdisciplinary research can be identified:
Fundamental understanding. Although this aim is shared with disciplinary research, the difference is that information, data, techniques, tools, perspectives, concepts and/or theories from a set of different disciplines are combined to advance fundamental understanding of an issue that requires collaboration;
Problem solving related to real-world issues that involves collaboration with non-academic participants;
Reflection-in-action involving critical reviews and changes to current practice and developments in the making, which is a bridging position between fundamental understanding and practical problem solving, typically involving heterogeneous disciplines.
Table 4 lists these questions and shows the way in which they were applied, in accordance with the purpose. There are two initial questions and, subsequently, the questions on integration (Q3, Q4, Q5), on the elaboration of the approach to learning (Q6) and on management and social and leadership skills (Q10, Q11) are all common to all the three types of interdisciplinary research that are distinguished (fundamental understanding, problem solving and reflection-in-action). In addition, for the purpose of reflection-in-action (Q7) as well as for problem solving (Q8, Q9), the possible impact can be assessed (Pohl et al., 2011). As an example, Di Iacovo et al. (2015) used this instrument to assess trandisciplinarity in a social farming project.
Questions for interdisciplinary research evaluation depending on the purpose.
Source: Pohl et al. (2011).
Conclusions
There are at present no generally accepted criteria and procedures for the evaluation of interdisciplinary research (Pohl et al., 2011), but since research is increasingly organized into time-bound, externally funded projects (König et al., 2013), the issue of research assessment is a crucial one, particularly in the case of interdisciplinary research, where the necessary integration, a defining feature of this type of research, sets specific challenges for its assessment.
Integration is a key concept in interdisciplinary research. A valid assessment of interdisciplinary research requires some indication of the degree or the extent of knowledge integration taking place as the research progresses. However, integration raises difficulties, as it is basically a cognitive process, whether it takes place within an individual’s mind or within a group (Wagner et al., 2011).
Porter et al. (2006) identified systematic factors grouped into contextual forces and factors that impact on interdisciplinary research while it is underway. The first contextual force was funding. Generally speaking they proposed to review the mechanisms that overcome disciplinary barriers to the understanding and the evaluation of interdisciplinary research proposals.
As Huutoniemi (2012) states, interdisciplinary goals are better served by adjusting the review process to panels formed of members who wish to promote innovative approaches and socially relevant research. These panels are composed of more broadly defined constituencies, instead of more narrowly defined established experts who concern themselves more with upholding high standards of technical merit. Therefore, ‘the intersubjective context of evaluation should be designed in a way that facilitates interdisciplinary dialogue’ (Huutoniemi, 2012: 198).
In this literature review, a practical example and a proposal have been described as a means of improving the evaluation of interdisciplinary research grant proposals within the peer-review process. Recently, from an empirical point of view, Fogelhom et al. (2012), in a comparison of two sets of scores from expert panels, reviewed the same grant proposals to show that considerable inter-reviewer disagreement was reduced by discussion in view of reaching a consensus, although the costs were quite high in the evaluation process, and the time constraints quite demanding. Fogelhom et al. (2012) believe that the mean score of the individual reviewers appears to be a good alternative, as the reliability of the results was not compromised.
The role of peer review therefore remains a key defining component of research evaluation. So, the design of the evaluation and its implementation should be carried out by scientists holding knowledge in the relevant field, particularly if it is considered that scientific activity should be steered from inside the scientific community, even though research can be funded with a strategic end in mind (Molas-Gallart, 2012).
In the case of the evaluation of interdisciplinary research projects, it is worth asking whether it should be done within specific programs, given the growing trend of including interdisciplinary elements in regular programs. However, the principles that guide good interdisciplinary research as well as the effective approaches to their evaluation are equally valid for monodisciplinary research and can even strengthen it (Strang & McLeish, 2015). Funding agencies would, in this way, not incur (human, time-related and any other type of resource) costs that entail the generation of different evaluation schemes for monodisciplinary research and for interdisciplinary research.
External assessment is a necessary component of the ‘strategic science’ notion that combines the long-term view of socio-economic returns from research investments with expectations of its relevance. It takes the relevance of the scientific activities into account within evaluation systems that combine scientific peers (expert evaluation), professionals (evaluation experts) and other stakeholders (Molas-Gallart, 2012).
In both cases, the Collaborative Research Network experience and the Td-net proposal can provide practical guidance. The evaluation criteria of the proposals in the field of strategic science might need to be adapted to the strategic objectives pursued, in particular with regard to the desired impacts. In this sense, Langefeldt and Scordato (2015) present an interesting review of methods and practices used by funding agencies of international reference for research impact evaluation.
Their comparative study maps the assessment processes at four worldwide leading funding agencies (Natural Environment Research Council, United Kingdom; The National Science Foundation, United States of America; The Research Council of Norway and Horizon 2020, European Union). All these bodies trust in peers for their research project evaluation processes, although they incorporate users in the evaluation process in some modalities (or at least marginally so, insofar as the users are reviewers from public agencies).
The agencies, entrusted with the design of the evaluation process, the selection of the evaluators and even the evaluation criteria, should consider various points in the evaluation of interdisciplinary research.
In the first place, the training of funding agency personnel will contribute to their awareness of the particular demands of interdisciplinary research evaluation (Lamont, 2009) and will help to build the interdisciplinary capacity of funders that is recommended by Lyall et al. (2013). In addition, the joint participation of applicants and reviewers in the preparation of evaluation criteria should be encouraged by funding agencies, which should guarantee that the selected reviewers have a minimal prior understanding of what interdisciplinary research entails (Laudel, 2006).
In second place, a funding agency cannot ignore that revision through panels of reviewers is a process of social interaction, in which the achievement of consensus is necessary (Lamont, 2009; Van Arensbergen et al., 2014). Thus, the panels should evidently be heterogeneous (including specialists and generalists, and both internal and external reviewers) with a certain degree of overlap in competences, to facilitate a fruitful discussion among their members (Huutoniemi, 2012; Pohl et al., 2011; Van Arensbergen et al., 2014).
An environment should be generated in which good communication and a collegial atmosphere prevails (without it muddling up divergent opinions, given that they stimulate information exchange), while reducing temporal pressure as much as possible (Van Arensbergen et al., 2014). In this sense, in the same way as an excessive cognitive distance should not exist between the reviewers and the applicants, to avoid bias and maintain fairness (Wang & Sandström, 2015), neither should there be an excessive cognitive distance between panel members, although this point needs further empirical research to be corroborated.
Finally, making panelists more accountable by reporting on the review process favors the search for and the sharing of information, and the explanation of the evaluation criteria that were used. It also avoids making premature decisions (Van Arensbergen et al., 2014). On this point, the proposals of the Td-net Network (Pohl et al., 2011) presented in this work are useful references, as are those of Strang and McLeish (2015) and Defila and Digiulio (1999).
However, for an appropriate and fair evaluation of interdisciplinary research projects, not only is the role played by funding agencies fundamental, but so too is the role of the reviewers themselves. These reviewers should collaborate to generate shared background knowledge (through discussion groups and participation in scientific meetings, for example) and should be ready to learn from the projects they evaluate (Fogelhom et al., 2012; Laudel, 2006). Skills related to communication and consensus (Huutoniemi, 2012; Laudel, 2006; Lyall et al., 2011) as well as awareness of personal subjective bias and its possible implications for evaluation (Lyall et al., 2011) are all characteristics the evaluators of interdisciplinary research projects should possess.
Footnotes
Funding
This work was supported by the European COST Targeted Network TN 1301 Next Generation of Young Scientist: Towards a Contemporary Spirit of R&I – Sci-Generation (grant number TN1301).