Abstract
Recent years have seen the growing use of systematic literature reviews within the social sciences. Despite some reservations over the adoption of an approach originally popularised within clinical and health sciences, the literature in the area has contributed some clear benefits to accounts of existing research. It is surprising, therefore, that political scientists have tended to ignore, or at best marginalise, reviews of this kind. This article outlines a number of features of systematic reviews which might be suitable for the kinds of questions political scientists ask of their data. Throughout, it highlights both the value and the potential complications of the approach, raising a number of questions which should be considered if systematic reviews are to be adopted more widely.
For a number of years, systematic literature reviews have been an accepted (perhaps the accepted) method of research synthesis in the clinical and health sciences. Clinicians and researchers have relied on a pool of good quality systematic reviews to identify emerging trends in research and intervention, reinforce their practice, and inform areas of future work. In recent years, systematic reviews have also become prominent in the social sciences and researchers interested in education, social work, policy studies and criminology have adopted the approach, with considerable success (Davies, 2000; Gough and Elbourne, 2002; Petticrew and Roberts, 2006; Wallace et al., 2004). It is surprising, therefore, that political scientists have tended to ignore, or at best marginalise reviews of this kind.
This is problematic, as systematic reviews have a number of features which might be useful to the study of political phenomena. The synthesis of a large amount of data while minimising bias, the ability to account for the quality of existing research and the potential for meta-analysis of the findings of different studies will all appeal to most political scientists. Systematic reviews might also allow claims to be made about the effectiveness of political innovations, or identify areas where existing research lacks depth and rigour. However, despite this potential, little has been said about the suitability of reviews of this kind to political research.
In this article, I outline a number of features of systematic reviews which might be suitable for the kinds of questions political scientists ask of their data, with the aim of prompting discussion of the suitability of the approach across the discipline. Throughout, I highlight both its value and potential complications, raising a number of questions which should be considered if systematic reviews are to be more widely adopted. The piece begins with an introduction to systematic reviewing and its use in the social sciences. It then discusses three common objections to the method and possible responses, before highlighting its potential to enhance research in political studies by contributing to scoping and problem-formation and to meta-analysis.
Systematic Reviews and the Social Sciences
Systematic reviews are:
a specific methodology that locates existing studies, selects and evaluates contributions, analyses and synthesises data, and reports the evidence in such a way that allows reasonably clear conclusions to be reached about what is known and what is not known (Denyer and Tranfield, 2009: 672).
This broad definition includes a wide range of different approaches to systematic literature reviews. Every systematic review will be clear about the questions addressed, the boundaries of the review and the criteria for selecting material for inclusion. Some (but not all) may include a meta-analysis of the findings of existing studies.
Systematic reviews are much like any other form of research in political science. They require clearly defined topics and research questions. Appropriate methodologies need to be selected and clearly stated. Data are collected in order to address the research question and subjected to interpretation and analysis. Findings are presented and claims and generalisations may be made. The fundamental difference between most conventional political research and the work carried out under a systematic review is that, in the latter case, the unit of analysis is existing studies, which can then be subject to secondary observations (Petticrew and Roberts, 2006).
Clearly, the idea that we might include some kind of synthesis of existing research in our work is familiar to political scientists. Everyone working in the discipline will, at some point, have carried out a literature review. Indeed, the importance of presenting one’s work in the light of previous research is an academic convention which is enforced by the peer review process, by questions asked at seminars and conferences, even by the examiners of undergraduate essays. However, systematic reviews are potentially important to political science because they question the underlying approach to these activities. By conforming to the standards usually expected of primary research, systematic reviews are able to demonstrate significant advantages by reducing bias (Davies, 2000), increasing transparency (Petticrew, 2001) and recording results and synthesis in a manner which is transparent enough to allow replication (Hansen and Rieper, 2009).
The basic justification for taking this approach to literature reviews is simple: individual studies ‘are limited in the generalisability of the knowledge they produce about concepts, populations, settings and times’ and ‘frequently illuminate only one part of a larger explanatory puzzle’ (Cook et al., 1992: 3). Rather, proper understanding of the findings of a particular study lies in considering them alongside the results of similar studies testing the same kinds of hypotheses, using similar populations. By making explicit their approach to the identification, inclusion and assessment of literature, researchers undertaking systematic reviews are able to speak with far greater certainty about the wisdom of existing research, highlighting common findings – as well as the shortcomings – in previous studies.
The translation of this kind of insight into the social sciences is well-established. Harris Cooper’s (1982) seminal account of the stages of a systematic review marked something of a watershed in the way in which research synthesis was viewed by social researchers. Cooper noted that, despite a general acceptance of the importance of providing an account of existing research, in the social sciences there seemed to be little consideration of the ways in which literature reviews were constructed. In response, he presented an attempt to conduct them in a manner which was both logical and allowed for replication, presenting five stages of research synthesis which allowed for an evaluation of quality and analysis of results:
Problem formulation. Clarifying the question(s) addressed by the review and preparing a protocol governing the review process.
Data collection. Literature searching and information gathering.
Data evaluation. Exclusion of studies, quality assessment.
Data analysis and interpretation. Interpreting the results of collected studies, meta-analysis (where appropriate).
Report preparation. Presentation of findings in a clear and accessible manner.
From this starting point, a wide range of different approaches have emerged, and the variety of methods available reflects the different kinds of research problem to which systematic reviews can be applied (Petticrew and Roberts, 2006). Rapid reviews are limited in scope and time, intended to investigate pressing research problems. Scoping reviews assess the extent and quality of existing research, highlighting potential areas for future work, and are often preliminary to a full systematic review. Realist reviews aim to produce generalisable theories, rather than reach a synthesis of research outcomes, as would be the case in a conventional systematic review. Approaches to research synthesis vary too (Hansen and Rieper, 2009). This is rarely a simple task, and several techniques to deal with methodological heterogeneity, diverse conceptual foundations and varying quality have been developed (Pawson et al., 2005).
Regardless of the approach taken, the structure of the review process is critical. Although it might seem that the latter stages of analysis are the most significant, without paying adequate attention to the initial stages of the review, it is unlikely that any kind of accurate synthesis can take place. The framing of the research question to be addressed is also essential to each of the approaches outlined above. As David Gough and Diana Elbourne (2002: 228) suggest, ‘any reviewing strategy must come from the precise questions driving the review’ and approaches which are too broad, or lack sufficient clarity and focus, can hamstring the process from the outset. Indeed, it is this structured and transparent approach to the entire process which marks synthesis in systematic reviews as distinct from less rigorous meta-analytical approaches.
The influence of this perspective on the wider social sciences is clear, and numerous accounts of the development of a systematised approach to research synthesis attest to its importance (Davies, 2000; Gough and Elbourne, 2002; Petticrew and Roberts, 2006; Victor, 2008; Wallace et al., 2004). This approach has been formalised with the establishment of an international research network, the Campbell Collaboration, which disseminates good practice in the conduct and development of reviews of social interventions in education, criminal justice, social work and welfare. Founded in 2000, the organisation is a sibling to the well-established Cochrane Collaboration (which works on clinical interventions) and has emerged as one of the clearest signals of the development of systematic reviewing in the social sciences.
Research syntheses of this kind have also become highly valued by policy-makers. From the late 1990s, the concept of evidence-based policy moved into the mainstream of public policy and this period famously saw the establishment of numerous centres within government espousing the systematic use of research evidence as a policy tool (Boaz et al., 2002; Davies, 2000; see also Cabinet Office, 1999). The application of systematic reviews to support public policy has generated new challenges, and the approach to reviewing has been adapted and new techniques developed as a result (see, for instance, the development of Rapid Evidence Assessment as a response to some of the difficulties in carrying out timely systematic reviews to support policy-making).
Quite why political scientists have not embraced systematic reviews in the same way is unclear. Most of the research problems tackled in the discipline would benefit from a forensic account of existing knowledge. Indeed, many of the claims made by empirical research might be made in a far clearer way if they are put across in the context of a comprehensive account of the findings of previous work. In the latter stages of this article, I explore some of the ways in which this might work. First, however, I raise three potential problems which should be considered before systematic reviews can be usefully adopted by political scientists.
Three Objections to Systematic Reviews
One explanation for the reluctance of political scientists to embrace systematic reviews lies in a set of objections raised in the literature examining their use in the social sciences (see Petticrew, 2001; Wallace et al., 2004). Some of these are methodological and relate to the perceived narrowness of the approach, highlighting its limitations when dealing with qualitative work and suggesting a positivist prejudice. Other issues are concerned with the relationship between systematic reviews and theory. Interestingly, on closer inspection, these objections may not be quite so problematic as we might expect and indeed, have been successfully addressed elsewhere in the social sciences.
Systematic Reviews Are Unhelpfully Positivist
It is a common misconception that systematic reviews take a rigidly positivist view of the kinds of methodologies used in studies which are suitable for inclusion, and also of the way in which those studies should be treated during analysis (Davies, 2000; Wallace et al., 2004). In many cases, reviewers construct a hierarchy of evidence 1 in order to identify the studies which are most reliable and therefore most suitable for inclusion. For example, the Cochrane Collaboration suggests (correctly) that ‘not all evidence is created equal’, 2 and develops a ranking of evidence which prioritises randomised control trials (RCTs), and then cohort studies, as the approaches best suited to producing high-quality evidence due to their focus on minimising bias. However, the practical and ethical issues of utilising RCTs in social research mean that they are rare within the pages of social science journals, and indeed, may not be desirable (see Davies, 2000). Indeed, concern around the application of a methodology developed in the medical sciences to social research means that they are not always emphasised by social scientists conducting reviews, given the danger of ‘losing’ important evidence through the exclusion of methodologically imperfect studies (Boaz et al., 2002; Victor, 2008).
However, such concerns miss the point of the role played by hierarchies of evidence. Mark Petticrew and Helen Roberts (2006: 58) note that their aim is ‘not to produce a definitive hierarchy of methodological purity for all purposes’ but simply to aid researchers in prioritising the kinds of study gathered when identifying relevant research evidence. Although it is tempting to characterise systematic reviews as holding a ‘methodolatory’ idolisation of RCTs and the like, in fact, sophisticated techniques for the appraisal of a wide range of different kinds of evidence have emerged, and in any case, any hierarchy adopted is dependent on the question addressed by the review (Petticrew, 2001). Even if this were not strictly the case, this kind of approach need not be particularly problematic to many political scientists. Evidence standards can be helpful in identifying ‘what works’ in political interventions. Beyond this, the importance of a clear account of the selection and assessment of the different studies included in a systematic review is central to the value of the approach. It is difficult to carry out a convincing meta-analysis without first establishing the terms on which existing research will be included, and the standard of rigour which can be imposed through the systematic review process enables syntheses of earlier studies to be presented with a greater degree of confidence.
Systematic Reviews Undervalue Qualitative Work
A second, related critique is concerned with the ways in which systematic reviews deal with qualitative research. The issue here is twofold. Some approaches to assessing evidence struggle to cope with small-n studies and consequently position qualitative case studies near (or at) the bottom of hierarchies of evidence (in some cases such evidence is described as ‘anecdotal’; see Petticrew and Roberts, 2006). Beyond this, the intellectual basis of qualitative work needs to be reconciled with the aims (and potential benefits) of systematic reviews. Particularly, the concern with minimising bias can be problematic; as Alison Wallace et al. (2004) highlight, there is a danger that research studies are abstracted from any kind of social context because of the focus of the systematic review process on generalisability and large-n studies, which may potentially result in distorted findings. Martyn Hammersley (2001) takes this further, questioning the value of abstract judgements over methodological validity which prioritises some methods of research design over others.
Again, existing work on systematic reviewing recognises this and presents an ongoing discussion of the practicalities and value of reconciling different research methodologies. Fundamentally, the use of qualitative work depends on the review question, and primary qualitative studies are regularly prioritised in studies in the health sciences. Gough and Elbourne (2002: 228) suggest that the systematic synthesis of qualitative work can be done effectively, so long as the studies included adhere to pre-determined quality standards. They suggest that qualitative data:
can help to clarify the ways in which different issues and events are interpreted and responded to by participants in a study and thus provide clues to the process by which an intervention did or did not have various different outcomes.
Elsewhere, Margarete Sandelowski and Julie Barroso (2006) recognise that qualitative research is under-exploited in most systematic reviews and propose an approach to synthesis which ‘preserve[s] the integrity and enhance[s] the utility of qualitative research’ (Barroso, 2006: 9–10) that has proved highly influential in recent work in the social sciences (Ludvigsen et al., 2015).
Systematic Reviews Are Theoretically Vacant
A final theme of criticism suggests that the theoretical complexity in the social sciences raises a number of difficult questions over the focus and conduct of systematic reviews. Ann Oakley et al. (2005) note the criticisms of systematic reviewing as being somehow intellectually redundant, sacrificing craft and conceptual nous in favour of rigid technique. Concerns abound that studies might be excluded on the basis of theoretical assumptions which could be acceptable to many scholars familiar with the field. Equally, the empirical claims investigated by different pieces of research might be rooted in very different theoretical starting points, even if this is not made clear in the original studies. In this context, it is understandable that some scholars will be nervous over the theoretical implications of this kind of work: systematic reviews aim to be, as far as possible, value-free, and so questions over the relationship between the method and theory need careful consideration.
Once more, there have been some sophisticated responses to these issues. Ray Pawson et al. (2005) outline the realist review as a response to some of the theoretical complexity at large in the social sciences, and in policy interventions in particular. They demonstrate that theoretical differences between studies can be resolved by a careful focus on the theoretical integrity of the synthesis, and adopting an orientation towards systematic reviewing which is both pluralist and flexible. Similarly, as Petticrew (2001) notes, theory has an essential role to play in all systematic reviewing techniques, informing both the selection and evaluation of the studies included. Beyond this, systematic reviews have an obvious contribution to make in testing theoretical assumptions. In political science, questions of theory are rarely settled, and important insights into the validity of normative claims can be reached through empirical synthesis.
Systematic Reviews in Political Science
If objections like these can be resolved, there seems to be little reason why systematic reviews should not be embraced by political science. Indirectly, political scientists have been reaching towards the rigour of systematic reviewing for some time, with scholars regularly attempting either some rudimentary account of the scope of existing material, or explicitly calling for the adoption of the technique. For example, Ben Tonra (2013: 11) notes that ‘a rudimentary search of Google Scholar […] reveals that [Europeanization], linked to “foreign policy”, was cited in just over 200 scholarly publications in 2000, in 800 such publications by 2005 and in nearly 1500 academic publications in 2012’. Elsewhere, Stephen Elstub (2013: 389) highlights the need to ‘systematically review’ the available research evidence related to deliberative democracy. Claims like this are frequently made in the literature and make the absence of systematic reviews from the prominent journals in the field all the more puzzling.
It is rarer to see explicit attempts at actually carrying out systematic reviews, despite a few notable exceptions (e.g. Lutz et al., 2014). Instead, tentative steps towards meta-analysis have been taken in recent years (e.g. Imbeau et al., 2001; Lau, 1999). Valuable as these studies are, it is rare to see direct attention paid to the structured process required under a systematic review: few studies publish review protocols, and clear accounts of search strategies and evidence hierarchies are hard to find. Instead, most attempts at meta-analysis pay close attention to the latter stages of analysis at the expense of preliminary concern over transparency and bias in case selection.
Consequently, there are areas where, much like in other disciplines, systematic reviews can be of great benefit to political science. The potential here is compelling. While we have seen that there are theoretical and empirical questions which need to be considered in any systematic review, these need not be a decisive reason to avoid the approach. Particularly, there are two main areas where the systematic reviews can be of immediate use to political scientists: scoping and problem-formation, and meta-analysis.
Scoping and Problem Formulation
Systematic accounts of existing literature and research evidence can offer a number of benefits in the early stages of empirical work. For instance, researchers can be certain that their studies will not replicate previous work. Furthermore, the early stages of a systematic review require careful thought to be given to the particular research problem addressed by the work. The ability to speak with certainty over the quality and reach of existing research evidence can help to sharpen both the focus and the methodological position taken. Conventionally, the establishment of a research problem has relied upon a particular scholar’s knowledge of the field and their impression of the most pressing issues requiring further research. However, this approach is not always capable of providing a solid foundation, particularly in areas where there is little existing work.
What is needed, in cases like these, is a review which can account for the quantity and quality of existing research studies. Rather than attempting to develop a synthesis of the findings of existing evidence, which in any case may not be possible, the task of the systematic review becomes one of defining a field. Scoping reviews of this kind might be particularly useful in political science. The scarcity of systematic accounts of the findings of research in particular fields means that prior to any kind of meta-analysis being undertaken, an appraisal of existing research is required, and suitable research protocols must be devised with this in mind. In essence, there is a benefit to asking what we know and how we know it.
A consideration of this process asks some interesting questions over the ways in which political scientists establish the foundations to their work. Many of the major assumptions held by researchers in the discipline may not be as solidly established as we think. For instance, there is a widely held wisdom that people on lower incomes are less inclined to vote in elections, and that political participation has declined in recent years. This has formed the basis of a number of major studies (Dalton, 1996; Rosenstone and Hansen, 1993), even providing the subject of a Presidential Address to the American Political Science Association (Lijphart, 1996). However, recent extensive (although not systematic) analysis of existing data has revealed that although a wide range of persuasive evidence exists, the situation may be rather more complex than is commonly assumed, and participation may not be declining in the ways previously thought (Norris, 2011). Clearly, the potential for research synthesis to clarify these, and other issues, should not be ignored.
There are other benefits to this kind of work. Beyond testing the stability of the empirical assumptions at large in the discipline, an important function of systematic reviews is to prevent unnecessary replication of research effort. The dangers of missing important research evidence through a limited or poorly conducted review, and of repeating work already undertaken, can be avoided through a careful analysis of existing research. More advanced reviews may be even more beneficial, providing new insights into established areas of research through the comparison or combination of existing evidence.
Meta-Analysis
Meta-analysis involves the quantitative synthesis of the results of different studies. It is a distinct activity and does not imply a full systematic review (Littell et al., 2008). Originated by Gene Glass (1976), the development of statistical meta-analysis proved to be the catalyst for the adoption of systematic reviewing within the social sciences, and many of the features are relevant to political studies. Meta-analytical techniques have proved valuable tools in reducing statistical imprecision and identifying common findings (and errors) among existing studies. However, their primary value lies in allowing inferences to be made from the pooled findings of different studies; in effect, to look across the results of a wide range of research and identify the most significant and compelling results.
Some limited use is made of meta-analysis in political science, even if it is rarely discussed in terms of the conventions of systematic reviewing. Hristos Doucouliagos and Mehmet Ulubaşoğlu (2008) focus their meta-analysis on the relationship between democracy and economic growth. Taking in 84 published studies, they are careful to provide a clear methodology, including an explicit statement of the criteria for excluding studies from the analysis. Kaat Smets and Carolien Van Ham (2013) review the findings of 90 studies of individual-level voter turnout in order to identify the most salient factors affecting electoral turnout, limiting their sample to studies published in ‘top ten journals’, a list derived by the authors themselves. Kim Strandberg (2008) utilises qualitative comparative analysis, rather than a conventional meta-analysis of existing data on party competition. However, analyses like these can hardly be seen as commonplace and, while their methodologies might be clear and comprehensive, none fulfil the criteria for a systematic review.
Systematic reviews can enhance this kind of research. In particular, the attention paid to the early stages of the review, before analysis takes place, can have a significant role in reinforcing both the rigour and value of meta-analysis. Like any kind of research technique, meta-analysis relies on the formulation of a suitable research question, a comprehensive literature search, and the quality of the studies included. Clearly, there is a role for systematic reviewing here, for instance in mitigating the bias involved in selecting studies for inclusion. Equally, the transparency standards enforced through the review process would enable the results of meta-analysis to be presented with far more certainty and explanatory power. As others have noted, meta-analysis is under-exploited in the political science literature, and the adoption of systematic reviewing techniques provides one way to demonstrate its value (Doucouliagos and Ulubaşoğlu, 2008).
Conclusion
Of course, I am not arguing that every literature review has to be systematic: there are many circumstances where conventional reviews are valuable in formulating an argument and underpinning empirical work. However, literature reviews which claim to be authoritative accounts of existing research surely have to be substantiated by a clear set of conventions which allow for methodological replication and transparency. The great strength of systematic reviews – that they provide an authoritative and transparent means of approaching research synthesis – has much to contribute here and can also enhance the current use of meta-analysis in the discipline. Of course, there may be legitimate objections to applying empirical standards developed in the applied sciences to political studies, but it is clear that there are many areas where approaching the synthesis of existing research can strengthen current work, as has been demonstrated in other social science disciplines.
As a discipline, it is incumbent on political scientists to take systematic reviewing seriously. The precarious place of research like this is made clear in those few accounts which exist in the literature, with the approach marginalised and undervalued (see Daigneault et al., 2012). But the advantages are clear. An account of the existing research evidence which is transparent, systematic and minimises bias can provide solidity to debates over existing knowledge. Where research synthesis is possible, systematic reviews can provide a stable foundation for meta-analysis. For Gough and Elbourne (2002: 234), ‘if research per se is considered a useful endeavour, then bringing together the results of individual pieces of research addressing similar questions should be seen as even more useful and largely unproblematic’. Surely, it is time to consider the benefits which such an outlook might bring.
Footnotes
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.