Judgment,Resources,and Complexity

Abstract

Systematic reviews play an increasingly important role in decision making in health promotion and public health. However, little has been published on how systematic reviewers acquire necessary knowledge and skills, and on the challenges they face in producing reviews. Semistructured interviews were conducted with a purposive sample of 17 systematic reviewers of health promotion. They described practice, training, and mentoring as being key ways that they learned reviewing skills, often in combination. Practice-based learning was considered to be particularly beneficial. Training was generally easy to access, though questions were raised about the feasibility of training stakeholders such as health professionals to become reviewers. It was suggested that an understanding of research methods is beneficial for novice reviewers. While funding opportunities for doing reviews are available, long-term investment is needed to support an infrastructure for the production of high-quality systematic reviews of important health promotion priorities.

Keywords

health promotion public health systematic reviews evidence synthesis research capacity

Introduction

Systematic reviews have come to play an integral role in decision making in health promotion and public health (Oliver, Thomas, Harden, & Oakley, 2006; Waters, 2009). They aim to ensure that policy and practice is influenced by sound and reliable evidence, maximizing health benefits and minimizing potential harm. An infrastructure for the promotion of evidence-based health has been evolving since the early 1990s, with the establishment of international organizations such as the Cochrane Collaboration, the Campbell Collaboration, and in the United Kingdom the Centre for Reviews and Dissemination (Clarke, 2006; Sowden & Glanville, 2006).

Advantages of systematic reviews include defining the impact of an intervention more precisely than a single primary evaluation, and the summary of large volumes of literature, providing a convenient resource for busy policy makers and practitioners. Many health promotion and public health interventions are multifaceted, involving a range of people with different skills and backgrounds, occurring in multiple settings, and covering a range of activities. To be meaningful, systematic reviews need to adequately describe these interventions, assess their effects, and illuminate the factors contributing to success or failure (Pettman et al., 2011; Sweet & Moynihan, 2007; Waters et al., 2011). To achieve this, systematic reviews are increasingly addressing broader questions which, in turn, require more innovative methods. There are published examples of systematic reviews that have included process evaluations (Shepherd et al., 2010) or qualitative studies (Brunton et al., 2005; Garside, Pearson, & Moxham, 2010; Rees et al., 2006; Shepherd et al., 2006) or other forms of evidence (Pawson, Greenhalgh, Harvey, & Walshe, 2005), thus attempting to answer “Does it work?” alongside “Why does it work? ‘For whom? And ‘Is it appropriate?”

The shortcomings of the public health evidence base as seen by policy makers have been investigated, including poor availability of evidence for effectiveness and cost-effectiveness of approaches to tackling health inequalities (Petticrew, Whitehead, Macintyre, Graham, & Egan, 2004). This supports the need for systematic reviews that address important public health priorities, such as assessing complex multicomponent interventions and use different types of evidence. Such interventions may be evaluated by experimental or observational evaluation designs, and may also use qualitative data.

Despite advances in methodology (Egan, Bambra, Petticrew, & Whitehead, 2009; Harden et al., 2004; Mays, Pope, & Popay, 2005), adequately meeting this demand will be challenging for systematic reviewers. In particular, they will need sufficient training and development opportunities, which specifically addresses this type of evidence. The U.K. Government’s review of inequalities and the wider determinants of health in England affirmed the value of systematic reviews in establishing an evidence base for action, but noted a lack of capacity within public health to produce them (Wanless, 2004). Yet, there are very few published research studies documenting the training needs of people learning to conduct systematic reviews, and their experiences and reflections of applying their knowledge and skills in practice (Nind, 2006; Oakley, 2003; Wallace et al., 2006). There are unanswered questions such as What are the key challenges that reviewers face? How do they deal with them, and with what success? What are the barriers, to and facilitators of, learning reviewing skills? What are reviewers’ experiences of helping others to learn systematic reviewing? And, what can we do to facilitate effective learning to ensure adequate capacity for systematic reviewing? Given the value placed on evidence from systematic reviews to inform policy and practice in health promotion and public health (Killoran, Swann, & Kelly, 2006; Petticrew et al., 2009), it is important these questions are addressed so that future systematic reviews can meet the needs and expectations of those who need them.

The aim of this qualitative study, therefore, was to conduct a series of semistructured interviews with systematic reviewers of health promotion (and more broadly public health) to investigate their experiences of doing reviews, their reflections on the learning process, and their recommendations for future training and conduct of reviews.

Method

An interview schedule was devised, piloted, and refined before being used. In general, the questions were openended, with probes used as necessary to elicit further information. Since the nature of this research was exploratory, a combination of purposive and opportunistic sampling strategies was used (Bowling & Ebrahim, 2005; Green & Thorogood, 2004). Sampling was purposive in the sense that interviewees were sought with the experience of producing systematic reviews of health promotion topics. To be eligible, a person had to have been a lead-author or coauthor on at least one systematic review of effectiveness of a health promotion topic (a broad definition of health promotion was employed, consistent with that of Green & Kreuter, 1999). Interviewees with experience of systematic reviewing specific types of health promotion intervention (e.g., one-to-one interventions, multicomponent interventions), and different health topics were also purposively sought.

The majority of the interviews were planned to be conducted at an international conference on evidence-based health (the Cochrane Collaboration’s annual Colloquium). This conference facilitates debate, discussion, and reflection on issues relevant to the methodology for producing systematic reviews, and more broadly, the relationship between evidence and policy and practice. The rationale for choosing this particular conference was to access people with experience of the methodological issues associated with conducting systematic reviews—the central focus of this study. It was considered important to access a range of views and experiences, particularly among systematic reviewers from countries outside of Europe and North America where research cultures and infrastructures are likely to be different. Collectively, delegates attending the conference represented 45 countries. Furthermore, given the work of the Cochrane Public Health Group in supporting systematic reviews, the conference was an ideal opportunity to interview people with experience of reviewing in this area.

The sampling frame was the subset of 145 systematic reviews published in the Cochrane Database of Systematic Reviews at that time classified as being within the scope of health promotion and public health. The intention was to sample up to 20 lead-authors or coauthors of the 145 systematic reviews, representative in terms of topic, country, and type of intervention. An e-mail was sent directly to the contact author of a purposive subsample of the 145 systematic reviews prior to the Colloquium to enquire if they were attending, and if so to request an interview. E-mails were also relayed on behalf of this project by Cochrane Review Group Coordinators, and also from the Cochrane Public Health Group, to their contact databases encouraging eligible people attending the Colloquium to volunteer to be interviewed. These e-mails generated interest from some eligible individuals who were attending the conference, plus from those who were not attending. Some of those in the latter group were subsequently interviewed face-to-face in the United Kingdom (n = 5), or by telephone for those overseas (n = 2). Furthermore, at the conference itself opportunistic sampling took place of individuals who, during informal contact, appeared to be eligible. The total number interviewed at the conference was 10, bringing the total number of interviewees in this study to 17. At this point, sampling ceased as preliminary data analysis indicated saturation in terms of types of responses elicited. All of the interviewees provided informed verbal consent and were assured of confidentiality and anonymity in the reporting of results.

The interviews lasted, on average, for around 45 min (note that the interview schedule included questions on other aspects of systematic reviews not reported here because they are not within the scope of this current article). The interviewer (the author of this article) had participated in a number of systematic reviews of both health care and health promotion interventions, and was trained and experienced in conducting qualitative interviews. It was unlikely that interviewer bias occurred as the interviewer was unfamiliar with the majority of the interviewees prior to the interview (and vice versa), and had not formally collaborated with any of them academically.

All interviews, with the exception of one of the telephone interviews, were tape recorded. Each interview was transcribed by a secretary and checked immediately by the interviewer for accuracy. Following any necessary amendments to the transcript, a content analysis of the data was performed. This method was chosen as it is a standard and relatively straightforward technique used in the social sciences to categorize and explore qualitative data from interviews. The interview transcripts were read and reread by the interviewer and the frequency with which key terms and issues were mentioned was recorded in relation to each question on the interview schedule (Low, 2007). These issues were then grouped into themes and the relevant dialogue coded according to each theme. Once this was complete, all of the themes were examined and higher order themes, which summarized the key findings from the study, were generated (e.g., subjectivity, time, and resources, etc.). It was considered important to seek a second interpretation of the themes to arise from the data to ensure a degree of objectivity. The themes were therefore reviewed by, and discussed with, the project director and following discussion minor amendments to the themes were made. Specialist NVivo computer software was used to analyze the interview data (Version 2.0, QSR international).

Basic data analysis took place during the course of the interviews in order to assess initial findings and the degree of saturation. This involved reading the interview transcripts and making notes on the general findings. More detailed analysis, as described above, took place once the interviews had been completed. As many of the interviews took place within a short space of time (i.e., at the conference), it was not practical to further explore initial findings during subsequent interviews.

Results

The majority of those interviewed were based in the United Kingdom (n = 8), followed by Australia (n = 4), Canada (n = 2), the United States (n = 1), South Africa (n = 1), and Nigeria (n = 1). Eight of the 17 interviewees were classified as academics, most employed in universities and ranging in position from junior research fellows to professors. Five were academic health professionals. Interviewees not classed as academics (n = 4) included three people employed by the Cochrane Collaboration (e.g., Review Group Coordinator; training officer) and a hospital-based ophthalmologist. Despite not being employed by an academic institution, they all had research experience.

Learning to Do Systematic Reviews

The interviewees were asked how they learned to do systematic reviews. Learning through practice (n = 11/65%), support from colleagues (n = 11/65%), and training courses (n = 10/59%) were commonly cited ways of learning, often in combination (Table 1). Other methods mentioned were literature and written resources (n = 9/53%); applying existing research skills (n = 5/29%); academic courses (n = 3/18%); and supervising and teaching others (n = 2/12%).

Table 1.

How the Interviewees Learned to Do Systematic Reviews?

Method of Learning	Number (%) of Interviewees
Practice	11 (65)
From colleagues and mentors	11 (65)
Training courses	10 (59)
Literature and written resources	9 (53)
Applied existing research skills	5 (29)
Academic course	3 (18)
Supervising and teaching others	2 (12)

When probed about how useful they thought learning through practice to be, the interviewees considered that it was particularly beneficial, and that other methods such as reading about systematic review methodology had limitations. For example, Interviewee 9 remarked:

I don’t know how else you could explain, I mean you can certainly read about reviews, until you do a review. . . You don’t know what it’s like. I mean you don’t know what you’re getting into. . .

Two-thirds of the interviewees mentioned that colleagues and mentors with experience of systematic reviews had helped them to learn. Although opinions on mentorship were generally positive not all interviewees considered that it was beneficial for them, but they did not elaborate on why this was.

Learning from peers was also mentioned. Two interviewees remarked that they began systematic reviewing at a time when there were few training opportunities available and self-teaching was necessary. Interviewee 11 described this as being “the blind leading the blind,” and told how she and two colleagues completed their first systematic review in 1996 largely on a “trial and error” basis. The other interviewee made reference to learning by “muddling along” (Interviewee 15).

The extent of training received, by those who reported having been trained, varied. Some had only attended one or two brief training sessions, while others had participated in longer, more detailed, courses. Some interviewees mentioned that it was only after participating in systematic reviews that they received training. None of the interviewees reported major problems in accessing training and tended to make use of courses whenever available, particularly if held locally. In the opinion of one interviewee, training opportunities were adequate although not everyone may be able to afford them. Interviewee 5, resident in Nigeria, took advantage of training while on a scholarship-funded sabbatical in the United Kingdom. However, he questioned whether he would have been able to access this training if he had not received the scholarship.

The majority of comments on the training received were positive. For example, Interviewee 16 noted:

I always find training really helpful, I mean it’s never, it’s often not immediately obvious how it’s helping you but you know you can get something out of what I’ve been doing in the long run.

However, when asked if training equipped her for doing systematic reviews, Interviewee 14 remarked that it could not be a substitute for practical experience:

I don’t think it ever adequately prepared before you start doing something because you learn so much from doing

Helping Others to Learn Systematic Reviewing

Thirteen of the 17 interviewees (76%) reported that they had provided some form of training and support on doing systematic reviews to others. The proportion of their time spent doing this varied. In four cases it was, or had been, their full-time role. In all other cases it had been only one aspect of their work, alongside other activities such as doing systematic reviews.

Nine (69%) of the 13 interviewees mentioned that they had taught systematic reviewing to professionals. In general, these were health professionals, including doctors, nurses, nutritionists, physiotherapists, health service managers, and policy makers. The training tended to cover most of the stages of the production of a systematic review, with variations in terms of length and level of detail. The shortest courses tended to last around a day, and covered the principles of evidence-based health and key stages of a systematic review.

Eight (62%) interviewees had taught systematic reviews as part of an academic degree. In most cases, the interviewees taught postgraduate students studying subjects such as epidemiology or public health. However, some taught at undergraduate level to nursing and medical students, and students studying health sciences. In most cases, systematic reviews and evidence-based health were reported to be only one component of the syllabus. The detail in which the training could cover these topics was therefore considered to be limited. However, in at least two cases interviewees described teaching a whole course dedicated to systematic reviewing, including practical experience of conducting a review via dissertation.

The interviewees were asked whether there were any issues that people they had trained or supported had found difficult to comprehend. The most common issue was the statistics that are sometimes used in systematic reviewing (n = 7; 54%). The difficulties fell into three categories. First, two of the interviewees commented that it was difficult to describe the general principles of quantitative evidence synthesis (meta-analysis) without getting into the complexities of the statistics involved (Interviewees 1 and 10). They felt that this was daunting for the trainees, particularly in the context of short training courses where there was little time to discuss the issues in more detail. Second, one interviewee mentioned that trainees find it difficult to recognize which statistical tests are appropriate for which scenarios (Interviewee 10).

Third, discussing how the results of statistical tests can be interpreted often caused confusion. One interviewee remarked that an element of subjectivity is involved when interpreting results of a meta-analysis and that this was a difficult issue to explain to trainees (Interviewee 4).

Another issue that trainees were said to have difficulties with was critical appraisal of study methodology (n = 4/31%). Interviewee 6, referring specifically to training health professionals, commented that trainees are not always aware of the need to think critically about research:

A lot of our students have not been studying for many many years, they’re kind of blinded by the whole thing, and so they don’t, they don’t have an ability to understand what they’re not noticing. . . .they take everything far too much at face value. . .or the name on the paper is famous so it must be good.

An academic grounding was mentioned as being an advantage when learning critical appraisal, specifically in relation to understanding different study designs. Those who lacked prior training in research or statistics were said to find this difficult.

There were mixed views on the utility of structured instruments in helping people to develop critical appraisal skills. One interviewee felt that some instruments are overly structured and lull users into the false perception that they do not have to use their judgment (Interviewee 16). This issue was echoed by Interviewee 4 who was concerned about an overly procedural approach to systematic reviewing, devoid of a good understanding of the underlying methodology and any consideration of whether the results are externally valid:

I think you’ve got to avoid ‘the cookbook.’

However, another interviewee was more positive about the contribution of structured instruments as a way of developing critical appraisal skills (Interviewee 6). In her opinion, having a framework was an effective way of helping people to think critically, particularly people from practice backgrounds who are formally engaging with critical appraisal for the first time, as it provides a tool for disentangling their thoughts. Efforts to help people intimidated by the technical aspects of quality assessment to develop their skills were described. Interviewee 16 remarked that trainees attending her course are encouraged to start by applying the skills they use to weigh-up the strengths and weaknesses of evidence in everyday life to research evidence.

The interviewees were asked for their suggestions for the conduct of future training and support for systematic reviewing. Three (23%) suggested that effective learning should involve a strong element of practical experience. Of these, Interviewee 4 commented that short courses that lack any practical activities may not be adequate:

Well I do think you have to practice it, and I do think that just saying, ‘Ah I’ve done a short course on systematic reviews and therefore I can do them’. . . that you don’t really understand bias and confounding and study design from a few short courses, unless, I really think you have to have some practice.

Four (31%) interviewees commented that there would be greater uptake of training if more time and funding were available. It was suggested that this would enable people to take periods of time away from work necessary to learn systematic reviewing. The funds would need to cover the costs of the training course itself, plus costs to cover their absence from work. Interviewee 14 commented that encouraging health professionals to learn how to conduct systematic reviews is desirable, but constrained by a lack of time and funding:

I mean the reality is that if you’re a full-time health professional you haven’t got time for a systematic review, much as we’d like it to be done by full-time practising health professionals because we might get different questions and different answers. And maybe that’s one of the things that we need, is more investment in buying-out health professionals to do things like this, to give them the time and the support to do it

Five (38%) interviewees suggested that learning to do systematic reviews could be improved with the use of mentors. This was influenced by the fact that, as mentioned earlier, many of the interviewees had found mentors to be very helpful in their own learning.

Challenges in Doing Systematic Reviews

The interviewees were asked what challenges they have faced in doing systematic reviews in health promotion (Table 2).

Table 2.

Challenges in Doing Reviews.

Theme	Challenge	Number (%) of Interviewees
External factors	Lack of resources and time	11 (65)
Analysis and methods	Analyzing complex interventions	5 (29)
	Locating the evidence	4 (24)
	Analyzing observational studies	3 (18)
	Critical appraisal	3 (18)
	Dealing with heterogeneity and statistics	3 (18)
	Developing the review question	2 (12)
	Lack of flexibility	1 (6)
Evidence limitations	Reporting bias	2 (12)
Evidence limitations	Poor evidence	1 (6)
Miscellaneous	Lack of motivation + tedium	1 (6)
Miscellaneous	Advisory group	1 (6)
	Interpretation and presentation	1 (6)
	Supporting external groups	1 (6)

A lack of resources and time was a key external issue (n = 11/65%). For some interviewees, this was their biggest challenge and they elaborated on it at length. One interviewee reported that he had received little funding to do systematic reviews (Interviewee 12). Part of the problem, he noted, stemmed from difficulties in finding time to write funding proposals, as well as the perception that there are few funding opportunities.

Some interviewees found it even harder to obtain funds to do reviews of some of the more complex types of intervention. One commented that, to be successful in receiving support, you have to tailor the scope of the review to appeal to the funders (Interviewee 11). In her experience, this can be at the expense of doing a review which is truly public health in its outlook, and which is within the research interests of the reviewer. However, she commented that public health is such a broad discipline that it usually can be “made to fit.”

Three of the interviewees reported that they had managed to undertake systematic reviews with little or no funding, as Interviewee 12 (an academic and professional practitioner) commented:

Nearly all our review work. . .is un-funded. So it’s. . . like a hobby I guess. I’ve never had any funding to do any of my, for any of my involvement in the reviews

He also commented that to get the reviews completed he had relied to a large extent on the goodwill of collaborators, though he acknowledged that there are risks involved in relying on them too much.

However, one interviewee (Interviewee 13) suggested that a “free” systematic review would not necessarily be acceptable in today’s academic environment:

For the first review, which was published in 1997, we just did it gratis and perhaps. . .10 years ago things weren’t as goal oriented as they are now

Three interviewees talked about their experiences of being given unrealistic time scales for completing reviews. Interviewee 7 commented specifically on the time pressures of doing a public health systematic review, as opposed to a review of a health care topic:

We do it in six months, but I don’t think we can do a public health review with that shortness of time, because. . .what takes your time is the searching, obtaining reports and clarification of authors, which takes, in my experience it takes months

Another commonly mentioned challenge was reviewing complex interventions (n = 5/29%), specifically interventions that aim to create changes in the physical environment to promote physical activity (Interviewee 14), and policy interventions to promote nutrition (Interviewee 9). One interviewee (Interviewee 5) commented on his experience of analyzing community-based interventions in which cluster randomized trials tend to be used. These had presented particular statistical challenges to him. Fortunately at the time he was on a sabbatical at a center for evidence-based health in the United Kingdom and was able to seek advice from experienced statisticians resident there. In contrast, Interviewee 2 mentioned a lack of support which she felt hampered in her efforts. Even though guidance was available on complex interventions, she nevertheless described the feeling of “working in a vacuum.”

Discussion

Three key themes emerge from the results of this study: the pressure on resources and time, the complexity of the evidence base, and subjective judgment and confidence to appraise and interpret evidence. These were recurring and interlinked themes that permeate many of the issues discussed.

Resources and Time

The results illustrate that it can be time consuming both to obtain funding for systematic reviews and to produce the reviews themselves. The time required to write a successful grant proposal for a review was considered a barrier, even for some academics. It was also perceived that adequate funding opportunities for systematic reviews were scarce. The solution to the problem of securing funding, for some, was to self-finance systematic reviews, doing them largely in their spare time on a “shoe-string” budget. While this may have been feasible in the past the feeling was that this is no longer realistic, given the increased expectations placed on the conduct and reporting of systematic reviews. The increasing complexity of the methods for systematic reviewing, coupled with higher expectations of their quality, means that reviews conducted using limited resources may fall short of the mark by current standards (Centre for Reviews and Dissemination, 2009; Higgins & Green, 2011). Academics are under ever increasing pressure to publish their research in high impact peer-reviewed journals (Raddon, 2011). Only reviews that are adequately funded and resourced are likely to be considered credible and worthy of publication. The drive for methodological rigor within evidence-based health appears to have created barriers to the production and dissemination of systematic reviews.

Increased investment into funding systematic reviews (National Institute for Health Research, 2010a, 2010b) will likely resolve the tensions between standards and expectations. Availability of funding, including opportunities to applying for program grants, will also provide researchers with a structured career pathway in which they can specialize in doing reviews, and further develop review methodology to address complex topics.

In the main, access to training did not appear to be difficult, and this is reflective of the investment that has gone into widening access to training and increasing capacity to meet the demand for systematic reviews (National Institute for Health Research, 2010a, 2010b). It was remarked that the utility of reviews may be enhanced by encouraging the participation of health practitioners, as they may have different perspectives on evidence. However, it was questioned whether financial support is available to enable them to undergo training.

The upshot of all this is that there may be disincentives for practitioners to consider participating in systematic reviews, particularly those with limited time and resources, but who by virtue of their expertise in a particular topic, may be in a particularly appropriate position to do so. There are, however, encouraging signs that provision of funding for evidence synthesis is increasing (National Institute for Health Research, 2010b; Walley & Thakker, 2008). The existence of such opportunities will hopefully cultivate a new generation of health professionals with awareness and experience of systematic reviewing in health promotion, to ensure that key policy and practice questions are answered appropriately. It will be important to monitor the availability of funding opportunities and training for evidence synthesis.

Complexity

Another significant challenge identified by this study was systematically reviewing complex interventions. While in some cases support was on hand, in others help had not always been available. There has been increasing interest in the evaluation of complex interventions over recent years, in what is an evolving area. Discussion of complex interventions has taken place in the literature (Michie, Fixsen, Grimshaw, & Eccles, 2009; Shepperd et al., 2009), empirical research has been conducted on how complex interventions are reported (Egan et al., 2009), and evaluation frameworks have been published (Craig et al., 2008; Pawson, 2006). Furthermore, there are also examples of process evaluation conducted within randomized controlled trials to shed light on the contributory factors in the effectiveness or failure of complex health promotion interventions (Elford, Sherr, Bolding, Serle, & Maguire, 2002; Oakley, Strange, Bonell, Allen, & Stephenson, 2006; Stephenson et al., 2004). These examples suggest increased intellectual investment into methods of assessing and analyzing complex interventions, and better reporting in primary studies.

Recommendations have been made for practical strategies that systematic reviewers can use to address complexity, including: designing “logic models” to explore the processes through which a particular intervention might be expected to achieve outcomes; specifying the key study characteristics that reviews need to report in order to maximize generalizability of findings; and ways of structuring the synthesis of complex interventions to be most relevance to users of the review (e.g., adopting a practitioner or policy maker lens; Waters et al., 2011). Hopefully, the challenges faced by systematic reviewers in addressing complexity will be lessened as the result of continued methodological development in this area.

Judgment and Confidence

The need to exercise judgment was mentioned as a being a difficult message to communicate to novice reviewers, particularly those lacking in confidence to do anything other than follow explicit instructions (e.g., following the cookbook as one interviewee put it). Some degree of subjectivity is inevitable particularly in judging methodological quality. Nonetheless, systematic reviewers, while committed to minimizing bias, can be mindful of their subjectivity and take steps to try and account for it (Nind, 2006; Pawson et al., 2005; Petticrew et al., 2004). The challenge, it would seem, is to communicate this to trainees without undermining their confidence. A recommendation from this study is to encourage trainees to use available systematic reviewing tools such as critical appraisal instruments, but to record the reasons for their judgments, and to discuss and reflect on these with coreviewers, acknowledging where subjectivity is needed.

It was commented that those with some grounding in research methods tended to do better at exercises to develop critical appraisal skills than those without. This suggests that a prerequisite for systematic reviewing is possession of a higher education qualification. Yet, little research has been conducted on the necessary prerequisites for systematic reviewing. Systematic reviews are a form of research, and research is commonly done by academics. However, if involving practitioners and other stakeholders can enhance the utility of a review, as suggested by one of the interviewees in this study, then what is the most effective way to train and support them? Some of the approaches to training identified in this study appear promising. For example, one interviewee commented that the MSc course she coordinates is designed to encourage students to apply the skills they use to appraise phenomena in everyday life to critically evaluate research evidence. This might give trainees greater confidence to develop their critical appraisal skills within the context of evidence-based health. Formal evaluation of innovative approaches such as this would be useful for informing future training.

The interviewees learned systematic review skills through a combination of different methods with a strong emphasis on practice-based learning. That practical experience should be a significant method of learning seems intuitive, and can be explained by theories of reflective practice (Schön, 1983), experiential learning (Kolb, 1984), and the practice of action learning (McGillI & Brockbank, 2004) as applied to professional education. The interviewees also tended to learn from working with more experienced systematic reviewers. Reflections from interviewees who were involved in systematic reviewing in the mid-1990s illustrated the trial and error approach undertaken and the lack of support and expertise available at that time. Similarly, a study of U.K. academics’ accounts of learning about research skills found that training and development was informal and often based on learning through immersion with trial and error (Raddon, 2011). A lack of a formal structure for research training, and poor recognition by research managers and researchers themselves of the need to develop and refine their skills, has been described as a culture of “nondevelopmentalism” (Evans, 2009; Raddon, 2011).

The phenomenon of being “left to get on with it” as described by the academics interviewed by Raddon, may now be less of an issue. Today, it would appear, there is a larger pool of people with skills in systematic reviewing from whom guidance and mentorship can be sought. This increase in capacity has almost certainly developed as a result of increasing demand for systematic reviews which, in turn, has created greater demand for skilled reviewers. The more people that learn to do reviews, the more they can pass on their knowledge and skills to others. Raddon (2011) likens this to a “scaffolded” approach to learning in which there is a structure in place to develop skills and knowledge, including the opportunity to work alongside experienced researchers. However, the potential downsides of this could be a lack of engagement between novice and mentor, and with the novice unquestioningly imitating poor practice. It could be suggested that, by its very nature, the systematic review lends itself to a scaffolded approach. Many of the tasks, such as critical appraisal, need to be done in duplicate or in consultation with a second researcher to maintain validity and reliability. This provides a built-in opportunity for novice reviewers to engage in critical discussion of the evidence, and to reflect on their skills in processing and interpreting it. The risks of being left to get on with it, and lack of engagement, seem to be reduced.

The findings of the current study therefore lend support to a scaffolded approach to learning systematic review skills. It is recommended that people are given the opportunity to have hands-on experience of reviewing (in conjunction with other learning strategies, such as training, as appropriate) with adequate support from more experienced reviewers. For example, they could be given a role in a review team and assigned tasks such as eligibility screening, data extraction, critical appraisal, and data synthesis—liaising with coreviewers at each stage to discuss judgments and reflect on their learning. This could be preceded by induction training in the principles of systematic reviewing to provide important context.

Strengths and Limitations of this Study

One of the strengths of this study was that an international sample of systematic reviewers was included, thereby providing a flavor of how experiences of reviewing vary in different parts of the world. However, despite best efforts to ensure representation from low- and middle-income countries, the final sample included a disproportionate number of reviewers from high-income countries, particularly the United Kingdom. Interviews were negotiated in principle with some reviewers from low- and middle-income countries, however, due to the busy nature of the conference where the majority of the research interviews were conducted, it was not possible to find a mutually convenient time to interview all of them.

Another strength was that many of the interviews were done at a conference on evidence-based health, which featured discussions and scientific presentations on all aspects of the methodology of evidence synthesis. The issues covered by this study were therefore at the forefront of the interviewees’ minds at the time, which will likely have helped facilitate discussion and recall. However, the fact that they were interviewed at such a conference suggests that they may be considered a select group and not necessarily representative of the broader field of systematic reviewers. While they were chosen specifically because of their experience, the sample was reasonably diverse in terms of role (e.g., from junior researcher to consultant physician), experience of reviewing (e.g., from novice reviewer to author of numerous reviews), and participation in methodological research and policy setting in evidence synthesis (e.g., an awareness and interest in methodology to directing methodological research projects).

One of the potential disadvantages of recruiting interviewees specifically at a Cochrane conference is that, by attending such a conference, it could be assumed that they are supportive of Cochrane’s methods, policies, and general approaches to systematic reviewing. The views and experiences of those interviewed at the Colloquium may not be wholly representative of the wider systematic reviewing community. This was borne out, to some extent, by some of the comments made by those interviewed following the conference in other settings. Cochrane methods were noted as being distinct from other approaches used in other contexts, and in some cases comments were mildly critical. However, some of those interviewed at the Colloquium were, on occasion, also critical and commented on inconsistencies in approaches and policies within the Collaboration. This suggests a balance of views and that sample selection bias was unlikely.

Conclusions

This is one of the few studies that has sought the views and experiences of systematic reviewers, focusing specifically on the issues inherent in health promotion research. There is an increasing demand for systematic reviews to answer different types of policy questions, necessitating the development of innovative methods. In turn, this has created the need for skilled reviewers, including health professionals, to produce reviews that meet these expectations. An infrastructure to support systematic reviewing has been developed, but this will require long-term investment and maintenance to ensure sufficient skilled capacity, and adequate funding to produce reviews in a timely fashion, which are meaningful to policy and practice. This research recommends a combination of course-based and practice-based training for systematic reviewers, with structured support, discussion, and reflection at all stages of the review process. Evaluation of innovative approaches to learning systematic review skills, particularly for those from practice backgrounds, is needed.

Footnotes

Authors’ Note

This study was approved by the School of Education, University of Southampton.

Acknowledgements

Thanks to Professor Katherine Weare, School of Education, University of Southampton for expert guidance in the conduct of this research. Thanks to all of the interviewees who generously gave up their time to be interviewed, and to the Cochrane Collaboration Review Groups that provided help in recruiting interviewees.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

References

Bowling

Ebrahim

(2005). Handbook of health research methods: Investigation, measurement and analysis. Maidenhead, England: Open University Press.

Brunton

Thomas

Harden

Rees

Kavanagh

Oliver

Oakley

. (2005). Promoting physical activity amongst children outside of physical education classes: A systematic review integrating intervention studies and qualitative studies. Health Education Journal, 64, 323–338.

Centre for Reviews and Dissemination. (2009). Systematic reviews: CRD's guidance for undertaking reviews in health care (3rd ed.) York, England: Author.

Clarke

(2006). The cochrane collaboration. In Killoran

Swann,

Kelly

(Eds.), Public health evidence: Tackling health inequalities (pp. 115–124). Oxford, England: Oxford University Press.

Craig

Dieppe

Macintyre

Michie

Nazareth

Petticrew

(2008). Developing and evaluating complex interventions. London, England: Medical Research Council.

Egan

Bambra

Petticrew

Whitehead

(2009). Reviewing evidence on complex social interventions: Appraising implementation in systematic reviews of the health effects of organisational-level workplace interventions. Journal of Epidemiology and Community Health, 63, 4–11.

Elford

Sherr

Bolding

Serle

Maguire

(2002). Peer-led HIV prevention among gay men in London: Process evaluation. AIDS Care, 14, 351–360.

Evans

(2009). Developing research capacity in the social sciences: A professionality based model. International Journal For Researcher Development, 1, 134–149.

Garside

Pearson

Moxham

(2010). What influences the uptake of information to prevent skin cancer? A systematic review and synthesis of qualitative research. Health Education and Research, 25, 162–182.

10.

Green

Thorogood

(2004). Qualitative methods for health research. London, England: SAGE.

11.

Green

Kreuter

(1999). Health promotion planning: An educational and ecological approach (3rd ed.). Boston, MA: McGraw-Hill.

12.

Harden

Garcia

Oliver

Rees

Shepherd

Brunton

Oakley

(2004). Applying systematic review methods to studies of people's views: An example from public health research. Journal of Epidemiology and Community Health, 58, 794–800.

13.

Higgins

J. P.

Green

. (2011). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Retrieved from www.cochrane-handbook.org

14.

Killoran

Swann

Kelly

M. P.

(2006). Public health evidence: Tackling health inequalities. Oxford, England: Oxford University Press.

15.

Kolb

(1984). Experiential learning : Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall.

16.

Low

(2007). Unstructured interviews and health research. In Saks

Allsop

(Eds.), Researching health: Qualitative, quantitative and mixed methods (pp. 74–91). London, England: Sage.

17.

Mays

Pope

Popay

(2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services and Research Policy, 10, 6–20.

18.

McGillI

Brockbank

. (2004). The action learning handbook : Powerful techniques for education, professional development and training. London, England: RoutledgeFalmer.

19.

Michie

Fixsen

Grimshaw

J. M.

Eccles

M. P.

(2009). Specifying and reporting complex behaviour change interventions: The need for a scientific method. Implementation Science, 4, 40.

20.

National Institute for Health Research. (2010a). Embedding Health Research: National Institute for Health Research Annual Report 2009/10. Retrieved from http://www.nihr.ac.uk/files/pdfs/400891_NIHR_AnnualReport2010_acc3.pdf

21.

National Institute for Health Research (2010b). Systematic reviews: Knowledge to support evidence-informed health and social care. Retrieved from http://www.nihr.ac.uk/files/pdfs/NIHR%20Systematic%20Reviews%20Infrastructure.pdf

22.

Nind

(2006). Conducting systematic review in education. London Review of Education, 4, 183–195.

23.

Oakley

(2003). Research evidence, knowledge management and educational practice: Early lessons from a systematic approach. London Review of Education, 1, 21–33.

24.

Oakley

Strange

Bonell

Allen

Stephenson

(2006). Process evaluation in randomised controlled trials of complex interventions. British Medical Journal, 332, 413–416.

25.

Oliver

Thomas

Harden

Oakley

(2006). Accumulating evidence to bring policy, practice and research together. In Killoran

Swann

Kelly

(Eds.), Public health evidence: Tackling health inequalities (pp. 125–140). Oxford, England: Oxford University Press.

26.

Pawson

(2006). Simple principles for the evaluation of complex programmes. In Killoran

Swann

Kelly

(Eds.), Public health evidence: Tackling health inequalities (pp. 223–238). Oxford, England: Oxford University Press.

27.

Pawson

Greenhalgh

Harvey

Walshe

(2005). Realist review–A new method of systematic review designed for complex policy interventions. Journal of Health Services and Research Policy, 10, 21–34.

28.

Petticrew

Tugwell

Welch

Ueffing

Kristjansson

Armstrong

Doyle

Waters

(2009). Better evidence about wicked issues in tackling health inequities. Journal of Public Health, 31, 453–456.

29.

Petticrew

Whitehead

Macintyre

S. J.

Graham

Egan

(2004). Evidence for public health policy on inequalities: 1: The reality according to policymakers. Journal of Epidemiology & Community Health, 58, 811–816.

30.

Pettman

T. L.

Hall

B. J.

Waters

de Silva-Sanigorski

Armstrong

Doyle

(2011). Communicating with decision-makers through evidence reviews. Journal of Public Health, 33, 630–633.

31.

Raddon

(2011). A changing environment: Narratives of learning about research. International Journal For Researcher Development, 2, 26–45.

32.

Rees

Kavanagh

Harden

Shepherd

Brunton

Oliver

Oakley

(2006). Young people and physical activity: A systematic review matching their views to effective interventions. Health Education Research, 21, 806–825.

33.

Schön

(1983). The reflective practitioner : How professionals think in action. London, England: Maurice Temple Smith.

34.

Shepherd

Harden

Rees

Brunton

Garcia

Oliver

Oakley

(2006). Young people and healthy eating: A systematic review of research on barriers and facilitators. Health Education Research: Theory and Practice, 21, 239–257.

35.

Shepherd

Kavanagh

Picot

Cooper

Harden

Barnett-Page

Price

. (2010). The effectiveness and cost-effectiveness of behavioural interventions for the prevention of sexually transmitted infections in young people aged 13 to 19: A systematic review and economic evaluation. Health Technology Assessment, 14, 1–206.

36.

Shepperd

Lewin

Straus

Clarke

Eccles

M. P.

Fitzpatrick

Sheikh

. (2009). Can we systematically review studies that evaluate complex interventions? PLoS Medicine, 6, e1000086.

37.

Sowden

Glanville

(2006). The centre for reviews and dissemination. In Killoran

Swann

Kelly

(Eds.), Public health evidence: Tackling health inequalities (pp. 95–114). Oxford, England: Oxford University Press.

38.

Stephenson

J. M.

Strange

Forrest

Oakley

Copas

Allen

Johnson

A. M

. (2004). Pupil-led sex education in England (RIPPLE study): Cluster-randomised intervention trial.[see comment]. Lancet, 364, 338–346, 330.

39.

Sweet

Moynihan

(2007). Improving population health: The uses of systematic reviews. New York, NY: Milbank Memorial Fund and Centers for Disease Control and Prevention.

40.

Wallace

Croucher

Bevan

Jackson

O'Malley

Quilgars

(2006). Evidence for policy making: Some reflections on the application of systematic reviews to housing research. Housing Studies, 21, 297–314.

41.

Walley

Thakker

R. V.

(2008). Developments for funding clinical research in the UK. Lancet, 372, 518–519.

42.

Wanless

(2004). Securing good health for the whole population. London, England: Her Majesty's Stationery Office (HMSO).

43.

Waters

(2009). Building an evidence base to meet the needs of those tackling obesity prevention. Journal of Public Health, 31, 300–302.

44.

Waters

Hall

B. J.

Armstrong

Doyle

Pettman

T. L.

de Silva-Sanigorski

(2011). Essential components of public health evidence reviews: Capturing intervention complexity, implementation, economics and equity. Journal of Public Health, 33, 462–465.