Abstract
Objective
To describe systematic review production in 41 countries in Africa, the Americas, Asia and the eastern Mediterranean to understand one dimension of the climate for evidence-informed health systems and to provide a baseline for an evaluation of knowledge translation initiatives.
Methods
Our focus was systematic reviews published between 1996 and 2008 that had a corresponding author based in, or that appeared to target, one of the countries in these regions. We searched both Medline and Embase using validated search strategies, identified citations with a country name in the corresponding author's institutional affiliation or as a textword (i.e., an explicit mention in the title or abstract) or keyword, and coded articles describing a systematic review. We followed the same citation identification procedure for Health Systems Evidence, a database containing systematic reviews about health systems.
Results
Systematic review production increased between three-fold (for Africa in Medline) and 110-fold (for Asia in Embase) between the first period (1996-2002) and second period (2003-2008). In the second period, China was more often the home of corresponding authors and the target of reviews than any other country. No systematic reviews were produced by a corresponding author based in nine countries, or appeared to target five countries. Only 48 reviews identified through Medline and Embase addressed health systems, and 35 health systems reviews identified through Health Systems Evidence addressed these countries.
Conclusion
In many countries, those seeking to support evidence-informed health systems cannot turn to experienced local systematic reviewers to help them to find and use systematic reviews or to conduct reviews on high priority topics when none exists. These findings suggest the need for local capacity-building initiatives.
Introduction
A profile of the systematic reviews being conducted by authors based in a given country can provide helpful insights to policy-makers, stakeholders, researchers and journalists about the climate for evidence-informed health systems, and can help them to support such systems. 1 Knowing who is actively engaged in conducting locally relevant systematic reviews can be a boon to policy-makers and stakeholders, who are increasingly realizing the many advantages of systematic reviews (compared to single studies) when answering a broad range of policy questions. 2 Policy-makers and stakeholders may seek out local systematic reviewers to assist with finding and using systematic reviews in the course of preparing policy briefs, or more generally in informing or influencing decisions about health systems.3,4 They may also commission local systematic reviewers to address a focused policy question. Local health systems researchers may partner with local systematic reviewers to build their own capacity to conduct reviews. Journalists may seek out systematic reviewers who can help to put the findings of a single study in the context of the research literature available on a topic. Increases in the production of systematic reviews over time can suggest that investments in capacity-building or in review production are having an impact. On the other hand, a limited volume of, or a decline in, review production can suggest the need for a proactive effort to build capacity for, or support, review production.
A profile of the systematic reviews being targeted at a given country can also provide helpful insights and support. While policy-makers and stakeholders would ideally be interested in learning from the global pool of research literature on a given topic, they must always judge whether and, if so, how the local context in which the included studies were conducted differs from their own and what this means for the local applicability of a review's findings. 5 Policy-makers and stakeholders in a given country might be very interested in a systematic review that includes studies that were conducted locally. Alternatively, they might only be interested in a systematic review that includes studies that were conducted in contexts that share key realities and constraints, health system arrangements and baseline conditions. 5 While the pros and cons of such targeted reviews have yet to be fully examined, they are likely to be of significant interest to policy-makers and stakeholders.
Profiles of systematic review production can contribute to the monitoring and evaluation of the knowledge translation (KT) initiatives called for in the 2004 Mexico Statement on Health Research, the 2005 World Health Assembly resolution, and the 2008 Bamako Call to Action on Research for Health.6–8 For example, the WHO-sponsored Evidence-Informed Policy Networks (EVIPNet) have been established to facilitate evidence-informed policy-making at country level. A description of review production can inform an understanding of the context in which these KT initiatives are operating, and this context may influence whether their activities and outputs translate into outcomes and impacts. Review production can also complement measures of related activities (namely priority-setting processes and policy dialogues, which can draw on and spur the production of systematic reviews) and outputs (namely policy briefs, which draw on systematic reviews as a key input), 4 as well as provide an independent assessment of outcomes (namely that health research evidence – in this case synthesized research evidence – about high-priority policy issues is more readily available, and that partnerships between policymakers and researchers – in this case systematic reviewers – are strengthened). These measures are all part of the monitoring and evaluation framework for EVIPNet and its ‘sister’ KT initiatives (Figure 1). 9
Relationship between systematic review production profiles and the monitoring and evaluation framework
In order to understand one dimension of the climate for evidence-informed health systems and to provide a baseline for an evaluation of KT initiatives, we described the production of systematic reviews published between 1996 and 2008 that: had a corresponding author based in one of 41 countries in Africa, the Americas, Asia and the eastern Mediterranean that host (or have signalled their intent to host) a local EVIPNet or similar type of KT facility;10,11 or appeared to target one of these countries. Because Burundi, Mali and Rwanda have only recently signalled their intent to each host a local KT facility, they are not included in this analysis. In total we studied 44 jurisdictions because we included both China as a whole, and the two provinces (Shandong and Sichuan) and one municipality (Beijing) that host a local EVIPNet. We provided each KT facility with the citations for all systematic reviews that were produced in or targeted at their jurisdiction so that they could identify systematic reviewers who could be engaged in their activities, and thereby improve the climate for evidence-informed health systems in their jurisdiction.
Methods
We searched both Medline and Embase using the OVID interface to identify systematic reviews published and indexed between January 1996 and December 2008. To identify systematic reviews in Medline we used a validated search strategy, 12 typically a three-term strategy that balanced sensitivity and specificity, while emphasizing sensitivity (meta-analysis.mp, pt OR review.pt OR search:.tw). However, for searches that returned more than 2,000 results (i.e., for Brazil, China, and Mexico), we used a more specific five-term search strategy that balanced sensitivity and specificity, while emphasizing specificity (Cochrane Database of Systematic Reviews.jn. OR search.tw. OR meta-analysis.pt. OR Medline.tw. OR Systematic Review.tw.). To identify systematic reviews in Embase we also used a validated search strategy, typically a three-term search strategy that optimized sensitivity and specificity (meta-analys:.mp. OR search:.tw. OR review.pt.). 13 For searches that returned more than 2,000 results (i.e., for Argentina, Brazil, China, and Mexico), we used an alternative three-term search strategy that offered high specificity and precision at the cost of some sensitivity (meta-analysis.tw. OR systematic review.tw. OR Medline.tw.).
Identifying jurisdiction-specific citations
We generated two jurisdiction-specific citation listings: one by identifying those citations with a jurisdiction name in the institutional affiliation field (corresponding authors are the only authors for whom institutional affiliations are always available, but any other affiliations also appear in this field), and the other by identifying citations where the jurisdiction name was present as a textword (i.e., an explicit mention in the title or abstract) or keyword (i.e., MeSH in Medline and Subject Heading in Embase). We used common English language jurisdiction spellings, with the exception of Brazil (also Brasil), the Central African Republic (also Republique Centrafricaine), Kyrgyzstan (also Kyrgyz), and Laos (also Lao PDR).
Selecting systematic reviews
After excluding a small proportion (0.06) of citations for which we could obtain neither an abstract nor full-text article online (2,143/36,084), we used a two-stage approach to select systematic reviews, with different selection criteria for each stage (Figure 2). For the first stage, we used one inclusion criterion – the abstract employed terms that suggested a database (e.g., Medline, PubMed) search had been conducted or a full systematic review had been conducted (meta-analysis or ‘systematic review’) – and one exclusion criterion (the abstract indicated that the article described only original research). For the second stage, we used two inclusion criteria: the abstract or full-text article indicated that a database search had been conducted (e.g., Medline, PubMed, or description of a search strategy); and the abstract or full-text article indicated that explicit inclusion or exclusion criteria had been used to select studies. A statement such as ‘all relevant studies were included’ was not considered to be an explicit inclusion criterion. A second reviewer independently verified the eligibility of all articles that had passed through both stages, and any disagreements between the first and second reviewer were resolved by consensus.
Selection process
Coding systematic reviews by health systems topic
Two reviewers independently categorized all systematic reviews according to a taxonomy of health systems topics described in detail elsewhere. 14 The taxonomy distinguishes among governance arrangements (i.e., who has the authority to make policy, organizational, commercial and professional decisions and whether and how consumers and stakeholders are involved in decisionmaking), financial arrangements (i.e., how revenue is raised, organizations are funded, professionals are remunerated, financial incentives are set for patients, and resources are allocated) or delivery arrangements (i.e., patient targeting to improve access or quality, setting of care, information and communication technology, quality and safety monitoring systems).
Identifying additional systematic reviews addressing health systems topics
We followed the same citation-identification procedure for Health Systems Evidence (www.healthsystemsevidence.org), a database then containing more than 830 systematic reviews about health systems. 14 That is, we identified citations with a country name in any author's institutional affiliation (not just the corresponding author) or as a textword (i.e., an explicit mention in the title or abstract). We also identified citations where the country name was mentioned in the research question, or the region name was mentioned as a textword or in the research question. All reviews contained in the database (except those labelled health technology assessments) had already been assessed for eligibility using the same two inclusion criteria used in stage 2 (above) and coded by health systems topics. Two independent reviewers were involved in all steps, and any disagreements were resolved by consensus. Abstracts written in any language were included.
Results
Systematic review production increased between threefold (for Africa in Medline) and 110-fold (for Asia in Embase) between the first (seven-year) period (1996-2002) and second (six-year) period (2003–2008) (Table 1). A similar number of systematic reviews was retrieved from both Medline (n = 1419) and Embase (n = 1616). China accounted for more systematic reviews than other countries in Asia (n = 837), while Brazil did the same in the Americas (n = 628), Iran in the eastern Mediterranean (n = 99) and Nigeria in Africa (n = 98).
Systematic reviews produced by a corresponding author based in or that appear to target a participating jurisdiction, by database (1996-2002 and 2003-2008) *
Systematic reviews can appear in both databases
China does not host a local EVIPNet per se but three Chinese jurisdictions (Beijing, Shandong, and Sichuan) host local EVIPNets. These numbers do not exclude Beijing, Shandong or Sichuan
KT = knowledge translation
Systematic reviews produced by corresponding authors based in one of the countries
In the second period, China was more often the home of corresponding authors than any other country, which contributed significantly to Asia being more often the home of corresponding authors than any other region (Table 2, available online only at: http://www.jhsrp.rsmjournals.com/cgi/content/full/jhsrp.2011.010109/DC1). Within China, most systematic reviews were produced by corresponding authors in Beijing municipality and in Sichuan and Shandong provinces (n = 123, 113 and 58, respectively), who together produced more than half of all systematic reviews produced in China during this period. Among the jurisdictions in the Americas, corresponding authors based in Brazil produced the most systematic reviews (n = 395), and among African jurisdictions, corresponding authors in Nigeria produced the most systematic reviews (n = 53). In the eastern Mediterranean, corresponding authors in Iran produced the most systematic reviews (n = 67). No systematic reviews were produced by a corresponding author based in nine countries.
Systematic reviews identified in Health Systems Evidence that were produced by authors based in, or that appear to target, participating jurisdictions or both
Systematic reviews that appear to target one of the countries
In the second period, China was more often the target of reviews than any other country (n = 609), meaning that China was explicitly mentioned in the title or abstract or that the keyword China was assigned to the review more frequently than other jurisdiction names (Table 3, available online only at: http://www.jhsrp.rsmjournals.com/cgi/content/full/jhsrp.2011.010109/DC1). Within the Americas, Brazil was targeted most frequently (n = 207). Within Africa, Nigeria was targeted the most (n = 65), and within the eastern Mediterranean, Pakistan was targeted most frequently (n = 23). No systematic reviews appeared to target five countries (one in Africa, one in the Americas, and three in Asia).
Systematic reviews coded by health system topics
Of the 2,063 reviews identified through Medline and Embase, only 48 addressed health system topics. Most addressed delivery arrangements, while only a few addressed governance and financial arrangements (43, 6 and 3, respectively, with 4 addressing more than one type of arrangement). The only sub-categories within delivery arrangements that accounted for more than three reviews were case management (i.e., packages of care, care pathways, and disease management) and timely access to care (both from the category ‘to whom care is provided and with what efforts to reach them’), substitution (from the category ‘by whom care is provided’), and site of service delivery (from the category ‘where care is provided’) (16, 4, 4 and 7, respectively). Of the six reviews addressing governance arrangements, three focused on policy authority.
Of the more than 830 health systems reviews identified through Health Systems Evidence, only 16 had a lead author, and 26 had a co-author, who was based in one of the jurisdictions (Table 4). Five jurisdictions hosted a lead author, and four jurisdictions hosted a co-author, of a health system-focused review in the last year (2008). Just two health system-focused reviews mentioned a participating jurisdiction, and four mentioned a region containing participating jurisdictions, in the title or abstract. Only two mentioned one of the regions in the research question, while no jurisdictions were mentioned in the research questions.
Discussion
Principal findings
While there has been a significant increase in systematic review production in the most recent six-year period compared to the preceding seven-year period, the overall number of reviews being produced is relatively low and only a small fraction of these reviews addressed health system topics, which are more likely to be the focus of policy-makers and the stakeholders, researchers and KT initiatives seeking to inform them. Moreover, while countries such as China and Brazil have a significant pool of corresponding authors on which to draw, and these same countries appear to be targeted by many systematic reviews, no systematic reviews were produced by a corresponding author based in nine countries and no systematic reviews appeared to target five countries.
Strengths and limitations
Our study has four main strengths: we searched two databases – Medline and Embase – using validated search strategies and we searched a third database – Health Systems Evidence – focused specifically on systematic reviews of health systems evidence; we focused on the large and diverse array of countries that host (or have signalled their intent to host) a local EVIPNet or similar type of KT initiatives and over a long (13-year) time period; we used moderately strict inclusion criteria to select systematic reviews; and we used a newly developed taxonomy to categorize systematic reviews that addressed health system topics. Our study has three main limitations: we did not search regional databases (such as LILACS for the Americas), which could be important sources of locally conducted or targeted systematic reviews; we excluded citations for which we could obtain neither an abstract nor a full-text article (although only 6% of initial search results were excluded for this reason); and a second reviewer screened only those reviews that had passed through a first and second review, meaning that while we can be confident about the included reviews, some may have been missed. Our focus on institutional affiliation (which always includes corresponding authors but may not include all others), may have led us to underestimate local engagement in the production of systematic reviews. On the other hand, our focus on the use of jurisdiction names as text-words and keywords, and not analyzing each review to ensure that this reflected some degree of country targeting, may have led us to overestimate country targeting.
Findings in relation to other studies
We are aware of only two reports that have addressed a similar objective to ours. We drew on the first of the two reports in developing the study method, while introducing broader inclusion criteria and expanding the list of participating countries beyond those located in Africa. 15 We cannot compare our findings directly to those reported in the second of the two reports, which focused on articles published in Latin American and Caribbean journals indexed in Medline, LILACS or both from 2000–2005, because the report did not distinguish between clinical trials and systematic reviews. 16
Implications for policy and research
Our study provides a window onto systematic review production in low- and middle-income countries, and our method can be used in the future to describe how production changes over time. In a number of countries, those seeking to support evidence-informed health systems can turn to experienced local systematic reviewers who can help them to find and use systematic reviews or who can conduct systematic reviews on high-priority topics when none exists. However, the nine jurisdictions with no corresponding authors and five jurisdictions that were not targeted by any systematic reviews will require additional support to develop local capacity to produce systematic reviews and/or partnerships with jurisdictions that have existing review capacity, face similar challenges, and (possibly) share key health system features. Intraregional cooperation between high- and low-volume jurisdictions should be encouraged, and collaboration between low-producing jurisdictions on topics of mutual interest would be valuable. Also, despite searching Medline, Embase, and Health Systems Evidence, few health systems-focused systematic reviews relevant to these jurisdictions were identified, indicating that synthesized health system research is especially lacking. WHO, research funding agencies, and national policy-makers can all support the strengthening of systematic review capacity and partnership building in these countries.
Footnotes
Acknowledgements
The authors thank members of the KT Platform Evaluation Team, specifically Pierre Ongolo-Zogo, Tomas Pantoja, Nordin Saleh and Nelson Sewankambo. The authors thank Jason Akerman, Randall Lau, Jonathan Sachs, and Mike Wilson for acting as second reviewers for many of the articles retrieved through the Medline and Embase searches. The authors also thank Daniel Fortino and Nathan Mendes Souza for screening articles retrieved through Health Systems Evidence and for assistance with coding.