Epidemiology,Quality,and Reporting Characteristics of Systematic Reviews of Acupuncture Interventions Published in Chinese Journals

Abstract

Background:

Systematic reviews (SRs) of acupuncture have become increasingly popular in China in recent years and have been published in large numbers. This review provides the first examination of epidemiological characteristics of these SRs as well as compliance with the PRISMA and AMSTAR guidelines.

Objectives:

The study objectives were to examine epidemiological and reporting characteristics as well as methodological quality of SRs of acupuncture published in Chinese journals.

Methods:

Four (4) Chinese databases were searched (CBM, CSJD, CJFD, and Wanfang Database) for SRs of Traditional Chinese Medicine, from January 1978 through to December 2010. Data were extracted into Excel spreadsheets. The PRISMA and AMSTAR checklists were used to assess reporting characteristics and methodological quality, respectively.

Results:

A total of 88 SRs were identified; none of the reviews had been updated. Less than one third (27.3%) were written by clinicians and one third (35.2%) were reported in specialty journals. The impact factor of 53.4% of the journals published was 0. Information retrieval was not comprehensive in more than half (59.1%) of the reviews. Less than half (36.4%) reported assessing for publication bias. Though 97.7% of the reviews used the term “systematic review” or “meta-analysis” in the title, no reviews reported a protocol and none were updated even after they had been published after 2 or more years.

Conclusions:

Although many SRs of acupuncture interventions have been published in Chinese journals, the reporting quality is troubling. Thus, the most urgent strategy is to focus on increasing the standard of SRs of acupuncture interventions, rather than continuing to publish them in great quantity.

Introduction

T he first systematic review (SR) of acupuncture interventions published in a Chinese journal is believed to have been by Li et al in 2002.¹ Since that time, SRs of acupuncture have become increasingly popular in China and have been published in large numbers. As SRs play an important role in the assessment of interventions and are important tools in guiding policy and practice, it is critical that their results be unbiased and of the highest possible quality.

Utilization of newer assessment tools is recommended since Quality of Reporting of Meta-analyses (QUOROM) was replaced by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist in 2009.² An additional assessment tool, Assessment of Multiple Systematic Reviews (AMSTAR),³ released in 2007, was based on the Oxman-Guyatt Overview Quality Assessment Questionnaire⁴ and Sack's Quality Assessment Checklist.⁵ AMSTAR is considered the best way to assess methodological quality of SRs by the Canadian Agency for Drugs and Technologies in Health.⁶

Over the past 10 years, only one study⁷ has described the SR of acupuncture interventions published in Chinese journals reporting characteristics based on the QUOROM checklist, but it did not include information about epidemiological characteristics or methodological quality. In addition, even though Linde et al. described the characteristics and quality of SRs of acupuncture, it was only based on 39 SRs of acupuncture that did not include literature published in Chinese journals.⁸

The purpose of this review is to describe the epidemiological and reporting characteristics, as well as the methodological quality of SRs of acupuncture published in Chinese journals, using the most up-to-date assessment tools.

Materials and Methods

Inclusion criteria

All SRs about acupuncture published in Chinese journals were included. Acupuncture interventions may have been administered alone or in combination with conventional Western medicine. There was no limitation in acupuncture manipulation and moxibustion. Publications described as SRs or those that provided an overview of evidence from multiple studies and where authors described their methods in explicit detail were included.

Search strategy

Four (4) Chinese databases (Chinese Biomedicine Literature Database, Chinese Scientific Journal Full-text Database, Chinese Journal Full-text Database, and Wanfang Database) were searched from January 1978 through to December 2010. The search terms included systematic review, meta-analysis, acupuncture, electroacupuncture, acupuncture manipulation, and moxibustion. Reference lists of retrieved articles were also searched for.

Screening

Two (2) researchers independently screened the titles and abstracts of identified studies. One (1) reviewer (GQ) subsequently screened the full text articles of potentially included studies, while a second reviewer (BM) independently screened a 20% random sample. Disagreement was resolved by discussion.

Data collection and analysis

Variables extracted included publication and reporting characteristics as well as items from the PRISMA and AMSTAR checklists. Conditions studied were classified using the International Classification of Diseases. Data were collected using a standardized form and summarized using descriptive statistics (frequency, median, interquartile range [IQR]). Analyses were performed using Excel (version Microsoft Excel 2003; http://office.microsoft.com/zh-cn/) and SPSS (version 13.0; www.spss.com).

Results

Search

A total of 693 publications met the inclusion criteria. Figure 1 shows the process used to select potentially relevant studies for inclusion in this study. Of the 693 references, 310 were duplications. A further 277 were excluded, 18 of which were letters or editorials, 88 were nonacupuncture SRs, and 171 were narrative reviews. A further 18 reviews were excluded because they were quality assessments of systematic reviews and meta-analyses. This left 88 reviews, which were all written in Chinese (See supplemental data with 88 systematic reviews in Chinese journals at www.liebertpub.com/acm <http://www.liebertpub.com/acm>).

FIG. 1.

Flowchart illustrating the systematic review (SR) articles identified, included and excluded. CBM, Chinese Biomedicine Literature Database; CJFD, Chinese Journal Full-text Database; CSJD, Chinese Scientific Journal Full-text Database.

Epidemiological characteristics

The 88 reviews were all written in Chinese and were published in 40 different Chinese journals. Frequency of citation of each review ranged from 0 to 32, more than half of reviews (53.4%) had not been cited, and only 5.4% had been cited more than 15 times. More than one third (37.5%) of the reviews were written by graduate students. The most common conditions studied were diseases of mental and behavioral disorders (23.9%) and diseases of the musculoskeletal system and connective tissue (15.9%) (Table 1).

Table 1.

Epidemiology of Systematic Reviews of Acupuncture Intervention

Category	Characteristic	Number (%) of n=88
Number of times cited	0	47 (53.4)
	1–5	30 (34.1)
	6–10	3 (3.4)
	11–15	4 (4.5)
	>15	4 (4.5)
Role of first author	Clinician	24 (27.3)
	Graduate student	33 (37.5)
	Other	31 (35.2)
Condition focused on in review (Common ICD-10)	Mental and behavioral disorders	21 (23.9)
	Diseases of the musculoskeletal system and connective tissue	14 (15.9)
	Diseases of the nervous system	11 (12.5)
	Diseases of the circulatory system	10 (11.4)
	Symptoms, signs and abnormal clinical and laboratory findings, not classified elsewhere	8 (9.1)
	Diseases of the genitourinary system	7(8.0)
	Diseases of the skin and subcutaneous tissue	7 (8.0)
	Diseases of the digestive system	2 (2.3)
	Diseases of the respiratory system	2 (2.3)
	Endocrine, nutritional and metabolic diseases	1 (1.1)
	Neoplasms	1 (1.1)
	Infectious and parasitic diseases	1 (1.1)
	Pregnancy, childbirth, and the puerperium	1 (1.1)
	Diseases of the ear and mastoid process	1 (1.1)
	Diseases of the eye and adnexa	1 (1.1)
	Diseases of the blood and blood-forming organs and immune mechanism	0

ICD-10, International Classification of Diseases 10.

Descriptive characteristics

The reviews included a median of four authors (IQR: 3–5). Only one-third of reviews (35.2%) were published in specialty journals, less than half of the reviews (40.9%) were published in journals cited by Chinese Science Citation Database (CSCD). The reviews included a median of 9 studies each, involving 911 participants. Meta-analysis was conducted in almost all the reviews (92%). None of the reviews had been updated from a previous review (Table 2).

Table 2.

Descriptive Characteristics of Included Systematic Reviews

Category		All SRs n=88
Number of authors, median (IQR)		4 (3–5)
Journal type	General, n (%)	57 (64.8)
	Specialty, n (%)	31 (35.2)
Indexed in CSCD Yes n (%)		36 (40.9)
Number of included studies: Median (IQR)		9 (6–14)
Number of participants in included studies: Median (IQR)		911 (591–1368)
Meta-analysis: Yes (%)		81 (92)
Update of a previous review: Yes n (%)		0 (0)

IQR, interquartile range; CSCD, Chinese Science Citation Database.

AMSTAR Checklist Assessment

Compliance with AMSTAR checklist items ranged from 0 to 96.6%. Most reviews were compliant with the following checklist items (Table 3):

Table 3.

AMSTAR Assessment of Methodological Characteristics

Category (yes)	Overall, N=88 n (%)
1. Was an a priori design provided?	0 (0.0)
2. Was there duplicate study selection and data extraction?	76 (86.4)
3. Was a comprehensive literature search performed?	36 (40.9)
4. Was the status of publication (i.e., gray literature) used as an inclusion criterion?	3 (2.3)
5. Was a list of studies (included and excluded) provided?	50 (56.8)
6. Were the characteristics of the included studies provided?	81 (92.0)
7. Was the scientific quality of the included studies assessed and documented?	85 (96.6)
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?	78 (88.6)
9. Were the methods used to combine the findings of studies appropriate?	78 (88.6)
10. Was the likelihood of publication bias assessed?	32 (36.4)
11. Was the conflict of interest stated?	0 (0.0)

• reported that there were duplicated study selection and data extraction (86.4%)

• provided the characteristics of included studies (92.0%)

• assessed and documented the scientific quality of the included studies (96.6%)

• appropriately addressed the quality of included studies in formulating conclusions (88.6%)

• used appropriate methods to combine the findings of studies (88.6%)

More than half of the reviews provided a list of studies (56.8%).

Less than half of the reviews were compliant with the following checklist items:

• reported that a comprehensive literature search was performed (40.9%)

• assessed the likelihood of publication bias (36.4%)

Few studies reported the status of publication used as an inclusion criterion (2.3%). No studies provided an a priori design, or stated whether there was a conflict of interest.

PRISMA checklist assessment

Compliance with PRISMA checklist items ranged from 0 to 98.9%. Almost all reviews (97.7%) described themselves using the terms “systematic review” or “meta-analysis” and included a clear rational (98.9%). Most reviews were compliant with the following checklist items (Table 4):

Table 4.

PRISMA Assessment of Reporting Characteristics

Category	Item (Yes)	Overall, N=88 n (%)
Title	1. Title	86 (97.7)
Abstract	2. Structured summary	82 (93.2)
Introduction	3. Rationale	87 (98.9)
	4. Objectives	87 (98.9)
Methods	5. Protocol and registration	0 (0.0)
	6. Eligibility criteria	80 (90.9)
	7. Information sources	73 (83.0)
	8. Search	60 (68.2)
	9. Study selection	62 (70.5)
	10. Data collection process	81 (92.0)
	11. Data items	72 (81.8)
	12. Risk of bias in individual studies	81 (92.0)
	13. Summary measures	82 (93.2)
	14. Synthesis of results	81 (92.0)
	15. Risk of bias across studies	32 (36.4)
	16. Additional analyses	38 (43.2)
Results	17. Study selection	53 (60.2)
	18. Study characteristics	79 (89.8)
	19. Risk of bias within studies	82 (93.2)
	20. Results of individual studies	80 (78.4)
	21. Synthesis of results	73 (83.0)
	22. Risk of bias across studies	49 (55.7)
	23. Additional analysis	20 (22.7)
Discussion	24. Summary of evidence	20 (22.7)
	25. Limitations	79 (89.8)
	26. Conclusions	81 (92.0)
Funding	27. Funding	50 (56.8)

PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

• provided a structured summary (93.2%)

• reported eligibility criteria (90.9%)

• described information sources (83.0%)

• described data collection process (92.0%)

• gave lists of data items (81.8%)

• described the method used for assessing risk of bias of individual studies (92.0%)

• stated the principle of summary measures (93.2%)

• described synthesis of results (92.0%)

• described characteristics of included studies (89.8%)

• described risk of bias within studies (93.2%)

• presented synthesis of results (83.0%)

• discussed limitations at study and outcome level (89.8%) and

• presented an available conclusion (92.0%).

More than half of the reviews were compliant with the following checklist items:

• presented search strategy (68.2%)

• stated the process for selecting studies (70.5%)

• described how the studies were selected (60.2%)

• described the results of individual studies (78.4%)

• presented results of any assessment of risk of bias across studies (55.7%), or

• described sources of funding and other support (56.8%).

Less than half of reviews were compliant with the following checklist items:

• specified any assessment of risk of bias that may affect the cumulative evidence (36.4%)

• described additional analysis (43.2%)

• gave results of additional analysis (22.7%), or

• provided a summary of evidence in the discussion (22.7%).

None of the studies provided protocol or registration information.

Discussion

This study identified 88 SRs of acupuncture interventions published in Chinese journals. It updates previous reviews of this topic⁷ by including an additional 50 SRs. This study is also the first to examine compliance of Chinese SR authors with the PRISMA and AMSTAR tools for assessing reporting and methodological quality.

The range of diseases addressed in this study is similar, with Xiong et al.'s reviews⁷ focusing on the Chinese literature. However, this review did not report details about the epidemiological and study characteristics of SRs of acupuncture, including number of times cited, journal type, and whether it is an update of a published review. In addition, it was found that, despite the increasing use of the terms “systematic review” and “meta-analysis” in the title and subsequent article sections, the quality of reporting remains poor.

More than half of the Chinese SRs (53.4%) were not referenced by other researchers working in the same field. There may be two potential reasons. First, the overall quality of this body of work is poor, because only 40.9% of the reviews were indexed in the Chinese Science Citation Database (CSCD), which is similar to the Science Citation Index, in that indexed journals are considered of higher quality than nonindexed journals. Second, so far, many clinicians and nurses still have not heard of or do not understand the meaning of evidence-based medicine (EBM).^9,10 Not all medical colleges in China have introduced EBM curricula, even though the Chinese Cochrane Center was established in 1997 by the Ministry of Health of the People's Republic of China.

It is well known that results from reviews are most useful when they are up to date,¹¹ but none of the SRs included in the current study reported being an update of a previous review. This may be due to lack of relevant policies in China to encourage updates and reluctance of Chinese journals to publish updated reviews that are not substantially different from previous publications. This problem is not restricted to Chinese journals. Moher et al.¹² reported that 18% of the English SRs were updates; however, only those SRs published in the Cochrane Library had been updated. If these SRs are to retain their currency, updating them needs to be a much higher priority. This issue is likely to become increasingly important in coming years.

This analysis demonstrated that the main methodology shortcomings of the Chinese SRs were that more than half (59.1%) did not perform comprehensive literature searches and almost all (97.7%) did not include searches of gray literature or ongoing studies. Publication bias was considered or assessed in only a minority of reviews despite some evidence for its existence and potential influence on the results of reviews.^13,14 This important methodology component must be considered in future research.

As for the quality of reporting, half of the Chinese reviews failed to report being funded by commercial sources, which has been associated with bias in the results of clinical trials.¹⁵ It was also disappointing that most reviews did not specify and present results of any assessment of risk of bias that may affect the cumulative evidence despite the fact that many studies had shown that assessment of the heterogeneity of risk of bias across studies was an essential part of meta-analysis.^16,17

In addition, many Chinese reviews did not report key aspects of SR methodology, thus impairing confidence in their results and conclusions. For example, none of the reviews reported working from a protocol to complete their review. Likewise, no reviews published in Chinese journals were registered, or had a registration number. Although SRs registration is not yet widely available,¹⁸ the participating journals of the International Committee of Medical Journal Editors¹⁹ have required all clinical trials to be registered in an effort to increase transparency and accountability.²⁰ Similar behaviors may well affect SRs, although currently there are little data to confirm this belief. However, the reasons for registering SRs are likely different from that of registering clinical trials.

Based on the results as well as on experiences and impressions collected during our analysis, the authors would like to suggest three measures to improve the quality of SRs published in Chinese journals. First, the use of reporting guidelines by authors is strongly recommended. Second, it is also recommended that editors of medical journals recognize and promote the use of reporting guidelines in their publications. Lastly, medical schools should introduce reporting guidelines into medical education as early as possible.

This study acknowledges the following. First, the study included SRs published only in Chinese journals, whereas Chinese investigators increasingly publish articles in international journals. Second, in this study, the terms “systematic review” and “meta-analysis” were included. Some potentially eligible SRs may, however, have not used these terms in their publications. Third, this analysis relied on reporting from authors. It is possible that authors conducted their SRs but omitted important details from their report, or the peer-review process resulted in the removal of key information this review sought.

Conclusions

The purpose of this review was to provide readers with a broad overview of the reporting and methodological characteristics of SRs of acupuncture published in Chinese journals. Although many SRs of acupuncture interventions have been published in Chinese journals, the reporting quality is troubling. Thus, the most urgent strategy is to focus on increasing the value of SRs of acupuncture interventions, rather than continuing to publish them in great quantity.

Footnotes

Acknowledgments

The authors thank the library of Lanzhou University for database access and acquiring full texts.

Disclosure Statement

No competing financial interests exist.

References

, Feng

, Zou

, Liu

et al. Meta-analysis of acupuncture for semiplegia caused by stroke. J Chengdu Univ Trad Chin Med, 2002; 25:37–39.

Moher

, Liberati

, Tetzlaff

et al. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med, 2009; 151:264–269.

Shea

, Grimshaw

, Wells

et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol, 2007; 7:10.

Oxman

. Checklists for review articles. BMJ, 1994; 309:648–651.

Sacks

, Berrier

, Reitman

et al. Meta-analyses of randomized controlled trials. NEJM, 1987; 316:450–455.

Bessa-Nogueira

, Vasconcelos

, Niederman

. The methodological quality of systematic reviews comparing temporomandibular joint disorder surgical and non-surgical treatment. BMC Oral Health, 2008; 8:27.

Xiong

, Du

, Li

et al. Assessment of methodology and report quality of systematic evaluation and meta-analysis of acupuncture-moxibustion in China. Chin Acupunct Moxibust, 2009; 29:763–768.

Linde

, ter Riet

, Hondras

et al. Characteristics and quality of systematic reviews of acupuncture, herbal medicines, and homeopathy. Forsch Komplementärmed Klass Naturheilkd, 2003; 10:88–94.

Wang

, Fan

. Investigation cognition about evidence-based medicine. Med Information, 2010; 8:2537–2539.

10.

Liu

, Guo

, Zhang

. Investigation into nursing staff's cognition status of evidence-based medicine. J Liao Ning Med Univ, 2010; 31:259–262.

11.

Moher

, Tsertsvadze

. Systematic reviews: When is an update an update? Lancet, 2006; 367:881–883.

12.

Moher

, Tetzlaff

, Tricco

et al. Epidemiology and reporting characteristics of systematic reviews. PLoS Med, 2007; 4:e78.

13.

Dickersin

. Publication bias: Recognizing the problem, understanding its origins, scope, preventing harm. Rothstein

, Sutton

, Borenstein

. Publication Bias in Meta-Analysis-Prevention, Assessment and Adjustments. West Sussex (UK): John Wiley & Sons, 2005; 356.

14.

Sutton

. Evidence concerning the consequences of publication, related biases. Rothstein

, Sutton

, Borenstein

. Publication Bias in Meta-Analysis-Prevention, Assessment and Adjustments. West Sussex (UK): John Wiley & Sons, 2005; 356.

15.

Lexchin

, Bero

, Djulbegovic

et al. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ, 2003; 326:1167–1170.

16.

Thompson

. Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated. BMJ, 1994; 309:1351–1355.

17.

Higgins

, Thompson

, Deeks

et al. Measuring inconsistency in meta-analyses. BMJ, 2003; 327:557–560.

18.

University of York Centre for Reviews and Dissemination. www.york.ac.uk/inst/crd/. 2011 February 20.

19.

De Angelis

, Drazan

, Frizelle

et al. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. Lancet, 2004; 364:911–912.

20.

Whittington

, Kendall

, Fonagy

et al. Selective serotonin reuptake inhibitors in childhood depression: Systematic review of published versus unpublished data. Lancet, 2004; 363:1341–1345.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB