Randomized Controlled Trials In Endourology: A Quality Assessment

Abstract

Purpose:

To analyze the quality of studies reporting randomized clinical trials (RCTs) in the field of endourology.

Materials and Methods:

RCTs published in the Journal of Endourology from 1993 until 2011 were identified. The Jadad scale, van Tulder scale, and Cochrane Collaboration Risk of Bias Tool (CCRBT) were used to assess the quality of the studies. The review period was divided into early (1993–1999), mid (2000–2005), and late (2006–2011) terms. Studies were categorized by country of origin, subject matter, single- vs multicenter setting, Institutional Review Board (IRB) approval and funding support, and blinding vs nonblinding.

Results:

In total, 3339 articles had been published during the defined review period, of which 165 articles were reporting a RCT. There was a significant increase in the number of RCTs published over time, with 18 (2.81%), 43 (4.88%), and 104 (5.72%) studies identified in the early, mid, and late term, respectively (P=0.009). Nevertheless, there was no difference in terms of quality of reporting, as assessed with the Jadad scale, van Tulder scale, or CCRBT, between the three study terms. On the other hand, significant differences were found in both the number of high qualitative RCTs that used blinding methodology and those that had IRB review, when comparing the early, mid, and late terms.

Conclusion:

There has been a growing number of Journal of Endourology publications reporting on RTC over the last two decades. The quality of reporting for these studies remains suboptimal, however. Researchers should focus on a more appropriate description of key features of any given RCT, such as randomization and allocation methods, as well as disclosure of IRB review and financial support.

Introduction

S ince the introduction of evidence-based medicine (EBM), the quality of medical care has continually improved and medical knowledge has advanced. EBM uses the results from randomized controlled trials (RCTs) because these results are based on an objective, scientific, and systematic study design.^1,2 RCTs, however, may still contain sources of bias that can lead to incorrect conclusions.³ Therefore, an assessment of the quality of evidence in the research literature plays an important role in the establishment of EBM, ultimately improving the quality of healthcare through implementation of key findings into clinical practice. In addition, EBM can lead to improved cost-effectiveness through the removal of unnecessary medical expenses.

Currently, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidelines for establishing high-quality RCTs, with studies that follow these guidelines considered as suitable for inclusion in EBM. Further, many international academic societies recommend compliance with the CONSORT statement for RCTs.^4
–6 One limitation of the CONSORT statement, however, is that it is not a tool to assess the quality of an RCT, simply a set of guidelines for its preparation.

By contrast, the Jadad scale is a tool that can be used to assess the quality of an RCT via three questions on randomization, double blinding, and dropout.⁷ Other tools used to assess the strength of evidence derived from an RCT include the van Tulder scale⁸ and the Cochrane Collaboration Risk of Bias Tool (CCRBT), both tools recommended by the Cochrane group. The quality of reporting of RCTs presented in the form of abstracts at the World Congress of Endourology has been previously studied.⁹

The aim of this study was to perform a quality assessment of RCTs published in the Journal of Endourology, the most representative journal in the field of endourology.

Materials and Methods

Scope of review and selection criteria for studies

A total of 3339 original articles published in the Journal of Endourology, from 1993 (vol. 1) until 2011 (vol. 12), were identified through the Pubmed and Embase databases by using the following search terms: “randomized,” “randomization,” and “randomly.”

Quality assessment method

Two reviewers performed the analysis using the Jadad scale, the van Tulder scale, and the CCRBT as quality assessment tools.

The studies were also categorized by country of origin, subject matter, single- vs multicenter setting, Institutional Review Board (IRB) approval and funding support, and blinding vs nonblinding.

The results from the two reviewers were moderated by a third reviewer in the event of a dispute between them.

Jadad scale. The Jadad scale (also known as the Oxford quality scoring system) is composed of five points in total: two points related to randomization, two points related to blinding, and one point related to dropout.¹⁰ For randomization and blinding, when there are general comments without any description about randomization or blinding, one point each is assigned to each respective category. When there is a description of the proper method, one point is added to each respective category. When the description method is inappropriate, one point is deducted from each respective category. For dropout, when the number of dropouts for each subject group and the reasons for dropout are specified, one point is given. Even if there is no dropout, this fact should be stated. When the total for the Jadad score is ≥3 points, the study is considered as “high-quality”; when ≤2 points, as “low quality.” For studies in which double blinding is not impossible, a study with a total score ≥2 points is considered as “high-quality.”

van Tulder scale. The van Tulder scale evaluates 11 components, including randomization, allocation concealment, baseline characteristics, patient blinding, caregiver blinding, observer blinding, cointervention, compliance, dropout rate, end point assessment time point, and intention-to-treat analysis.⁸ The van Tulder assessment method involves the selection of “yes,” “no,” or “don't know” for each item. When ≥5 criteria for assessment items are satisfied (≥5 points), the quality is considered as “high.”

CCRBT. The CCRBT includes six items: Sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential threats to validity. For each domain, the CCRBT assessment involves the selection of “yes,” “no,” or “unclear,” which corresponds to a low, high, or uncertain risk of bias, respectively. After the six domains are assessed, if the first three questions are answered with “yes” and no important concerns related to the last three domains are identified, then the study is classified with a “low risk of bias.” Studies assessed as having ≤2 domains with “unclear” or “no” are classified with a “moderate risk of bias.” Studies assessed as having ≥3 domains with “unclear” or “no” responses are classified with a “high risk of bias.”

Statistical analysis

The studies were divided into three terms: Early (1993 to 1999), mid (2000 to 2005), and late (2006 to 2011) term. The scores obtained by each respective assessment tool (Jadad scale, van Tulder scale, CCRBT) were compared and analyzed using the one-way analysis of variance test whereas the comparison of high-quality article ratio and qualitative assessment by CCRBT were analyzed using the chi-square test. In addition, the comparisons of Institutional Review Board (IRB) approval, funding support, and use of blinding were analyzed using the Student t test. SPSS v.18.0 was used for all statistical analyses and a P value of <0.05 was considered statistically significant.

Results

Variations in the number of RCTs by term

From a total of 3339 Journal of Endourology publications, 165 were identified as RCTs. The number of RCTs published in the early, mid, and late terms were 18 (2.81%), 43 (4.88%), and 104 (5.72%), respectively (P=0.009; Table 1).

Table 1.

Characteristics of Randomized Controlled Trials According to Publication Year

Years	Original articles	RCT (%)	Ethics (IRB) approval(%)	Funding (%)	Blinding (%)	Multi/single (%)	Concealment of allocation (%)
1993–1999	640	18 (2.81)	2 (11.11)	0	3 (16.67)	2 (11.11)	0
2000–2005	881	43 (4.88)	15 (34.88)	1 (2.33)	17 (39.53)	2 (4.65)	0
2006–2011	1818	104 (5.72)	54 (51.92)	3 (2.88)	36 (34.62)	4 (3.85)	1 (0.96)
P value		0.009	0.002	0.763	0.221	0.415	0.744
Total	3339	165 (4.94)	71 (43.03)	4 (2.42)	56 (33.94)	8 (4.85)	1 (0.61)

Chi-square test.

RCT=randomized controlled trial; IRB=Institutional Review Board.

Qualitative variations of RCTs by term

The Jadad scale scores for the early, mid, and late terms were 1.50±0.86, 1.95±1.02, and 1.92±1.03, respectively (P=0.627). The number of high quality articles for the early, mid, and late terms were 2 (11.11%), 11 (25.58%), and 27 (25.96%), respectively (P=0.387).

The van Tulder scale scores for the early, mid, and late terms were 3.22±1.31, 3.72±1.32, and 3.87±1.46, respectively (P=0.646). The number of high-quality articles for the early, mid, and late terms were 2 (11.11%), 11 (25.58%), and 27 (25.96%), respectively (P=0.387).

Based on the CCRBT, there was no article with a low risk of bias in either the early or mid terms, but there were two articles (1.92%) with a low risk of bias in the late term (P=0.464; Table 2).

Table 2.

Quality Assessment of Randomized Controlled Trials According to Publication Year

	Jadad scale		van Tulder scale		Cochrane Assessment of Risk Bias
Years	Score	High quality (%)	Score	High quality (%)	Low risk (%)	Moderate risk (%)	High risk (%)
1993–1999	1.50±0.86	2 (11.11)	3.22±1.31	2 (11.11)	0	5 (27.78)	13 (72.22)
2000–2005	1.95±1.02	11 (25.58)	3.72±1.32	11 (25.58)	0	18 (41.86)	25 (58.14)
2006–2011	1.92±1.03	27 (25.96)	3.87±1.46	27 (25.96)	2 (1.92)	48 (46.15)	54 (51.92)
P value	0.627	0.387^a	0.646	0.387^a			0.464^a
Total	1.88±1.02	40 (24.24)	3.76±1.42	40 (24.24)	2 (1.21)	71 (43.03)	92 (55.76)

Analysis of variance test.

chi-square test.

Analysis of factors related to the quality of the articles

There were significant differences in both the country of origin (P<0.001; Table 3) and the subject matter of the studies (P=0.011; Table 4) by term. Similarly, there were significant differences in the number of trials approved by an IRB by term (P=0.002). It was noted that trials approved by an IRB were more likely to be classified as high-quality articles when using the Jadad and van Tulder scales (P<0.001). With regard to funding, blinding, single-center or multicenter setting, and concealment of allocation, there was no significant difference in any of these parameters (P=0.763, P=0.221, P=0.415, P=0.744, respectively; Table 1). It was noted that trials that included blinding were more likely to be classified as high-quality articles when using all tools (P<0.001). Studies with funding were more likely to be classified as high-quality articles when using all tools (P=0.045, P=0.045, P<0.001; Table 5).

Table 3.

Distribution of Randomized Controlled Trials (%)

Country of origin	1993–1999	2000–2005	2006–2011	Total	P value
United States	7 (33.33)	18 (35.29)	13 (13.98)	38 (23.03)	<0.001
Turkey	4 (19.05)	6 (11.76)	18 (19.35)	28 (16.97)
Iran	0	2 (3.92)	17 (18.28)	19 (11.52)
India	1	5 (9.80)	12 (12.90)	18 (10.91)
China	0	0	9 (9.68)	9 (5.45)
Italy	0	5 (9.80)	3 (3.23)	8 (4.85)
Canada	0	5 (9.80)	1 (1.08)	6 (3.64)
Others	11 (52.38)	9 (17.65)	20 (21.51)	40 (24.24)
Total	21	51	93	165

Chi-square test.

Table 4.

Distribution of Randomized Controlled Trials (Topics)

Topic	Stone	BPH	Prostate cancer	RCC	Kidney donor	Bladder tumor	Renal cyst	Urethral stenoses	Urachal remnant	Other
1993–1999	8	7	0	0	0	0	0	0	0	3
2000–2005	9	14	1	1	0	0	0	0	0	18
2006–2011	59	12	6	2	3	2	1	1	1	17
Total	76	33	7	3	3	2	1	1	1	38
P value	0.011

Chi-square test.

BPH=benign prostatic hyperplasia; RCC=renal-cell carcinoma.

Table 5.

Factors Associated with Quality of Randomized Controlled Trials

		Jadad scale		van Tulder scale		Cochrane's assessment of risk bias
Factors	No. of RCTs (%)	Score	High quality (%)	Score	High quality (%)	Low risk (%)	Moderate risk (%)	High risk (%)
Blinding
Yes	56 (33.94)	2.95±0.88	37 (66.07)	5.14±1.33	38 (67.86)	2 (3.57)	41 (73.21)	13 (23.21)
No	109 (66.06)	1.34±0.53	3 (2.75)	3.05±0.80	2 (1.83)	0	30 (27.52)	79 (72.48)
P value		<0.001	<0.001^a	<0.001	<0.001^a		<0.001^a
Funding source
Yes	4 (2.42)	3.00±1.41	3 (75.00)	6.00±2.16	3 (75.00)	1 (25.00)	2 (50.00)	1 (25.00)
No	161 (97.58)	1.86±0.99	37 (22.98)	3.70±1.36	37 (22.98)	1 (0.62)	69 (42.86)	91 (56.52)
P value		0.026	0.045^a	0.001	0.045^a		<0.001^a
Reviewed by IRB
Yes	71 (43.03)	2.20±1.12	27 (38.03)	4.11±1.54	14 (19.72)	2 (2.82)	36 (50.70)	33 (46.48)
No	94 (56.97)	1.65±0.86	13 (13.83)	3.49±1.26	26 (27.66)	0	35 (37.23)	59 (62.77)
P value		0.001	<0.001^a	0.005	0.001^a		0.043^a

Student t test.

chi-square test.

Discussion

The analysis of articles published in the Journal of Endourology showed that the number of RCTs published has increased over time, while there was no significant improvement over time with regard to quality of reporting. There were, however, changes over time with regard to the subject matter and country of origin of the articles. Scales and associates¹¹ reported that the number of RCTs increased over time when they compared RCTs reported by the Journal of Urology, Urology, European Urology, and British Journal of Urology International in 1996 and 2004. Lee and colleagues¹² performed a quality assessment using the Jadad scale on articles published in the Korean Journal of Urology over the past 20 years and observed an increase in the number of RCTs over time.

Similar to these previous reports, the present study showed that the number of RCTs published in the Journal of Endourology increased over time. This result is likely because methods providing a high level of evidence have increasingly been required since EBM has become important. Because of the recent advancement of medical research, there are currently 20,000 different medical academic journals being published globally, with approximately 400,000 articles being listed in Medline each year. The importance of EBM is now well recognized and can be seen as an approach to scientifically and systematically analyze the vast amount of studies that are now published.^13,14 The increase in RCTs observed for the Journal of Endourology is also reflected in the majority of other Science Citation Index journals. The quality of RCTs is also important, especially as the number of studies increases.

The quality assessment for RCT enables investigators to perform evaluation of bias likely to incur from design, performance, and analysis of a clinical trial, to determine the acceptability of the article's conclusion, and to recognize the needs of further studies and is required as the instrument for those activities.^15,16

When comparing the early vs the mid-late periods, an increase in the score and the high-quality ratio based on the Jaded and van Tulder scales was noticed. This increase trended toward statistical significance in terms of score for both scales (P=0.088 and P=0.082), whereas it was not statistically significant in terms of high-quality ratio. In CCRBT, a proportion of low risk for mid/late periods is higher than the early period, but it did not differ in statistics (Table 6).

Table 6.

Quality Assessment of Randomized Controlled Trials According to Publication Year (Early vs Mid/Late)

	Jadad scale		van Tulder scale		Cochrane's assessment of risk bias
Years	Score	High quality (%)	Score	High quality (%)	Low risk (%)	Moderate risk (%)	High risk (%)
1993–1999 (early)	1.50±0.86	2 (11.11)	3.22±1.31	2 (11.11)	0	5 (27.78)	13 (72.22)
2000–2011 (mid/late)	1.93±1.03	38 (25.9)	3.84±1.42	38 (26.0)	2 (1.4)	66 (45.2)	78 (53.4)
P value	0.088	0.168^†	0.082	0.164^†		0.303^†
Total	1.88±1.02	40 (24.24)	3.76±1.42	40 (24.24)	2 (1.21)	71 (43.03)	92 (55.76)

Analysis of variance test; ^†chi-square test.

Tools that have been validated and are used for quality assessment of RCTs include the Jadad scale, the van Tulder scale, the CCRBT, Newell scale, the Scottish Intercollegiate Guidelines Network, and the National Institute for Health and Clinical Excellence tool.¹⁷ The three representative tools used in this study enabled comprehensive analysis of several elements contained within the CONSORT statement.^18,19

The Jadad scale is advantageous because of its simplicity, but it does not include an assessment item for allocation concealment. By contrast, the van Tulder scale and the CCRBT contain assessment items for allocation concealment, increasing the effectiveness of quality assessment of a RCT. There is not, however, any standardized assessment tool to perform a quality assessment of an RCT. To this end, we performed three simultaneous analyses to compare the aforementioned RCT quality assessment tools. The use of these three assessment tools meant that a broad range of assessment items within the CONSORT statement were considered, which enabled a more objective assessment of the quality of the RCT articles.

There was a significant difference in the diversity of countries from which the RCTs were performed over time, indicating that the role of the Journal of Endourology as an international academic journal is expanding. Further, the diversity of RCT subject matter increased significantly over time, implying that the latest trends within the scope of the Journal of Endourology interests are being captured.

We also examined the incidence and role of IRB approval. No previous study has reported on the association between IRB approval and the quality of the articles. We found that there was a significant increase over time in the number of articles stating IRB approval, and that those with IRB approval were associated with higher quality. IRB approval is considered an international standard and can be an important stage to check the validity of the study design. In fact, it is thought that the increasing occurrence of IRB review of RCT study protocols has played a role in the elevated number of high-quality articles that are now appearing in the medical literature.

Hewitt and coworkers²⁰ assessed the quality of RCTs published in four high impact journals (the British Medical Journal, the Journal of the American Medical Association, Lancet, and the New England Journal of Medicine) and reported that the number of studies with inadequate or uncertain concealment of allocation was as high as 46%. Schulz and colleagues²¹ also reported that inadequate concealment of allocation tends to damage or bias the randomization process of a clinical trial, and that it could distort the effectiveness of intervention by more than 40%, despite initial randomization having been conducted.

We found that the percentage of RCT articles published in the Journal of Endourology with correct concealment of allocation was low, needing attention to be given to this important aspect of study design and implementation. Clifford and associates²² analyzed 100 RCTs published in five different peer-reviewed, general medical journals with high impact factor and reported that there was no association between the funding source and the quality of the article.

On the contrary, in the present study we found that there was a difference in the qualitative assessment of the quality of the article depending on the funding sources for the RCT. Jadad and coworkers¹⁰ suggested that blinding was necessary for qualitative improvement of a RCT. We also noted that there was a qualitative difference in the quality of the RCT depending on whether there was blinding in the study design. It was also identified that RCTs related to the use of drug products had a higher quality than RCTs related to surgery or operational procedures. The fact that a substantial number of RCTs published in the Journal of Endourology were of a surgical nature, with the inherent difficulties in implementing single or double blinding, meant that double blinding was uncommon overall, leading to an overall decrease of scores in quality assessment.

A limitation of this study was the subjective judgment used in the process of assessment. Two reviewers extracted the RCTs and performed the quality assessment independently to limit subjectivity; further, a third reviewer was used to moderate the outcome if there was a discrepancy between the two original reviewers. In addition, although we used well-known quality assessment tools for RCTs, not all items listed in the CONSORT statement were considered. Nevertheless, the use of three different validated tools meant that the majority of items were considered and the conclusions drawn from the analysis were strengthened.

Other studies' designs, including collaborative and large cohort studies, can also significantly contribute to medical knowledge. RCTs are usually better suited for drug testing rather than surgical procedures. It is important to understand, however, the quality of the RCTs in the field of endourology/minimally invasive surgery. Among the study limitations, one needs to be recognized. The Journal of Endourology was originally conceived as a main venue for the description of novel surgical techniques and their outcomes in the field of minimally invasive urologic surgery rather than a stage for clinical trials. This might have affected the features and profile of the studies submitted to this journal in the early period.

Conclusion

The numbers of original articles and RCTs published in the Journal of Endourology has steadily increased between 1993 and 2011, which testifies to the higher level of evidence provided by the authors contributing to the journal. Nevertheless, the quality of reporting of RCTs remains suboptimal. Areas of improvement are related to the reporting of key features of these studies, such as randomization and allocation methods and IRB approval and financial support. By more precisely reporting these relevant aspects of any RCT, investigators will deliver higher quality studies in the field of endourology/minimally invasive surgery.

Footnotes

Acknowledgment

This work was supported by the Medical Research Center (grant 2010-0029508) funded by the National Research Foundation (NRF) of the Ministry of Education, Science and Technology, Republic of Korea.

Disclosure Statement

No competing financial interests exist.

Abbreviations Used

References

Jadad

, Cook

, Jones

et al. Methodology and reports of systematic reviews and meta-analyses: A comparison of Cochrane reviews with articles published in paper-based journals. JAMA, 1998; 280:278–280.

Juni

, Altman

, Egger

. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ, 2001; 323:42–46.

Featherstone

, Donovan

. Random allocation or allocation at random? Patients' perspectives of participation in a randomised controlled trial. BMJ, 1998; 317:1177–1180.

Begg

, Cho

, Eastwood

et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA, 1996; 276:637–639.

Altman

. Endorsement of the CONSORT statement by high impact medical journals: Survey of instructions for authors. BMJ, 2005; 330:1056–1057.

Gagnier

, Boon

, Rochon

et al.

Improving the quality of reporting of randomized controlled trials evaluating herbal interventions: Implementing the CONSORT statement [corrected]

Explore (NY), 2006; 2:143–149.

Clark

, Wells

, Huet

et al. Assessing the quality of randomized trials: Reliability of the Jadad scale. Control Clin Trials, 1999; 20:448–452.

van Tulder

, Furlan

, Bombardier

, Bouter

. Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine, 2003; 28:1290–1299.

Autorino

, Borges

, White

et al.

Randomized clinical trials presented at the World Congress of Endourology: How is the quality of reporting?

J Endourol, 2010; 24:2067–2073.

10.

Jadad

, Moore

, Carroll

et al.

Assessing the quality of reports of randomized clinical trials: Is blinding necessary?

Control Clin Trials, 1996; 17:1–12.

11.

Scales

Jr. , Norris

, Preminger

, Vieweg

, Peterson

, Dahm

. Evaluating the evidence: statistical methods in randomized controlled trials in the urological literature. J Urol, 2008; 180:1463–1467.

12.

Lee

, Chung

, Kang

et al. Quality assessment of randomized controlled trials published in the korean journal of urology over the past 20 years. Korean J Urol, 2011; 52:642–646.

13.

Sutton

, Cooper

, Jones

. Evidence synthesis as the key to more coherent and efficient research. BMC Med Res Methodol, 2009; 9:29.

14.

Gill

, Dowell

, Neal

et al. Evidence based general practice: A retrospective study of interventions in one training practice. BMJ, 1996; 312:819–821.

15.

Chalmers

, Celano

, Sacks

, Smith

Jr . Bias in treatment assignment in controlled clinical trials. N Engl J Med, 1983; 309:1358–1361.

16.

Schulz

, Chalmers

, Hayes

, Altman

. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA, 1995; 273:408–412.

17.

Kim

, Choi

, Ahn

et al. Quantity and quality assessment of randomized controlled trials published in five Korean medical journals, from 1980 to 2000. J Korean Acad Fam Med, 2004; 25:118–125.

18.

Moher

, Schulz

, Altman

. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Clin Oral Investig, 2003; 7:2–7.

19.

Moher

, Jadad

, Nichol

et al. Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials, 1995; 16:62–73.

20.

Hewitt

, Hahn

, Torgerson

et al. Adequacy and reporting of allocation concealment: Review of recent trials published in four general medical journals. BMJ, 2005; 330:1057–1058.

21.

Schulz

, Grimes

. Allocation concealment in randomised trials: defending against deciphering. Lancet, 2002; 359:614–618.

22.

Clifford

, Barrowman

, Moher

. Funding source, trial outcome and reporting quality: Are they related? Results of a pilot study. BMC Health Serv Res, 2002; 2:18.