‘Allocation concealment’: the evolution and adoption of a methodological term

Abstract

Introduction

Random assignment of individual participants in clinical trials entails two separate steps: (1) generating an unbiased allocation schedule; and (2) using the schedule, without foreknowledge of upcoming allocations, to assign participants to treatment comparison groups. Both of these steps were implemented in two controlled trials done under the aegis of the British Medical Research Council in the 1940s¹: the trial of patulin for the common cold,^2,3 and the better known trial of streptomycin for pulmonary tuberculosis.^1,4 However, the two separate steps needed to achieve unbiased treatment assignment were not identified clearly until 1955, when Austin Bradford Hill distinguished them in the sixth edition of his book Principles of Medical Statistics.⁵ Shortly thereafter, David Cox⁶ and Peter Armitage⁷ emphasised the importance of implementing an unbiased allocation by using sealed envelopes to conceal an upcoming allocation until after a participant had been irrevocably entered into a trial. Despite early recognition of the importance of the two separate steps in unbiased allocation, however, no widely accepted term denoting the process of concealing upcoming allocations had been adopted more than half a century later.

Methodological terminology is often imprecise. Researchers may abandon the intended meaning, even with common terminology. For example, the terms ‘nomogram’ and ‘case-control’ are widely misused in research.^8,9 Jargon (for example, ‘gold standard’) leads to confusing and meaningless terms.¹⁰ Clearly, accurate communication requires unambiguous terminology.

The plethora of terminology surrounding randomised trials presents particular challenges. Trials suffer from a great deal of arcane terminology. Some examples include ‘random permuted blocks’, ‘participant retention’, ‘double- and triple-blinding’, ‘masking’, ‘interim analyses’, ‘alpha spending’, ‘urn randomisation’, ‘biased-coin randomisation’ and ‘group sequential trials’. While most of these represent important trial processes and methods, even the most common terms may confuse or obfuscate. For example, although blinding terminology seems well ensconced, even apparently simple terminology like ‘double-blinding’ elicits inconsistent connotations. When investigators examined physician interpretations and textbook definitions of ‘double-blinding’, they found 17 unique interpretations and nine different definitions.¹¹ Reporting only ‘double-blinding’ without proper elaboration leads to ambiguity.^12,13

Terminology serves another important function. It can indicate a critical methodological process that might otherwise be neglected in the conduct of a study unless highlighted by terminology. Because lack of attention to detail can impact on the conduct of trials, lack of appropriate terminology may cause harm if an important trial function has been inadequately addressed.

Despite the sheer volume of terminology in randomised trials, we discovered a terminological gap when we were assessing the quality of treatment allocation in reports of controlled trials published in journals of obstetrics and gynaecology in the 1990s.¹⁴ When writing up that research, we introduced the term ‘allocation concealment’¹⁴ to denote the second of the two essential steps in achieving unbiased allocation to treatment comparison groups. In this article, we have reviewed the precursors of the term ‘allocation concealment’ and its subsequent evolution and adoption from the mid-1990s onwards.

Recognising the need for randomisation terminology

Over the years, books on randomised trials provided excellent and detailed descriptions of methods to generate randomised allocation sequences. These sequences formed randomisation lists. Chapters and sections of chapters had titles such as ‘methods of randomisation’, ‘constrained randomisation’, ‘adaptive randomisation’, ‘unequal treatment allocation’, ‘permuted block designs’, ‘biased-coin design’, ‘urn design’ and ‘stratified randomisation’.

In books published before we suggested the term ‘allocation concealment’ in 1994, chapters describing the mechanisms for implementing allocation sequences were rare. Moreover, scant information was devoted to those mechanisms elsewhere in the books. For example, one of the most detailed discussions we found was a ‘Mechanics of Randomization’ section from a book published in 1985, but it was less than two pages in a 17-page chapter.¹⁵

As we have noted, some trialists recognised the importance of mechanisms for implementing allocation sequences long ago, including the use of sealed envelopes.^5–7,16 In meetings with IC and KFS in 1992, Richard Peto¹⁷ noted that he had emphasised that randomisation should incorporate a mechanism to prevent investigators, healthcare providers and participants from foreknowledge of upcoming assignments, but he had not come up with a term for the mechanism.

‘Randomisation blinding’

Thomas Chalmers et al.¹⁸ provided key inspiration. In the early 1980s, they had not only recognised the importance of the assignment mechanism, but had termed it ‘randomisation blinding’. They suspected that bias introduced into ‘…studies in which assignment of controls is less blinded may be explained by bias in the selection or rejection of patients when the treatment to be given is known or suspected at the time of assignment’.¹⁹ Thus, without adequate ‘randomisation blinding’, selection bias could occur. To investigate their suspicions, they used trials of treatments for acute myocardial infarction, using case-fatality rates as the outcome. They defined ‘blinded randomisation’ as ‘opaque envelopes’, ‘a telephone call to a statistical center’ or ‘a prearranged order of blinded medications labeled consecutively by the pharmacy’.¹⁹ They defined ‘unblinded randomisation' as:

‘assignment from an open table of random numbers, according to date of birth or chart number, or by some other variably random system in which patients could present for study in a chance order but be selected or rejected after the physician knew the treatment assignment’.¹⁹

They found that trials using unblinded randomisation yielded larger estimates of treatment effects, which were more often statistically significant, than those in trials using blinded randomisation. Their findings provided the strongest evidence at that time that ‘unblinded randomisation’ was associated with empirical evidence suggestive of bias in ‘randomised’ trials.

The ‘blinding’ terms ‘blinded randomisation’ and ‘unblinded randomisation’ are confusing because they address a mechanism within treatment allocation, not whether a trial is subsequently blinded to treatment identity. For example, a trial may have ‘blinded randomisation’ as defined by Chalmers et al.,¹⁹ but, after allocation, investigators and participants may not have been blinded to treatment assignments. Perhaps because of confusion between ‘randomisation blinding’ and ‘treatment blinding’ or ‘outcome measurement blinding’, adoption of the ‘blinded randomisation’ and ‘unblinded randomisation’ terminology remained rare. Moreover, their definitions of ‘blinded randomisation’ and ‘unblinded randomisation’ were insufficiently specific. For example, although referring to ‘opaque envelopes’, they did not mention additional precautions such as sequential numbering, and their definitions and analysis neglected the common occurrence of published reports that had not described the implementation mechanism of the random allocation schedule.

As noted by commentators,²⁰ the analysis used by Chalmers et al.¹⁹ could not take account of the different types of treatments in their blinded and unblinded randomisation categories, and thus leaving their results potentially confounded by treatment. For example, most beta-blocker trials were in the blinded randomisation group while most antithrombotic agent trials were in the unblinded randomisation group.²⁰ Second, they did not account for other trial characteristics, such as generation of the allocation sequence and blinding of treatments. Third, their study left unanswered the status of all those trials whose published reports did not provide any information about the implementation mechanism for random allocation. That was a critical issue, because poorly reported trials then predominated in the literature,^21–24 as they do now.²⁵

‘Bias-reducing allocation’

In 1990, Altman and Doré²³ improved terminology and definitional clarity. Instead of using the terms ‘blinded randomisation’ and ‘unblinded randomisation’, they termed the process ‘bias-reducing allocation’, using this definition: ‘The mechanism of treatment allocation should be designed to avoid bias: suitable methods are central randomisation, coded drugs prepared by the pharmacy, and use of a series of numbered opaque sealed envelopes’. Their categorisation of reports was based on these definitions as ‘yes’, ‘no’ or ‘not specified’.

The Altman and Doré terminology and analysis were improvements over that suggested by Chalmers et al.¹⁹ because they avoided derivatives of the term ‘blind’, specified that envelopes should be sequentially numbered and sealed, and included a ‘not specified’ option. The latter applied to almost half of the responses in their study and to three-quarters of the reports of randomised trials in PubMed.²⁵ For anyone appraising the quality of the literature, this ‘not specified’ category provides an important perspective for evaluating quality.

‘Allocation concealment’

We needed a replacement term for ‘randomisation blinding’. Although Altman and Doré had improved the terminology by introducing ‘bias-reducing allocation’, some confusion remained. For example, ‘bias-reducing allocation’ could be confused with the process of generating a random allocation sequence, or indeed, the entire randomisation process. Accordingly, we sought further terminological improvement in our review of reports of trials in obstetrics and gynaecology.¹⁴

Without any recognised method for introducing terminology, we relied upon a subjective, iterative, discussion process. We avoided any terminology associated with ‘blinding’ and explored the dictionary for words to capture the assignment process. After long deliberations, KFS, IC, DGA and DAG proposed ‘allocation concealment’. This implied a mechanism to prevent foreknowledge of upcoming assignments which avoid any reference to ‘blinding’. We first used the term ‘allocation concealment’ in 1994.¹⁴ Because the term was new, we explained our rationale for preferring ‘allocation concealment’ in detail:

The reduction of bias in trials depends crucially upon preventing foreknowledge of treatment assignment. Concealing assignments until the point of allocation prevents foreknowledge, but that process has sometimes been confusingly referred to as ‘randomization blinding’. This term, if used at all, has seldom been distinguished clearly from other forms of blinding (masking) and is unsatisfactory for at least three reasons. First, the rationale for generating comparison groups at random, including the steps taken to conceal the assignment schedule, is to eliminate selection bias. By contrast, other forms of blinding, used after the assignment of treatments, serve primarily to reduce ascertainment bias. Second, from a practical standpoint, concealing treatment assignment up to the point of allocation is always possible, regardless of the study topic, whereas blinding after allocation is not attainable in many instances, such as in trials conducted to compare surgical and medical treatments. Third, control of selection bias pertains to the trial as a whole, and thus to all outcomes being compared, whereas control of ascertainment bias may be accomplished successfully for some outcomes, but not for others. Thus, concealment up to the point of allocation of treatment and blinding after that point address different sources of bias and differ in their practicability. In light of those considerations, we refer to the former as ‘allocation concealment’ and reserve the term ‘blinding’ for measures taken to conceal group identity after allocation.¹⁴

We deemed the following approaches as adequate allocation concealment: ‘central randomisation (e.g. by telephone to a trials office); a pharmacy (drugs or study products prepared by the pharmacy); numbered or coded containers and sequentially numbered, opaque, sealed envelopes’.¹⁴ We considered non-random, transparent (often called ‘systematic’) approaches, such as ‘alternate assignment and assignment by odd/even birthdate or hospital number’ as inadequate allocation concealment.¹⁴ We deemed as ‘unclear allocation concealment’ those approaches in which authors had not reported any allocation approach or reported an approach that was not captured by the ‘adequate’ or ‘inadequate’ concealment categories. Of 206 trial articles, only 23% used adequate allocation concealment, while 48% had not described any mechanism to allocate treatments.^14,26

Showing that ‘allocation concealment’ matters

Having defined allocation concealment, our next line of inquiry was to investigate whether this construct, as implemented in trials, was associated with indicators of bias. Prior work by Chalmers et al.¹⁹ had shown that ‘unblinded randomisation’ was associated with larger estimates of effect compared to ‘blinded randomisation’. However, as noted above, their study had several weaknesses.

We addressed those weaknesses by using multiple logistic regression statistical models to analyse 250 trials in 33 meta-analyses published in the Cochrane Pregnancy and Childbirth Database.²⁷ To address confounding by treatment we examined the association between ‘allocation concealment status’ and estimates of treatment effects within the same treatments across the 33 meta-analyses. To address confounding from other trial characteristics, we controlled for the method used to generate the allocation sequence, exclusions after allocation and blinding of outcome assessments. Lastly, we analysed trials with unclear allocation concealment (largely those trials for which the published reports had not provided any description of allocation concealment).

Using the substantial amount of relevant data generated by the Cochrane Pregnancy and Childbirth Group,²⁸ using software that had become available to facilitate our approach, we were able to address weaknesses in the analyses reported by Chalmers et al.¹⁹ more than a decade earlier. Our more detailed analyses showed that allocation concealment was indeed associated with trial results: on average, estimates of treatment effects were 41% larger in trials with inadequate allocation concealment and 33% larger in those with unclear allocation concealment compared to trials with adequate allocation concealment.²⁷ Moreover, as a further indication of likely bias, we found that the results of trials with reports of inadequate allocation concealment were more heterogeneous than those with adequate allocation concealment.²⁷ Our study of randomised trials in pregnancy and childbirth provided empirical evidence that poor allocation concealment was likely associated with bias. We were pleased that the importance of our study was recognised explicitly by others (Figure 1).

Figure 1.

Plaque recording the 1996 CDC and ATSDR Statistical Science Award presented to Schulz, Chalmers, Hayes and Altman for the Best Applied Paper.

Moher et al.²⁹ replicated our study in additional medical specialties and found similar results. Repeatedly, evidence was found that allocation concealment mattered. This empirical evidence gave substantial impetus to calls for increased attention to improved reporting, especially on critical items such as allocation concealment. Others had called for improved reporting, but this research provided tangible evidence that indications of poor methods were indeed likely associated with bias. As Drummond Rennie wrote in a 1995 editorial in JAMA commenting on the empirical evidence of bias in our paper, ‘These admonitions are not new; what is new is the demonstration of the consequences of their neglect’.³⁰

Establishment and adoption of allocation concealment terminology

Having created the term and validated the concept of ‘allocation concealment’, launching the terminology extended beyond journal publications. We introduced the terminology and concepts into various organisations and groups. Indeed, scientific overlap among all of us aided the process, initiated the Cochrane Centre, which spawned the Cochrane Collaboration. Iain Chalmers enlisted Schulz as the first visiting research fellow at the Cochrane Centre, where they collaborated with Grimes, Altman and Hayes. Altman and Schulz led the initial development of the Cochrane Statistical Methods Group.

Independently, Moher was organising a meeting to discuss evaluation of the quality of randomised trials, and he invited Chalmers to attend. However, because our analysis had not been published at the time, Chalmers suggested that Schulz should attend and promote adoption of the term ‘allocation concealment’ and present the evidence that bias is likely to be associated with inadequate allocation concealment. Moher’s meeting resulted in a reporting guideline entitled Standards Of Reporting Trials (SORT).³¹ Following a further meeting to which Moher and Schulz contributed, SORT morphed into another trial reporting guideline – the Consolidated Standards Of Reporting Trials (CONSORT) Statement.³² This in turn blossomed into a worldwide initiative to improve the reporting of medicial research – the EQUATOR Network.³³

Schulz³⁴ promoted appreciation of the importance of allocation concealment with personal accounts of problems he had encountered with assignment schemes and suggested methods for improving allocation concealment. Altman joined Moher and Schulz as members of the CONSORT Executive and coauthors of succeeding CONSORT Statements,^35–37 and Grimes became part of the CONSORT Group. We published a paper setting out the rationale for allocation concealment and providing examples of good reporting.³⁸ And in his book on cluster randomised trials, Hayes extended the need for allocation concealment in those trials.³⁹ The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guidelines for the content of protocols for randomised trials^40,41 drew heavily on the CONSORT Statement.

A PubMed search for the term ‘allocation concealment’ in ‘any field’ during the 22 years before our 1994 paper in JAMA yielded no citations, compared with 1471 citations over the 22 years between 1995 and 2016. The annual number of citations climbed steadily before plateauing around 2012 (Figure 2).

Figure 2.

Number of ‘allocation concealment’ citations retrieved using PubMed, by year, 1994 to 2016.

Using Google Scholar to search for ‘allocation concealment’ anywhere in articles published during the 22 years before our 1994 JAMA paper yielded 25 matches, most of which were mistakes. Thirty thousand matches were retrieved from articles published after 1994, and the annual number of matches has increased through 2016 (Figure 3).

Figure 3.

Number of ‘allocation concealment’ citations (anywhere in article) retrieved using Google Scholar, by year, 1994 to 2016.

Concluding reflections

The term ‘allocation concealment’ has been widely adopted by authors and editors. We have less evidence that rigorous definitions are being used for allocation concealment. Although the Cochrane Collaboration uses highly precise definitions,⁴² we suspect that many authors and editors define allocation concealment imprecisely, similar to their imprecise use of other trial terminology.¹¹ Furthermore, ‘allocation concealment’ could refer to blinding of outcome assessors. For example, Cox⁶ noted in 1958 that ‘The final stage in which concealment may be advisable is in the making of the observation itself’. Although the term ‘allocation concealment’ might still be improved to avoid occasional misconceptions about its meaning (for example, extending it to ‘allocation schedule concealment’), we assume that it has been widely adopted by authors and editors because they find the two-word term useful.

Our modelling and methodological approach²⁷ to examine the associations between dimensions of methodological quality and estimates of treatment effects has gained recognition and has led to confirmatory replications. As stated by two prominent methodological researchers:⁴³

In more recent years, the debate has shifted from anecdotal evidence of bias in single trials to more sophisticated ‘meta-epidemiological’ research, based on many trials and meta-analyses.⁴⁴ Schulz and colleagues²⁷ pioneered this approach when they assessed the methodological quality of 250 trials from 33 meta-analyses from the Cochrane Pregnancy and Childbirth Database and examined the association between dimensions of trial quality and estimated treatment effects.⁴³

This general approach has even been used by Savovic et al.⁴⁵ to incorporate multiple meta-epidemiological studies in an analysis. Although these investigations of associations of allocation concealment extended to other subject areas have yielded the same directions of association as our 1995 analysis,²⁷ the strength of the associations has varied.⁴⁵

Our research on empirical evidence of bias related to allocation concealment provided impetus for the initial development of the SORT and CONSORT reporting guidelines for randomised trials,^{31,32,35–37} and the adoption of CONSORT has been associated with improved reporting of randomised trials.^46,47 We believe that heightened attention to the process of allocation concealment has also improved the conduct of randomised trials. Indeed, compared with randomised trials published before 1990 those published between 2006 and 2012 were more likely to have reported adequate allocation concealment.^48,49 However, much room for improvement remains.^25,50

Our identification and validation of allocation concealment represents one step towards better conduct and reporting of randomised trials. Further steps must include expanded and enhanced medical research training in design, conduct and reporting of trials.

Footnotes

Declarations

References

Chalmers I. Why the 1948 MRC trial of streptomycin used treatment allocation based on random numbers. JLL Bulletin: Commentaries on the history of treatment evaluation. See www.jameslindlibrary.org/articles/why-the-1948-mrc-trial-of-streptomycin-used-treatment-allocation-based-on-random-numbers/ (2010, last checked 28 April 2018).

Medical Research Council. Clinical trial of patulin in the common cold. Lancet 1944; 2: 373–375.

Clarke M. The 1944 patulin trial of the British Medical Research Council: an example of how concerted common purpose can get reliable answers to important questions very quickly. JLL Bulletin: Commentaries on the history of treatment evaluation. See www.jameslindlibrary.org/articles/the-1944-patulin-trial-of-the-british-medical-research-council-an-example-of-how-concerted-common-purpose-can-get-reliable-answers-to-important-questions-very-quickly/ (2004, last checked 28 April 2018).

Medical Research Council. Streptomycin treatment of pulmonary tuberculosis: a Medical Research Council investigation. BMJ 1948; 2: 769–782.

Chalmers

Statistical theory was not the reason that randomisation was used in the British Medical Research Council’s clinical trial of streptomycin for pulmonary tuberculosis. In: Jorland

Opinel

Weisz

(eds). Body Counts: Medical Quantification in Historical and Sociological Perspectives, Montreal: McGill-Queens University Press, 2005, pp. 309–334.

Cox

. Planning of Experiments, New York: Wiley, 1958.

Armitage

The construction of comparable groups. In: Hill

(eds). Controlled Clinical Trials, Oxford: Oxford University Press, 1960, pp. 14–18.

Grimes

. The nomogram epidemic: resurgence of a medical relic. Ann Intern Med 2008; 149: 273–275.

Grimes

. “Case-control” confusion: mislabeled reports in obstetrics and gynecology journals. Obstet Gynecol 2009; 114: 1284–1286.

10.

Conti

. Confusing and sometimes meaningless terms used in cardiovascular medicine. Clin Cardiol 1994; 17: 53–54.

11.

Devereaux

Manns

Ghali

Quan

Lacchetti

Montori

et al.

Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001; 285: 2000–2003.

12.

Schulz

Chalmers

Altman

. The landscape and lexicon of blinding in randomized trials. Ann Intern Med 2002; 136: 254–259.

13.

Schulz

Grimes

. Blinding in randomised trials: hiding who got what. Lancet 2002; 359: 696–700.

14.

Schulz

Chalmers

Grimes

Altman

. Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 1994; 272: 125–128.

15.

Friedman

Furberg

Demets

. Fundamentals of Clinical Trials, Littleton, MA: PSG Publishing, 1985.

16.

Hill

. Statistical Methods in Clinical and Preventive Medicine, Edinburgh: E&S Livingstone, 1962.

17.

Peto

. Why do we need systematic overviews of randomized trials? Stat Med 1987; 6: 233–240.

18.

Chalmers

Smith

Blackburn

Silverman

Schroeder

Reitman

et al.

A method for assessing the quality of a randomized control trial. Control Clin Trials 1981; 2: 31–49.

19.

Chalmers

Celano

Sacks

Smith

. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983; 309: 1358–1361.

20.

Gillman

Runyan

. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1984; 310: 1610–1612.

21.

Mosteller

Gilbert

McPeek

. Reporting standards and research strategies for controlled trials: agenda for the editor. Control Clin Trials 1980; 1: 37–58.

22.

Dersimonian

Charette

Mcpeek

Mosteller

. Reporting on methods in clinical trials. N Engl J Med 1982; 306: 1332–1337.

23.

Altman

Doré

. Baseline comparisons in clinical trials. Lancet 1990; 335: 149–153.

24.

Tyson

Furzan

Reisch

Mize

. An evaluation of the quality of therapeutic studies in perinatal medicine. J Pediatr 1983; 102: 10–13.

25.

Hopewell

Dutton

Chan

Altman

. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 2010; 340: c723–c723.

26.

Schulz KF, Chalmers I, Altman DG, Grimes DA and Dore CJ. The methodologic quality of randomization as assessed from reports of trials in specialist and general medical journals. Online J Curr Clin Trials 1995; 4 (197).

27.

Schulz

Chalmers

Hayes

Altman

. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–412.

28.

Chalmers I, ed. The Oxford Database of Perinatal Trials. Oxford: Oxford University Press, 1988.

29.

Moher

Pham

Jones

Cook

Jadad

Moher

et al.

Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?

Lancet 1998; 352: 609–613.

30.

Rennie

. Reporting randomized controlled trials. An experiment and a call for responses from readers. JAMA 1995; 273: 1054–1055.

31.

The Standards Of Reporting Trials Group. A proposal for structured reporting of randomized controlled trials. JAMA 1994; 272: 1926–1931.

32.

Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D and Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT Statement. JAMA 1996; 276: 637–639.

33.

Altman DG and Simera I. A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. JLL Bulletin: Commentaries on the history of treatment evaluation. See www.jameslindlibrary.org/articles/a-history-of-the-evolution-of-guidelines-for-reporting-medical-research-the-long-road-to-the-equator-network/ (2015, last checked 28 April 2018).

34.

Schulz

. Subverting randomization in controlled trials. JAMA 1995; 274: 1456–1458.

35.

Moher

Schulz

Altman

. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357: 1191–1194.

36.

Schulz

Altman

Moher

. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c332–c332.

37.

Schulz

Altman

Moher

Fergusson

. CONSORT 2010 changes and testing blindness in RCTs. Lancet 2010; 375: 1144–1146.

38.

Altman

Schulz

Moher

Egger

Davidoff

Elbourne

et al.

The Revised CONSORT Statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 663–694.

39.

Hayes

Moulton

. Cluster Randomised Trials, Second Edition. Boca Raton: Chapman and Hall, 2017.

40.

Chan

Tetzlaff

Altman

Laupacis

Gotzsche

Krleza-Jeric

et al.

SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med 2013; 158: 200–207.

41.

Chan

Tetzlaff

Gotzsche

Altman

Mann

Berlin

et al.

SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 2013; 346: e7586–e7586.

42.

Higgins

Altman

Gotzsche

Juni

Moher

Oxman

et al.

The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343: d5928–d5928.

43.

Juni

Egger

. Commentary: Empirical evidence of attrition bias in clinical trials. Int J Epidemiol 2005; 34: 87–88.

44.

Sterne

Juni

Schulz

Altman

Bartlett

Egger

. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 2002; 21: 1513–1524.

45.

Savovic

Jones

Altman

Harris

Juni

Pildal

et al.

Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012; 157: 429–438.

46.

Turner

Shamseer

Altman

Schulz

Moher

. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev 2012; 1: 60–60.

47.

Turner

Shamseer

Altman

Weeks

Peters

Kober

et al.

Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012; 11: MR000030–MR000030.

48.

Peters

JPM

Stegeman

Grolman

Hooft

. The risk of bias in randomized controlled trials in otorhinolaryngology: hardly any improvement since 1950. BMC Ear Nose Throat Disord 2017; 17: 3–3.

49.

Reveiz

Chapman

Asial

Munoz

Bonfill

Alonso-Coello

. Risk of bias of randomized trials over time. J Clin Epidemiol 2015; 68: 1036–1045.

50.

Chan

Altman

. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005; 365: 1159–1162.