Envelope use and reporting in randomised controlled trials: A guide for researchers

Abstract

Introduction

To produce robust evidence RCTs need to be rigorously conducted as poorly performed studies introduce bias and can mislead clinicians and policy makers. Poor allocation concealment has the largest single impact on bias in RCTs than other methodological aspects. Envelopes are frequently used as a method of allocation concealment and can be associated with increased risk of bias. This paper aims to review envelope use in RCTs published in 2017–2018 and create a guide as a reference for researchers when planning and publishing RCTs when using envelopes as an allocation concealment method.

Methods

RCTs that used envelopes as a form of allocation concealment that were published in BMJ, JAMA, NEJM and The Lancet in 2017 and 2018 were identified and methodological data on their envelope use extracted and authors were contacted to ascertain reasons for using envelopes in their research.

Results

338 RCTs were identified that were published in 2017 and 2018. 8% (n = 29) of the RCTs published used envelopes as an allocation concealment method. 24.1% (n = 7) of studies reported envelope studies robustly with all required methodological information stated to enable an assessment of quality. Budget was the most frequent reason given for envelope use (41.7%).

Discussion

Only 24% of published RCTs, that used envelopes, contained robust methodological information to enable the reader to judge whether the randomisation and allocation concealment method was adequate.

Conclusion

RCTs are not reporting envelope use well. RCTs using envelopes should be designed and reported clearly ensuring all necessary methodological information is included.

Keywords

Validity reliability bias evidence-based medicine methods and methodology planning the research designing a randomised blinded trial randomised trials clinical trials meta-analysis

Introduction

Randomised Controlled Trials (RCTs) are considered to be the gold standard in assessing the effectiveness of interventions. To produce robust evidence RCTs need to be rigorously conducted as poorly performed studies introduce bias and can mislead clinicians and policy makers. Probably the single design element associated with biased findings in trials is poor or absent allocation concealment.^1,2

Allocation concealment

Allocation concealment is defined as the method used to conceal the randomisation sequence from all study personnel until after the patient has been recruited into the study. This stops the randomisation sequence being subverted and the study having a high risk of bias. It has been shown that having an inadequate allocation concealment method can exaggerate the effect size by 41%.^1,2 There are multiple ways that the randomisation sequence can be concealed, such as web-based or telephone systems. Traditionally, before web and telephone systems were available envelopes were used. The use of sealed envelopes as a method still lingers on as a concealment method for a significant proportion of RCTs. For instance, Yelland et al found in 2015 that 9% of RCTs employed sealed envelopes as a method of concealment.³

Advantages and disadvantages of envelope use as a method of allocation concealment

There are significant disadvantages to using envelopes for allocation concealment. They can be opened in advance for example,⁴ trans illumination can determine the allocation^5–8 such methods allow subversion of the randomisation. On the other hand they are relatively cheap and logistically practical in remote areas that are internet or telephone free or in emergency medicine situations.

In this paper we aim to describe the types of trials that continue to use envelopes and the quality of the envelope concealment used and to provide advice on their safer usage.

Methods

RCTs published in BMJ, JAMA, NEJM and The Lancet in 2017 and 2018 were identified. Two reviewers extracted data from each paper on the randomisation and allocation concealment methods. Those RCTs that used envelopes to conceal the randomisation sequence were identified.

Envelope concealment

We used the approach described by Doig and Simpson⁹ to define high quality envelope concealment. There are three areas that were assessed as follows:

If the person who created the envelope was stated

Best practice for the use of envelopes in RCTs would be that a randomisation sequence would be generated and personal not involved in the RCT would create the envelopes for the RCT.

2. Whether the envelopes had an additional security measure.

Envelopes should have an additional security measure rather than just being closed and be opaque and sequentially numbered. This order can then be checked and anomalies will be identified if the randomisation sequence has been violated. Other additional security measures include the person who has created the envelope signing the back of the envelope when sealed so it is obvious if it has been tampered with. Inserting foil and or carbon paper into the envelope prevents trans-illumination and the carbon paper allows an additional audit trail as the participants name and date of recruitment can be written on the envelope at the point of recruitment before the envelope has been opened and the carbon paper prints this information to the allocation insert. The envelopes should be kept securely and not with the research team who are responsible for recruiting participants into the study.

3. If the person who opened the envelope (recruited participants) was stated

The person who created the envelope should not be the same person who recruits participants to prevent the ordering of participants into one treatment arm or another.

We identified these and other quality factors and extracted them from each RCT that used envelopes: the envelope description, whether who created the envelope and who opened the envelope was stated. We emailed each corresponding author of the RCTs that used envelopes as an allocation concealment method and inquired as to why they have chosen envelopes as an allocation concealment method.

Results

A total of 338 RCTs were identified that were published in 2017 and 2018. 7.5% of RCTs published in 2017 used envelopes and 9.5% in 2018. Combined, 8.6% (n = 29) of the RCTs published in 2017 and 2018 used envelopes as an allocation concealment method. We emailed each author of the RCTs using envelopes and received responses from 12 (41%) of them. Figure 1 shows the flow of studies.

Figure 1.

Flow of trials through study.

Table 1 shows the description from the paper of each envelope trial with the necessary quality factors and trial setting. 24.1% (n = 7) of studies that use envelopes for allocation concealment, reported envelope use robustly with all required methodological information stated to enable an assessment of quality. 44.8% (n = 13) reported who created the envelope, 44.8% (n = 13) reported who opened the envelope and 62.1% (n = 18) reported the envelope description adequately.

Table 1.

Quality factors and setting reported in included RCTs.

Author	Envelope description	Additional security measures stated on envelope	Person who created envelope stated	Who opened envelope stated	Envelope descriptions and methods adequately reported	Setting
Andrews	Sealed opaque envelopes	Non reported	No	No	No	Emergency department
Smits	Closed, opaque envelopes	Non reported	No	No	No	Hospital
Landoni	sealed, opaque, sequentially numbered envelopes	Sequentially numbered	No	No	No	Operating theatre or ICU.
Dwivedi	sequentially numbered, sealed, opaque envelopes	Sequentially numbered	By persons not involved in the trial	No	No	Referral center for epilepsy surgery in northern India, New Delhi
Kulkarni	sealed, opaque envelopes,	Non reported	no	No – ‘opened in the preoperative holding area just before the patient entered the operating room’	No	Operating theatre in Uganda
Kaufman	Opaque envelopes concealing the allocation, within sealed individual study packs.	Study packs were kept available from a locked study box from which they could only be taken sequentially.	Yes: independent statistician	No	No	A tertiary paediatric emergency department (Melbourne)
Patel	sealed in sequential numbered opaque envelopes	sequential numbered	Independent support staff	trained health assistants	Yes	Primary health centres in Goa
Nadkarni	sequential numbered opaque sealed envelopes	Sequential numbered	No	trained health assistants based	No	Primary health centres in Goa
Brockman	sequentially assigned sealed randomisation envelopes,	Sequentially assigned	Person independent of the research team and who had no further role in the trial	Envelopes were opened immediately before induction of anaesthesia by the attending anaesthetist	Yes	Department of Anaesthesia and Pain Management, Perth
Chan	sealed opaque envelopes	Non reported	independent staff member assigned the treatments	no	No	Prince of Wales Hospital of The Chinese University of Hong Kong
Mundle	sequentially numbered, sealed, opaque envelope	sequentially numbered	The envelopes were generated by staff at Gynuity Health Projects	Research staff	Yes	Two public hospitals in India
Stocker^a	sequentially numbered sealed opaque envelopes.	Sequentially numbered	no	no	No	Hospital - Neonates born >34 weeks who had suspected early-onset sepsis in the first 72 h of life and who required antibiotic therapy
Molloy	sequentially drawing sealed envelopes	Sequential	No	Yes: trial pharmacist and clinician	No	9 African hospitals
Franklin	Sequentially numbered, sealed, opaque envelopes	Sequentially numbered, sealed, opaque envelopes	No	No	No	Emergency departments and general pediatric inpatient units in 17 tertiary and regional hospitals in Australia and New Zealand
Boden	sequentially numbered sealed opaque envelopes containing allocation cards wrapped in aluminium foil.	Sequentially numbered, sealed opaque envelopes. Foil used to wrap allocation cards. Patient details were marked on envelopes to record that randomisation was in order of recruitment.	Yes: independent administrator	Yes: physiotherapist	Yes	Multidisciplinary preadmission clinics at three tertiary public hospitals in Australia and New Zealand.
Firanescu	sealed randomisation envelope	Non reported	No	No	No	Four community hospitals in the Netherlands
Mason	opaque, sequentially numbered, sealed envelopes	Opaque, sequentially numbered	No	No	No	Recruitment from workplaces, social media platforms, and schools in Birmingham UK
Salimen	opaque, sealed, and sequentially numbered randomization envelopes were shuffled and then distributed to each participating hospital	opaque, sealed, and sequentially numbered randomization envelopes	No	Yes: surgeon	No	Three hospitals in Finland
Peterli	sealed envelopes	Non reported	No	No	No	Four bariatric centres in Switzerland
Jabre	sealed envelopes	Non reported	No – implied that it is not the same person who opened the envelope	No	No	20 prehospital emergency medical services (EMS) centers: 15 in France and 5 in Belgium.
Labhart	sealed, sequentially numbered, opaque envelopes.	Sealed Sequentially numbered	Yes: ‘a separate person not involved in the study’	Yes: study nurse	Yes	Six health care facilities in northern Lesotho.
Huttner	Opaque sealed envelopes	Non reported	No	No	No	hospital units and outpatient clinics in Switzerland, Poland and Israel.
Driver	sequentially numbered, opaque envelopes	sequentially numbered, opaque envelopes	No	Yes: research associate	No	Emergency department at Hennepin County Medical Center, anurban,
Fossat	sealed, opaque, and numbered envelopes	sealed, opaque, and numbered envelopes	Yes: a clinical research assistant	Yes: investigator	Yes	ICU (hospital) in France
Cooper	Sealed opaque envelopes and permuted variable block sizes (2 and 4).	Sealed opaque envelopes and permuted variable block sizes (2 and 4).	No	No	No	Patients both out-of-hospital and in emergency departments in Australia, New Zealand, France, Switzerland, Saudi Arabia, and Qatar
Montaigne	The code sequence was computer generated and kept in sealed envelopes at a central location by non-medical staff not involved in the study.	Non reported	No	Yes: staff cardiologists	No	Hospital in France
Farquhar	Allocations were concealed in sequentially numbered, sealed, opaque envelopes	sequentially numbered, sealed, opaque envelopes	Yes: an independent statistician	Yes: study coordinator	Yes	Fertility clinics in New Zealand
Heinemann	Each study site received sealed envelopes with the respective group allocation. After successful completion of the baseline phase, the respective envelope was opened. .	Non reported	Not explicit but states ‘Randomisation was done centrally at the study coordinating centre by staff who were not involved with recruitment or treatment of study participants’	No	No	Diabetes practices in Germany
Blumberger	The randomisation tables were used by staff outside the study team to produce opaque, sealed envelopes, labelled with a participant specific randomisation identification number and containing a treatment allocation	Non reported	Yes: staff outside the study team	Yes: study staff	No	Three hospitals in Germany

^aTwo types of randomisation – SNOSE and drawing cards at random in different centres.

Table 2 shows the author stated reasons for envelope use, it was found that the most frequent reason given was budget (41.7%).

Table 2.

Author stated reasons for envelope use.

	Author stated reasons for envelope use
Author	Budget	Can randomise quickly	Removes issues of equipment malfunction	Maintain optimal blindness	Geographical Location	Unreliable internet and phone signal	Pilot of SMS service highlighted error, SNOSE chosen instead	Rapid randomisation needed due to setting and intervention^a	Size of study	No telephone or online system available at start of study recruitment	Ease of use	Setting	Practical	Pragmatic within study design
Bellomo	●	●	●	●
Patel and Nadkarni							●
Weeks					●	●
van Herk								●	●
Salminen	●									●	●
Jabre								●				●
Huttner	●												●
Driver								●
Farquhar	●
Heinemann													●	●
Blumberger	●

^aOne study stated within 1-2 minutes and another <12 hour.

Discussion

Envelope use is similar to 2015 where Yelland found, in a similar group of journals, that 9% of RCTs used envelopes as a form of allocation concealment [2]. Reasons for envelope use were all appropriate for the trial design and setting that the research was being conducted.

It was disappointing that only 24% of the published RCTs in this sample contained robust methodological information to enable the reader to judge the RCT as adequate and low risk of bias when assessing the randomisation and allocation concealment methods. This therefore means that 76% of RCTs cannot enter into systematic reviews with a low risk of bias, there will be a higher level of uncertainty of the validity of the systematic review. All resources used to perform this research is wasted as the published report is not clear.

Many envelopes were not described as having any additional security measures. They were simply stated as being ‘sealed’, they may have been sequentially numbered however this was not stated so it cannot be assumed. Only one study (Boden – see Appendix 1 for a list of all included studies) stated that they used very secure envelopes with the addition of foil to wrap the allocation cards within the envelopes. Foil prevents the trans-illumination of the envelopes and further protects an RCT from subversion.^1,9 It has been found that RCTs employing the use of envelopes without additional security measures are associated with an exaggerated effect size.¹⁰ Results from insecure envelopes will be treated with caution by policy makers – thus highlighting the important of using secure envelopes and ensuring that if secure envelopes were used the details are reported comprehensively to enable policy makers to have confidence in the reported results. The use of foil with sequentially numbered sealed opaque envelopes in the opinion of the authors of this paper should be the gold standard way to set up an envelope for use within an RCT.

One point of interest is there are two RCTs in this sample that describe the same methodology from the large scale RCT but are reporting different results. Interestingly one study was deemed methodologically robust (Patel) and one was not (Nadkarni) as they did not report who created the envelope. This small omission has resulted in a study not being classed as having robust methodology.

There are a variety of valid reasons why envelopes are used within RCTs. We would urge researchers to carefully consider their research budget and assess whether they should allocate additional funds to cover the cost of a more robust and secure randomisation and allocation concealment method if their research design allows. Envelopes are inexpensive but if not executed and published robustly the entire research cannot contribute meaningfully to the evidence base.

After analysing the data gained in this research we can see that there is still a two-fold issue with envelope use. Envelopes are not being prepared in a rigorous manner with additional security measures and they are not reported in a transparent robust way ensuring all methodological information is provided. There is unclear information given to ascertain whether the envelope had additional security measures and whether the person creating the envelope is separate to the person who opens the envelope at the point of recruitment.

Future recommendations

Figure 2 shows the recommendation we have to the following when performing research with envelopes as an allocation concealment method.

Figure 2.

Flow diagram stating methodological steps to creating a robust envelope as an allocation concealment method in an RCT and essential methodological information to be reported in the publication of an RCT.

It is also pertinent to discuss the evolving nature of technology and allocation concealment methods. For rapid randomisation envelopes are a sensible choice, there are however apps being created that can rapidly randomise participants. These are also a relatively inexpensive method of randomisation and allocation concealment and may be used more widely in the future. Even with the use of innovative technology methodology will still need to be published thoroughly for methodological quality judgements to be made.

Moving forward we would urge authors to plan for and create secure envelopes when using envelopes as a form of allocation concealment are the only option for their RCT and to write their research transparently to include all the methodological information stated in Figure 2. Journals should ensure that any RCT published that uses envelopes as a form of allocation concealment should be reported robustly.

Conclusions

Allocation concealment methods are one of the most influential methodological factors on the validity of an RCT. Envelopes can be used as a robust method of allocation concealment. However, they are the most insecure method associated with subverting an RCT. If they are used within a research design they should be created robustly and reported clearly ensuring all necessary methodological information is included.

Supplemental Material

sj-pdf-1-rmm-10.1177_2632084320957204 - Supplemental material for Envelope use and reporting in randomised controlled trials: A guide for researchers

Supplemental material, sj-pdf-1-rmm-10.1177_2632084320957204 for Envelope use and reporting in randomised controlled trials: A guide for researchers by Laura Clark, Alexandra Dean, Alex Mitchell and David J Torgerson in Research Methods in Medicine & Health Sciences

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Laura Clark

Alexandra Dean

Alex Mitchell

David J Torgerson

References

Swingler GH and Zwarenstein M. An effectiveness trial of a diagnostic test in a busy outpatients department in a developing country Issues around allocation concealment and envelope randomization. Journal of clinical epidemiologyt 53(7): 702–706.

Schulz

Chalmers

Hayes

, et al. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–412.

Yelland

Kahan

Dent

, et al. Prevalence and reporting of recruitment, randomisation and treatment errors in clinical trials: a systematic review. Clin Trials 2018; 15: 278–285.

Kennedy

Torgerson

Campbell

, et al. Subversion of allocation concealment in a randomised controlled trial: a historical case study. Trials 2017; 18: 204.

Schulz

Unbiased research and the human spirit: the challenges of randomized controlled trials. CMAJ 1995; 153: 783–786.

Schulz

KF.

Subverting randomization in controlled trials. JAMA 1995; 274: 1456–1458.

Schulz

KF.

Randomised trials, human nature, and reporting guidelines. Lancet 1996; 348: 596–598.

Schulz

Chalmers

Hayes

, et al. Empirical evidence of bias. JAMA 1995; 273: 408.

Doig

Simpson

Randomization and allocation concealment: a practical guide for researchers. J Crit Care 2005; 20: 187–191.

10.

Herbison

Hay-Smith

Gillespie

WJ.

Different methods of allocation to groups in randomized trials are associated with different levels of bias. A meta-epidemiological study. J Clin Epidemiol 2011; 64: 1070–1075.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB