When randomization is not random: Allocation bias in small sample,group sequential randomized clinical trials

Abstract

Even in rare diseases, where the sample size is limited and blinding is less frequently implemented, randomized controlled trials are considered the gold standard to prove efficacy. Randomization is used to mitigate bias and regulatory guidance recommend the investigation of the impact of bias on the test decision. We quantified how allocation bias affects the test decision in small-sample two-arm group sequential trials under a biasing policy based on the Blackwell–Hodges convergence strategy. Type I error and power were evaluated under Lan–DeMets spending (Pocock-, O’Brien–Fleming-, Wang–Tsiatis-type functions), with and without futility (non-binding, binding), varying interim timing, number of looks and stage-wise restarting of randomization. Allocation bias inflated type I error most for more restrictive randomization procedures, especially permuted blocks with small block sizes. Spending more alpha at interim reduced inflation. Non-binding futility reduced type I error, while binding increased type I error inflation for more aggressive stopping boundaries. Stage-wise restarting modestly reduced inflation for most procedures. Overall, group sequential choices had secondary effect and did not rescue a predictable randomization scheme. When allocation bias cannot be ruled out (e.g. open-label trials), we recommend less restrictive randomization procedures (e.g. big stick design) or, if using permuted blocks, large block sizes.

Keywords

Selection bias interim analyses Lan and DeMets O’Brien and Fleming Pocock restricted randomization type I error convergence strategy

1. Introduction

Randomized controlled trials (RCTs) are the gold standard in clinical trials, primarily for their unparalleled effectiveness in minimizing biases. Despite safeguards in RCTs such as randomization, concealment, and blinding, bias can still affect their results. The US Food and Drug Administration (FDA) and European Medicines Agency caution that any clinical trial may be subject to systematic biases and urge active steps to reduce them, particularly when blinding is not feasible.^1,2 One such threat is allocation bias, which is referred to as selection bias in earlier research papers.³ It can be classified as a subtype of selection bias, due to predictable randomization sequences.⁴ For example, with permuted block randomization (PBR) with block size $4$ , knowledge of the first three assignments within a block determines the fourth with certainty. Even without knowledge of the randomization scheme, investigators can raise the chance of correctly predicting the next assignment above $50 %$ when restricted randomization procedures are used, with strategies such as the convergence strategy.³ Blinding helps to prevent guessing strategies, but is not always possible (e.g. surgery vs. drug intervention). Furthermore, treatment side effects or changes in vital parameters can still reveal prior assignments.⁵ A review of open-label trials found baseline imbalances between groups in trials using block randomization, which might be an indicator for allocation bias.⁶

Assessments of allocation bias are mostly descriptive, as in the RoB 2 tool,⁷ which rates the bias from domains including the randomization process, deviations from intended interventions, missing data, outcome measurement, and selective reporting via signaling questions. Metrics such as the proportion of correct treatment guesses quantify predictability of a randomization procedure (RP),^3,8 but they do not show how much this predictability distorts the test decision. The impact of such bias depends on the trial design,^9–11 but group sequential designs (GSDs) have not been systematically evaluated.

GSDs are appealing in rare diseases, where sample sizes are small and efficient use of patient data is critical.^12,13 These trials are more often open label than conventional trials,¹⁴ increasing the risk of allocation bias. Classic formulations of GSDs assume exact 1:1 allocation at each look.^15,16 In practice, the choice of RP can create interim or final imbalances that interact with stopping rules. RP choices that allow imbalance at interim analyses require corresponding choices of stopping boundaries to preserve type I error rates (T1E). Because the Lan–DeMets (LDM) framework recalibrates boundaries using observed information and is flexible under deviations from the planned allocation ratio, we adopt LDM here.¹⁷ We focus on a set of representative RPs that span key properties: unrestricted allocation (complete randomization), terminal balance within blocks (permuted block randomization, with fixed and randomized block sizes and the random allocation rule (RAR)), bounded imbalance via a maximum-tolerated imbalance rule (big stick design (BSD)), probabilistic control around balance (Efron’s biased coin (EBC)), and the combination of bounded imbalance and probabilistic control (Chen’s procedure). We do not consider minimization or covariate-adaptive methods, which target different mechanisms and require modeling of prognostic covariates.

Our objective is to quantify how allocation bias affects T1E and power across representative RPs and how group sequential choices modify that effect. Section 2 defines the biasing policy, GSDs, RPs and the simulation setup. Section 3 is organized as follows: We begin with a fixed-sample design as a benchmark without interim analysis to show how allocation bias alone affects an one-sided $t$ -test. We then add a single interim look with futility stopping only. Next, we study efficacy stopping by varying the alpha spent during the interim analysis, the timing of the interim analysis and we evaluate power under allocation bias. Finally, we combine efficacy and futility stopping, presenting both non-binding and binding futility stopping and we assess stage-wise restarting of randomization process with different numbers of stages. Section 4 outlines practical challenges and safeguards to mitigate allocation bias and the conclusion summarizes the key takeaways. A list of abbreviations used throughout the article is provided at the end of the article. To our knowledge, this is the first systematic assessment of allocation bias in GSDs across representative classes of RPs.

2. Methods

We consider a two-arm randomized controlled clinical trial using GSDs to prove the superiority hypothesis with a continuous normal distributed endpoint $H_{0} : μ_{E} \leq μ_{C}$ against the alternative hypothesis $H_{1} : μ_{E} > μ_{C}$ using a one-sided $t$ -test at level $α$ with an unknown variance $σ^{2}$ . It is common to investigate one sided hypothesis with GSDs.^18,19

2.1. Biasing policy

Our biasing policy will be based on the following assumptions:

Patients can be categorized into groups with positive, neutral, and negative expected responses to the outcome.

The investigator has a preference to demonstrate that the experimental treatment is superior to the control and can selectively decline patient enrollment.

The investigator knows all past treatment allocations but has no knowledge of future assignments and does not know the RP (or its parameters, such as block size). He is aware only of the target allocation ratio.

In this setting, a natural choice is the Blackwell–Hodges convergence strategy, which guesses that the next allocation will go to the currently underrepresented arm. The investigator uses this rule to influence enrollment and admit a patient with a positive expected response when the experimental arm is predicted next and a patient with a negative expected response when the control arm is predicted next. When the arms are balanced he admits a patient expected to be neutral.

These assumptions model a clinical trial, where the inclusion and exclusion criteria involve clinical judgment, allowing the investigator to decline patient participation until a candidate favorable to him is identified and that the investigator can classify candidates as likely good, neutral, or bad responders, for example based on a holistic assessment of the patient’s clinical situation.

According to ICH E9,² the investigator should not be aware of the technical specifications of the RP used. However, when using this strategy, the guessing strategy would be the same independent of whether the investigator knows the RP or not.

Under these assumptions, the investigator can change the composition of patients in both arms by guessing the upcoming allocations. For example, under PBR with block size $4$ and a total sample size of $48$ , our simulations show the following expected composition for the experimental arm: 12 positive, 10 neutral, and 2 negative responders, while in the control arm: 12 negative, 10 neutral, and 2 positive responders (see the Supplemental Appendix). Suppose positive responders have a mean outcome that is $0.4$ higher than the population mean and negative responders a mean outcome that is $0.4$ lower than the population mean (which is about $50 %$ of the treatment effect for a trial with $80 %$ power), independent of treatment assignment. Then, under $H_{0}$ , the outcome distributions for a randomly selected participant from each arm are as shown in Figure 1, conditional on fixing the counts of positive, neutral, and negative responders at the above mentioned expected values for PBR( $4$ ).

Figure 1.

Illustration of how outcomes in a two-arm randomized controlled trial are affected using an allocation biasing policy for permuted block randomization with block size $4$ and a total sample size of $n = 48$ . Expected compositions (from simulation, see the Supplemental Appendix) are $12$ positive, $10$ neutral, and $2$ negative responders in the experimental arm, and $12$ negative, $10$ neutral, and $2$ positive responders in the control arm. Positive (negative) responders have a $+ 0.4$ ( $- 0.4$ ) better (worse) expected response under $H_{0}$ , independent of treatment allocation. Distributions shown are for a randomly selected participant, conditional on fixing counts of the patient type at these expected values.

We now present a mathematical formulation of this biasing policy. Let $n$ be the maximum number of patients and $K$ denote the total number of stages in the GSD. For example, $K = 2$ means that there is one pre-planned interim analysis and a final analysis. Let $Z_{i} = 1$ indicate an allocation to group $E$ and $Z_{i} = 0$ denote an allocation to group $C$ . The cumulative number of past assignments to groups $E$ and $C$ after $i - 1$ assignments is given by:

n_{(i - 1) E} = \sum_{j = 1}^{i - 1} Z_{j}, n_{(i - 1) C} = \sum_{j = 1}^{i - 1} (1 - Z_{j})

Let

η

represent the size of the bias introduced by the positive and negative responders in the patient population. Then the biasing policy can be expressed as:

τ_{i} = {\begin{cases} η, & n_{(i - 1) E} > n_{(i - 1) C} \\ 0, & n_{(i - 1) E} = n_{(i - 1) C} \\ - η, & n_{(i - 1) E} < n_{(i - 1) C} . \end{cases}

This can be equivalently expressed as:

τ_{i} = η (sign (n_{(i - 1) C} - n_{(i - 1) E}) .

To quantify the overall response from each patient, we use a continuous, normally distributed endpoint, denoted as

y_{i}

y_{i} = μ_{E} Z_{i} + μ_{C} (1 - Z_{i}) + τ_{i} + ε_{i},

where

ε_{i} \sim N (0, σ^{2})

1 \leq i \leq n

2.2. Design choices for group sequential trial

We consider LDM alpha spending with Pocock- and O’Brien–Fleming (OBF)-type functions,^19,20 because it updates critical values at each look using observed information fraction. This matters here since several RPs do not keep exact $1 : 1$ balance at interim analyses. To vary how much alpha is spent early, we also include the Wang–Tsiatis family, which has OBF and Pocock-like boundaries as special cases.²¹ Early stopping is allowed for efficacy and for futility. Futility is implemented in two ways, as non-binding, where crossing the futility boundary allows, but does not require stopping and leaves efficacy boundaries unchanged, and binding, where crossing forces stopping and the efficacy boundaries are recalculated to preserve the overall alpha level. While the FDA recommends the use of non-binding futility boundaries when determining efficacy boundaries in group sequential and adaptive designs,²² we also evaluate binding futility to examine how boundary recalculation interacts with allocation bias.

Analyses use a one-sided $t$ -test. For the $t$ -test we apply the quantile-substitution method,¹⁵ which transforms $z$ -scale boundaries into $p$ -values and then into $t$ -quantiles based on the stage-specific degrees of freedom.^20,23 Other methods have been proposed in the literature,^19,24,25 but would be computationally prohibitive for our study, as we need different boundaries for every randomization sequence.

All RPs target an overall 1:1 allocation but can be imbalanced at interim (and sometimes at the final) depending on their constraints. We study the following representatives:

Complete Randomization (CR): Randomization achieved by flipping a fair coin.

Random Allocation Rule (RAR): Randomization assigning the same proportion of patients to each treatment.

Permuted Block Randomization (PBR( $b$ )): Allocation in blocks of length $b$ , with randomization within each block according to RAR.

Randomized Permuted Block Randomization (RPBR( $b_{1}, b_{2}, \dots$ )): Permuted block randomization with block sizes for each block randomly selected from $b_{1}, b_{2}, \dots$ .

Efron’s biased coin (EBC( $p$ )): Randomization using a biased coin with probability $p$ in favor of the treatment with fewer allocations and a fair coin in case of equal allocations.

Big stick design (BSD( $a$ )): Complete randomization with deterministic assignment to the underrepresented treatment group when a maximum-tolerated imbalance $a$ is reached.

Chen’s design (CHEN( $p, a$ )): EBC( $p$ ) with deterministic assignment to the underrepresented treatment group when a maximum-tolerated imbalance $a$ is reached.

Note, here “RAR” denotes the random allocation rule, not response adaptive randomization. We classify all RPs that limit the number of possible randomization sequences as restricted RPs, that is, all except for CR. To isolate design effects, we evaluate the following scenarios in sequence:

Fixed-sample design (no interim analysis).

One interim look with futility only (non-binding).

Efficacy stopping:

With Wang–Tsiatis varying alpha spent during interim.

With LDM Pocock and LDM OBF varying timing of the interim look.

Power under allocation bias for LDM Pocock and LDM OBF.

Varying bias magnitude for LDM Pocock and LDM OBF.

Combined efficacy (LDM Pocock/LDM OBF) and futility stopping (non-binding and binding).

Stage-wise restart of randomization (new list in each stage) versus a single full list, varying the number of stages $K$ for LDM Pocock/LDM OBF when targeting $1 : 1$ by stage.

2.3. Simulation

For each RP, we generated 16,000 randomization sequences and simulated 1,000 clinical trials per sequence. Outcomes $y_{i}$ had variance $σ^{2} = 1$ . Critical values for group sequential testing were obtained with gsDesign²⁶ and rpact,²⁷ randomization sequences were generated with RandomizeR.²⁸ We focused on a small-sample setting typical of rare disease trials with open-label designs, fixing the maximum sample size at $n = 48$ . We focused on two levels of allocation bias, $η \in {0.04, 0.40}$ , corresponding to roughly $5 %$ and $50 %$ of the effect size in a trial powered at $80 %$ . For the biased-coin procedures we set $p = 0.67$ for both EBC and CHEN, which is common in practice. For BSD and for Chen’s maximum-tolerated imbalance we used $a = 3$ , following recommendations in the literature.²⁹ For PBR we used block size $b = 4$ , which is widely used in GSDs.¹⁷ As performance measures we report the mean T1E and power.

3. Results

3.1. Investigation of allocation bias strategy for a $t$ -test without interim analysis

As a benchmark, we first examined allocation bias in a fixed-sample trial (no interim analyses) using a one-sided $t$ -test with a total sample size of $n = 48$ . Figure 2 displays the T1E across varying bias levels $η$ . T1E increased monotonically with $η$ , with the largest inflation for more restrictive procedures. For positive bias, the ordering (highest inflation to lowest) was: PBR( $4$ ) > CHEN( $2 / 3, 3$ ) > EBC( $2 / 3$ ) > RAR $\geq$ BSD( $3$ ) > CR. Notably, even CR showed a small T1E inflation for the one-sided test, in contrast to the two-sided case, where CR does not inflate T1E.⁹ Conversely, under negative bias, T1E fell below the nominal one-sided $2.5 %$ level. Here, negative bias represents a systematic misclassification within our biasing policy, meaning that the investigator’s prognostic judgment used for selective enrollment is applied in the other direction and therefore works against the investigational treatment. This could arise, for example, when the investigator favors the standard-of-care arm or is skeptical about the new treatment in a comparison of two active treatments.

Figure 2.

Mean type I error rates under different randomization procedures in a trial with total sample size $n = 48$ , analyzed using a one-sided $t$ -test. The $y$ -axis is restricted to the range $[0, 0.20]$ to highlight the differences. The horizontal dashed lines highlight the values $η = 0.04$ and $η = 0.4$ , equivalent to about $5 %$ and $50 %$ of the treatment effect required for $80 %$ power, respectively.

In what follows, we focus on two representative bias levels: $η = 0.04$ (about $5 %$ of the treatment effect for a trial with $80 %$ power) and $η = 0.4$ (about $50 %$ of the treatment effect for a trial with $80 %$ power). The latter was included as the differences between RPs are more pronounced allowing for a better discrimination. These reference values are marked by the vertical dashed lines in Figure 2.

Figure 3.

Mean type I error rates under different randomization procedures in a group sequential trial with a maximum sample size of $n = 48$ and one non-binding futility analysis. The trial can stop for futility at interim if $p_{1} > α_{0}$ , where $p_{1}$ denotes the first-stage $p$ -value. A one-sided $t$ -test was used at the final analysis. Note, the $y$ -axis is restricted to $[0, 0.035]$ for the low-bias scenario and to $[0, 0.20]$ for the high-bias scenario to highlight the differences. The smallest $α_{0}$ is $α_{0} = 0.653$ , corresponding to a futility boundary at test statistic $- 0.4$ .

Figure 4.

Mean type I error rates under different randomization procedures in a group sequential trial with a maximum sample size of $n = 48$ with one interim efficacy analysis. Wang–Tsiatis boundaries where used with varying $Δ$ values. Note, the $y$ -axis is restricted to $[0, 0.035]$ for the low-bias scenario and to $[0, 0.20]$ for the high-bias scenario to highlight the differences.

Figure 5.

Mean type I error rates under different timings of the (efficacy) interim analysis for a maximum sample size of $n = 48$ . A Lan–DeMets group sequential design with O’Brien–Fleming- and Pocock-type alpha spending functions was used. The values $0$ and $1$ on the $x$ -axis represent trials without an interim analysis. Note, the $y$ -axis is restricted to $[0, 0.035]$ for the low-bias scenario and to $[0, 0.20]$ for the high-bias scenario to highlight the differences.

3.2. Futility stopping only

We next considered a design with a maximum sample size of $n = 48$ and $K = 2$ stages, incorporating a single non-binding futility analysis but no efficacy stopping. Figure 3 shows the T1E across RPs, as a function of the futility boundary $α_{0}$ . Smaller values of $α_{0}$ correspond to a more aggressive futility rule, as a trial will be stopped if the $p$ -value based on the interim data is above the futility bound $α_{0}$ , that is, the lower $α_{0}$ , the higher the probability of stopping for futility becomes. The T1E decreased with a more aggressive futility boundary $α_{0}$ . The reduction in T1E with a more aggressive futility boundary arises because the probability of stopping for futility at interim increases. Under the null hypothesis, this stopping probability would usually equal $1 - α_{0}$ , as $p$ -values are uniformly distributed. Under allocation bias, however, this relationship no longer holds, and the futility stopping probability depends on the RP, see Figure 2 in the Supplemental Appendix. For example, with $α_{0} = 0.28$ the stopping probability for PBR( $4$ ) was $0.69$ in the low-bias setting and $0.42$ in the high-bias setting, versus $0.72$ without bias. In the low-bias setting PBR achieved nominal $2.5 %$ T1E at $α_{0} = 0.13$ , however such a strict futility rule reduces power and only restores T1E for that assumed bias level. More restrictive RPs generally showed greater inflation. RAR and BSD( $3$ ) behaved similarly in this setting.

3.3. Efficacy stopping only

We then evaluated one interim analysis for efficacy using Wang–Tsiatis boundaries with different $Δ$ values. The $Δ$ parameter determines how the significance level is distributed between interim and final analyses. Figure 4 shows that spending more significance level at the interim analysis reduces the T1E elevation under allocation bias for restrictive RPs, particularly in the high-bias scenario. In the low allocation bias scenario, for example, under PBR, T1E was $3.29 %$ with $Δ = 0$ (OBF-type boundary) and $3.21 %$ for $Δ = 0.5$ (Pocock-type boundary). To have the two most extreme spending functions, we therefore focus on OBF- and Pocock-type boundaries in the following. Generally, the more significance level is kept for the final analysis, the higher the T1E when keeping all other design parameters the same (such as number and timing of interim analyses, stage-wise and maximum sample sizes).

We next examine how the timing of the interim analysis influences T1E under these boundaries. The timing axis values $0$ and $1$ represent trials without an interim analysis. As shown in Figure 5, timing had only modest influence in the low-bias setting. However, in the high allocation bias scenario, differences are more visible: early interim analyses led to elevated T1E across all RPs. This inflation is however attributable to constraints in our simulation setup: randomization sequences were skipped in which an interim analysis $t$ -test could not be conducted, because less than two patients were allocated to any treatment group, which was more likely the earlier the interim analysis was and made the randomization process restrictive by construction. As observed previously, Pocock-type boundaries generally resulted in slightly lower T1E than OBF-type boundaries under restrictive RPs (i.e. all except for CR). For PBR, we can observe the biggest difference for an interim analysis when the outcome of half of the patients has been observed. In the following, we fix the timing of the interim analysis to $50 %$ for the subsequent simulations, highlighted by the dashed line in Figure 5.

We also evaluated power under allocation bias. Figure 6 presents the results for OBF- and Pocock-type spending functions. For reference, we included the power of PBR without bias (the black line with a square in it). Differences between randomization procedures were most pronounced in the mid-power range (around 50–60%). For example, for OBF-type boundaries, at an effect size of $0.6$ , PBR without bias showed $52.7 %$ power. Under bias, power for BSD(3) was at $54.4 %$ , and highest for PBR(4) at $57.3 %$ . At an effect size of $0.8$ , the differences for the RPs were lower, for example, for OBF-type boundaries, PBR without bias showed $77.3 %$ power, while BSD(3) power was at $78.5 %$ and PBR(4) power was at $80.6 %$ . Overall, it can be seen, that CR has about the same power as PBR without bias, while all other RPs have an increased power with the caveat of no T1E control.

Figure 6.

Mean power under different randomization procedures in a group sequential trial with a maximum sample size of $n = 48$ and one interim efficacy analysis, using a Lan–DeMets design with an O’Brien–Fleming-type alpha spending function. An allocation bias of $η = 0.04$ (equivalent to $5 %$ of the treatment effect required for $80 %$ power) was assumed. For comparison, the power under permuted block randomization with block size 4 and no allocation bias is also shown, representing exact $1 : 1$ allocation at each analysis.

Next, we investigated the effect of efficacy stopping boundaries under varying levels of allocation bias. Figure 7 shows the T1E as a function of allocation bias $η$ . OBF-type designs behaved much like the fixed sample $t$ -test without interim analyses (Figure 2), which is expected, as most alpha is spent at the final stage. In contrast, Pocock-type boundaries consistently showed lower T1E than OBF boundaries under the same conditions (Figure 7). The Supplemental Appendix also includes block size effects. Smaller blocks led to much stronger T1E inflation, larger blocks reduced it, and the T1E inflation for RPBR values fell between their component block sizes (i.e. RPBR( $4, 8$ ) between PBR( $4$ ) and PBR( $8$ )).

Figure 7.

Mean type I error rates under different randomization procedures in a group sequential trial with a maximum sample size of $n = 48$ with one interim efficacy analysis with O’Brien–Fleming- and Pocock-type boundaries. Note, the $y$ -axis is restricted to the range $[0, 0.20]$ to highlight the differences.

3.4. Combined futility and efficacy stopping

We then analyzed designs with both efficacy and futility stopping. As shown in Figure 8, under non-binding futility stopping, T1E decreased with stricter futility boundaries (i.e. using lower $α_{0}$ ), consistent with the futility-only setting (cf. Figure 3). For less stringent futility boundaries (i.e. using larger $α_{0}$ ), OBF-type boundaries produced higher T1E than Pocock-type boundaries under restrictive RPs, in line with earlier observations (cf. Figure 4). However, as the futility threshold became more aggressive, T1E decreased faster under OBF-type boundaries. For example, in the low-bias setting under PBR( $4$ ), Pocock-type boundaries resulted in higher T1E than OBF-type boundaries when $α_{0} < 0.35$ . This pattern reflects the delayed alpha spending of OBF designs, where more significance level is reserved for the final analysis and may be pre-empted by early futility stopping.

Figure 8.

Mean type I error rates under different randomization procedures in a group sequential trial with a maximum sample size of $n = 48$ and one (non-binding) futility analysis and a Lan–DeMets design with O’Brien–Fleming- and Pocock-type alpha spending functions. The trial can stop for futility at interim if $p_{1} > α_{0}$ , where $p_{1}$ denotes the first-stage $p$ -value. Note, the $y$ -axis is restricted to $[0, 0.035]$ for the low-bias scenario and to $[0, 0.20]$ for the high-bias scenario to highlight the differences. The smallest $α_{0}$ is $α_{0} = 0.653$ , corresponding to a futility boundary at test statistic $- 0.4$ .

Figure 9 shows the impact of binding futility stopping. Compared with the non-binding case, T1E inflation was even greater. This occurs because futility stopping is incorporated into the efficacy boundaries, which means that the adjusted alpha levels for both interim and final analysis becomes larger compared to the non-binding setting. As a result, when using more aggressive (i.e. lower) binding futility boundaries $α_{0}$ , the adjusted efficacy levels become more liberal as well, resulting in more rejections. The efficacy boundaries increase to preserve overall alpha, but do not account for allocation bias.

Figure 9.

Mean type I error rates under different randomization procedures in a group sequential trial with a maximum sample size of $n = 48$ and one binding futility analysis and a Lan–DeMets design with O’Brien–Fleming- and Pocock-type alpha spending functions. The trial stops for futility at interim if $p_{1} > α_{0}$ , where $p_{1}$ denotes the first-stage $p$ -value. Note, the $y$ -axis is restricted to $[0, 0.15]$ for the low-bias scenario and to $[0, 0.5]$ for the high-bias scenario to highlight the differences. The smallest $α_{0}$ is $α_{0} = 0.653$ , corresponding to a futility boundary at test statistic $- 0.4$ .

3.5. Stage-wise restarting of the randomization process

GSDs often assume an exact $1 : 1$ allocation ratio at each stage,^15,19 implying that randomization may effectively restart at interim stages. We therefore investigated how stage-wise restarting of the RPs affects T1E across different number of stages under allocation bias.

To allow a fair comparison between stage-wise and full randomization, we assumed that the investigator is aware of the restart and resets the allocation biasing policy at the beginning of each stage. In this setup, the number of previous allocations to each arm is reset to zero whenever randomization restarts. Figure 10 shows the results for OBF- and Pocock-type spending functions, comparing full randomization sequences (solid lines) with stage-wise restarts (dashed lines). The case $K = 1$ represents a single-stage design with no interim analysis, where all significance level is spent at the only final analysis. Here, stage-wise and full randomization coincide. PBR produced identical results under both approaches, since PBR( $4$ ) enforces exact $1 : 1$ allocation within each stage, regardless of whether there are 2, 3, 4, or 6 stages. By contrast, RAR showed a marked increase in T1E under stage-wise randomization in both low and high bias scenarios. This occurs because stage-wise RAR behaves like PBR with a block size equal to the stage size, becoming more restrictive as the number of stages increases.

Figure 10.

Mean type I error rates under different randomization procedures in group sequential trials with maximum sample size $n = 48$ and varying numbers of stages. Results are shown for O’Brien–Fleming- and Pocock-type alpha spending functions. Solid lines represent full randomization (one sequence generated for the entire trial), and dashed lines represent stage-wise restarting of randomization at each interim analysis. An allocation bias of $η = 0.04$ (low bias, $5 %$ of the treatment effect for $80 %$ power)) and $η = 0.4$ (high bias, $50 %$ of the treatment effect for $80 %$ power) was assumed. Note, the $y$ -axis is restricted to $[0, 0.035]$ for the low-bias scenario and to $[0, 0.20]$ for the high-bias scenario to highlight the differences.

For BSD, CHEN, and EBC, T1E decreased slightly for stage-wise randomization compared to full randomization in the high-bias setting as the number of interim analyses increased. BSD and CHEN, which share the same maximum-tolerated imbalance, become less restrictive with more stages because the imbalance tolerance applies independently to each stage. This property does not apply to EBC. Interestingly, CR also exhibited lower T1E under stage-wise randomization. The reductions for both CR and EBC can be explained by the reset in the investigator’s biasing policy, which alters the effective patient population. For example, each restart introduces a neutral patient at the beginning of the stage. Details on the distribution of neutral, positive, and negative responders under both full and stage-wise randomization are provided in the Supplemental Appendix.

Finally, under high bias, increasing the number of stages reduced T1E for most restrictive RPs (with the exception of RAR), because more significance level is spent earlier through additional interim analyses. This effect was especially pronounced for Pocock-type boundaries, which allocate more alpha to interim looks than OBF-type boundaries. In particular, T1E dropped substantially under Pocock-type boundaries as the number of stages increased for restrictive RPs with the exception of RAR. By contrast, designs without interim analyses consistently showed higher T1E.

In summary, here are the key observations of our simulation:

When using a biasing strategy, the T1E was inflated for all RPs evaluated. Only the magnitude differed by the restrictiveness of the procedure.

The gains of power observed depend on the RP at the expense of T1E control.

Generally allowing for early stopping (efficacy, non-binding futility) decreases the inflation. The larger a trial, the more biased the test statistic will become. Therefore, if the likelihood to stop early increases, this implies that on average less time is left to bias the test statistic. If a trial allows for early stopping for efficacy, less alpha is left for the final analysis. If early stopping for futility using non-binding futility boundaries is implemented, it means less time is left to “fully” bias the test statistic which would have led to a rejection otherwise.

Under our investigated simulation scenarios and design settings, the RP was the main driver for allocation bias. Restrictive RPs, like PBR with small block sizes which has a higher predictability, produce the largest inflation of T1E.

OBF-type boundaries resulted in more T1E inflation than Pocock-type boundaries, as more alpha is reserved for the final analysis.

More stages reduced the impact of bias on the T1E. Stage-wise restarting lowered T1E for most RPs (BSD, CHEN, EBC, CR), had no effect for PBR, and increased T1E for RAR. The inflation of T1E reduced with more stages, especially under Pocock-type boundaries, as more significance is spent earlier.

With non-binding futility, T1E decreased as the futility threshold became more aggressive (smaller $α_{0}$ ). In contrast, binding futility increased T1E due to the increased efficacy boundaries accounting for the futility stopping, but which do not correct for the bias.

Larger block sizes for PBR reduced T1E inflation, randomized block sizes (e.g. RPBR( $4, 8$ )) did not help in eliminating bias, but resulted in a weighted average of the block sizes used.

4. Discussion

The purpose of this research was to investigate the effect of allocation bias on different design choices for a two-arm RCTs, focusing specifically on the impact of different RPs in conjunction with different types of interim analyses. Our findings revealed differences in the T1E across various RPs in all scenarios considered. PBR remains the most widely used RP in GSDs.¹⁷ However, under our biasing policy, PBR with small block sizes produced the greatest T1E inflation among the evaluated procedures, as was also observed in other trial designs.^9–11 If interim analyses both for futility and efficacy are performed, then the impact on the inflation on the T1E is less compared to designs without interim analysis. The general tendency is that the higher the likelihood for stopping, the less the T1E inflation is. For example, if all design parameters such as timing of interim analysis and maximum sample size are the same, then when using OBF-type efficacy boundaries, there is a higher inflation of the T1E compared to Pocock-type boundaries. The latter have a higher probability to stop early. The results of our study should not be misinterpreted that we suggest to use Pocock designs and many interim analysis. The reason why Pocock-type spending shows less inflation is simply because less alpha is left for the final analysis (where the test-statistic is most biased). The more patients are included, the more biased the test statistic will be. Similarly, using more interim analyses leaves less alpha for the final analysis. The simulation also show that using binding futility boundaries, that is, increasing the efficacy boundaries for potential stopping is not a good idea, as it further increases T1E inflation. In general, early stopping should require convincingly large effects, which aligns more closely with OBF-type spending. Stage-wise restarting of the RP might be useful, for example, when balance by stage is important, but it also alters the RP itself. For example, RAR becomes much more restrictive (behaving like PBR with block size equal to stage size) and BSD allows for a larger overall imbalance, as the imbalances are additive for each stage. In our simulations, stage-wise restarting slightly reduced T1E for most procedures, the exceptions were PBR( $4$ ) (unchanged by construction) and RAR (increased inflation due to added restrictiveness). Therefore, stage-wise restarting can be helpful in protecting against allocation bias, but it should be adopted with the understanding that it alters the operating characteristics of the chosen RP.

Whenever feasible, blinding should be ensured. Open-label settings increase predictability. From a bias perspective, unrestricted randomization is preferable, but if there is no bias, this results in a loss of power¹⁷ and, in extreme cases with very early interim looks, could even leave one group without observations at interim, preventing analysis (or with only one observation, in which case a $t$ -test could still not be conducted). In open-label trials, the GSD and RP combination that is most robust to bias will not be the most powerful under a no-bias assumption. Reducing predictability with less restrictive RPs will require an increase in sample size even when no bias effect is present.

Regarding the biasing policy we used the concept of good, neutral, and bad responders, which could be considered like a single, unknown prognostic factor. However, in practice, selective enrollment may be driven by the investigator’s overall clinical impression, which may not be directly pictured by observable prognostic factors and may even evolve during trial conduct, as clinicians gain experience with the disease course independent of treatment allocation. A second important consideration is feasibility in rare diseases. We do not assume that investigators can defer enrollment indefinitely until a favorable or unfavorable patient appears. Instead, the biasing policy should be interpreted as a worst-case model used to assess the robustness of different designs. When the inclusion and exclusion criteria leave room for judgment, enrollment decisions may be influenced by the investigator’s expectation of a successful treatment rather than reaching a specific sample size. Even if the total population is very limited, an investigator would not include a patient in a clinical trial if he expects that the patient will not profit from the treatment. Moreover, rare disease trials are often conducted in specialized centers where investigators may know patients well and they may have some influence over the timing and ordering of enrollment. Tamm et al. discuss the rejection of candidates and show that even an imperfect or constrained selection mechanism can affect the test decision.³⁰ These points underscore why we emphasize robustness at the design stage. In applied work, sensitivity analyses that explicitly model allocation bias mechanisms can be considered; for example, Wied et al. re-analyzed the EPISTOP trial and performed a bias-corrected analysis by including a term representing allocation bias in the statistical model.³¹ In multi-center trials, allocation bias can arise when stratifying by center.¹⁰ Central randomization reduces this risk but does not fully eliminate it if a center contributes consecutive patients to a central list. For example, with PBR(6) the next assignment can be correctly predicted, on average, $58 %$ of the time by simply guessing the opposite of the previous allocation³² and consecutive positions from the same center can introduce bias even under a central allocation schedule.³³

The scope of our simulation was necessarily limited and we followed the stepwise way adaptive designs are often planned. We did not aim to cover every setting. For example, we focused on a total sample size of $n = 48$ to reflect small-sample clinical trials, where allocation bias risk is higher. The accompanying code lets readers extend the analysis to other sample sizes, numbers/timing of looks, spending functions, futility policies (non-binding or binding), bias magnitudes, and RPs.

It could be interesting to examine allocation bias when the outcome is defined as a change from baseline. Future work should also evaluate multi-arm design, such as platforms trials, where staggered entry of arms, stage-wise restarting of randomization and unequal allocation ratios may alter the extent of bias. Such investigations will support the credibility of innovative trial designs.

5. Conclusion

Bias in clinical trials is a key concern for regulatory agencies.^1,2,34 Our findings provide insights into mitigating allocation bias in GSDs with small-sample sizes. The size of the problem depends mainly on the RP, with design choices regarding the GSD having only secondary effects in our investigated scenarios. Therefore, if allocation bias cannot be ruled out, a less restrictive RP, such as BSD, should be chosen over, for example, PBR with small block sizes. If PBR is used, large block sizes should be used. Though allocating more alpha at interim led to a lower T1E inflation compared to preserving more alpha for the final analysis, this should not be taken as a justification for such type of spending functions. In our simulation studies the maximum sample size was fixed when comparing different GSDs. It has to be noted that when using Pocock-type boundaries instead of OBF-type boundaries, this would also require much larger sample sizes.^18,19 Similarly, if futility stopping is implemented, the sample size has to be increased accordingly. Thus, the choice of design parameters in GSDs should not depend on how they will potentially mitigate the impact of potential biases strategies. Our work demonstrates that the choice of the RP is key.

List of abbreviations

BSD( $a$ )

big stick design with maximum-tolerated imbalance $a$

CHEN(

p, a

)

Chen’s design with probability $p$ and maximum-tolerated imbalance $a$

complete randomization

EBC(

p

)

Efron’s biased coin with probability $p$

GSD

group sequential design

LDM

Lan and DeMets

OBF

O’Brien and Fleming

PBR(k)

permuted block randomization with block length $k$

PBR(

k_{1}, k_{2}, \dots

)

randomized permuted block randomization with block length $k_{1}, k_{2}, \dots$

RAR

random allocation rule

RCT

randomized controlled trial

randomization procedure

T1E

type I error rate

Supplemental Material

sj-r-1-smm-10.1177_09622802261442914 - Supplemental material for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials

Supplemental material, sj-r-1-smm-10.1177_09622802261442914 for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials by Daniel Bodden, Ralf-Dieter Hilgers and Franz König in Statistical Methods in Medical Research

Supplemental Material

sj-r-2-smm-10.1177_09622802261442914 - Supplemental material for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials

Supplemental material, sj-r-2-smm-10.1177_09622802261442914 for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials by Daniel Bodden, Ralf-Dieter Hilgers and Franz König in Statistical Methods in Medical Research

Supplemental Material

sj-pdf-3-smm-10.1177_09622802261442914 - Supplemental material for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials

Supplemental material, sj-pdf-3-smm-10.1177_09622802261442914 for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials by Daniel Bodden, Ralf-Dieter Hilgers and Franz König in Statistical Methods in Medical Research

Footnotes

ORCID iDs

Daniel Bodden

Ralf-Dieter Hilgers

Franz König

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: R-DH is coordinator, and DB and FK are members of RealiseD supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under grant agreement No. 101165912. The JU receives support from the European Union’s Horizon Europe research and innovation program and COCIR, EFPIA, Europa Bío, MedTech Europe, and Vaccines Europe. Views and opinions expressed are those of the author(s) only. This publication reflects the author’s views. They do not necessarily reflect those of the Innovative Health Initiative Joint Undertaking and its members, who cannot be held responsible for them. R-DH received funding from European Rare Disease Research Coordination and Support Action consortium (ERICA) funded by the European Union’s Horizon 2020 research and innovation program under Grant Agreement. no. 964908. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Code availability

All code related to the simulation study is available in the following repository: Bodden, Daniel (2026). Simulation code for “When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials” is available at figshare software: .

Supplemental material

Supplemental material for this article is available online.

References

US Food and Drug Administration. Providing clinical evidence of effectiveness for human drug and biological products. Guidance for Industry, 1998.

International Conference on Harmonisation (ICH). ICH harmonised tripartite guideline: E9 statistical principles for clinical trials, https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-9-statistical-principles-clinical-trials-step-5_en.pdf (1998).

Blackwell

Hodges Jr

. Design for the control of selection bias. Ann Math Stat 1957; 28: 449–460.

Nunan

Heneghan

Spencer

. Catalogue of bias: allocation bias. BMJ Evid Based Med 2018; 23: 20–21.

International Conference on Harmonisation (ICH). ICH Harmonised Tripartite Guideline E10: choice of control group and related issues in clinical trials, https://database.ich.org/sites/default/files/E10_Guideline.pdf (2000). Step 4 version.

Clark

Burke

Margaret

, et al. A review found small variable blocking schemes may not protect against selection bias in randomized controlled trials. J Clin Epidemiol 2022; 141: 90–98.

Sterne

JAC

Savović

Page

, et al. RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ 2019; 366: l4898.

Shao

Rosenberger

. Properties of the random block design for clinical trials. In: Kunert J, Müller CH and Atkinson AC (eds) mODa 11—advances in model-oriented design and analysis. Contributions to Statistics. Cham: Springer, 2016, pp. 207–215. DOI: 10.1007/978-3-319-31266-8_26.

Hilgers

Uschner

Rosenberger

, et al. ERDO—a framework to select an appropriate randomization procedure for clinical trials. BMC Med Res Methodol 2017; 17: 159.

10.

Hilgers

Manolov

Heussen

, et al. Design and analysis of stratified clinical trials in the presence of bias. Stat Methods Med Res 2020; 29: 1715–1727.

11.

Schoenen

Heussen

Verbeeck

, et al. The impact of allocation bias on test decisions in clinical trials with multiple endpoints using multiple testing strategies. BMC Med Res Methodol 2024; 24: 223.

12.

EMA/CHMP. Guideline on clinical trials in small populations, https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-clinical-trials-small-populations_en.pdf (2006).

13.

Day

Jonker

Lau

LPL

, et al. Recommendations for the design of small population clinical trials. Orphanet J Rare Dis 2018; 13: 195.

14.

Bell

Tudur Smith

. A comparison of interventional clinical trials in rare versus non-rare diseases: an analysis of clinicaltrials.gov. Orphanet J Rare Dis 2014; 9: 170.

15.

Pocock

. Group sequential methods in the design and analysis of clinical trials. Biometrika 1977; 64: 191–199.

16.

O’Brien

Fleming

. A multiple testing procedure for clinical trials. Biometrics 1979; 35: 549–556.

17.

Bodden

Hilgers

König

. Randomization in clinical trials with small sample sizes using group sequential designs. PLoS One 2025; 20: e0325333.

18.

Wassmer

Brannath

. Group sequential and confirmatory adaptive designs in clinical trials. Springer Series in Pharmaceutical Statistics. Springer International Publishing, 2016.

19.

Jennison

Turnbull

. Group sequential methods with applications to clinical trials. Chapman & Hall/CRC Interdisciplinary Statistics. CRC Press, 1999.

20.

Lan

KKG

DeMets

. Discrete sequential boundaries for clinical trials. Biometrika 1983; 70: 659–663.

21.

Wang

Tsiatis

. Approximately optimal one-parameter boundaries for group sequential trials. Biometrics 1987; 43: 193–199.

22.

Lin

Lee

Zhen

, et al. CBER’s experience with adaptive design clinical trials. Ther Innov Regul Sci 2016; 50: 195–203.

23.

Nikolakopoulos

Roes

van der Tweel

. Sequential designs with small samples: evaluation and recommendations for normal responses. Stat Methods Med Res 2018; 27: 1115–1127.

24.

Shao

Feng

. Group sequential

t

-test for clinical trials with small sample sizes across stages. Contemp Clin Trials 2007; 28: 563–571.

25.

Rom

McTague

. Exact critical values for group sequential designs with small sample sizes. J Biopharm Stat 2020; 30: 752–764.

26.

Anderson

. gsDesign: group sequential design, https://keaven.github.io/gsDesign/ (2023).

27.

Wassmer

Pahlke

. rpact: confirmatory adaptive clinical trial design and analysis, https://www.rpact.org (2025).

28.

Uschner

Schindler

Hilgers

, et al. randomizeR: an R package for the assessment and implementation of randomization in clinical trials. J Stat Softw 2018; 85: 1–22.

29.

Berger

Bour

Carter

, et al. A roadmap to using randomization in clinical trials. BMC Med Res Methodol 2021; 21: 168.

30.

Tamm

Cramer

Kennes

, et al. Influence of selection bias on the test decision. A simulation study. Methods Inf Med 2012; 51: 138–143. Erratum in: Methods Inf Med. 2014;53(5):343.

31.

Wied

Hilgers

Heussen

, et al. Methodological insights from the EPISTOP trial to designing clinical trials in rare diseases—a secondary analysis of a randomized clinical trial. PLoS One 2024; 19: e0312936.

32.

Hills

Gray

Wheatley

. Balancing treatment allocations by clinician or center in randomized trials allows unacceptable levels of treatment prediction. J Evid Based Med 2009; 2: 196–204.

33.

Krisam

Ryeznik

Carter

, et al. Understanding an impact of patient enrollment pattern on predictability of central (unstratified) randomization in a multi-center clinical trial. Stat Med 2024; 43: 3313–3325.

34.

International Council for Harmonisation (ICH). ICH E20: guideline on adaptive designs for clinical trials – step 2b (draft), 2025.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.01 MB

0.25 MB

0.01 MB

0.00 MB

When randomization is not random: Allocation bias in small sample,group sequential randomized clinical trials

Abstract

Keywords

1. Introduction

2. Methods

2.1. Biasing policy

2.3. Simulation

3. Results

3.1. Investigation of allocation bias strategy for a t -test without interim analysis

3.3. Efficacy stopping only

5. Conclusion

List of abbreviations

Supplemental Material

sj-r-1-smm-10.1177_09622802261442914 - Supplemental material for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials

Supplemental Material

sj-r-2-smm-10.1177_09622802261442914 - Supplemental material for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials

Supplemental Material

sj-pdf-3-smm-10.1177_09622802261442914 - Supplemental material for When randomization is not random: Allocation bias in small sample, group sequential randomized clinical trials

Footnotes

ORCID iDs

Funding

Declaration of conflicting interests

Code availability

Supplemental material

References

Supplementary Material

3.1. Investigation of allocation bias strategy for a $t$ -test without interim analysis