Response to: Comment About Statistical Analysis of a Cluster-Randomized Trial About Clustering and Nesting (DOI: 10.1089/chi.2019.0142)

The article this Commentary refers to was officially retracted on September 15, 2020. See 10.1089/chi.2016.0216.retract for full details.

We certainly share the authors' concern regarding the importance of the intracluster correlation coefficient (ICC) ¹ in cluster-randomized clinical trials, as it is well known that this design might lead to biased results in case of unbalanced clusters. ²

However, in our study, clusters were public schools within a very limited geographic area (within one town in southern Catalonia with a population of <110,000 inhabitants), sharing the same socioeconomic characteristics and with the same profile of students. Schools were chosen as randomization units instead of individuals to avoid (or minimize the chances of) contamination between students in control and intervention groups within the same school. Students in the control group having information about the intervention provided by peers in the intervention group are an issue that has been previously detected in similar studies. ³ For this reason, we used the school as the randomization unit with no need of analyzing differences between schools (no school-level information was collected).

As the authors of the present editorial comment state, several studies have found relatively high ICC in obesity studies, mostly for continuous responses and when the clusters within intervention or treatment groups are very different in relevant variables. In this sense, the studies referenced by the comment authors ¹ have important differences with respect to the work presented in Aceves-Martins et al. ⁴ For instance, Gray et al. ⁵ included schools all over New York area (where the population comprises >8 million of inhabitants), which can be expected to have very different student profiles that could lead to high ICC. Even more clearly, Masood et al. ⁶ included schools from several countries. In addition, the method suggested by Hedges ⁷ and adopted by the authors to correct p-values from our study is intended for continuous variables, so how it was adapted to correct p-values from generalized linear models ⁸ with dichotomous responses like those considered in Aceves-Martins et al. ⁴ is unclear.

In contrast, in our setting, it seems reasonable to think that the impact of this design on the results and, moreover, on the interpretation of the conclusions will be very limited. This is confirmed by actual ICC values and confidence intervals for main response variables, reported in Table 1. These values were computed using the ICCbin ⁹ R ¹⁰ package, considering the moment estimate with weights proportional to cluster size method by Kleinman. ¹¹ The reported 95% confidence intervals are based on Smith's method, ¹² and it can be seen that null ICC is included in the interval for all variables.

To evaluate the potential impact of ICC on the results, an additional analysis based on generalized linear mixed models was conducted, including a random intercept for school, ¹³ and the direction of the conclusions was in line with the reported conclusions. Certainly, other alternative approaches might have been used, like multilevel analysis as suggested in Omar and Thompson, ¹⁴ but since ICC was negligible, and given the scope of this study, ¹⁵ such approaches were not considered.

The main change in the final analysis described in Aceves-Martins et al. ⁴ with respect to the published protocol ¹⁵ is the complementary analyses based on multiple imputations of missing data, due to reviewers' comments after submission of the first version of the article. In the case of Aceves-Martins et al., ⁴ these modifications actually improved the robustness of the reported conclusions. Undertaking additional analyses as suggested by reviewers is not uncommon, and the authors felt it reasonable to complete and publish results of the complementary analyses.

Data used in this study ⁴ are part of a broader project that is not yet over, and as mentioned in our protocol, we will guarantee public access to the full protocol, participant database, and statistical code as soon as it is finished and after acceptance from all partners.