Estimates of Intraclass Correlation Coefficients From Longitudinal Group-Randomized Trials of Adolescent HIV/STI/Pregnancy Prevention Programs

Abstract

Introduction. Group-randomized trials (GRTs) are one of the most rigorous methods for evaluating the effectiveness of group-based health risk prevention programs. Efficiently designing GRTs with a sample size that is sufficient for meeting the trial’s power and precision goals while not wasting resources exceeding them requires estimates of the intraclass correlation coefficient (ICC)—the degree to which outcomes of individuals clustered within groups (e.g., schools) are correlated. ICC estimates vary widely depending on outcome, population, and setting, and small changes in ICCs can have large effects on the sample size needed to estimate intervention effects. This study addresses a gap in the literature by providing estimates of ICCs for adolescent sexual risk-taking outcomes under a range of study conditions. Method. Multilevel regression analyses were applied to existing data from four federally funded GRTs of school-based HIV/STI/pregnancy prevention programs to obtain a variety of ICC estimates. Results. ICCs ranged from 0 to 0.15, with adjustment for covariates and repeated measurements reducing the ICC in the majority of cases. Minimum detectable effect sizes with 80% power and 0.05 significance levels ranged from small to medium Cohen’s d (0.13 to 0.53) assuming 20 schools of 100 students each. Conclusions. This study provides the first known set of ICC estimates for investigators to use when planning studies of school-based programs to prevent sexual risk behaviors in youth. The results provide further evidence of the importance of using the appropriate adjusted ICC estimate at the design stage to maximize resources in costly GRTs.

Keywords

adolescent health health behavior research design school health instruction sex behavior

Group-randomized trials (GRTs), sometimes called cluster-randomized trials, are common in behavioral health intervention research in youth. They are considered one of the most scientifically rigorous methods for evaluations in which intervention and control conditions must be assigned to groups (Donner, Birkett, & Buck, 1981; Donner & Klar, 2000; Murray et al., 2004). Many GRTs are conducted in schools, which are seen as an important venue for interventions because most youth attend schools, making them efficient sites for program delivery (Davis & Bauman, 2013).

In school-based trials of behavioral health interventions, the GRT is used when interventions are delivered to whole schools or classrooms rather than at the individual level. This is commonly done when it is not feasible or cost effective to deliver the intervention at the individual level or when the intervention is inherently group based (Donner & Klar, 2000; Murray, 1998; Murray & Blitstein, 2003; Varnell, Murray, Janega, & Blitstein, 2004). While GRTs randomize groups to intervention and control conditions, most are focused on assessing the impact of the intervention on the outcomes of individuals, so that analyses are performed on observations measured at the individual, not group level. However, observations on individual members within the same group may be correlated, violating the assumption of independence of observations required by the statistical theories underlying classical techniques for trial planning and analysis. The correlation between outcomes on individual members of the same group must be accounted for to ensure the study being designed is not underpowered, and/or the study being analyzed does not have inflated Type I error rates (Donner et al., 1981; Donner & Klar, 2000; Goldstein, 1995; Murray, 1998). In particular, classical sample size determinations must be inflated by a factor of 1 + (n − 1) * ICC, where n is the size of each group, and ICC, or intraclass correlation coefficient, is a measure of the correlation between observations within groups (Cornfield, 1978; Donner et al., 1981; Kish, 1965). Application of this factor, termed the “design effect” (Snijders & Bosker, 1999) or “variance inflation factor” (Donner et al., 1981), is necessary to ensure that a GRT will have adequate power and precision for detecting and estimating intervention effects. The ICC can be thought of as a measure of the proportion of total variance in the outcome attributable to variation between groups (e.g., schools); for this reason some authors prefer the name “variance partition coefficient” to denote the ICC (Rasbash, Steele, Browne, & Goldstein, 2009).

A small change in an ICC estimate can have a large impact on the required sample size needed to power a study with the desired accuracy to detect the effect size of interest. For example, with an ICC of 0.001, a researcher planning a school-based GRT would need to recruit nine schools, assuming average group sizes of 100 students recruited per school, 80% power, a .05 significance level and a 0.2 Cohen’s d minimum detectable effect size (equivalent to a change in a proportion outcome from 0.5 to 0.4); with an ICC of 0.01, the number of schools needed increases to 16 under the same assumptions; an ICC of 0.05 increases the number of schools to 63. Therefore it is important for researchers planning GRTs to have access to ICCs that best match their study design, primary outcomes, population, and setting; this will help ensure that a sample size is selected that is sufficient for meeting the trial’s power and precision goals while not wasting resources exceeding them.

Studies have demonstrated that ICCs can vary substantially depending on the outcome, population, setting and intervention (Donner & Klar, 2000; Murray, 1998; Resnicow et al., 2010; Rotondi & Donner, 2009). Additionally, estimates of ICCs vary depending on the model or method of analysis. For example, ICCs often decrease with the inclusion of covariates in an outcome model (Murray & Blitstein, 2003; Pals, Beaty, Posner, & Bull, 2009; Zhang et al., 2014), and may differ depending on whether repeated measures or cross-sectional analyses are applied (Murray & Blitstein, 2003). Published ICC estimates found for trials of community and school-based interventions focused on tobacco and alcohol prevention, and physical activity and nutrition promotion ranged from 0 to 0.03 (Murray, Birnbaum, Phillips, & Glenn, 2001; Murray & Blitstein, 2003; Murray et al., 2004; Murray et al., 2006; Murray & Short, 1996, 1997; Siddiqui, Hedeker, Flay, & Hu, 1996), while those found for community-based HIV/STI prevention trials ranged from 0.001 to 0.16 (Feldblum et al., 1999; Pals et al., 2009; Zhang et al., 2014); no ICC estimates were found for school-based HIV/STI/pregnancy prevention trials.

Estimates of ICCs for outcomes involving sexual risk-taking behaviors under a larger variety of conditions are sorely needed in the field of sexual and reproductive health (Pals et al., 2009; Resnicow et al., 2010; Zhang et al., 2014). The present study obtained estimates of ICCs for sexual risk-taking behavior outcomes using data from four published, school-based HIV/STI/pregnancy prevention trials for adolescents, and examined how the ICCs varied depending on the inclusion of covariates in analyses and whether the analyses accounted for repeated measures across time. We also examined the impact of these issues on sample size/power considerations. The purpose was to contribute needed data for trial planning in the field of school-based HIV/STI/pregnancy prevention interventions, providing researchers with a new array of parameter estimates from which to choose depending on their population, study design, outcomes, and planned analyses.

Method

Samples and Study Designs

The data sets were from four completed, federally funded GRTs of theory-based HIV/STI/pregnancy prevention programs. All the studies collected student self-report survey data on several sexual risk behaviors including sexual initiation, unprotected sex in the past 3 months, and condom and birth control use at last sex. Each study is described briefly and summarized in Table 1.

Table 1.

Study Descriptions.

Descriptor	Study 1 (All4You2!)	Study 2 (All4You!)	Study 3 (Safer Choices)	Study 4 (Draw the Line)
Sample age group	Ages 13-20 years	Ages 14-18 years	Grades 9-11	Grades 6-9
Sample setting	Alternative high schools in urban and suburban northern California counties	Alternative high schools in urban northern California counties	Mainstream public high schools in urban California and Texas	Mainstream public middle schools in urban northern California
Racial composition	Black = 22.3%	Asian = 15.3%	Asian = 18.2%	Asian = 15.9%
	Latino = 37.9%	Black = 27.7%	Black = 16.1%	Latino = 53.3%
	White = 18.5%	Latino = 29.1%	Latino = 27.3%	White = 16.5%
	Other = 5.9%	White = 12.2%	White = 31.0%	Other = 6.8%
	Mixed = 15.4%	Other = 15.6%	Other = 7.3%	Mixed = 7.5%
Research design	Longitudinal randomized controlled trial with blocked randomization	Longitudinal randomized controlled trial with restricted randomization	Longitudinal randomized controlled trial with restricted randomization	Longitudinal randomized controlled trial with restricted randomization
Group conditions	1. Control (Ctr)	1. Control	1. Control	1. Control
	2. HIV lessons	2. Treatment (Tx)	2. Treatment	2. Treatment
	3. Service learning
	4. HIV lessons + Service learning
Level of randomization	Classroom	School	School	School
Number of schools (Ctr, Tx)	11	24 (Ctr = 11, Tx = 13)	20 (Ctr = 10, Tx = 10)	19 (Ctr = 9, Tx = 10)
Number of classrooms (Ctr, Tx)	47 (HIV only = 12; Service only = 12; HIV+ Service = 11; Control = 12)	n/a	n/a	n/a
Number of students	765	988	3,869	2,829
Measurement times	0, 6, 18 months	0, 6, 12, 18 months	0, 7, 19, 31 months	0, 12, 24 months^a
Study years	2006-2010	2000-2001	1993-1996	1997-2000

The original article also presents a fourth wave of survey data at 36-month follow-up. This final survey was not part of the original study and has more than 30% attrition. To minimize the effect of such large attrition on the ICC estimates, the 36-month follow-up was not analyzed for this article.

Study 1 (All4You2!) evaluated the individual and combined effects of a HIV/STI/pregnancy prevention classroom curriculum and a service learning component on sexual risk outcomes of high school-aged youth attending district-run alternative high schools in northern California; the classroom curriculum significantly reduced unprotected sex in the treatment group relative to control at 6-month follow-up (Coyle et al., 2013). Study 2 (All4You!) evaluated the combined effects of an intervention involving a classroom curriculum and a service-learning component on sexual risk-taking in youth attending county-run alternative high schools; the combined intervention had a statistically significant effect on condom use behaviors at the 6-month follow-up (Coyle et al., 2006). Study 3 (Safer Choices) evaluated an intervention involving classroom and school-level components to reduce sexual risk-taking behaviors among inner city high school students in Texas and California, and found a significant reduction in unprotected sex (Coyle et al., 2001). Finally, Study 4 (Draw the Line/Respect the Line) evaluated a classroom curriculum to reduce sexual risk behaviors in middle school youth and found a significant reduction in sexual initiation among boys but not girls (Coyle et al., 2004).

Measures

Outcome measures in this study were the main behavioral outcomes reported for the original studies. For the three high school studies (Studies 1-3) they included (a) number of times had unprotected sex in the past 3 months, (b) number of partners with whom had unprotected sex in the past 3 months, (c) condom used at last sex, (d) effective method of birth control (condom or hormonal) used at last sex. Given convergence problems associated with Poisson models in some of the original outcome studies, we treated count variables as continuous, assuming an underlying normal distribution. When count variables were highly skewed, we applied transformations to reduce skew or dichotomized to 0 times/partners and 1 or more times/partners.

For the first two outcomes, analyses were performed on “full samples” of all study participants, with respondents assigned a value of 0 if they had not had sex in the past 3 months or had never had sex. For the last two outcomes, analyses were conducted only on filtered samples of sexually active youth as in the original studies, to understand the impact of condom use separate from abstinence.

For Study 4 (Draw the Line/Respect the Line) in which the prevalence of sexual behaviors was much lower because of the younger population, the following outcome variables were selected: sexual initiation (was sex initiated between baseline and a given follow-up time point); a Likert-type scale measuring self-efficacy to refuse sex; and a Likert-type scale measuring sexual limits (e.g., “Imagine you are alone with someone you like very much. Would you let them . . . touch your private parts below the waist?”).

The covariates examined in the studies included age, gender, race, parent lived with most of the time (mother, father, both), whether cohabitated with partner, probation status, number of suspensions, language spoken at home (English primary or not), language spoken with friends, whether born in United States, self-reported grades, parent education level, years lived in United States, and student’s educational goal level. A large subset of these were available for each study.

Analysis

Linear and logistic multilevel regression models (MLMs) were used to obtain all ICC estimates. MLMs partition the error variance into components associated with the data hierarchy and yield estimates of the standard errors of regression parameters that are adjusted for the presence of intraclass correlation (Goldstein, 1995). These, or equivalent models, should be used when analyzing data from GRTs (Murray 1998; Murray et al., 2004).

Because the four studies described above were longitudinal GRTs, three-level repeated measures MLMs, with time (including baseline) as Level 1, student as Level 2, and school as Level 3 would make the most efficient use of all of the available data while accounting for all three levels of nesting (Baumler, Harrist, & Carvajal, 2003; Murray & Blitstein, 2003). However, ICCs estimated from these models would be most applicable to the planning of future GRTs that also are longitudinal with the same or a similar number of follow-up time points and that intend to utilize repeated measures MLMs for data analysis. Therefore, to broaden the utility of our results, we applied both two-level and three-level MLMs. The two-level models, referred to as “endpoint analysis” models, analyzed data from a single follow up time point (the method of analysis currently required for Teen Pregnancy Prevention replication evaluations funded by the Office of Adolescent Health), with student as Level 1, school as Level 2, and the baseline value of the outcome as a covariate (Coyle et al., 2006). Random intercept MLMs were applied in all cases given the trial aims were to examine whether there was a difference in mean outcomes across study arms.

To assess the impact on ICCs of including an intervention indicator and various covariates as independent variables, the following models were run successively for each outcome (at each time point, for the endpoint analyses): (a) a “means model,” in which the outcome variable was the dependent variable and a constant was the independent variable; (b) the previous model with the addition of a fixed effect treatment indicator as an independent variable; and (c) the previous model with the addition of a set of covariates significantly related to the outcome variable at p < .15 added as fixed effect independent variables.¹ Stata Version 12.1 was used to obtain estimates of all ICCs and their standard errors.

Computation of School-Level ICCs: Continuous Outcomes

Following the terminology of Hox (2002), Baumler et al. (2003), and Rasbash et al. (2009), for the two-level case, school-level ICC was defined as the school-level error variance (i.e., variance component) divided by the total error variance: ICC_school = σ_u²/(σ_u² + σ_e²), where σ_u² is the component of variance in the outcome attributable to differences in school-level means (the Level 2 variance), and σ_e² is the component of variance in the outcome attributable to differences in student-level means (the Level 1 variance). For the three-level case, school-level ICC was defined as ICC_school = σ_u²/(σ_u² + σ_v0² + σ_e²), where σ_u² is the component of variance in the outcome attributable to differences in school-level means (the Level 3 variance), σ_v0² is the component of variance attributable to differences in student-level means across time but within schools (a component of the Level 2 variance), and σ_e² is the component of variance attributable to differences in repeated observations within student (the Level 1 variance).

Computation of School-Level ICCs: Binary Outcomes

Given that the lowest level of error variance is fixed for binary outcomes, we used a latent variable formulation (sometimes referred to as a threshold model formulation) to calculate ICCs. Under this formulation, a school-level ICC for a two-level model is defined as ICC_school = σ_u²/(σ_u² + π²/3), and for a three-level model is ICC_school = σ_u²/(σ_u² +σ_v0² +π²/3) (Hox, 2002; Rodriguez & Elo, 2003).

Sample Size/Minimal Detectable Effect Size

Minimal Detectable Effect Size (MDES) estimates were computed instead of required sample size because, like the four studies included in the analyses, school-based GRTs are generally constrained by budgets and feasibility to a fixed sample size of schools, often 20 or less. Thus, for a range of ICCs we obtained a range of MDES estimates, using Cohen’s d effect size measure, which represents the standardized difference between group means (e.g., mean for treatment group minus mean for control group divided by pooled estimate of standard deviation; Cohen, 1992). We assumed a sample size of 20 schools, an average of 100 students per school, and 80% power to detect significance at p < .05, all conditions that are likely to occur in a real-world research setting of this type. Optimal Design Version 3.0 (Raudenbush, Bryk, Cheong, & Congdon, 2011; Spybrook, Bloom, Hill, Martinez, & Raudenbush, 2011) was used to compute MDES estimates.

Results

Table 1 shows demographic characteristics of students in each of the studies. Tables 2 and 3 show the ICC estimates for the two-level endpoint and three-level repeated measures models, respectively, under the various conditions and across the four studies. While standard errors are reported in the tables, it should be noted that for small ICCs they may not be reliable and should be interpreted with caution (Murray et al., 2006). Table 4 shows the MDESs across various conditions.

Table 2.

School-Level ICCs for Two-Level Endpoint Models.

Primary outcome	Measurement time	Intercept only model ICC (SE)	Model with Tx indicator ICC (SE)	Model with Tx and covariates ICC (SE)^a
Study 1: All4You2!
Number of times had unprotected sex in past 3 months (dichotomous)^b,c	Baseline	0.078 (0.043)	0.088 (0.048)	0.055 (0.035)
	6 months	0.049 (0.032)	0.055 (0.035)	0.004 (0.013)
	12 months	0.048 (0.030)	0.050 (0.031)	0.018 (0.019)
Number of partners had unprotected sex with in past 3 months (dichotomous)^b,c	Baseline	0.128 (0.065)	0.139 (0.070)	0.103 (0.058)
	6 months	0.045 (0.029)	0.052 (0.034)	0^d
	12 months	0.046 (0.032)	0.048 (0.033)	0.002 (0.015)
Condom used at last sex (among sexually active in past 3 months)	Baseline	0.092 (0.054)	0.076 (0.046)	0.063 (0.044)
	6 months	0.003 (0.014)	0.003 (0.014)	0⁴
	12 months	0.011 (0.018)	0.010 (0.018)	0^d
Birth control used at last sex (among sexually active in past 3 months)	Baseline	0.025 (0.026)	0.019 (0.022)	0.018 (0.023)
	6 months	0^d	0^d	0^d
	12 months	0.015 (0.022)	0.016 (0.023)	0.000 (0.015)
Study 2: All4You!
Number of times had unprotected sex in past 3 months^b	Baseline	0.029 (0.017)	0.028 (0.016)	0.041 (0.023)
	6 months	0.029 (0.020)	0.021 (0.019)	0.007 (0.028)
	12 months	0.049 (0.029)	0.043 (0.027)	0.001 (0.026)
	18 months	0^d	0^d	0^d
Number of partners had unprotected sex with in past 3 months^b	Baseline	0.032 (0.017)	0.028 (0.016)	0.027 (0.021)
	6 months	0.035 (0.021)	0.031 (0.021)	0.032 (0.032)
	12 months	0.033 (0.026)	0.033 (0.026)	0^d
	18 months	0^d	0^d	0^d
Condom used at last sex (among sexually active in past 3 months)	Baseline	0.072 (0.033)	0.069 (0.033)	0.069 (0.039)
	6 months	0.024 (0.031)	0.013 (0.030)	0^d
	12 months	0.050 (0.040)	0.049 (0.041)	0.020 (0.056)
	18 months	0.151 (0.079)	0.151 (0.079)	0.154 (0.092)
Birth control used at last sex (among sexually active in past 3 months)	Baseline	0.040 (0.046)	0.041 (0.046)	0.060 (0.053)
	6 months	0^d	0^d	0^d
	12 months	0^d	0^d	0^d
	18 months	0.096 (0.066)	0.094 (0.066)	0.084 (0.077)
Study 3: Safer Choices
Number of times had unprotected sex in past 3 months (dichotomous)^b,c	Baseline	0.140 (0.048)	0.139 (0.047)	0.024 (0.018)
	7 months	0.078 (0.031)	0.078 (0.031)	0.019 (0.016)
	19 months	0.029 (0.014)	0.027 (0.013)	0^d
	31 months	0.033 (0.014)	0.032 (0.014)	0.018 (0.011)
Number of partners had unprotected sex with in past 3 months (dichotomous)^b,c	Baseline	0.133 (0.045)	0.132 (0.045)	0.024 (0.017)
	7 months	0.084 (0.032)	0.084 (0.032)	0.018 (0.015)
	19 months	0.027 (0.013)	0.025 (0.013)	0^d
	31 months	0.034 (0.014)	0.033 (0.014)	0.018 (0.011)
Condom used at last sex (among sexually active in past 3 months)	Baseline	0.014 (0.018)	0^d	0^d
	7 months	0.020 (0.017)	0.014 (0.016)	0^d
	19 months	0.002 (0.011)	0^d	0^d
	31 months	0.029 (0.016)	0.026 (0.016)	0.016 (0.014)
Birth control used at last sex (among sexually active in past 3 months)	Baseline	0.016 (0.020)	0^d	0^d
	7 months	0.016 (0.016)	0.011 (0.015)	0^d
	19 months	0.003 (0.012)	0.000 (0.012)	0^d
	31 months	0.039 (0.020)	0.038 (0.020)	0.021 (0.015)
Study 4: Draw the Line
Initiated sex since baseline	12 months	0.005 (0.005)	0.004 (0.005)	0.006 (0.005)
Initiated sex since baseline	24 months	0.003 (0.004)	0.002 (0.004)	0.006 (0.007)
Self-efficacy to refuse sex (continuous, 1-4)	Baseline	0.017 (0.009)	0.016 (0.009)	0.004 (0.004)
	12 months	0.006 (0.004)	0.005 (0.004)	0.011 (0.007)
	24 months	0.004 (0.004)	0.004 (0.004)	0.006 (0.005)
Sexual limits (continuous, 1-4)	Baseline	0.006 (0.005)	0.006 (0.004)	0.002 (0.003)
	12 months	0.008 (0.006)	0.009 (0.006)	0.008 (0.006)
	24 months	0.011 (0.007)	0.010 (0.006)	0.012 (0.008)

Note. ICC = intraclass correlation coefficients; SE = standard error.

Covariates include baseline value of the outcome variable unless model is for baseline time point or outcome is initiation since baseline. ^bStudents who reported not being sexually active in the 3 months prior to the survey were given a score of 0, allowing the entire sample to be included in the analysis (i.e., full sample), unless otherwise indicated. ^cBecause this variable was highly skewed, it was dichotomized to 0 times/partners and 1 or more times/partners. ^dICCs are listed as 0 if the estimate obtained from Stata was <1.0e⁻¹⁰. In this case the accompanying SE and confidence interval limits were also always <1.0e⁻¹⁰.

Table 3.

School-Level ICCs for Three-Level Longitudinal (Repeated Measures) Models.

Primary outcome	Intercept only model ICC (SE)	Model with Tx indicator ICC (SE)	Model with Tx and covariates ICC (SE)
Study 1: All4You2!
Number of times had unprotected sex in past 3 months (dichotomous)^a,b	0.060 (0.031)	0.070 (0.036)	0.023 (0.017)
Number of partners had unprotected sex with in past 3 months (dichotomous)^a,b	0.081 (0.040)	0.092 (0.046)	0.030 (0.023)
Condom used at last sex (among sexually active in past 3 months)	0.028 (0.020)	0.025 (0.018)	0.008 (0.013)
Birth control used at last sex (among sexually active in past 3 months)	0.010 (0.013)	0.010 (0.012)	0^c
Study 2: All4You!
Number of times had unprotected sex in past 3 months^a	0.019 (0.010)	0.018 (0.010)	0.016 (0.011)
Number of partners had unprotected sex with in past 3 months^a	0.025 (0.012)	0.023 (0.012)	0.028 (0.015)
Condom used at last sex (among sexually active in past 3 months)	0.057 (0.024)	0.056 (0.024)	0.062 (0.029)
Birth control used at last sex (among sexually active in past 3 months)	0.015 (0.013)	0.013 (0.013)	0.030 (0.022)
Study 3: Safer Choices
Number of times had unprotected sex in past 3 months (dichotomous)^a,b	0.040 (0.014)	0.045 (0.016)	0.016 (0.007)
Number of partners had unprotected sex with in past 3 months (dichotomous)^a,b	0.042 (0.015)	0.048 (0.016)	0.017 (0.008)
Condom used at last sex (among sexually active in past 3 months)	0.015 (0.008)	0.009 (0.007)	0.001 (0.004)
Birth control used at last sex (among sexually active in past 3 months)	0.018 (0.009)	0.013 (0.008)	0.003 (0.005)
Study 4: Draw the Line
Self-efficacy to refuse sex (continuous, 1-4)	0.004 (0.003)	0.004 (0.003)	0.003 (0.003)
Sexual limits (continuous, 1-4)	0.005 (0.003)	0.006 (0.004)	0.001 (0.002)

Note. ICC = intraclass correlation coefficients; SE = standard error.

Students who reported not being sexually active in the 3 months prior to the survey were given a score of 0, allowing the entire sample to be included in the analysis (i.e., full sample), unless otherwise indicated. ^bBecause this variable was highly skewed, it was dichotomized to 0 times/partners and 1 or more times/partners. ^cICCs are listed as 0 if the estimate obtained from Stata was <1.0e⁻¹⁰. In this case the accompanying SE and confidence interval limits were also always <1.0e⁻¹⁰.

Table 4.

Minimum Detectable Effect Size.^a

ICC	MDES
0.000	0.13
0.02	0.23
0.15	0.53

Note. MDES = minimum detectable effect size; ICC = intraclass correlation coefficients.

The MDES for each ICC assumed a sample size of 20 schools, an average of 100 students per school, and 80% power to detect significance at p < .05.

ICC Estimates

Endpoint MLMs

Unadjusted ICCs from two-level endpoint MLMs ranged from 0 to 0.15 depending on the outcome, follow-up time point and study population. The range for covariate-adjusted ICCs across outcomes, time points and study populations was the same, although the majority of ICCs were substantially reduced after adjusting for covariates. In Studies 1 and 3, adjusting for covariates reduced the ICC for all outcomes at all follow-ups; in Study 3, ICCs were at least halved by the addition of covariates. In Study 4, ICCs remained relatively the same (with a few slight increases) with the addition of covariates. In Study 2, in several cases—mostly for baseline models—the ICCs actually increased with the addition of covariates to the MLM. This result is examined further in the Discussion section. The range of ICCs for the four sexual behavior outcomes measured across multiple studies were similar: 0 to 0.14 for both number of times unprotected sex in past 3 months and number of partners with whom had unprotected sex in past 3 months, 0 to 0.15 for condom use at last sex, and 0 to 0.10 for birth control at last sex.

Repeated Measures MLMs

The unadjusted ICCs from the three-level repeated measures MLMs ranged from 0.004 to 0.092 while covariate-adjusted ICCs ranged from 0 to 0.062. In both Study 1 and Study 3, ICCs were more than halved by the addition of covariates to the models. In Study 4, the addition of covariates to the MLMs also reduced ICCs but not by as much. Finally, similar to the ICCs estimated from the endpoint models, for Study 2, for three of the four outcomes examined, ICCs increased with the addition of covariates to the repeated measures MLMs.

Minimum Detectable Effect Sizes

Table 4 shows the minimum detectable effect sizes across the range of ICCs found. The MDES ranged from a small Cohen’s d of 0.13 to a medium d of 0.53.

Discussion

This study found that the range of ICCs from several group-randomized trials of school-based HIV/STI/pregnancy prevention interventions in the United States was 0 to 0.15. This is similar to the range (0.003 to 0.16) found in the only other published reports of ICC estimates for HIV/STI community prevention interventions, albeit with different populations and in different settings (Feldblum et al., 1999; Pals et al., 2009). This range is slightly wider than the range of ICCs reported for tobacco, alcohol, and nutrition health-related outcomes (range 0.01 to 0.03; Murray & Blitstein, 2003), and has much lower bounds than those reported for educational outcomes (range 0.15 to 0.25; Hedges & Hedberg, 2007). The fact that the results of this study, and others focused on sexual risk outcomes, differed from those found in studies of other health and education outcomes provides further evidence of the importance of utilizing ICC estimates from studies with similar outcomes. Furthermore, ICC estimates from this study differed depending on the type of sexual risk outcome, the population, and the type of analysis performed, indicating the need to select ICC estimates from studies closely aligned in population, design, primary outcome, and intended analysis method to maximize efficiency when planning GRTs.

In the majority of cases, estimates of ICCs at baseline were larger than those at follow-up time points. This may be because as the studies progressed in time the original student groupings within schools or classrooms shifted because of real-world factors such as attrition and migration, increasing the within-school variation. For two outcomes in the high school studies—condom use and birth control use at last sex—the ICC was appreciably larger at a later follow-up time point. However, for these variables analyses were conducted on filtered samples of only students who were sexually active in the past 3 months—a sample that would likely have changed in composition substantially as students get older. These results suggest that not only should an ICC estimate be from an outcome as closely matched as possible to that of the study being planned, but the specific follow-up time point also should be considered.

For a given study and outcome, the ICC estimated using a repeated measures model appeared to be roughly somewhere in the middle of the ICCs estimated using endpoint models for the individual time points. Again, results from this study further confirm the importance of ensuring that the type of analysis planned is taken into account when selecting ICCs at the design stage of a study.

In most cases, the addition of just the treatment indicator as an independent variable to the means model did not result in appreciable differences to the ICC estimates. This could be because in these studies the intervention did not account for school-level variation. On the other hand, including a set of demographic/risk-related covariates lowered the ICC in the vast majority of cases for both the endpoint and repeated measures analyses. While Murray and Blitstein (2003) noted the importance of understanding the distribution of covariates across the groups being randomized before assuming that the addition of covariates will decrease between-group variance, in the four studies examined here adjustment for a set of covariates associated with the outcome variable was almost always beneficial in terms of reducing the ICC. The one exception was for Study 2—the All4You! study involving county-run alternative schools, for which the ICCs increased appreciably with the addition of the set of covariates, particularly in the repeated measures cases. County-run alternative schools are often used as an option for students who are not succeeding in district-run alternative schools or for students with more serious behavioral issues; they tend to serve very specific communities, such as youth from a certain geographic area, and their relatively homogeneous composition may affect covariate-adjusted ICCs differently than they affect ICCs for other school enviornments.

Limitations

One of the major limitations of this study was the relatively large standard errors seen in the ICC estimates, in part a reflection of the relatively low number of schools available for estimating between-school variation in each study. Additionally, the reliability of the standard error estimates of ICCs is low (Murray et al., 2006). Another limitation of this study is that it obtained estimates of ICCs only for studies that were published; studies for which there were no significant impacts on any sexual behavior outcomes often are not published and may have different ICCs associated with them. However, this limitation is mitigated by the fact that we found ICC estimates were negligibly affected by the inclusion of treatment indicators in the models. Furthermore, we chose to present ICC estimates for the most commonly addressed outcomes in the literature regardless of whether these four programs positively affected those particular outcomes. Finally, we did not look at whether ICCs differed for treatment and control groups through separate subgroup analyses, but the finding that the addition of a treatment indicator variable in the models did not change ICCs suggests they would not differ. Despite these limitations, this study offers the first published data on ICC estimates for school-based HIV/STI/pregnancy prevention interventions, an important contribution to the literature.

Implications for Practice

This study provides one of the first sets of ICC estimates for researchers to use when planning school-based studies involving sexual behavior outcomes among adolescents. ICC estimates differed across studies, outcomes, follow-up time points, and method of analysis, providing further evidence of the importance of selecting ICC estimates that reflect the characteristics of the study being planned.

In general, ICCs ranged from 0 to 0.15, with adjusted ICCs for unprotected sex in the past 3 months—often the target outcome for studies of this type (e.g., replication studies for the Office of Adolescent Health’s Teen Pregancy Prevention Initiative)—estimated at 0.023 or less. With such an ICC, studies able to recruit at least 20 schools could detect a relatively small effect size of d = 0.23 or more. In the vast majority of cases examined across the four studies, controlling for a set of demographic and related covariates reduced the intraclass correlation coefficient; further research needs to be conducted to understand which particular covariates result in the biggest reductions in ICCs.

Footnotes

Authors’ Note

The authors dedicate this work to the memory of Dr. Ronald Harrist, whose warm optimism and practicality affirmed the value of our work.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was supported by the National Institute of Child Health and Human Development (HD# 068173).

Notes

References

Baumler

E. R.

Harrist

R. B.

Carvajal

(2003). Analysis of repeated measures data. In Reise

S. P.

Duan

(Eds.), Multilevel modeling: Methodological advances, issues, and applications (pp. 140-1560). Mahwah, NJ: Lawrence Erlbaum.

Cohen

(1992). A power primer. Psychological Bulletin, 112, 155-159.

Cornfield

(1978). Randomization by group: A formal analysis. American Journal of Epidemiology, 108, 100-102.

Coyle

K. K.

Basen-Engquist

Kirby

Parcel

Banspach

Collins

Harrist

(2001). Safer choices: Reducing teen pregnancy, HIV, & STDs. Public Health Reports, 116(Suppl. 1), 82-93.

Coyle

K. K.

Glassman

J. R.

Franks

H. M.

Campe

S. M.

Denner

Lepore

G. M.

(2013). Interventions to reduce sexual risk behaviors among youth in alternative schools: A randomized controlled trial. Journal of Adolescent Health, 53, 68-78.

Coyle

K. K.

Kirby

D. B.

Marin

B. V.

Gomez

C. A.

Gregorich

S. E.

(2004). Draw the line/respect the line: A randomized trial of a middle school intervention to reduce sexual risk behaviors. American Journal of Public Health, 94, 843-851.

Coyle

K. K.

Kirby

D. B.

Robin

L. E.

Bansbach

S. W.

Baumler

Glassman

J. R.

(2006). All4You! A randomized trial of an HIV, other STDs, and pregnancy prevention intervention for alternative school students. AIDS Education and Prevention, 18, 187-203.

Davis

Bauman

(2013). School enrollment in the United States: 2011: Population characteristics (Census Report No. P20-571). Retrieved from http://www.census.gov/prod/2013pubs/p20-571.pdf

Donner

Birkett

Buck

(1981). Randomization by cluster: Sample size requirements and analysis. American Journal of Epidemiology, 114, 906-914.

10.

Donner

Klar

(2000). Design and analysis of cluster randomization trials in health research. London, England: Arnold.

11.

Feldblum

P. J.

Chen-Mok

Bwayo

J. J.

Omari

Kuyoh

Ryan

K. A.

(1999). Intracluster correlation of STD prevalence in a community intervention trial in Kenya. Lancet, 354, 1356-1357.

12.

Goldstein

(1995). Multilevel statistical models. London, England: Arnold.

13.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60-87.

14.

Hox

(2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.

15.

Kish

(1965). Survey sampling. New York, NY: John Wiley.

16.

Murray

(1998). Design and analysis of group-randomized trials. New York, NY: Oxford University Press.

17.

Murray

D. M.

Birnbaum

D. M.

Phillips

A. S.

Glenn

(2001). Intraclass correlation for measures from a middle school nutrition intervention study: Estimates, correlates, and applications. Health Education & Behavior, 28, 666-679.

18.

Murray

D. M.

Blitstein

J. L.

(2003). Methods to reduce the impact of intraclass correlation in group-randomized trials. Evaluation Review, 27(1), 79-103.

19.

Murray

D. M.

Cattellier

D. J.

Hannan

P. J.

Treuth

M. S.

Stevens

Schmitz

K. H.

Conway

T. L.

(2004). School-level intraclass correlation for physical activity in adolescent girls. Medicine and Science in Sports and Exercise, 36, 876-882.

20.

Murray

D. M.

Short

B. J.

(1996). Intraclass correlation among measures related to alcohol use by school aged adolescents: Estimates, correlates and applications in intervention studies. Journal of Drug Education, 26, 207-230.

21.

Murray

D. M.

Short

B. J.

(1997). Intraclass correlation among measures related to tobacco use by adolescents: Estimates, correlates and applications in intervention studies. Addictive Behaviors, 22(1), 1-12.

22.

Murray

D. M.

Stevens

Hannan

Catellier

D. J.

Schmitz

K. H.

Dowda

Yang

(2006). School-level intraclass correlation for physical activity in sixth grade girls. Medicine and Science in Sports and Exercise, 38, 926-936.

23.

Pals

S. L.

Beaty

B. L.

Posner

S. F.

Bull

S. S.

(2009). Estimates of intraclass correlation for variables related to behavioral HIV/STD prevention in a predominantly African American and hispanic sample of young women. Health Education & Behavior, 36, 182-194.

24.

Rasbash

Steele

Browne

W. J.

Goldstein

(2009). A user’s guide to MLwiN. Bristol, England: Centre for Multilevel Modelling, University of Bristol.

25.

Raudenbush

S. W.

Bryk

A. S.

Cheong

Y. F.

Congdon

R. T.

(2011). Optimal design software for multi-level and longitudinal research (Version 3.01) [Computer software]. Retrieved from http://sitemaker.umich.edu/group-based/home

26.

Resnicow

Zhang

Vaughan

R. D.

Reddy

S. P.

James

Murray

D. M.

(2010). When intraclass correlation coefficients go awry: A case study from a school-based smoking prevention study in South Africa. American Journal of Public Health, 100, 1714-1718.

27.

Rodríguez

Elo

(2003). Intra-class correlation in random-effects models for binary data. The Stata Journal, 3(1), 32-46.

28.

Rotondi

M. A.

Donner

(2009). Sample size estimation in cluster randomized educational trials: An empirical bayes approach. Journal of Educational and Behavioral Statistics, 34, 229-237.

29.

Siddiqui

Hedeker

Flay

B. R.

F. B.

(1996). Intraclass correlation estimates in a school-based smoking prevention study: Outcome and mediating variables, by sex and ethnicity. American Journal of Epidemiology, 144, 425-433.

30.

Snijders

Bosker

(1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.

31.

Spybrook

Bloom

Hill

Martinez

Raudenbush

S. W.

(2011). Optimal design for longitudinal and multilevel research: Documentation for the “Optimal Design” software [Computer software manual]. Ann Arbor: University of Michigan, School of Education, Hierarchical Models Project. Retrieved from http://sitemaker.umich.edu/group-based/home

32.

Varnell

S. P.

Murray

D. M.

Janega

J. B.

Blistein

J. L.

(2004). Design and analysis of group-randomized trials: A review of recent practices. American Journal of Public Health, 94, 393-399.

33.

Zhang

Pals

S. L.

Medley

Nichols

Bachanas

van Zyl

Juma

(2014). Parameters for sample size estimation from a group-randomized HIV prevention trial in HIV clinics in Sub-Saharan Africa. AIDS and Behavior, 18, 2359-2365. doi:10.1007/s10461-013-0631-9