Standardized Mean Differences in Two-Level Cross-Classified Random Effects Models

Abstract

Multilevel modeling techniques are becoming more popular in handling data with multilevel structure in educational and behavioral research. Recently, researchers have paid more attention to cross-classified data structure that naturally arises in educational settings. However, unlike traditional single-level research, methodological studies about multilevel effect size have been rare and those that have recently appeared had an emphasis on strictly hierarchical data structure. This article extends the work on multilevel standardized mean differences from strictly hierarchical structure to both fully and partially cross-classified structures. Analytically derived formulae for calculating effect sizes and the corresponding sampling variances (or standard errors) are presented, verified by simulation results, and illustrated with real data examples. Implications for primary research studies and meta-analyses are discussed.

Keywords

cross-classified random effects model effect size randomized trials multilevel standardized mean differece

The field of educational and psychological science has witnessed a movement from the obsession of statistical significance testing to the evaluation of effect size (Ferguson, 2009). However, for studies with multilevel data, effect size is still underreported, even though some effect size statistics have been recently developed for multilevel data with a nested structure (e.g., standardized mean difference; Hedges, 2007, 2011) and a cross-classified structure (e.g., proportion of variance accounted for; Luo & Kwok, 2010). As cross-classified random effects models (CCREMs) have gained increasing attention in educational research (e.g., Friedel, Cortina, Turner, & Midgley, 2010; Johnson, 2011), there is a need to develop relevant effect size measures for CCREMs. The intent of the present article is to (a) analytically develop the standardized mean difference measure for two-level CCREM, (b) verify the performance of the mathematically derived formulae using simulated data, and (c) illustrate the computation of the effect size statistic with real data example.

The American Educational Research Association (AERA, 2006, p. 37), in its Standards for Reporting on Empirical Social Science Research in AERA Publications, recommended the use of effect size statistics with the rationale that “[i]nterpretation of statistical analyses is enhanced by reporting magnitude of relations.” The American Psychological Association (2010, p. 34), in its Publication Manual, took a stronger stance to deem the reporting of effect size statistics as “almost always necessary.” The attention given to effect size can be attributed to three important advantages of reporting such statistics. First, effect sizes, rather than statistical significance tests, directly answer research questions such as how strong two variables are associated or how effective an intervention is (see Thompson, 2007). Second, to date, effect size is the element to be synthesized in almost all meta-analytic studies (Lipsey & Wilson, 2001). Third, effect size estimation also plays a critical role in research planning, such as power analysis (Cohen, 1988). Two of the commonly reported effect size families include the standardized mean difference (i.e., group difference divided by sample standard deviation [SD] or the d family) and the proportion of variance accounted for (or the r family; Grissom & Kim, 2012). Whereas the effect size statistics in single-level studies are already well developed, effect size in multilevel modeling has appeared only recently and is generally limited to strictly nested data. Therefore, more discussion on this topic is necessary and valuable.

Effect Size in Multilevel Analyses

Although techniques for handling data with cluster structures have been developed for several decades (e.g., Goldstein, 1986; Mason, Wong, & Entwisle, 1983), in the past 10 years, they have gained much more attention in educational and behavioral research. This is not surprising, given that in these fields much data collected have intrinsically nested structure. For example, in the field of education, students are naturally nested within classrooms, and classrooms are naturally nested within schools. Because traditional data analytic techniques ignore the multilevel structure and give incorrect standard errors (SEs; Hox, 2010), new methods are proposed that provide correct SEs and hence accurate statistical inference. One of the most popular approach is multilevel modeling (Goldstein 2011), which is synonyously called hierarchical linear modeling (Raudenbush & Bryk, 2002), linear mixed modeling (Littell, Milliken, Stroup, & Wolfinger, 1996), and other similar names.

Despite the rapid growth in the number of multilevel studies, rarely did researchers utilize effect size statistics in reporting multilevel results. Most of these studies used proportion of variance accounted for, or R ² (see Luo & Kwok, 2010; Snijders & Bosker, 2011). However, for studies with a binary covariate, such as treatment–control or male–female, the standardized mean difference is a more natural choice and is more easily understood by researchers.

In addition to the point estimates of an effect size, its sampling variance (or SE) is important. As commented by Cohen (1994), it is “far more informative to provide a confidence interval” (p. 1310), and the computation of (asymptotic) confidence interval (CI) requires the sampling variance of the effect size. This is particularly important for meta-analysts (Hedges, 1981; Lipsey & Wilson, 2001), because both point and variance estimates of effect size are required to get an overall average effect size and to understand the influence of study-level covariates including publication bias in the literature. Given the importance of the point estimate and the sampling variance of the standardized mean difference effect size, as well as the lack of discussion about them in complex multilevel models, research efforts to supplement methods for their calculation are warranted.

Recently, Hedges (2007, 2009, 2011) made a seminal effort in defining standardized mean difference statistics for data with two-level and three-level nested structures. Particularly, he suggested that, depending on the context, there could be different choices of SDs in computing the effect size. Hedges (2007) illustrated the calculation of effect size in two-level studies with an example about the effect of connected mathematics in classrooms. In that example, students were nested within classrooms, and the treatment (i.e., connected mathematics) was defined at the classroom level (i.e., Level 2). He showed that the overall effect size was 0.15 (95% CI [0.29, 0.59]) and the within-classroom effect size was 0.17 (95% CI [0.34, 0.69]). In a three-level cluster-randomized design, five possible effect size statistics can be computed depending on which variance component is invoked. The formulae given by Hedges (2007, 2009, 2011) do not require researchers to have the raw data to obtain an effect size estimate; instead, only the estimated treatment effect (i.e., grand mean difference between the treatment and the control arm), sample sizes for all levels of clustering, and the corresponding intraclass correlations (ICCs) are needed. In the context of meta-analysis, Ahn, Myers, and Jin (2012) have suggested methods to estimate ICCs when the original research report does not include the relevant information.

CCREMs

The number of published articles adopting the CCREM method, a more complicated structure than nested multilevel models, has increased dramatically in recent years. A simple search in the Educational Research Information Center database with the keyword “cross-classified” found only three articles during 2000–2005 but 32 articles during 2006–2012. One reason for the increasing adoption of CCREM is that multilevel data may not always have a strictly hierarchical structure. A typical example is given by Beretvas (2010), where students are nested within both primary schools (PSs) and high schools (HSs), but PS is not nested within HS nor vice versa. That is, not all students in one HS come from the same PS, nor do all students from one PS go to the same HS. In this case, PS and HS are labeled as crossed factors. If both PS and HS are assumed to be random effects, then CCREM can be used to analyze such kind of data. Luo and Kwok (2010) have discussed the R ² effect size for CCREMs. However, to the best of our knowledge, no discussion has taken place about standardized mean difference for CCREMs. Standardized mean difference would be suitable, for instance, in describing the effects of a school-based intervention on students’ learning, where students are nested in both schools and neighborhoods. Based on the framework of previous studies (Hedges, 2007, 2011), in this article, we develop effect sizes for CCREMs through mathematical derivation and evaluate their performances using both simulated and real data sets.

The purpose of this article is to analytically develop the standardized mean difference measure for two-level CCREMs for both balanced and unbalanced designs and to verify the performance of the mathematically derived formulae. Because of the complexity of the formulae, we also provide real data examples for pedagogical purposes, so that applied researchers can better understand how those formulae can apply to their research. In the following sections, we would (a) briefly introduce the notations for a two-level CCREM with two crossed factors; (b) discuss two estimation approaches to obtain the standardized mean difference, D, and the corresponding sampling variance, V(D) (where $V (\cdot)$ denotes the variance operator), for balanced design CCREMs; (c) empirically verify their performance through simulations; and (d) illustrate their calculations with real data having a cross-classified structure. The discussion of partially CCREMs (PCCREMs) included the same elements.

Model and Notation

In a balanced design with the cross-classification of two random effects A and B at Level 2, let J and K be the number of clusters in effect A and in effect B, respectively. In the context of education, A can be classrooms in a school and B can be neighborhoods. To make things more concrete, in the following sections, we would use an hypothetical example where effect A is the classroom effect and effect B is the neighborhood effect, although the notation is equally applicable to other contexts such as therapy grouping effect by classroom effect or in longitudinal settings with person effect by time effect. As a result, there are J classrooms and K neighborhoods, and J × K combinations of classroom and neighborhood or J × K cells. Further, let $n_{j k} = n$ be the number of students in each cell with index $i = 1, \dots, n$ . In addition, assume that classrooms are randomly assigned to treatment condition or control condition. Because the word group can refer to either people in one of the treatment conditions or people from one of the classroom, to avoid confusion, in subsequent discussions, the group receiving treatment is referred to as the treatment arm, whereas the group in the control condition is referred to as the control arm (Bauer, Sterba, & Hallfors, 2008). For example, a researcher can randomly assign half of the classrooms to adopt a new reading instruction and the other half to use the traditional approach. Thus, classrooms are nested within treatment arms, but neighborhoods and treatment arms are crossed. In this case, students from the same classroom must have the same treatment status, whereas students from the same neighborhood can have different treatment statuses if they come from different classrooms.

Let $j = 1, \dots, J^{T}$ and $j = 1, \dots, J^{C}$ be the index of classroom for the treatment (T) and the control (C) arms where $J^{T} + J^{C} = J$ , and $k = 1, \dots, K^{T}$ and $k = 1, \dots, K^{C}$ be the index of neighborhood. Furthermore, the sets of neighborhoods in the treatment arm and in the control arm may be completely overlapped, partially overlapped, or completely separated. Let $K_{o v e r l a p}$ be the number of overlapping clusters, and the three possible situations are then $K_{o v e r l a p} = K^{T} = K^{C} = K$ (complete overlapping), $K_{o v e r l a p} = (K^{T} + K^{C}) - K > 0$ (partial overlapping), or $K_{o v e r l a p} = 0$ and $K = K^{T} + K^{C}$ (complete separation). Therefore, the sample size for the treatment group is $N^{T} = J^{T} \times K^{T} \times n$ , that for the control group is $N^{C} = J^{C} \times K^{C} \times n$ , and the total sample size is $N = N^{T} + N^{C}$ . The model can then be specified as

Y_{i j k} = γ_{00} + γ_{10} ({T R E A T}_{j}) + μ_{0 j} + ν_{0 k} + ε_{i j k},

where $Y_{i j k}$ refers to the score of the ith student in the jth classroom and the kth neighborhood, and ${T R E A T}_{j}$ the treatment status variable dummy coded as 0 (control) and 1 (treatment). γ₀₀ is the grand mean of the control arm in the sample, γ₁₀ is the mean difference between the treatment arm and the control arm, μ_0j is the magnitude of the effect of the jth classroom, ν_0k is the magnitude of the effect of the kth neighborhood, and ∊ _ijk is the within-cell residual (i.e., the student effect). Usually, researchers do not estimate the interaction effect between random effects for simplicity (Shi, Leite, & Algina, 2010). Also, following Hedges (2007), it is assumed that the treatment effect does not interact with random effects A and B.

In a balanced design, the variance of Y can be partitioned into three independent components, which are denoted as $σ_{W}^{2}$ , the within cluster variance; $σ_{A}^{2}$ , the classroom-level variance or the variance due to the random effect A; and $σ_{B}^{2}$ , the neighborhood-level variance or the variance due to the random effect B. There are several methods to obtain an estimate of these variance components, such as the analysis of variance method, full maximum likelihood, and restricted maximum likelihood (REML; Searle, Casella, & McCulloch, 2006). As discussed in later sections, obtaining estimates of these variance components is the key to computing an effect size.

ICC

The ICC quantifies the degree to which two randomly drawn observations within a cluster are correlated. In CCREMs, there are different possible ICCs depending on how a cluster is defined. For instance, for observations in the same classroom (random effect A) but in different neighborhoods (random effect B), the ICC can be defined as follows:

ρ_{A} = \frac{σ_{A}^{2}}{σ_{A}^{2} + σ_{B}^{2} + σ_{W}^{2}} = \frac{σ_{A}^{2}}{σ_{T}^{2}},

where $σ_{T}^{2} = σ_{A}^{2} + σ_{B}^{2} + σ_{W}^{2}$ . Similarly, for observations in the same neighborhood but in different classrooms, the ICC can be defined as

ρ_{B} = \frac{σ_{B}^{2}}{σ_{A}^{2} + σ_{B}^{2} + σ_{W}^{2}} = \frac{σ_{B}^{2}}{σ_{T}^{2}} .

Standardized Mean Differences for Fully Cross-Classified Data

In educational research, the standardized mean difference is defined as the ratio of (a) the difference between the population means of the treatment arm and of the control arm to (b) an SD. Hedges (2009) defined different effect sizes associated with different levels. For example, a researcher may be interested in how an intervention is effective in group level and can use only the between-level SD, while ignoring the within-group variations. Similarly, in CCREM, one can consider using σ _W , σ _A , σ _B , $\sqrt{σ_{W}^{2} + σ_{A}^{2}}$ , $\sqrt{σ_{W}^{2} + σ_{B}^{2}}$ , or σ _T . Perhaps the issue can be made simpler by reminding that in single-level studies, standardized mean difference between the treatment and the control arms can be converted to an R ² effect size due to the binary treatment dummy variable (e.g., 0 = control, 1 = treatment). Because in education treatment is usually a variable at the second or higher level, as in the classroom–neighborhood example where treatment is on the classroom level, generally the treatment will not explain within-level (i.e., student level) variance (Snijders & Bosker, 1994). Therefore, in our opinions, $σ_{W}^{2}$ is in general not justified unless one assumes that the treatment effect stays the same, whether it is invidually randomized or cluster randomized. For meta-analysts, the decision often depends on the nature of the other studies. If there are single-site studies in the list, generally choosing variance components of classroom or neighborhood levels makes the comparison in meta-analysis difficult (Hedges, 2007). Because in education often data are cross-classified (e.g., Beretvas, 2010), and researchers are usually interested in generalizing the effect to a broader population of students (or other Level 1 units), σ _T is a more natural choice. Thus, in subsequent mathematical derivation, we focused on using σ _T .

On the population level, the effect size is defined as

δ_{T} = \frac{μ_{∙ ∙ ∙}^{T} - μ_{∙ ∙ ∙}^{C}}{σ_{T}},

where $μ_{∙ ∙ ∙}^{T}$ and $μ_{∙ ∙ ∙}^{C}$ are the population means of the treatment and of the control arm, respectively.

In a balanced design, the average of the cell means, ${\overset{ˉ}{Y}}_{∙ ∙ ∙}^{T} = \sum {\overset{ˉ}{Y}}_{∙ j k}^{T} / (J^{T} K)$ and ${\overset{ˉ}{Y}}_{∙ ∙ ∙}^{C} = \sum {\overset{ˉ}{Y}}_{∙ j k}^{C} / (J^{C} K)$ , are unbiased and efficient estimators of $μ_{∙ ∙ ∙}^{T}$ and $μ_{∙ ∙ ∙}^{C}$ . Thus, the difference between the two averaged cell means is an unbiased and efficient estimator of the numerator of δ _T . However, the observed total variance

S_{T}^{2} = \frac{\sum_{k = 1}^{K^{T}} \sum_{j = 1}^{J^{T}} \sum_{i = 1}^{n} {(Y_{i j k} - {\overset{ˉ}{Y}}_{∙ ∙ ∙}^{T})}^{2} + \sum_{k = 1}^{K^{C}} \sum_{j = 1}^{J^{C}} \sum_{i = 1}^{n} {(Y_{i j k} - {\overset{ˉ}{Y}}_{∙ ∙ ∙}^{C})}^{2}}{N - 2}

is in general a biased estimator of the total population variance $σ_{T}^{2}$ in random effect models. We would present two methods to obtain a consistent estimate of the population effect size δ _T , one by multiplying a correction factor to S_T and the other by utilizing the computer estimates of the variance components to obtain S_T . The first method is based on the expected mean squares of the random effects, and the estimated effect size is denoted as D ₁ in this article. It is both efficient and consistent on balanced data structure where cells have (roughly) equal size and will be useful for meta-analysts when the primary research studies did not present estimates of σ _T . As shown later in the simulation results, it is also robust to unbalanced design. The second method is based on the estimated variance components of the random effects, and the estimated effect size is denoted as D ₂. It is efficient for both balanced and unbalanced data and is easier to compute than the first method, provided that the point and variance (or SE) estimates of variance components are available. It will be useful for both researchers working with primary data and meta-analysts having access to the required information.

Estimation of D ₁

With a balanced data structure assumed, and when the sets of clusters of random effect B in the treatment arm and in the control arm overlap completely (e.g., students receiving treatment come from the same set of neighborhoods as those in the control arm), the sample estimator of δ _T , D ₁, and the corresponding sampling variance V(D ₁) are

D_{1} = \frac{{\overset{ˉ}{Y}}_{∙ ∙ ∙}^{T} - {\overset{ˉ}{Y}}_{∙ ∙ ∙}^{C}}{S_{T}} \sqrt{1 - \frac{2 (K n - 1) ρ_{A} + (J n - 2) ρ_{B}}{N - 2}},

and

\begin{aligned} V (D_{1}) = \frac{1}{\tilde{N}} [1 + (K n - 1) ρ_{A}] + D_{1}^{2} [K n (N - 2 K n) ρ_{A}^{2} \\ + J n (N - J n) ρ_{B}^{2} + (N - 2) {\overset{ˉ}{ρ}}^{2} + 2 (N - 2 K n) {\overset{ˉ}{ρ} ρ}_{A} \\ + 2 (N - J n) {\overset{ˉ}{ρ} ρ}_{B}] / \{2 {[(N - 2) - 2 (K n - 1) ρ_{A} - (J n - 2) ρ_{B}]}^{2}\}, \end{aligned}

where $\tilde{N} = N^{T} N^{C} / (N^{T} + N^{C})$ and $\overset{ˉ}{ρ} = 1 - ρ_{A} - ρ_{B}$ . See Appendix A for detailed derivations (see the online Appendix, available at http://jeb.sagepub.com/supplemental). On the other hand, if the sets of clusters of random effect B in the treatment arm are different to those in the control arm (e.g., students from certain neighborhoods are all in the treatment arm and students from some other neighborhoods are all in the control arm), the approximated sampling variance V(D) is given by

\begin{aligned} V (D_{1}) = \frac{1}{\tilde{N}} [1 + (K n - 1) ρ_{A} + (1 - r_{K}) (J n - 2) ρ_{B}] + D_{1}^{2} [K n (N - 2 K n) ρ_{A}^{2} + J n (N - J n) ρ_{B}^{2} + (N - 2) {\overset{ˉ}{ρ}}^{2} + 2 (N - 2 K n) {\overset{ˉ}{ρ} ρ}_{A} + 2 (N - J n) {\overset{ˉ}{ρ} ρ}_{B}] / \{2 {[(N - 2) - 2 (K n - 1) ρ_{A} - (J n - 2) ρ_{B}]}^{2}\}, \end{aligned}

where $r_{K} = \sqrt{(K_{o v e r l a p})^{2} / (K^{T} \times K^{C})}$ is the correlation of the random effect B between the treatment and the control arm, K^T and K^C are the numbers of effect of B clusters specific to the treatment and the control arm, and K _overlap is the number of overlapping clusters. Note that K is now defined as the total number of B-clusters such that K = K^T + K^C − K _overlap. Equations 6 –8 outline the influence of cluster size, the number of clusters, and the ICCs on the effect size estimates and its sampling variance.

Estimation of D ₂

The derivation of D ₁ is based on assumptions that (a) the cluster size is constant and (b) the ICCs are known or estimated with a reasonable accuracy. In real research, these assumptions may not hold. If either (a) or (b) or both (a) and (b) are violated, then D ₁ and V(D ₁) calculated from Equations 6 and 7 can be biased and inefficient. For unbalanced designs, the close forms of D ₁ and V(D ₁) are very complex and are functions of the cell sizes in addition to the components in Equations 6 and 7. Because information about the cell sizes is rarely available from published research reports, it is difficult to obtain efficient estimates of δ _T and V(δ_T) for unbalanced data starting from expected mean squares. However, if consistent estimates of the variance components (from maximum likelihood, REML, or Bayesian estimation, etc.) are available, researchers can use both the point estimates and the SEs of the random effects to calculate the effect size. Specifically, if estimates of the treatment effect and the variance components, ${\hat{γ}}_{10}$ , ${\hat{σ}}_{W}^{2}$ , ${\hat{σ}}_{A}^{2}$ , ${\hat{σ}}_{B}^{2}$ , and their corresponding variances (the squared values of the SEs), $V ({\hat{γ}}_{10})$ , $V ({\hat{σ}}_{W}^{2})$ , $V ({\hat{σ}}_{A}^{2})$ , and $V ({\hat{σ}}_{B}^{2})$ can be obtained, then we get

D_{2} = \frac{{\hat{γ}}_{10}}{\sqrt{{\hat{σ}}_{W}^{2} + {\hat{σ}}_{A}^{2} + {\hat{σ}}_{B}^{2}}},

V (D_{2}) = \frac{V ({\hat{γ}}_{10})}{{\hat{σ}}_{W}^{2} + {\hat{σ}}_{A}^{2} + {\hat{σ}}_{B}^{2}} + \frac{D_{2}^{2} [V ({\hat{σ}}_{W}^{2}) + V ({\hat{σ}}_{A}^{2}) + V ({\hat{σ}}_{B}^{2})]}{4 {({\hat{σ}}_{W}^{2} + {\hat{σ}}_{A}^{2} + {\hat{σ}}_{B}^{2})}^{2}} .

Derivations of Equations 9 and 10 can be found in Appendix A (see the online Appendix, available at http://jeb.sagepub.com/supplemental).

If meta-analysts can obtain neither information about the degree of imbalance of the data nor unbiased estimates of the variance components and their variances, the best they can do is to compute the effect size assuming a balanced design and replace n in Equations 6 and 7 with the average cell size, N/(JK), to obtain D ₁. If information about ICC is not obtainable, they can put in a reasonable guess of ρ _A and ρ _B by referring to research with similar designs and variables. There are also several articles summarizing what typical ICCs are for various designs and areas (Hedges & Hedberg, 2007; Murray & Blitstein, 2003).

Monte Carlo Study for Evaluating the Two Effect Size Estimation Approaches Under Unbalanced Designs

We used a 3 × 2 × 2 × 2 full factorial simulation study to empirically check the performance of D ₁ and D ₂ with unbalanced designs. The design factors included (a) the population effect size (δ _T = 0.2, 0.5, or 0.8), (b) the number of clusters in random effect A (e.g., classrooms) per treatment status (J^T = J^C =20 or 50), (c) the average cell size (n = 0.25 or 1), and (d) the ICC of random effect $A$ (ρ _A = .10 or .25, which are common values used in previous simulation studies). In the first simulation, we generated data such that the treatment arm and the control arm share the same set of neighborhoods. In other words, K^T = K^C = K = K _overlap. K was set to equal to J^T and J^C , and ρ _B was fixed to .1. The imbalance of data structure was similar across conditions in which 20% of the cells had an expected cell count that was 10 times larger than that of the remaining 80% of cells (see Appendix B, available at http://jeb.sagepub.com/supplemental, for more details). Such a data structure is similar to that described in Beretvas (2008), where students are nested within the cross-classification of schools and neighborhoods. Across conditions Pearson’s contingency coefficients ranged from .84 to .92, showing that the degree of imbalance was quite strong. R 3.0.1 (R Core Team, 2013) was used to generate 500 data sets for each conditions, with $μ_{∙ ∙ ∙}^{C} = 0$ and $σ_{W}^{2} = 1.0$ . All random effects were normally distributed.

The estimation of D ₁ and V(D ₁) was performed in R with ρ _A and ρ _B being fixed to the population value. On the other hand, D ₂ and V(D ₂) were obtained using MODEL CONSTRAINT with the TYPE=CROSSCLASSIFIED procedure in Mplus 7.0 (Muthén & Muthén, 1998–2012) with the default noninformative prior Bayesian estimation.

As shown in Table 1, both effect size estimators had relative bias less than 5%. The relative SE bias was stronger, but the impact was small as the percentage coverage of the 95% symmetric CI¹ was close to nominal value and ranged from 93.6% to 95.6%. This is within the expected interval [92.7%, 96.6%] when the true coverage is 95% with 500 replications, and thus, we conclude that the performance of both estimators was satisfactory. As expected, D ₁ was less efficient under unbalanced structure, but the loss of efficiency was little. Specifically, the relative efficiency, RE = V(D ₂)/V(D ₁), of D ₁ was lowest when both J^T and n were small and when ρ _A was large, but it was still acceptable as RE = 86%. In summary, both approaches to obtain δ _T and V(δ _T ) performed well in unbalanced designs.

Table 1

Simulation Results for Unbalanced CCREMs (With Clusters in Random Effect B Completely Overlapped)

				Coverage		Relative Bias		Relative SE Bias
δ _T	J^T	N^T	ρ _A	D ₁	D ₂	D ₁	D ₂	D ₁	D ₂	RE(D ₁, D ₂)
0.2	20	100	.10	94.8	96.0	4.3	1.9	−5.8	3.5	89.2
			.25	94.4	94.8	4.4	−4.1	−9.1	0.3	87.2
		400	.10	96.4	94.2	−2.9	−0.7	2.2	2.2	95.0
			.25	94.8	93.8	−3.2	1.3	0.1	−6.1	93.6
	50	625	.10	94.0	93.0	−0.9	−3.4	−7.4	−8.2	93.0
			.25	93.2	94.0	−1.9	−3.5	−10.1	−3.4	91.2
		2,500	.10	94.6	92.4	0.4	0.3	−0.1	−14.2	97.1
			.25	94.0	91.4	0.2	2.2	−1.3	−20.2	96.4
0.5	20	100	.10	95.0	95.4	2.0	−1.2	−7.3	2.3	89.5
			.25	94.2	94.2	2.1	−3.7	−10.0	−0.1	87.4
		400	.10	96.0	95.0	−1.1	−1.7	2.8	4.0	95.1
			.25	94.8	93.4	−1.1	−1.1	0.1	−5.3	93.6
	50	625	.10	94.4	93.4	−0.3	−1.7	−5.9	−6.1	93.1
			.25	93.4	94.0	−0.6	−1.8	−8.9	−1.8	91.4
		2,500	.10	94.8	92.2	0.3	−0.0	−0.7	−14.1	97.1
			.25	94.2	91.8	0.3	0.6	−1.4	−19.3	96.1
0.8	20	100	.10	94.6	94.6	1.4	−2.0	−9.0	1.0	90.0
			.25	93.8	93.8	1.5	−3.6	−11.0	−0.6	87.8
		400	.10	96.2	95.0	−0.6	−1.9	3.2	5.5	95.5
			.25	95.2	93.4	−0.6	−1.7	−0.1	−4.3	93.7
	50	625	.10	94.4	93.6	−0.1	−1.3	−4.4	−3.8	93.1
			.25	94.2	94.2	−0.3	−1.4	−7.7	0.7	90.9
		2,500	.10	94.8	92.4	0.2	−0.2	−1.4	−13.4	97.1
			.25	94.0	92.6	0.3	0.1	−1.6	−18.7	96.5

Note. Based on 500 replications for each condition. Pearson’s contingency coefficients for all conditions ranged from .84 to .92, indicating strong associations between the clusterings of random effects A and B. δ _T = population effect size. J^T = J^C = number of clusters of random effect A in the treatment (control) arm; Number of clusters of random effect B = J^T . N^T = N^C = total sample size of the treatment (control) arm. ρ _A = intraclass correlation of effect A; ρ _B = 0.1 for all conditions. Coverage refers to the percentage of replications in which the 95% confidence interval includes δ _T . For 500 replications, the Monte Carlo coverage percentage have a confidence interval of [92.7%, 96.6%] if the true coverage percentage is 95%. RE(D ₁, D ₂) = relative efficiency of estimator D ₁ to the Mplus (version 7.0) estimation of D ₂ using TYPE=CROSSCLASSIFIED and ESTIMATOR=BAYES, which was computed by dividing the sampling variance of D ₂ by that of D ₁.

Next, we generated data with K _overlap > 0. We kept K = J^T = J^C , but set K^T = K^C = 3J^T /5 and so K _overlap = K/5. In other words, 20% of the clusters in random effect B were overlapped between the treatment and the control arms, with a correlation approximately equal to $r_{k} = \sqrt{(1 / 5)^{2} / [(3 / 5) \times (3 / 5)]} \approx 0.33$ . The results of the simulation are shown in Table 2. For all conditions, the coverage of 95% CI of both D ₁ and D ₂ was acceptable and ranged from 91.8% to 97.4%, which is not too far away from the expected CI when the true coverage is 95%. However, compared to occasions with complete overlapping, here the RE of D ₁ relative to D ₂ dropped to 70.8% to 87.9%, indicating that D ₁ is substantially less efficient than D ₂. Contrary to the pattern in occasions with sets of clusters completely overlapped, here RE was lowest when n was large and ρ _A was small.

Table 2

Simulation Results for Unbalanced CCREMs (With Clusters in Random Effect B Partially Overlapped)

				Coverage		Relative Bias		Relative SE Bias
δ _T	J^T	N^T	ρ _A	D ₁	D ₂	D ₁	D ₂	D ₁	D ₂	RE(D ₁, D ₂)
0.2	20	100	.10	92.6	94.4	3.2	−5.4	−9.4	−0.7	81.0
			.25	93.8	92.6	5.0	−1.4	−9.6	−11.5	81.6
		400	.10	95.8	97.4	1.9	−5.6	9.9	18.8	72.5
			.25	96.0	95.0	3.7	−9.2	9.5	6.4	77.5
	50	625	.10	94.4	94.2	0.6	−10.2	1.4	11.7	83.7
			.25	93.4	93.0	2.5	−10.2	−7.2	3.0	87.2
		2,500	.10	94.2	93.4	0.9	−8.4	1.0	−3.1	70.8
			.25	93.8	92.8	−0.1	−15.7	−5.4	−17.0	80.4
0.5	20	100	.10	92.8	94.4	1.5	−5.8	−8.6	0.5	80.8
			.25	94.0	93.4	2.4	−4.1	−8.7	−9.6	81.0
		400	.10	96.0	97.2	0.9	−3.9	10.0	19.6	72.8
			.25	95.8	95.0	1.7	−5.8	9.4	7.0	77.6
	50	625	.10	94.6	94.0	0.3	−4.9	2.3	13.2	84.3
			.25	93.2	93.0	1.1	−4.7	−7.3	3.2	87.7
		2,500	.10	94.6	93.2	0.5	−3.7	0.8	−2.7	71.0
			.25	94.0	91.8	0.1	−6.8	−5.5	−16.4	80.7
0.8	20	100	.10	93.0	94.0	1.0	−6.0	−7.7	1.9	81.0
			.25	93.6	94.0	1.8	−4.8	−7.8	−7.5	80.8
		400	.10	95.6	96.8	0.7	−3.5	9.9	20.4	73.5
			.25	95.6	94.8	1.1	−5.0	9.2	7.3	78.3
	50	625	.10	95.2	94.0	0.3	−3.6	3.1	14.3	85.1
			.25	93.0	92.6	0.8	−3.3	−7.4	3.8	87.9
		2,500	.10	94.6	93.0	0.4	−2.5	0.5	−2.5	71.4
			.25	94.0	91.8	0.2	−4.7	−5.6	−15.7	81.0

Note. Based on 500 replications for each condition. Pearson’s contingency coefficients for all conditions ranged from .84 to .92, indicating strong associations between the clusterings of random effects A and B. δ _T = population effect size. J^T = J^C = number of clusters of random effect A in the treatment (control) arm. Total number of clusters of random effect B = J^T , with 20% of the sets of clusters in B overlapping across treatment arms. N^T = N^C = total sample size of the treatment (control) arm. ρ_A = intraclass correlation of effect A; ρ _B = 0.1 for all conditions. Coverage refers to the percentage of replications in which the 95% confidence interval includes δ _T . For 500 replications, the Monte Carlo coverage percentage have a confidence interval of [92.7%, 96.6%] if the true coverage percentage is 95%. RE(D ₁, D ₂) = relative efficiency of the estimator D ₁ to the Mplus (version 7.0) estimation of D ₂ using TYPE=CROSSCLASSIFIED and ESTIMATOR=BAYES, which was computed by dividing the sampling variance of D ₂ by that of D ₁.

Real Data Example

We would use part of the National Educational Longitudinal Study data set (NELS:88; Ingles et al., 1990) to illustrate the calculation of D ₁ and D ₂. This longitudinal study followed a nationally representative sample of students starting from their eighth grade and recorded students’ experiences in a variety of areas such as home and working. A hypothetical research question is whether availability of a mathematics club in middle school (MS) predicted students’ mathematics achievement at 10th grade. Using only cases with complete data on all the variables related to the analysis, the data consisted of 15,611 students cross-classified by 986 MSs (383 with math club) and 1,418 HSs (K^T = 625, K^C = 940, so K _overlap = 147). The average cell size was thus 15,611/(986 × 1,418 = 0.0112). Only 147 HS had both students from MSs with math club and those from MSs without math club, so correlation r_K of the HS effect was $\sqrt{147^{2} / (625 \times 940)} = .192$ for the two treatment arms. Using the Statistical Package for the Social Sciences (SPSS) mixed procedure, we estimated the grouping effect of availability of mathematics club, the variance components for within cluster, MS (random effect A), and HS (random effect B), as well as their SEs. The grouping effect was estimated as 0.731 (SE = 0.362), and the variance components were estimated as ${\hat{σ}}_{A}^{2} = 18.784$ (SE = 1.781), ${\hat{σ}}_{B}^{2} = 7.293$ (SE = 1.445), and ${\hat{σ}}_{W}^{2} = 76.296$ (SE = 0.902). Using Equations 9 and 10, the effect size D ₂ for the grouping effect is 0.0722 (SE = 0.0358, 95% CI [0.002, 0.142]), indicating a small effect size. The estimated value of D ₁ (with the sample estimated ρ _A = 0.183 and ρ _B = 0.0712) is 0.116 (SE = 0.0338, 95% [0.050, 0.182]). Under such an extreme unbalanced data structure, D ₂ is expected to be more accurate than D ₁, although the difference is not truly substantial when the 95% CI is also taken into account.

Standardized Mean Differences for Partially Cross-Classified Data

Thus far, we have considered cross-classified data, where observations in both the treatment and the control arms are cross-classified by effects A and B. However, there are designs where only observations in the treatment arm are cross-classified, but the observations in the control arm are nested only in effect A but not in effect B. In this article, we denote such a data structure as partially cross-classified. This is similar to the partially nested design discussed in Bauer, Sterba, and Hallfors (2008), where the observations in the treatment arm are nested within random effect A, whereas those in the control arm are not. The difference is that in partially cross-classified data, there is one more level of nested structure, random effect B, present in both the treatment and the control arms.

Consider a hypothetical example, where students from J^T classrooms are randomly assigned to K emotion management groups (i.e., the treatment), and those from other J^C classrooms did not receive any treatment. Further, assume that each emotion management group includes students from different classrooms to avoid situations where group members are very familiar with each other. Suppose that a researcher is interested in the effectiveness of the emotion management group on students’ life satisfaction (Y). Such a design can be represented by the model equation

Y_{i j k} = γ_{00} + γ_{10} ({T R E A T}_{j}) + μ_{0 j} + ν_{0 k} ({T R E A T}_{j}) + ε_{i j k},

where ${T R E A T}_{j}$ is the treatment status dummy coded as 0 (control) and 1 (treatment), μ_0j is the classroom effect, ν_0k is the emotion management grouping effect that is only present in the treatment arm, and ∊ _ijk is the student effect. γ₀₀ is the grand mean of Y of the control arm, and γ₁₀ is the treatment effect. Further assume that both treatment conditions share the same total variance $σ_{T}^{2}$ and the same variance of effect A $σ_{A}^{2}$ . The variance of effect B in the treatment arm is $σ_{B}^{2}$ , the within-cell variance of the treatment arm is $σ_{W | T R E A T}^{2}$ and that of the control arm is $σ_{W | C O N}^{2}$ . Let $n_{j k}^{T}$ be the size of the cells in the treatment arm and $n_{j}^{C}$ be the cluster size in the control arm. We denote such a model as a PCCREM. Finally, we assume that A and B have no interaction, and ρ _A and ρ _B are defined the same way as in Equations 2 and 3.

Estimation of D ₁

As shown in Appendix A (see the online Appendix, available at http://jeb.sagepub.com/supplemental), the sample estimators D ₁ and V(D ₁) of the effect size δ _T are given as

D_{1} = \frac{{\overset{ˉ}{Y}}_{∙ ∙ ∙}^{T} - {\overset{ˉ}{Y}}_{∙ ∙ ∙}^{C}}{{\overset{ˉ}{S}}_{T}^{2}} \sqrt{\frac{W^{T} β^{T} + W^{C} β^{C}}{W^{T} + W^{C}}},

V (D_{1}) = \frac{1 + (K n^{T} - 1) ρ_{A} + (J^{T} n^{T} - 1) ρ_{B}}{N^{T}} + \frac{1 + (n^{C} - 1) ρ_{A}}{N^{C}} + \frac{D^{2}}{2 (W^{T} β^{T} + W^{C} β^{C})},

where ${\overset{ˉ}{S}}_{T}^{2} = (W^{T} S_{T | T R E A T}^{2} + W^{C} S_{T | C O N}^{2}) / (W^{T} + W^{C})$ is the weighted average of the total variances of the treatment arm, $S_{T | T R E A T}^{2}$ , and of the control arm, $S_{T | C O N}^{2}$ , with weights

\begin{array}{l} W^{T} = {(N^{T} - 1)}^{2} [K n^{T} (N^{T} - K n^{T}) ρ_{A}^{2} + J^{T} n^{T} (N^{T} - J^{T} n^{T}) ρ_{B}^{2} + (N^{T} - 1) {\bar{ρ}}^{2} + 2 (N^{T} - K n^{T}) \bar{ρ} ρ_{A} + 2 (N^{T} - J^{T} n^{T}) \bar{ρ} ρ_{B}]^{- 1}, \\ W^{C} = {(N^{C} - 1)}^{2} {(N^{C} - 1) - 2 (n^{C} - 1) ρ_{A} + (n^{C} - 1) [N^{C} - (n^{C} - 1)] ρ_{A}^{2}}^{- 1}, \end{array}

where $\overset{ˉ}{ρ} = 1 - ρ_{A} - ρ_{B}, a n d$

\begin{array}{c} β^{T} = 1 - \frac{(K n^{T} - 1) ρ_{A} + (J^{T} n^{T} - 1) ρ_{B}}{N^{T} - 1}, \\ β^{C} = 1 - \frac{(n^{C} - 1) ρ_{A}}{N^{C} - 1} . \end{array}

Estimation of D ₂

When maximum likelihood or other unbiased estimates of the fixed effect, the variance components, and their sampling variances are available, one can calculate the standardized mean difference and its sampling variance as

D_{2} = \frac{{\hat{γ}}_{10}}{\sqrt{{\hat{σ}}_{T}^{2}}},

V (D_{2}) = \frac{{V (\hat{γ}}_{10})}{{\hat{σ}}_{T}^{2}} + \frac{D_{2}^{2} [V ({\hat{σ}}_{T}^{2})]}{4 {({\hat{σ}}_{T}^{2})}^{2}},

where ${\hat{σ}}_{T}^{2}$ is the weighted average of the total estimated variances of the two treatment arms by their respective sampling variances (i.e., ${\hat{σ}}_{T | T R E A T}^{2}$ and ${\hat{σ}}_{T | C O N}^{2}$ , with ${\hat{σ}}_{T | T R E A T}^{2} = {\hat{σ}}_{W | T R E A T}^{2} + {\hat{σ}}_{A}^{2} + {\hat{σ}}_{B}^{2}$ for the treatment arm and ${\hat{σ}}_{T | C O N}^{2} = {\hat{σ}}_{W | C O N}^{2} + {\hat{σ}}_{A}^{2}$ for the control arm), and $V ({\hat{σ}}_{T}^{2})$ is given by

\frac{1}{{[V ({\hat{σ}}_{W | T R E A T}^{2}) + V ({\hat{σ}}_{A}^{2}) + V ({\hat{σ}}_{B}^{2})]}^{- 1} + {[V ({\hat{σ}}_{W | C O N}^{2}) + V ({\hat{σ}}_{A}^{2})]}^{- 1}} .

If the weighted average of the two variance components is difficult to obtain, researchers can replace ${\hat{σ}}_{T}^{2}$ by ${\hat{σ}}_{W | C O N}^{2} + {\hat{σ}}_{A}^{2}$ , which results in some loss of efficiency, but the loss is in general minor unless the sample size of the treatment arm is much larger than that of the control arm. Derivations of Equations 12 and 13 can be found in Appendix A (see the online Appendix, available at http://jeb.sagepub.com/supplemental).

Monte Carlo Study for Evaluating the Two Effect Size Estimation Approaches Under Unbalanced Designs

Similar to what we did for fully cross-classified designs, we used simulations to empirically check the performance of D ₁ and D ₂ for PCCREMs. The simulation conditions were the same as those used for the previous CCREM simulation study except that the random effect B was not present in the control arm, and the associated variances were added to $ρ_{W | C O N}$ . For each condition, 500 data sets were generated in R, where D ₁ for each condition was also computed. For the calculation of D ₂, the PROC MIXED (Littell et al., 1996) procedure in SAS 9.3 was used.

As shown in Table 3, both D ₁ and D ₂ have relative bias less than 5% and coverage percentage of the 95% CI ranging from 92.8% to 95.2%, so their performances are satisfactory. In general, D ₁ was less efficient under unbalanced structure when ρ _A was large and the average cell size was small, but the loss of efficiency was negligible (minimum relative efficiency being 96.4%). In summary, both approaches to estimate the effect size δ _T and V(δ _T ) perform similarly well in the chosen unbalanced designs.

Table 3

Simulation Results for Unbalanced PCCREMs

				Coverage		Relative Bias		Relative SE Bias
δ _T	J^T	N^T	ρ _A	D ₁	D ₂	D ₁	D ₂	D ₁	D ₂	RE(D ₁, D ₂)
0.2	20	100	0.10	94.2	94.6	1.9	2.3	−6.4	0.4	98.8
			0.25	94.0	95.0	2.1	3.0	−8.4	0.2	96.0
		400	0.10	95.0	93.0	−3.5	−3.3	−1.2	−0.4	100.5
			0.25	94.6	93.6	−3.9	−3.2	−1.6	−1.8	99.9
	50	625	0.10	93.2	92.8	−0.5	0.3	−13.1	−9.3	97.9
			0.25	93.6	93.2	−1.5	−0.1	−14.5	−10.0	96.1
		2,500	0.10	94.8	94.0	1.0	0.9	−4.3	−3.7	98.7
			0.25	94.8	93.2	0.9	0.7	−3.1	−2.2	98.3
0.5	20	100	0.10	94.6	95.2	0.9	1.6	−7.2	−0.4	98.8
			0.25	94.8	95.6	1.0	1.7	−9.1	−0.9	96.2
		400	0.10	94.8	93.6	−1.3	−1.1	−0.9	−0.1	100.2
			0.25	94.4	93.8	−1.3	−1.0	−1.7	−2.5	99.9
	50	625	0.10	93.4	93.2	−0.1	0.3	−11.6	−7.9	97.7
			0.25	92.8	93.4	−0.5	0.1	−13.2	−9.0	96.0
		2,500	0.10	94.4	94.4	0.5	0.5	−4.6	−4.1	98.5
			0.25	94.6	94.0	0.5	0.4	−2.9	−2.5	98.2
0.8	20	100	0.10	95.0	95.0	0.7	1.4	−7.9	−1.1	99.0
			0.25	94.8	95.2	0.8	1.4	−9.6	−2.1	96.5
		400	0.10	94.6	93.6	−0.8	−0.6	−0.4	0.1	99.9
			0.25	93.8	93.8	−0.7	−0.5	−1.7	−3.4	100.0
	50	625	0.10	93.4	93.4	0.0	0.3	−9.9	−6.5	97.5
			0.25	93.4	93.6	−0.2	0.2	−11.6	−8.1	96.0
		2500	0.10	95.0	94.0	0.3	0.3	−4.8	−4.5	98.2
			0.25	95.0	94.6	0.4	0.3	−2.5	−2.9	98.2

Note. Based on 500 replications for each condition. δ _T = population effect size. Pearson’s contingency coefficients for all conditions ranged from .84 to .92, indicating strong associations between the clusterings of random effects A and B for the treatment arm. J^T = number of clusters of random effect A in the treatment arm; number of clusters of random effect B for the treatment arm = J^T . N^T = N^C = total sample size of the treatment (control) arm. ρ _A = intraclass correlation of effect A; ρ _B = 0.1 for the treatment group for all conditions. Coverage refers to the percentage of replications in which the 95% confidence interval includes δ _T . For 500 replications, the Monte Carlo coverage percentage have a confidence interval of [92.7%,96.6%] if the true coverage percentage is 95%. RE(D ₁, D ₂) = relative efficiency of the estimator D ₂ to the estimator D ₂ computed from SAS 9.3 with DDFM=SATTERTHWAITE and METHOD=REML, which was computed by dividing the sampling variance of D ₂ by that of D ₁.

Real Data Example

For illustrative purpose, the data used in Coyne et al. (2013) were analyzed. The data consisted of 103 kids receiving Early Reading Intervention. The treatment arm consisted of 70 kids who have changed group membership based on their performance over the course of the study, which created a cross-classified data structure, given that the initial group membership of each kid was different from the final group membership. In other words, kids were cross-classified by the initial and final group memberships in the treatment arm. On the other hand, the control arm consisted of 33 kids who were randomly assigned to groups at the beginning of the study and were kept in the same group over the course of study, which created a strictly hierarchical structure for this group of kids. The dependent variable is the score on a word identification test at the final stage. There were 19 groups among those receiving intervention (i.e., J^T = 19) and 10 groups among those receiving regular reading instruction (i.e., J^T = 10) at the the initial stage. At the final stage, those in regular reading stayed in their original groups, but some of those receiving intervention had moved to different groups. In summary, students receiving regular reading instruction were nested within initial groupings, whereas those receiving intervention were cross-classified by the initial and final groupings (K = 19). The design was not balanced, and the average cell size was 0.187.

Using SPSS mixed with an approach analogous to Bauer et al. (2008), the grouping effect was estimated as 9.202 (SE = 3.492), and the variance components were estimated as ${\hat{σ}}_{A}^{2} = 3.401$ (SE = 15.217), ${\hat{σ}}_{B}^{2} = 100.140$ (SE = 42.621), ${\hat{σ}}_{W | T R E A T}^{2} = 97.723$ (SE = 20.549), and ${\hat{σ}}_{W | C O N}^{2} = 176.504$ (SE = 50.411). Assuming that the total variances of both treatment conditions are comparable, ρ _A and ρ _B were estimated as .018 and .550, respectively. For the estimation of D ₁, the additional inputs were ${\overset{ˉ}{Y}}_{∙ ∙ ∙}^{T} = 105.06$ , ${\overset{ˉ}{Y}}_{∙ ∙}^{C} = 95.73$ , $S_{T | T R E A T}^{2} = 181.83$ , and $S_{T | C O N}^{2} = 199.19$ . The computed D ₁ = 0.675 (SE = 0.264, 95% CI [0.157, 1.192]), which can be interpreted such that on average students receiving intervention scored .67 SD higher on word identification than those receiving regular instructions. Using Equations 14 and 15, the effect size D ₂ for the grouping effect is 0.682 (SE = 0.261, 95% CI [0.171, 1.193]). Both approaches gave similar point and interval estimates, and both indicated a moderate to large intervention effect with the 95% CI not including zero.

Conclusion

Unlike single-level research studies in which effect sizes are regularly reported, effect size statistics for multilevel studies, in particular standardized mean difference, are still not fully investiaged. Effect size is extremely important because it directly quantifies the effect of interest (e.g., the effect of the treatment and gender difference), regardless of whether the study consists of single-level or multilevel data.

Our article has included analytically derived formulae of the standardized mean difference for fully and partially cross-classified treatment–control arm designs, as well as methods for obtaining the effect size when reliable and consistent estimates of variance components are available. Although the analytical formulae for D ₁ are tedious to use and can lose efficiency when the design is unbalanced or when the sample size is small, they are nevertheless important. In secondary analyses and meta-analyses where the clustering is not taken into account in the original analyses or when information about the variance components is not available, D ₁ can still be computed when the following information is available: number of clusters, cluster size, and intraclass correlations are available. For occasions where intraclass correlations are not available, Hedges (2007) provided an example of substituting values reported from other studies or with an educated guess, and Ahn et al. (2012) suggested quantitative procedures to estimate the ICCs. In addition, one can perform sensitivity analyses to examine whether different choices of ICCs result in substantial differences in the estimated effect size and the corresponding SEs (Hedges, 2007, 2011).

We have also suggested a method to estimate effect size D ₂ using maximum likelihood or Bayesian estimates of variance components. It is easier to implement, and we thus recommend its use when raw data are available. To facilitate future replication and research synthesis, we also recommend researchers analyzing primary data to report the effect size and its sampling variance, or at least the estimated values and the sampling variances (or the SEs) of the variance components.

Given the complexity associated with the effect size estimation equation for CCREMs, a logical question would be when a researcher can ignore one level of clustering (i.e., random effect B) but still get a good estimate of the effect size and the sampling variance. We have reanalyzed the two real data examples by ignoring one level of clustering, and it appears that when the two crossed random effects are highly correlated, omitting one random effect does not lead to substantial differences in point and interval estimates of effect size. This makes sense because when the two effects share a lot of common information, most of the information is still preserved when one effect was omitted (see Luo & Kwok, 2009). On the other hand, if the crossed random effects were only weakly correlated or uncorrelated (such as when the design is balanced), in general, the bias on the estimated sampling variance increases when number of clusters K and the ICC ρ _B of the omitted random factor are large, based on Equations 7 and 8. For example, assuming a balanced design, when δ _T = 0.5, J = K = 30, n =1, ρ _A = 0.25, ρ _B = 0.1, ignoring the clustering of B results in an underestimation of V by 19.1% (from 0.046 to 0.037); when n = 10 and other things being unchanged but ρ _B = 0.2, then V(D ₁) is underestimated by 32.2% (from 0.055 to 0.037); and when n = 1 but K is doubled to 60, V(D ₁) is underestimated by 32.8% (from 0.028 to 0.019); This would result in CIs of the effect size that are too narrow and not valid (see Hedges, 2011). Nevertheless, because CCREMs are complex models, further studies are needed to fully understand the impact of ignoring one or more levels of clustering on effect size estimation in real research.

This article is limited to only two-level CCREM with two crossed random effects, which is an extension of two-level multilevel models. However, the framework can be extended to CCREMs with three or more levels and with three or more random effects, or to CCREMs where treatment is defined as a Level 1 variable (i.e., the treatment is individually randomized). Future research can investigate perhaps effect size estimations in more complicated designs, as well as in other models in the multilevel family such as the multiple membership models. Simulation studies comparing different variance component estimation methods (e.g., Bayesian vs. REML) in the process of computing effect size are also highly encouraged. Also, in this article, we assumed that the effect size of interest has σ _T as the denominator for standardization. There are occasions where researchers may be interested in effect size with σ _B or σ _W or other alternatives as the denominator, but they are left for discussions in future studies. Finally, procedures to convert standardized mean difference effect sizes with multilevel structure into proportion of variance accounted for effect sizes (see Luo & Kwok, 2010) will be highly valuable for research synthesis methodology.

Footnotes

Acknowledgments

We thank Dr. Deborah Simmons’s data from the Early Reading Intervention Project (supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324E060067 to Texas A&M University; the opinions expressed are those of the authors and do not represent views of the U.S. Department of Education). We are also grateful to Dr. Victor Willson for his invaluable suggestions on this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

References

Ahn

Myers

N. D.

Jin

(2012). Use of the estimated intraclass correlation for correcting differences in effect size by level. Behavior Research Methods, 44, 490–502. doi:10.3758/s13428-011-0153-1

American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35, 33–40. doi:10.3102/0013189X035006033

American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.

Bauer

D. J.

Sterba

S. K.

Hallfors

D. D.

(2008). Evaluating group-based interventions when control participants are ungrouped. Multivariate Behavioral Research, 43, 210–236. doi:10.1080/00273170802034810

Beretvas

S. N.

(2008). Cross-classified random effects models. In O’Connell

A. A.

McCoach

D. B.

(Eds.), Multilevel modeling of educational data (pp. 161–197). Charlotte, NC: Information Age.

Beretvas

S. N.

(2010). Cross-classified and multiple membership models. In Hox

J. J.

Roberts

J. K.

(Eds.), Handbook of advanced multilevel analysis (pp. 313–334). New York, NY: Routledge.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Cohen

(1994). The earth is round (p < .05). American Psychologist, 49, 997–1003. doi:10.1037/0003-066X.49.12.997

Coyne

M. D.

Simmons

Hagan-Burke

Simmons

L. E.

Kwok

Kim

Rawlinson

D. M.

(2013). Adjusting beginning reading intervention based on student performance: An experimental evaluation. Exceptional Children, 80, 25–44.

10.

Ferguson

(2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532–538. doi:10.1037/a0015808

11.

Friedel

J. M.

Cortina

K. S.

Turner

J. C.

Midgley

(2010). Changes in efficacy beliefs in mathematics across the transition to middle school: Examining the effects of perceived teacher and parent goal emphases. Journal of Educational Psychology, 102, 102–114. doi:10.1037/a0017590

12.

Goldstein

(1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73, 43–56. doi:10.2307/2336270

13.

Goldstein

(2011). Multilevel statistical models. Hoboken, NJ: Wiley.

14.

Grissom

R. J.

Kim

J. J.

(2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.). New York, NY: Routledge.

15.

Hedges

L. V.

(1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational and Behavioral Statistics, 6, 107–128. doi:10.3102/10769986006002107

16.

Hedges

L. V.

(2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32, 341–370. doi:10.3102/1076998606298043

17.

Hedges

L. V.

(2009). Effect sizes in nested designs. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 337–355). New York, NY: Russell Sage Foundation.

18.

Hedges

L. V.

(2011). Effect sizes in three-level cluster-randomized experiments. Journal of Educational and Behavioral Statistics, 36, 346–380. doi:10.3102/1076998610376617

19.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi:10.3102/0162373707299706

20.

Hox

J. J.

(2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.

21.

Ingles

Abraham

S. Y.

Rasinski

K. A.

Karr

Spencer

Frankel

(1990). National Educational Longitudinal Study of 1988: Base year student component data file user’s manual. Washington, DC: U. S. Department of Education.

22.

Johnson

I. Y.

(2011). Contingent instructors and student outcomes: An artifact or a fact? Research in Higher Education, 52, 761–785. doi:10.1007/s11162-011-9219-2

23.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta-analysis. Thousand Oaks, CA: Sage.

24.

Littell

R. C.

Milliken

G. A.

Stroup

W. W.

Wolfinger

R. D.

(1996). SAS system for mixed models. Cary, NC: SAS.

25.

Luo

Kwok

(2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182–212. doi:10.1080/00273170902794214

26.

Luo

Kwok

(2010). Proportional reduction of prediction error in cross-classified random effects models. Sociological Methods & Research, 39, 188–205. doi:10.1177/0049124110384062

27.

Mason

W. M.

Wong

G. Y.

Entwisle

(1983). Contextual analysis through multilevel linear model. Sociological Methodology, 14, 72–103. doi:10.2307/270903

28.

Murray

D. M.

Blitstein

J. L.

(2003). Methods to reduce the impact of intraclass correlation in group-randomized trials. Evaluation Review, 27, 79–103. doi:10.3102/0162373707299706

29.

Muthén

L. K.

Muthén

B. O.

(1998–2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

30.

R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

31.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

32.

Searle

S. R.

Casella

McCulloch

C. E.

(2006). Variance components. Hoboken, NJ: Wiley.

33.

Shi

Leite

Algina

(2010). The impact of omitting the interaction between crossed factors in cross-classified random effects modelling. British Journal of Mathematical and Statistical Psychology, 63, 1–15. doi:10.1348/000711008X398968

34.

Snijders

T. A. B.

Bosker

R. J.

(1994). Modeled variance in two-level models. Sociological Methods & Research, 22, 342–363. doi:10.1177/0049124194022003004

35.

Snijders

T. A. B.

Bosker

R. J.

(2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London, England: Sage.

36.

Thompson

(2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44, 423–432. doi:10.1002/pits.20234

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB

Standardized Mean Differences in Two-Level Cross-Classified Random Effects Models

Abstract

Keywords

Effect Size in Multilevel Analyses

CCREMs

Model and Notation

ICC

Standardized Mean Differences for Fully Cross-Classified Data

Estimation of D 1

Estimation of D 2

Monte Carlo Study for Evaluating the Two Effect Size Estimation Approaches Under Unbalanced Designs

Real Data Example

Standardized Mean Differences for Partially Cross-Classified Data

Estimation of D 1

Estimation of D 2

Monte Carlo Study for Evaluating the Two Effect Size Estimation Approaches Under Unbalanced Designs

Real Data Example

Conclusion

Footnotes

Acknowledgments

Declaration of Conflicting Interests

Funding

Note

References

Supplementary Material

Estimation of D ₁

Estimation of D ₂

Estimation of D ₁

Estimation of D ₂