Performance of Random Effects Model Estimators Under Complex Sampling Designs

In this article, we consider estimation of parameters of random effects models from samples collected via complex multistage designs. Incorporation of sampling weights is one way to reduce estimation bias due to unequal probabilities of selection. Several weighting methods have been proposed in the literature for estimating the parameters of hierarchical models, of which random effects models are a special case. We evaluate the bias of the weighted analysis of variance (ANOVA) estimators of the variance components for a two-level, one-way random effects model. For these estimators, analytic bias expressions are developed and the accuracy of the expressions is evaluated through Monte Carlo simulation. The expressions are used to examine the impact of sample size, the size of the intraclass correlation coefficient (ICC), and the sampling design on the estimators' performance. The sampling designs considered are two-stage, with a general probability design at Level 2 and simple random sampling without replacement (SRS) at Level 1. The study shows that variance component estimators using only first-order weights perform well when both cluster size and ICC are moderate. However, this weighting method should be used with caution for small cluster sizes (less than 20), particularly when ICC is small (less than 0.2). In such scenarios, scaled first-order weighted (SFW) estimators provide an alternative to the difficult-to-use second-order weighted estimators for designs in which SRS is used at the ultimate sampling unit level (Level 1). This article is discussed in the context of large educational survey assessments.