Chapter 5: Population Inferences and Variance Estimation for NAEP Data

In the National Assessment of Educational Progress (NAEP), population inferences and variance estimation are based on a randomization-based perspective where the link between the observed data and the population quantities of interest is given by the distribution of potential values of estimates over repeated samples from the same population using the identical sample design. Because NAEP uses a complex sample design, many of the assumptions underlying traditional statistical analyses are violated, and, consequently, analysis procedures must be adjusted to appropriately handle the structure of the sample. In this article, we discuss the use of sampling weights in deriving population estimates and consider the effect of nonresponse and undercoverage on those estimates. We also discuss the estimation of sampling variability from complex sample surveys, concentrating on the jackknife repeated replication procedure—the variance estimation procedure used by NAEP—and address the use of a simple approximation to sampling variability. Finally, we discuss measures of the stability of variance estimates.