Journal of Educational and Behavioral Statistics

A model-based approach to rater reliability for essays read by multiple readers is presented. The approach is motivated by the generalizability theory. Variation of rater severity (between-rater variation) and rater inconsistency (within-rater variation) is considered in the presence of between-examinee variation. An additive variance component model is posited and the method of moments for its estimation described. The models involve no distributional assumptions other than variance homogeneity and independence of certain random variables. Minimum mean squared error estimators of examinees’ true scores and readers’ severities are derived. Model diagnostic procedures are an integral component of the approach. The methods are illustrated on data from standardized educational tests.

This article has two objectives. The first is to investigate in greater detail the finding of Rogosa and Willett that the quasi-Markov simplex model fits a linear growth curve covariance structure. It is found that under various circumstances the quasi-Markov simplex model is rejected. Furthermore, the procedure is reversed by fitting the linear growth curve to quasi-Markov simplex covariance structure. It is found that the linear growth curve, like the quasi-Markov simplex, is not always rejected even though the model is formally incorrect. The second objective of this article is to present a quasi-Markov simplex model with structured means. This model, like the linear growth curve model with structured means, is based on the assumption that the variation in means and individual differences are attributable to the same causal agents. We argue that this assumption should be tested explicitly. An example is given.

In completely randomized experimental designs where population variances are equal under the null hypothesis, it is not uncommon to have multiplicative treatment effects that produce unequal variances under the alternative hypothesis. Permutation procedures are presented to test for (a) median location and scale shifts, (b) scale shifts only, and (c) mean location shifts only. Corresponding multivariate extensions are provided. Location-shift power comparisons between the parametric Bartlett-Nanda-Pillai trace test and three alternative multivariate permutation tests for five bivariate distributions are included.

The ANOVA F and several nonparametric competitors for two-way designs were compared for empirical α and power. Simulation of 2 × 2, 2 × 4, and 4 × 4 designs was done with cell sizes of 5 and 10 when sampling from normal, exponential, and mixed normal distributions. Conservatism of both α and power in the presence of other nonnull effects was seen in the tests due to Puri and Sen (1985) and, to a lesser degree, in the rank transform tests (Conover & Iman, 1981). Tests by McSweeney (1967) and Hettmansperger (1984) had liberal α for some designs and distributions, especially for small n. The ANOVA F suffers from conservative α and power for the mixed normal distribution, but it is generally recommended.

Type I error rates were estimated for three tests that compare means by using data from two independent samples: the independent samples t test, Welch’s approximate degrees of freedom test, and James’s second-order test. Type I error rates were estimated for skewed distributions, equal and unequal variances, equal and unequal sample sizes, and a range of total sample sizes. Welch’s test and James’s test have very similar Type I error rates and tend to control the Type I error rate as well or better than the independent samples t test does. The results provide guidance about the total sample sizes required for controlling Type I error rates.

The scaling constant d = 1.702 used in item response theory minimizes the maximum difference between the normal and logistic distribution functions. The theoretical and numerical derivation of d given by Haley (1952) is briefly recapitulated to provide access to curious researchers, instructors, and students.

The popular One-Sample Runs Test is used to identify a nonrandom pattern in a sequence of dichotomous elements. Although the test is generally effective in the identification of patterns, it is demonstrated to be incapable of signaling departures from randomness with run lengths of two. Further-more, with run lengths of two, increasing the sample size reduces the power of the test. Run lengths strictly of two, therefore, generate a unique category of anomaly in the test’s overall performance.