Educational and Psychological Measurement

The G* index of agreement, introduced by Lienert as a generalization of the G index is shown to have some deficiencies. It cannot, for example, be used in a Q-factor analysis. An alternative index, r_oz , is proposed for ordinal scales with fixed neutral (zero-) points. The r_oz index can be used for measuring both person similarity and item similarity. It fulfills the demands of a normalized scalar product and can thus be applied in a Q- or an R-factor analysis.

If each of k predictor variables is correlated with a criterion variable, the use of multiple t tests to assess pairwise differences between r's will compound Type I error. This note describes an overall F test for determining the significance of differences between r ²'s which provides an error term for use in conventional ANOVA follow-up procedures.

ANOVA computed intraclass reliability coefficients are often underestimated because ANOVA assumptions are ignored. Assumptions of an alternate technique are explored. Biasedness of the intraclass reliability coefficient is discussed and a table of corrections presented.

It is indicated that the Mean Square Successive Difference test is not always suitable for testing the slow change assumption of the Shine-Bower error term. A more suitable alternate statistic is presented. The two statistics are compared.

Recent research has indicated that the initial position in which a group of entities is presented can spuriously affect the subsequent rankings of these entities along some prescribed dimension. It has also been demonstrated that the location of this "ranking error" is variable and may occur anywhere across a set of rankings. It was suggested that a methodological variation of the Friedman Two-Way Analysis of Variance could be used for detecting this type of error.

If variables are scaled so that they have equal errors of measurement, then the maximally reliable composites found in a canonical reliability analysis have highly desirable properties. Under this scaling transformation, canonical factor analysis on all nonerror variance, principal components analysis and canonical reliability analysis all yield equivalent results. Thus, the resulting components are successively maximally reliable and simultaneously provide a least squares fit to both the covariance matrix of rescaled observed scores and of rescaled true scores. In addition, the proportion of true to observed score variance retained in an s-dimensional least squares fit is equal to the ratio of the s-dimensional canonical reliability to the canonical reliability of the entire battery. Based on the above, it is argued that data reduction (when there is no a priori reason to keep the original units of measurement) should proceed by doing a components analysis on the covariance matrix rescaled so as to yield equal errors of measurement.

Spearman's rho is a special case of the product-moment r. As such, a principal components analysis of inter-rater rhos among k raters on n cases may be performed. The size of the first principal component is proportional to the amount of inter-rater agreement and can be equated to Kendall's W. The principal components procedure can also reveal the presence of differing rating policies and can isolate subgroups of raters.

Validity scores for items were obtained by comparing the rank order of the correlations between each item and six criterion measures with the corresponding rank order for the test as a whole. For samples of about 85 cases, the validity pattern for each of two tests showed limited consistency from sample to sample and the item validity scores had low correlations between samples. While sample size contributed to these undependabilities, the additional undependability of item responses themselves raises strong doubts about the practicality of selecting items on the basis of their validity patterns.

Traditional multidimensional scaling is a two-stage procedure. Thurstone's comparative judgment model is often used in the initial stage to determine the perceived distance between pairs of objects, under the assumption that the perceived distance between each pair is normally distributed across a population of subjects. These distances are used in the second stage to construct a geometric configuration of the objects in multidimensional space. A stimulation method was employed to study the effect of nonnormally distributed distances on the accuracy of the recovered configuration. Accurate configurations were generated from a variety of nonnormal distributions. Thus, the multidimensional scaling procedure appears to be robust with respect to violations of the normality assumption inherent in the comparative judgment model.

In this paper, an exact test for use with certain types of Likert scale is presented. The test may be used on the assumption that the points of the scale are equally-spaced, that subjects respond independently of each other, and that each point of the scale has an equal likelihood of response for each subject. A table of critical values is presented for use with small samples.

When samples are large, it is shown that a normal approximation may be used. This approximation is described and illustrated.

Two models for the scaling of paired comparison data are compared to the Thurstone case III model. One model is based upon parameter estimates derived from the normal approximation of the binomial distribution, and the other from the beta density function defined over the unit interval. Procedures are given for estimating the parameters for each model, and arguments are presented in favor of a skewed characteristic distribution function in cases where the scale includes extreme stimuli.

Two indices of goodness of fit are presented for each of five data sets. The results generally illustrate the inability of the Thurstone case III model to adequately account for the data when the scale includes one or more extreme stimuli. The beta model performed well for all five data sets but fit best under just those conditions where the Thurstone model performed poorly. The binomial approximation performed adequately on three of the five data sets and would seem to be a simpler alternative estimation procedure to the Thurstone procedures when the normality assumption is tenable.

A procedure is specified for testing the significance of predicted trends in k independent correlations. An example is also provided for illustrative purposes.

This article illustrates the use of a recent technique for selecting alpha based on the particular circumstances of a hypothesis-testing problem. Rather than selecting alpha arbitrarily, this technique takes into account the prior probability that H_o is true and the penalties associated with Type I and Type II errors.

This research was directed at an examination of the pyschometric properties of the Ross Educational Philosophical Inventory (REPI), an instrument designed to investigate underlying philosophical beliefs influencing teaching behaviors and policy decisions of educational administrators. The REPI claims to measure individual attitudes with respect to four philosophical systems—Idealism; Realism; Existentialism; Pragmatism. Content evaluation and second-order factor analyses resulted in the dimensionalizing of two higher-order factors empirically fitting the REPI categories of Idealism and Existentialism. Reliability estimates for these two dimensions suggest further research directed at improving these two scales.

Profile stability of the Kuder Occupational Interest Survey (1966) for 729 persons was assessed by computing rank order correlations between the sets of scales on two profiles obtained 12 or 19 years apart, with first administrations at five different ages from 13 to 20. Median rhos varied from just over .40 for the youngest to near .80 for the oldest age at administration. Reliabilities were generally better for the male-normed than female-normed scales, and the difference between 12 and 19 year intervals was slight. These figures are comparable with similar reliabilities reported for the Strong Vocational Interest Blank, the only other interest inventory reporting long term reliability.

This study investigated student points of view in their ratings of specific courses and instructors by analyzing student responses within each of three classes separately, and then sought to generalize the results further by additional analyses with 300 students randomly selected from 402 classes in five colleges. Different points of view were found for student ratings of course examinations, textbooks and supplementary readings, and class discussions. These various points of view were moderately related to such student characteristics as grades and sex, although not in all three of the classes studied. This last point underscores the importance of the context (i.e., the particular course) in understanding or interpreting the meaning of student ratings.

A very simple and economical method for estimating initial parameter values for the normal ogive or logistic latent trait mental test model is outlined. The accuracy of the method in comparison with maximum likelihood estimation is investigated through the use of Monte-Carlo data. The study yields a number of observations concerning how sample size and true item parameter values of the data influence both estimation methods.

The computer program described contains various measures of attitudinal polarizations, vis. K, introduced by McCall and Simmons (1966), and V ₁, V ₂ and V ₃, suggested by Edvardsson and Vegelius (1974, 1975). In the latter three measures, two parameters, C and D, may be chosen by the user. The choice C= D= 0.5 is recommended.

The two major purposes of this paper were: (1) to describe the theoretical background underlying the mathematical development of different measures of the degree of attitudinal polarization applicable to bipolar questions and (2) to outline a small FORTRAN IV program POLARI that provides a rapid means for calculation of four of these measures.

A computer program written in FORTRAN IV is presented which will unambiguously partition the explained variance of a dependent variable into these parts due uniquely to each independent variable and to all possible combinations of independent variables through commonality analysis. Tests of the significance of the unique contribution due to each independent variable are also output. Documentation is provided.

Description of a computer program performing rotated canonical variate analysis was presented. The program was written as compatible with the IBM Scientific Subroutines Package. Examples of rotated and unrotated solutions of a sample problem were given, together with their interpretations illustrating the advantages of the rotated solution.

This article presents CONGR, a FORTRAN IV program which computes either of the coefficients of congruence (psi or phi) used in comparison of factors in factor analysis. Input consists of a title card, a parameter card, variable input card(s), and data cards. Output is a non-symmetric matrix of factor coefficients. Listings of the program, results, and test data are available.

In recent educational research discriminant analysis has been prominently emphasized as a technique for classifying subjects into predetermined categories. In practice a common experience is that the "best" discriminating variables do not satisfy the assumptions underlying classical discriminant analysis; for example, sex, age, and race. Discrete discriminant analysis is a technique developed to deal with such data. A FORTRAN Program is available from the authors that performs discrete discriminant analysis. It handles a maximum of 50 predictor variables and 12 outcome groups.

A computer program for two-group nonparametric discriminant analysis is described here. Based on Bayes' Theorem for probability revision, the statistical rationale for this program makes use of the calculation of maximum likelihood estimates of group membership. A unique feature of the program is the comparison of the classificatory ability of the Bayesian procedure relative to that of the standard Linear Discriminant Function.

A computer program is described that constructs and visually displays isodensity ellipses of a bivariate normal distribution for a maximum number of 100 pairs of values. The numbers of actual points within the ellipses can be compared with the theoretically expected frequencies by a chi-square analysis.

A computer program is described which performs the Bowker test of symmetry in a square contingency table, the Light-Margolin and Pearson tests for homogeneity in non-ordered contingency tables, the Kruskal-Wallis test for ordered dependent variables, and the Kendall and Maxwell tests for monotonic trend in ranks. The description includes the input, output, limitations, and availability of the program.

Goodman and Kruskal's t_b test statistic is frequently employed in measuring association between nominally-scaled polynomies. Because of an error in the published large sample standard error, the utility of t_b has been severely circumscribed for many years. A recent published correction of the large sample standard error promises to increase its use. This paper describes a Fortran Extended computer program which computes t_b , with column variable dependent, and its asymmetrical counterpart t_a , with row variable dependent. The program also provides three sets of confidence limits for each coefficient under both full multinomial and proportional stratified sampling.

A Fortran IV multiple-choice test scoring program for an IBM 370 computer is described that computes minimally acceptable performance levels and compares student scores to these levels. The program accommodates up to 500 items with no more than nine alternatives from a group of examinees numbering less than 10,000.

Internal consistency as one rationale for item selection from the universe of possible test items is discussed. Formulae, derived by Lord (1958), are presented which relate the maximum internal consistency of a test to the largest eigenvalue of the interitem correlation matrix. A computationally feasible stepwise procedure is suggested which selects a subset of the original test items that yields maximum internal consistency. The characteristics of a computer program designed to perform these calculations for an input score matrix are described.

This program computes both the interjudge reliability of individual measurements and the extent to which the judges are biased in their ratings vis-à-vis each other. The methods proposed are recommended on the basis of recent developments in statistical research.

A round-robin experiment involves observation of all possible pairs of subjects within each experimental condition. A program is described which performs analyses of variance for data arising from such experiments. Output of the program includes an ANOVA summary table, exact or quasi- F statistics for tests of various hypotheses, and least squares estimates of all relevant parameters.

A microcomputer program for two-way analysis of variance is described. Both the program and cell entries are stored on tape. Output consists of both intermediate and final products. Advantages of using a tape unit with a microcomputer are convenience, capacity, and flexibility. Limitations are found in not having data on cards and in the inability of the program to handle nested or mixed designs.

A microcomputer program is described for multiplying two matrices as large as 41 × 42 apiece. The major advantages of using a tape drive are single entry rather than multiple entry of data elements of matrices A and B and increase of memory capabilities. The major limitation is slowness of tape search. Verbal description of logic facilitates use of the program on different brands of computers.