Educational and Psychological Measurement

This study examined three statistical methods for selecting items for mastery tests. One is the pretest-posttest method due to Cox and Vargas (1966); it is computationally simple, but has a number of serious limitations. The second is a latent trait method recommended by van der Linden (1981); it is computationally complex, but has a number of theoretical advantages. The third method, proposed herein, parallels the latent trait method in many respects; but it is computationally simple, like the pretest-posttest procedure. A number of distinct data sets were simulated; and the three item selection methods were applied to each data set for the purpose of studying relationships among the methods. The correlation between the latent trait method and the one proposed herein was substantial, suggesting that the latter might be recommended as a practical alternative to the former. The results for the pretest-posttest method tended to confirm its reputed limitations.

Critics have occasionally asserted that the number of factors in the 16PF family of tests is too large and that a more parsimonious number of factors adequately explains human personality. This paper discusses areas of misunderstanding with regard to factor-analytic methodology that account for much of this disagreement. It then reviews more than 50 published studies concerned with the dimensionality of the 16PF series of personality instruments. The conclusion is reached that the number of important primaries encapsulated in the 16PF series is no fewer than the stated number. That conclusion is bolstered by new evidence presented that is based on the independent, statistical logic of the maximum likelihood test.

In a recent article, Gardner and Erdle (1984) have argued inappropriately against prior standardization of scores when an aggregate score is formed. They demonstrate that an aggregate of raw scores may not correlate perfectly with an aggregate of standard scores, and that the correlation of an external criterion with the two types of aggregates may differ markedly. However, these effects are not a function of standardization. They are an artifact resulting from the differential weighting of components in the process of aggregation. In fact, standardization improves the ability to interpret and compare scores. A demonstration of the effects of differential weighting of aggregate components is presented that clarifies the need for prior standardization. The conceptual role of standardization in statistics and the use of aggregate scores in research are also briefly discussed.

This article evaluated criticisms by Stevens and Aleamoni (1986) of an article by Gardner and Erdle (1984) on aggregation using either raw or standard scores. It was demonstrated that their criticisms were unfounded.

The development of a nonparametric K-sample test for equality of slopes using Puri's generalized L statistic is presented. The test is recommended when the assumptions underlying the parametric model are violated. For this procedure original data are replaced by either ranks or normal scores. A discussion is presented on which data substitution procedure to use. Ranks are suggested when the data have heavy tails and normal scores for data with light tails. An example is carried out illustrating the test process.

A simple statistical test procedure for assessing questionnaire response validity is proposed. The technique assesses the joint probability that frequently reported behaviors are not reported and infrequently reported behaviors are reported.

Methods of estimating the standard error at different ability levels were compared. Overall, it was found that at a given ability level the standard errors calculated using different formulas are not appreciably different. Further, for most situations the traditional method of calculating a standard error probably provides sufficient precision.

Despite much evidence that the Jenkins Activity Survey Type A scale is lacking in essential psychometric properties, it continues to be widely used for measuring coronary-prone behavior in both research and applied settings. Four psychometric properties-internal consistency, construct validity, criterion validity, and generalizability-of the JAS Type A scale were assessed relative to coronary risk measurements in a sample of 227 male and female hospital employees. Even when the JAS Type A scale was psychometrically enhanced, it failed to satisfy accepted reliability and validity criteria. Researchers are cautioned against using the JAS Type A scale without performing routine tests of psychometric soundness and controlling for gender differences.

A recurrent problem in assessment work is to determine the degree of match for a client score profile to some known prototypic group patterns. Frequently, it is more accurate to conceptualize the "known" group pattern as a range of values than as a series of point estimates. Interval banded profile analysis represents a generalization of the method of centours to this problem.

To correct for the effects of measurement error on structural parameter estimates, many researchers are now estimating models of educational achievement with LISREL. In order to estimate such models it is desirable to obtain multiple manifest measures of the latent constructs. Many researchers restrict their models to two manifest measures per latent construct for reasons of economy, but doing so assumes in the absence of external information that all of the covariance between the within-variable measures is reliable covariance. Such an assumption may or may not hold in practice. Using data from High School and Beyond, the present study empirically investigates the extent of within-variable, between-occasion error covariances among variables typically included in models of educational achievement.

A simple quasi-experimental design is described which may have utility in a variety of applied and laboratory research settings where ordinarily the one-group pretest-posttest pre-experimental design might otherwise be the procedure of choice. The design achieves high levels of internal and external validity and produces independent control groups by independently randomly assigning subjects to measurement occasions. It has superior internal and external validity in comparison with the one-group pretest-posttest design, in particular, but also in comparison with a number of other designs. The design includes elements which help control the internal validity factors-history, maturation, testing, regression, selection and their interactions; and the external validity factors-interaction of testing and treatment, interaction of selection and treatment, reactive arrangements and multiple treatment interference. The design approaches the internal validity of true experimental designs while optimizing external validity. It is readily analyzable with ANOVA techniques. The design may be particularly appropriate for evaluations of ongoing instructional programs in the schools, or evaluations of programmatic interventions or therapies occuring on an ongoing basis.

A factor replication procedure (FACTOREP) which provides a solution to the number of factors problem for multiple scale questionnaires was evaluated using four psychometrically equivalent synthetic correlation matrices containing an imposed three-subscale structure. Comparisons of the structure revealed by two, three, four and nine-factor rotations using the FACTOREP procedure, clearly showed that only the three factor solutions were replicable across all four matrices. An examination of the patterns of factor loadings themselves confirmed the superior replicability of the three-factor solutions and showed that in every case the three factors found were those originally imposed on the matrices.

The Endler S-R Inventory of General Trait Anxiousness was analysed employing Smallest Space Analysis and factor analysis. The Smallest Space Analysis yielded a two-dimensional lattice partitioning the space into regions corresponding to the situations and the response clusters. The situations were the ones specified in the inventory: Interpersonal general situation, social evaluation situation, ambiguous or novel situation, and physical danger situation. The evaluative and the ambiguous situations merged in the solution. The responses were segregated into regions defined as approach, confidence, distress, and physiological responses. A general partitioning of the responses on a positive-negative dimension was indicated. The factor-analytic results were similar with respect to situations but were not as informative with respect to responses. Further specification of facets is discussed.

The study was designed to answer two questions: (1) Can cognitive strategies to increase testwiseness be taught? (2) Can the testwiseness strategies be generalized by students so they can improve their scores on examinations other than those measuring testwiseness? Fifty-four undergraduate education students served as subjects. The treatment consisted of a one-hour lecture/discussion presentation covering four cognitive strategies identified as comprising testwiseness. This was followed by a 48 item instrument developed to measure testwiseness on typical classroom examinations. The results demonstrate that it is possible to teach four cognitive strategies which are part of testwiseness. On all measures of testwiseness, the experimental group significantly outperformed the control group.

Statistical power analysis is becoming an increasingly routine procedure when planning experiments and assessing the validity of tests of hypotheses following statistical analysis. For those situations where Type I error rate is not set to .10, .05, or .01 the task is impeded by the unavailability of tables for computation. A BASIC computer program is described that computes power for any combination of effect size, degrees of freedom for hypothesis, degrees of freedom for error. and a level. As a consequence of the algorithm an approximation to the critical value of the Bonferroni F-test is also computed.

This paper presents a BASIC microcomputer program that constructs the Tukey (1977) stem and leaf graph. Options within the LEAF program include a modified stem and leaf where the stem is split, and a parallel stem and leaf graph where two separate sets of data are displayed from a common stem.

This paper describes a BASIC microcomputer program that computes all pairwise comparisons of means after analysis of covariance using the Tukey-Kramer test. Input to the program consist of the means of the covariate, the adjusted criterion means, the sample size, mean square error from a one-way analysis of covariance, and the 95th percentile point on the Studentized range distribution.

The Johnson-Neyman (J-N; 1936) technique is a parametric alternative to analysis of covariance that permits non-parallel regression lines. This article presents computer programs for J-N using the transformational languages of SPSS-X and SAS. The programs are designed for two groups and one covariate.

An algorithm and associated FORTRAN-77 computer subroutine are described for computing Goodman and Kruskal's tau-b statistic along with the associated nonasymptotic probability value under the null hypothesis Tb = 0.

A BASIC computer program is described which computes the Stuart-Maxwell test with its chi-square value and associated probability level. Following a significant chi-square test, the user has the option of performing multiple comparisons using McNe-mar's test for the significance of change.

This paper describes a set of four BASIC computer programs for computing the Rand (1971) index of cluster similarity. The routines calculate the unadjusted Rand index; the separate routines are designed to handle problems of different sizes and alternate methods of data storage.

This paper describes two BASIC computer programs that calculate Hotelling's T² statistic either for one sample or for two samples. Output of the program includes the Mahalanobis distance D² , the F ratio associated with T² , and its probability level.

A nonparametric Mokken scale analysis (a stochastic elaboration of the deterministic Guttman scale) and program is described. Focus has been placed on three procedures of scaling: a search procedure, evaluation of the whole set of items as a scale and extension of an existing scale. All procedures provided a coefficient of scalability for all the items together that satisfy the criteria of the Mokken model, and an item coefficient of scalability for every item and the statistic delta star which provides information whether the null hypothesis of random response may be rejected. The program is written in Pascal, and it is usable on many microcomputers and larger systems.

A description of a Mokken test program to test the robustness of nonparametric stochastic scales is presented. For each sample and the whole population, different scale statistics, both on scale and on item level, are computed recursively. A statistic T is presented for testing the robustness of the scales in the samples.

The Tests of General Educational Development (GED Tests) are used throughout the United States and in much of Canada as a basis for awarding high school level equivalency credentials to adults who did not graduate from high school. The paper summarizes much of the available information concerning the reliability and validity of the GED Tests. These data were gathered in a number of studies conducted between 1978 and 1984. Results are given for both internal consistency and parallel forms reliabilities as well as for many aspects of validity. The data suggest that the test results are sufficiently reliable for continued use and that the validity evidence generally supports the intended uses of the tests.

A battery of nine specific cognitive abilities tests to assess verbal, spatial, perceptual speed, and memory abilities in children at age 4 was constructed and administered to a sample of 267 children participating in the Colorado Adoption Project. Exploratory and confirmatory factor analyses revealed four correlated factors that adequately describe the covariation among the tests. Specific cognitive ability composites showed test-retest correlations ranging from .56 to .81. The reliable measurement of these specific cognitive abilities in preschool children facilitates analyses of familial resemblance and longitudinal comparisons.

Local norming data from the Brief Index of Adaptive Behavior (BIAB) are presented (Study 1), as are construct and concurrent validity data (Studies 1, 2, and 3). Participants were 430 children, 330 of whom were randomly selected within certain strata from one southeastern state. Factor analytic results from Study I yielded an easily interpreted three-factor solution, which accounts for 52% of the variance and corresponds to the three domains from the BIAB. Conventional item analysis techniques yielded significant item-domain (.66 to .87) and item-total (.64 to .79) coefficients. Reliability indices ranged from .93 to .97. Modest correlations emerged from comparisons of the BIAB to AAMD Adaptive Behavior Scale-School Edition, ranging from .25 to .55. Stronger correlations were obtained from comparisons of the BIAB to the new Vineland Adaptive Behavior Scale-Classroom Edition, ranging from .64 to .71 for like-named domains. Because the BIAB is conveniently administered and scored, and is particularly amenable to collection of local norms, it is recommended for use as a screening test. Also, collection of local norms for the BIAB is recommended.

The predictive validity of the Standard Progressive Matrices(SPM) was examined with a sample of 212 sixth grade students and 214 seventh grade students. Correlation coefficients between the SPM and the California Achievement Tests of Reading, Language, and Mathematics ranged from .34 to .60. Those coefficients were similar in range and magnitude to those coefficients reported by the SPM Manual.

Admissions test scores were available for a sample of 84 students who had taken both the Scholastic Aptitude Test (SAT) and the Israeli Inter-University Psychometric Entrance Test (IUPET). Concurrent validity was supported by the high level of intercorrelations between the two tests, including a correlation of .85 between IUPET and SAT total scores. Transformation equations were presented to estimate performance on one test from knowledge of performance on the other test.

This study investigated the effects of past computer experience on the computerized aptitude test performance of college students. Subjects were 92 undergraduates. All subjects completed an arithmetic reasoning test by computer and a Computer Experience Questionnaire. Past computer experience did account for a significant amount of variance on the arithmetic reasoning test. Suggestions regarding the future use of computerized aptitude tests are given.

For a sample 239 undergraduate university students (135 males and 104 females) a factor analysis of the intercorrelations of 20 4-item subtests of the 5 scales (4 subtests per scale) of the Dimensions of Self-Concept (DOSC), Form H, an academic self-concept measure, and of 6 subtests from the Intellectual Achievement Responsibility Questionnaire (IAR), a locus of control measure, revealed 5 distinct factors that correspond to the 5 a priori dimensions afforded by the DOSC but only one factor for the IAR. With respect to 3 self-report measures of academic achievement, the Level of Aspiration scale of the DOSC yielded the only 2 validity coefficients in excess of .30. It was concluded that (a) the DOSC reflects 5 distinct factor dimensions consistent with a theory of academic self-concept proposed by Michael and Smith (1976) and (b) the IAR is a unidimensional measure quite independent of the DOSC.

The Myers-Briggs Type Indicator (MBTI) is widely recognized as an important measure of normal or non-pathological variations in personality. However, the construct validity of the measure has not been clearly established using factor analytic techniques. The present study investigated the structure and item performance of the instrument using data from 359 college students. Factor analysis was applied to the 95 scored MBTI items. Factor adequacy and invariance coefficients were computed, and the appropriateness of the recommended item weights was examined. The results strongly supported the instrument's construct validity.

The Learning Style Inventory (LSI) and the newly revised Learning Style Inventory (LSI II) were examined for internal consistency, test-retest reliability and stability of the four classifications resulting from their scores. Internal consistency was much improved in the LSI II, but problems with low test-retest indices and classification stability continue to plague the instruments. The authors hypothesize that the observed improvement in internal consistency may be an artifact of the revised scoring scheme.

This study investigated the underlying structure of ratings of clinical competence. The study sample was comprised of 609 physicians graduating from Jefferson Medical College between 1980 and 1983. The rating instrument consisted of 33 statements on clinical behavior in Likert-type format filled out by directors of medical education programs at postgraduate training institutions. The data were subjected to factor analysis. Three factors emerged involving "data gathering and processing skills," "interpersonal and attitudinal," and "socioeconomic" dimensions. Correlations of factor scores with independent measures of conceptually related and unrelated constructs supported the appropriateness of the assigned factor titles. It was concluded that ratings of clinical competence represent a multidimensional construct involving at least three dimensions.

The Association of American Medical Colleges is conducting research on the incremental validity associated with adding an essay to the Medical College Admission Test (MCAT). The present research examined the reliability of holistic scoring for a sample of essays written during the Spring 1985 MCAT administration. Analysis of variance techniques were used to estimate the reliability of scoring and to partition score variance into that due to level differences between papers and to context-specific factors such as rater, time of scoring, etc. Recommendations for structuring and managing large-scale scoring sessions are presented based on the results.

The purpose of the present study was to analyze the internal consistency and factorial composition of the Job Stress Questionnaire (JSQ), a perceptual measure of job related stress, and to compare its factor structure across three occupational groups as well as comparing the empirical solutions to the a priori dimensions hypothesized for this scale. The results indicated acceptable reliabilities for three of the four subscales while factor analysis revealed limited support for the factorial validity of the instrument. Subgroup analyses by occupational category showed differential factor dimensionality for blue collar, white collar, and professional workers. The results were discussed in terms of their implications of job stress as a function of occupational class membership.