Educational and Psychological Measurement

This article examines the simple difference score (Y-X) as used in three different correlational analyses. As discussed in this paper, this difference score is viewed as a simple change from a pretest (X) to a posttest (1Y), but the generalizations apply to any difference score of the type (Y-X). The three classes of correlational analyses considered involve the correlation of initial score (X) with change (Y-X); the correlation of another variable (A) with change (Y-X); and the correlation of one change score (B-A) with another (Y-X). In each case, the complexity of the change score and the ambiguity of the resulting correlation are demonstrated. It is proposed that attention should be directed to the elements making up each correlation in addition to attempting to use and interpret the correlation itself.

In two experiments the writers investigated the effects of statistical and cognitive difficulty ordering of test items, gender, and explicit labeling of item type on test performance of college students. Students were given tests with items written on various statistical and cognitive difficulty levels. In addition, some students received tests on which the items were explicitly labeled as to the cognitive difficulty (knowledge, comprehension, application) of the item. Analysis of the results indicated significant effects for gender and use of labels. Item ordering effects were not consistent across the two studies.

Counts of rare events tend to have Poisson or J-shaped distributions that render parametric assumptions questionable. Six different methods for testing the significance of the difference between location parameters for two such distributions are evaluated in this article. A binomial test for the difference between means is known to be the most powerful unbiased test when population distributions are truly Poisson; however, it appears extremely non-robust against departures from the distributional assumption. Robust enough to provide appropriate protection against Type I error, Student's t test was among the most powerful of the tests when applied to counts of rare events in two treatment groups. The Mann-Whitney sum of ranks test also provided superior alpha protection and power even where a large number of tied ranks occurred in the zero-count category of a J-shaped distribution. Finally, a 1 df chi-square test for linear shift in proportional representation across frequency categories of one group as opposed to the other was the most powerful of three chi-square tests that were evaluated.

To verify whether or not frequency of use is an acceptable measure of word difficulty in vocabulary test construction and translation, three 24 word lists, one English and two Spanish, were compiled. The first list of Spanish words matched the meaning and frequency of use of the corresponding English words, whereas the second list of words matched the meaning but not the frequency of use. The English list was administered to 80 English-speaking eighth grade students and the Spanish lists to samples of 80 Spanish-speaking students (40 Mexicans and 40 Puerto Ricans) matched to the English counterparts in grade, age, sex, and academic achievement. Means were compared for statistical significance of differences. Most of the comparisons between the English and the Spanish frequency matched synonyms-either with Mexicans or Puerto Ricans or with boys and girls-were not statistically significant, whereas most English-Spanish comparisons when frequency of use was not matched were significant. The results support the hypothesis that frequency of use is a valid measure of word difficulty that could be utilized in vocabulary test construction and translation.

The power to detect main and interaction effects in a factorial design was determined when the Bonferroni method was used to control the overall rate of Type I error at a conventional five percent level. For sample sizes typical of educational research, the power of this procedure is shown to be considerably less than that of recommended standards. Alternative applications of the Bonfer-roni procedure are illustrated and discussed.

The Spearman-Brown formula for correction of the split-half coefficient of reliability was discussed as a conceptual precursor of modern formulations of internal-consistency reliability. An alternative expression of the Spearman-Brown formula was suggested. Derived from the principal components of the product moment coefficient of correlation, this formulation facilitates understanding of the meaning of internal-consistency reliability. It also is conducive to classroom discussion of the key concepts of the classical theory of tests and measurements.

A procedure proposed by Kaiser (1968) for averaging correlation coefficients using the first eigenvalue of an intercorrelation matrix was studied via Monte Carlo methods. The Kaiser average if applied to sample intercorrelations was found to be markedly biased toward overestimation of the population correlation when the population correlation was small, and to show a slight negative bias similar to average r when the population correlation was large. A modification of the Kaiser average substantially reduced the bias for correlations near zero and showed slightly smaller standard errors (greater efficiency) than the other averages for small correlations. The least biased averaging procedure still appears to be the backtransformed average Fisher's z, particularly when sample size is small.

The present paper demonstrates how LISREL can be used to examine measurement model assumptions and to assess the reliability of a scale. LISREL was used to investigate (a) the nature of the basic measurement model for a scale, (b) scale invariance across time, and (c) scale invariance across groups. Information obtained in such analyses helps to assess accurately the estimate of the reliability of the scale. As an illustration, responses of 722 elementary school students to the Coopersmith Self-Esteem Inventory (Form B) were analyzed. Results indicated that a congeneric measurement model with correlated errors was most appropriate. Internal-consistency estimates were not affected by correlated errors, within each of the two time points (pretest and posttest). However, correlated errors across time produced an overestimate of the stability coefficient. Further, the factor structure of the scale was invariant across three ethnic groups, an invariance that implied similar reliability estimates across groups.

In the process of fitting a quantitative synthesis model the assumption of homogeneous effect size variation is typically assessed through omnibus test statistics. Whether individual studies fit the model may be answered by testing the extent to which the sample estimates deviate from the modelled population estimate. It is proposed that graphical representations may facilitate the analysis of effect size residual variation. This strategy is applied to data concerning techniques for the teaching of writing composition skills.

This paper presents a Monte Carlo study which explores the reliability of the inverse scree test for cluster analysis. Two sets of distributions were generated, and 6000 "computer subjects" were cluster analyzed and graphed. Three raters were then asked to detect the critical point at which the inverse scree line showed a marked break. Results indicated that the inverse scree may be a useful and reliable cluster analytic technique for determining the number of true groups.

Common applications of the part correlation coefficient are in causal regression models and estimation of suppressor variable effects. However, there is no statistical test of the significance of the difference between a zero-order correlation and a part correlation, nor between a pair of part correlations. This note uses Hotelling's (1940) t for contrasting dependent rs for contrasting a zero-order and first order part correlation, and for contrasting first- order part correlations based on different predictors and different covariates, the same predictor but different covariates, and different predictors and the same covariate.

The Environmental Assessment Index was developed to assess the educational/developmental quality of home environments of 3-year-old through 11-year-old children. The long form of Index is composed of 44 home observation and/or maternal interview items. The short form consists of 24 of the 44 items. The Index has been used to assess the quality of 62 rural children's homes in a three-year longitudinal study of ecological influences on rural children's mental and social development. The long form had a Cronbach alpha reliability of .74 and concurrent and predictive validity correlations with IQ estimates of +.66 and +.61 (p <.01) for preschool children. The two-year test-retest correlation was .81 (p <.001). The short form had a Cronbach alpha reliability of .82 and concurrent and predictive validity correlations with IQ estimates of +.64 and +.66 (p <.01). The short form test-retest correlation was .86 (p < .001). The scale items are included.

Pass/fail decisions for all two-method combinations of 10 standard-setting procedures were analyzed. The most reliable or equivalent methods were the practitioners—borderline group, Ebel—contrasting groups, and 33rd percentile—Angoff approaches. The least reliable or equivalent were the chance/ideal mean—masters group and Nedelsky—masters group approaches. The most valid of the 10 methods, as evidenced by correlations with an external criterion, were the practitioner and borderline group approaches while the least valid were the nonmasters, Nedelsky, and chance/ideal mean methods. Results of a more detailed study of the reliability and validity of the item-judgment methods are also reported.

Based on data from 260 adults, the present study investigated the structure of stereotypically defined love, and the concurrent validity of a measure of beliefs about this love. The results from several analyses suggest that stereotypic love is predominantly a "g"' factor phenomenon in which obsessive thought processes are distinctive. The concurrent validity analyses suggest that the factors underlying responses to items from the Love Relationships Scale are meaningfully related to measures of Lee's (1973/1976) typology of love styles.

The development and psychometric characteristics of the Instructional Preferences Inventory, a measure of "classroom" personality, are described. This instrument was designed to measure preference for course structure (PCS) and preference for course difficulty (PCD). Statistical findings supported the instrument's reliability and factorial validity.

The purpose of this study was to investigate whether interval and time sampling methods of measuring student engaged time are valid for estimating the actual time students spend engaged in relevant motor performance in physical education classes. Two versions of the interval Academic Learning Time in Physical Education (ALT-PE) instrument and an equivalent time sampling method were compared with actual time students spent in motor performance. Thirty-six students enrolled in six different university physical education classes were the subjects. The actual student engaged time and engaged time as estimated by the time sampling instrument were significantly lower than the time as estimated with the two ALT-PE instruments. In addition, a class type by coding instrument interaction occurred indicating that the difference was present for all classes except where activity is continuous. This result suggests that the popularity of the ALT-PE instruments should be reexamined.

Four studies were conducted to examine the relationship between academic self-efficacy and performance in college courses. Measures of academic grade goals and ability were also obtained. The first study was used to select valid items from a larger pool of self-efficacy measures. The items were further revised and the new item set was then tested out on the sample from study 4. A final set of items was selected and the resulting self-efficacy scale replicated on two other samples (studies 2 and 3). Self-efficacy was found to be significantly related to academic performance and to self-set academic grade goals. Grade goals and ability were also related to course performance. These results are consistent with laboratory studies of the relation of self-efficacy and goals to task performance.

In an effort to provide additional information concerning the relationship between study habits and college grade-point average (GPA), the College Adjustment and Study Skills Inventory (CASSI), high school rank (HSR), and the Verbal as well as the Mathematics scores on the College Board Scholastic Aptitude Test (SAT-V and SAT-M) were intercorrelated. Three stepwise multiple regression analyses were computed, with GPA as the criterion measure. One multiple regression equation used as predictors six subscales of the CASSI; another employed HSR, SAT-V and SAT-M; and the third, used a combination of HSR and the Time Distribution scale of the CASSI. Also, differences between means of males and females on all measures were calculated.

Results showed that all the academic variables and all but one of the CASSI variables were correlated significantly with GPA; females were significantly higher than males on HSR and GPA; and males were significantly higher than females on SAT-M. The combination of SAT-V, SAT-M, HSR, and Time Distribution yielded a coefficient of multiple determination (R²) that was 16% higher than that obtained from HSR alone.

The predictability of Grade Point Average (GPA) from College Level Examination Program (CLEP) General Examination scores (in English, Natural Science, Social Science and History, and Humanities) was determined for a sample of 119 undergraduate continuing education students. In addition, correlations between each General Examination and GPA, and intercorrelations among General Examinations were obtained. A stepwise multiple regression procedure revealed that CLEP Social Science and History scores accounted for 19% of the variance in their prediction of GPA, followed by CLEP English scores which accounted for 3% additional variance. An intercorrelation matrix for all General Examinations and GPA revealed significance for all possible correlations. Implications of results are discussed.

The primary aim of this study was to test for age bias in the predictive validity of scholastic aptitude test scores. The analyses were based on the college admissions aptitude test scores of 795 students applying for admission to a major Northern Israeli university. The students were partitioned into the following four age groups for data analytic purposes: 18-21, 22-25, 26-29, and 30+. On the whole, composite aptitude test scores are shown to be poorer predictors of first year cumulative grade point average (= GPA) for students in the oldest age group (30+) in comparison to students in the remaining age groups. Furthermore, aptitude test scores were observed to interact significantly but modestly with age group membership in differentially predicting academic achievement for various age groups. Overall, the data are consistent with some previous research conducted among student samples in the U.S., attesting to the differential predictive validity of aptitude test scores by age group.

The difficulty of 548 sections of 55 courses taken by 2906 freshmen during their first two quarters at a university were examined in light of the influence of these sections on student attrition, and long-term and short-term grades. Approximately 18% of the courses (sections) were identified as difficult, and 18% were identified as easy. The difficulty of courses taken by students during these terms was found to influence long-term grades and attrition through their influence on short-term grades.

For a total sample of 418 students pursuing the diploma program in nursing and for each of four ethnic subgroups comprising 272 Whites, 67 Hispanics, 34 Blacks, and 20 Asians (an undeclared group of 25 students not being considered), this investigation was concerned with the validity of each of 12 predictor variables representing age, high school grade point average (HS-GPA), GPA in three prerequisite courses and in a composite of these three courses, scores in three reading measures, and scores in three mathematics measures relative to each of two criterion variables reflecting (a) cumulative grade point average in required courses taken in a school of nursing (SN-GPA) and (b) standing in the National Council Licensure Examination for Registered Nurses (NCLEX-RN). Parallel information also was sought for eight intermediate variables represented by scores on standardized achievement tests of the National League of Nursing (NLN) corresponding to the content of key academic courses and administered at the end of each course. Selected conclusions were that (a) SN-GPA is a highly valid predictor of NCLEX-RN scores for members of the White and Hispanic subgroups, but only a modestly valid predictor for those in the Black and Asian subgroups, (b) HS-GPA affords little, if any, predictive validity of either criterion measure for any ethnic group, (c) prerequisite coursework taken primarily in a community college affords moderate predictive validity for White, Hispanic, and Asian subgroups but high validity for the Black subgroup, in the prediction of SN-GPA, but reduced if not marginal validity for all ethnic subgroups relative to standing on the NCLEX-RN, (d) standardized achievement tests in reading and mathematics provide somewhat mixed validity promise across the various ethnic subgroups, (e) the NLN measures, with a few noteworthy exceptions, furnish consistently high validity coefficients across all subgroups for both criterion measures.

The present study evaluated the validity of a revised version of the Multidimensional Health Locus of Control (MHLC) Scales when the measure is employed with older elementary school children. Confirmatory maximum-likelihood factor analysis employed data from 876 students. The factor structure of the measure matched expected structure fairly closely, although a few deviations from expectation were noted. Thus, the results supported the validity of the measure when it is employed with children.

This study was designed to measure pupil disruptive behavior in schools. In order to measure pupil disruptive behavior, an instrument called the Pupil Disruptive Behavior (PDB) Scale was developed and tested. A 64-item instrument was administered to 227 public high school educators in Illinois, Iowa, and Wisconsin. A principal components analysis revealed three interpretable factors accounting for 45% of the variance. A 29-item PDB Scale is presented that shows moderately high reliability at the high school level.

The present study examined concurrent validity between the Nowicki-Strickland Locus of Control Scale and the Behavior Rating Profile: Home, School, and Peer Scales. In all, 268 children (131 males and 137 females) in grades five through eight participated. Results indicated significant negative correlations between locus of control and measures of home, school, and peers for both males and females. Implications are discussed.

This study compared the reliability and validity of two different measures of reading ability, the Degrees of Reading Power (DRP) and the Iowa Tests of Basic Skills (ITBS) Reading Comprehension test and the ITBS Vocabulary test. The data consisted of scores of 377 grade 5 and 6 students on these tests, assigned reading levels, and teacher-assigned grades. Intercorrelations, including disattenuated values, were the primary method of data analysis. The differences between the ITBS and DRP on the reliability and validity evidence presented were minor and appeared to be of limited practical significance.

The factor structure of the Career Decision Scale, but not total scores, showed gender differences in a sample of undergraduate college students. The factor structure of the CDS tended to be more stable for females than males over a six-week interval. The only item on the CDS that describes an interpersonal reason for being undecided was omitted from the factor structure for females, but not males. Reliance on total scores remains the most useful option because of the instability of the factor structure over time and across samples. In the search for variables which contribute to our understanding of career indecision, those which differentiate between the indecision experienced by males and females are still undetermined and are a useful area for further research.

In a longitudinal follow-up investigation spanning approximately five years with an initial sample of 135 freshman music majors, the Musical Aptitude Profile (MAP) (Gordon, 1965) revealed some statistically significant validity coefficients for Tonal Imagery subtest scores, Rhythm Imagery subtest scores, and the Composite Score of the MAP relative to course grades in music theory for the first three semesters of coursework and grades in the components of these music theory courses. Whereas the Tonal Imagery subtest was found to be the most valid predictor of success in the three components of each of the first two courses in music theory, it afforded no significant predictive validity for grades in any of the components of the third semester course. The Rhythm Imagery subtest of the MAP, however, continued to be modestly valid in predicting grades in the sight-singing and ear-training components of the third semester of music theory. No statistically significant validity coefficients were obtained for the MAP as a predictor of grades in the fourth and fifth semesters of music theory coursework or in applied music.

One hundred ninety-two students at an APA approved doctoral program in clinical psychology were examined with regard to the relationships between admissions criteria and evaluation of performance within the graduate program. Predictors for the study included GRE scores, UGPA, MMPI scores, indices from the SVIB, indices from Letters of Recommendation, and Biographical/Educational information. Performance criteria included Graduate GPA, Preliminary Oral Interview Scores, Comprehensive Examination Scores, Professional Qualifying Exam Scores, Practicum Evaluations, Internship Evaluations, Peer Ratings, and Faculty Ratings. Pearson Product Moment Correlations were performed to examine relationships between Pre-Admission predictor variables and each of the Within-Program performance criteria. A series of Factor Analyses were then performed on predictor and criterion variables and Pearson Product Moment Correlations were computed on the resultant factors. Results showed that the most meaningful findings are personality variables, which were useful predictors of Internship Evaluations. Several indices from Letters of Recommendation were found to be highly correlated with later graduate school performance measure. Factor analysis of variables contributed little to the understanding of relationships between predictors and criteria. Implications of these findings and recommendations for future prediction research in psychology are discussed.

The Revised Learning Style Inventory (LSI II) was examined for internal consistency, test-retest reliability and stability of the four classifications resulting from the scores of subjects employed in industry. The results from this study were compared to the results of similar research conducted on a student sample (Sims, Veres, Watson, and Buckner, 1986). The internal consistency of the industrial sample was comparable to that of the students. Test-retest indices and classification stability, however, remained low for the industrial sample, a fact failing to support the Sims et al. hypothesis that the learning styles of students were not fully developed and thus contributed to observed instability. The results of this preliminary study in industry indicate that problems noted with students continue to operate in the assumably "more stable" industrial setting. Further research is discussed to identify the source of LSI instability.

The present study was conducted to assess the relative sensitivity of three rating scale formats (graphic, BES, & BOS) to behavior and performance cue primes. Subjects were 72 college students who rated video-taped instructor performance on the three types of rating scales. The experimental design was a 2 (Low vs. High Behavior) × 2 (Negative vs. Positive Performance Cue). The relative sensitivity of each scale was defined as the proportion of variance explained by behavior divided by the proportion of variance explained by behavior plus performance cue variance. The results indicated that in terms of sensitivity to behavioral variance the Graphic was most sensitive followed by the BES and then BOS scales.

In this study, the process a school might use to build a "customized test" from an "on shelf' standardized achievement test was followed, as closely as possible. The main purpose was to assess the effects of using "customized test" results rather than full test results in estimating examinee ability levels. To this end, ability parameters were estimated for groups of students on the Quantitative Thinking subtest of a standardized achievement test battery as well as on customized versions of the test. It was found that the ability estimates from the full-length tests and those from the customized tests at times differed substantially.

Fifty undergraduate students (27 female, 23 male) completed measures of verbal intelligence and locus of control, together with the Responsibility scale of the California Psychological Inventory and a student-written scale of Personal Responsibility. On the basis of correlational and multiple-regression analyses, it was found that Intelligence and Personal Responsibility were jointly the most efficient predictors of a subject's mean course grade, and that the latter was the best single predictor. These results add further support to Jackson's (1971) contention that naive item writers can create personality scales of equivalent or greater validity than more formal empirically-derived scales.

The Experimental Minnesota Multiphasic Personality Inventory (EMMPI) consists of 566 nonpathological personality items, where each item is paired with a corresponding item in the Minnesota Multiphasic Personality Inventory (MMPI) such that both items have approximately the same social desirability scale values and the proportion of True responses. The results of the secondary analysis applied separately to the 21 MMPI trait scales and to the corresponding 21 EMMPI scales indicated that their first factor loadings were congruent and both correlated highly with the proportion of items in the scales keyed for socially desirable responses. It is proposed that both the EMMPI and the MMPI first factors be named the positive self description (PSD) factor.