Educational and Psychological Measurement

An analytic procedure is presented which casts sequential moderator analysis in the role of a multivariate test of parallelism of regressions. The procedure addresses a test for comparing predictor-criterion relationships for one set of measurements on multiple predictors and repeated measurements on a criterion. The application of the sequential moderation test to issues associated with dynamic criteria is discussed.

Formulas in the statistical theory of test scores have led some psychometricians to believe that measures of change and growth have questionable value in research. However, certain combinations of parameters, when substituted into these formulas, yield reliable change scores and high non-spurious correlations between change scores and independent criterion scores, even when pretest scores are not good predictors of either changes or the criterion. Because of mathematical constraints, these particular combinations of parameters are the ones to be expected in research designs if valid and reliable changes in individuals' test scores do occur. Accordingly, it is possible for measures of change and growth to have excellent predictive value, as investigators in many fields have taken for granted, and, conversely, independent variables can be excellent predictors of changes. Although criteria which are highly correlated with changes are difficult to discover empirically, their existence cannot be ruled out by statistical arguments alone.

Several authors have encouraged the use of strength of association measures as a step beyond tests of significance. However, it is still rare to see these measures reported in the literature. Since most journals do not publish complete ANOVA tables, it is generally not possible for readers to compute these measures for themselves. Practical formulas for several ANOVA designs and models are presented which make it possible for readers to compute these measures without the use of complete ANOVA tables.

Tables are presented giving the critical values of the Biserial and the Point Biserial correlation coefficients (when the null hypothesis assumes a value of zero for the coefficient) at the 0.05 and the 0.01 levels of significance.

It is often considerably easier to obtain the value of the point biserial correlation coefficient for a given set of data than it is to obtain the biserial correlation coefficient. A table is presented that directly converts any known point biserial coefficient to the biserial coefficient providing the largest proportion of the dichotomous variable is also known.

When several variates are measured on the individuals of a single sample, it is often of interest to test whether or not two population correlation coefficients are equal. For example, in the behavioral sciences, it is often desirable to know whether two mental tests have equal test-retest reliability. In the present paper, eight asymptotically robust tests are proposed and are studied along with two parametric tests recommended by previous studies. Monte Carlo simulation is used to compare the small sample performance of these ten tests under the null hypothesis H_o : ρ₁₂ = ρ₃₄ where ρ _ab is the population product moment correlation coefficient between variables X_a and X_b . Seven distributions are considered; a normal distribution, two mixed normal distributions, and four other nonnormal distributions.

This study investigates four widely used methods of collecting direct similarity judgments for nonmetric multidimensional scaling: Rating Scale, Triads, Rank Order of Pairs, and Conditional Rank Order. The results indicate that: (1) when class structure is very striking in the attributes to be scaled, respondents under the Rating Scale method place significantly more weight on class structure than on continuously varying attributes in determining similarity; (2) nonmetric multidimensional scaling solutions account for a greater proportion of the variance in Rank Order of Pairs data than in data collected by the other methods studied, although the differences are so small that they would have little operational impact; and (3) the four methods do not differ significantly in test-retest reliability.

The cross-lagged correlation technique (CLC) for assessing causality from passive observational data has been the subject of much recent criticism. CLC compares cross correlations between variables across time points of measurement and attributes differences in correlations to causal effects. Some of the major areas of criticism have to do with: (a) lack of a no-cause baseline, (b) spurious effects of mediating variables, and (c) obscuring effects of heterogeneous stabilities. The present author attempts to show that while these criticisms are valid, CLC should not be rejected outright as a tool in social science research. CLC has some utility as an exploratory technique. Furthermore, if certain assumptions are specified and these assumptions are correct, valid inferences can be made from CLC. Lastly, some assumptions for CLC are not as stringent as implied by critics and CLC is robust to minor violations of assumptions.

A common practice among clinicians and counselors is the administration of comprehensive personality inventories to their clients, followed by examination of the various inter-scale, intra-profile comparisons which are possible. The significant difference scores are then examined for theoretical significance, and diagnosis or placement directives may be based on them. A method is derived in which the magnitude of the critical difference (which an obtained difference score must equal or exceed in order to be considered statistically significant at α < .05) is a function of the number of comparisons being made and the average individual reliability of the K scales. A nomograph is provided from which the critical difference may be read directly.

It is argued that analysis of variance F-ratios which are significant in the left tail of the appropriate central F distribution should be interpreted as indicating some inadequacy in the analysis of variance model being used to represent the properties of the data. Procedures for testing left-tailed F-ratios are indicated.

Four methods of item bias detection-transformed item difficulty, item discrimination expressed as Clemans' lambda, ChiSquare, and the three-parameter item characteristic curve-were studied to determine the degree of correspondence among them in identifying biased and unbiased items in reading and math subtests of the 1978 SRA Achievement Series. Intercorrelations among the four methods were moderate at best, confirming previous research involving different item bias analysis techniques. The item discrimination method showed the least correspondence with the other three methods. The extent of overlap between the four item bias methods in identifying biased items depends on the extent of bias in the items comprising the initial pool of items. That is, except for the item discrimination method, the item bias procedures identify similar sets of most biased items.

A sample of 220 American college students completed both the Eysenck Personality Inventory and the Eysenck Personality Questionnaire. Consistent with the Eysencks' assumptions, the corresponding Extraversion scales from the two inventories correlated .80, and the two Neuroticism scales correlated .87. In addition, within the Eysenck Personality Questionnaire, Psychoticism was independent of both Extraversion and Neuroticism. With the exception of the Lie scales, test-retest reliabilities ranged from .85 to .94. Means and standard deviations were reported for all scales, both for the total sample and by sex.

A paper-and-pencil test of seven of Inhelder and Piaget's formal operational schemes was developed. This objective assessment of formal operational thought was cross-validated clinically. The multitrait-multimethod procedures indicated that the objective test is a valid and reliable measure of formal operational thought. It has the advantage of being administered in large groups, of being easily scored, and of not requiring special training for its administration.

Within the context of a causal modeling system involving use of Joreskog's confirmatory maximum likelihood factor analysis in conjunction with the LISREL IV computer program devised by Joreskog and Sorbom an evaluation was made of the construct validity underlying the higher order factor structure of a given correlation matrix of 46 structure-of-intellect (SOI) tests emphasizing the product of transformations. The tenability of four alternative factor models was examined relative to their accounting for the intercorrelations among the 46 tests that had been hypothesized to represent first-order abilities involving one kind of operation, one type of content, and one form of product. The major elements of each of the three postulated dimensions of the SOI model (the five operations, the five contents, and the six products) were defined as the 16 possible third-order factors. The following conclusions resulted: (a) the five third-order SOI factors comprising operations of cognition, convergent production, divergent production, memory, and evaluation do not appear to be highly differentiated (with the cognition and convergent production factors being substantially correlated); (b) the two third-order factors of symbolic and semantic content are relatively discrete; (c) the third-order product factor involving the ability to complete tasks requiring hypothesized transformations is an identifiable construct; and (d) evidence of the construct validity of the third-order factors in the SOI test battery analyzed appears to be at best only equivocal. The predominance of tests intended to reflect the product of transformations (32 out of 46) very possibly created inflated levels of intercorrelations among SOI tests that prevented the emergence of distinct higher order dimensions. New confirmatory maximum likelihood factor analyses of correlational matrices derived from SOI tests in which a careful balance has been achieved in the representation of hypothesized third-order factors across the three basic dimensions of the SOI model are expected to demonstrate differentiability among the higher order constructs.

Thirty-four tests measuring verbal ability, academic achievement, reading skills, attitudes, visual perception, spelling, and word decoding skills were compared for their correlational strengths with grade in a required freshman course in composition. In several modes of analysis scores from the English subtest of the ACT Assessment yielded the strongest and most pervasive relationship to composition grades. Several other language tests showed varying power to discriminate between groups of students with high, middle, or low grades in the composition course, but a strong correlation between scores measuring word-level skills and grade in freshman course in composition suggests their unique potential in test batteries designed for the differential placement of students into English composition courses.

This paper examines the concurrent validity of self-rating and interviewer rating of English as a second language proficiency for a stable population of rural Australian migrants. Ratings obtained for two versions of the Australian Second Language Proficiency Ratings Scale for the macroskills of listening, speaking, reading, and writing indicated that migrants tended to overestimate their proficiency with English. However, the strong correlations between their self-ratings and the interviewer ratings showed that self-rating technique is valid for the collection of large-scale research data.

The development and validation of the Course Structure Inventory (CSI), a measure of student attitude toward course structure and course difficulty, is described. A sample of 215 college students was given the CSI along with measures of arousal-seeking tendency, achievement motivation, and dogmatism. A significant negative correlation was found between scores on the arousal seeking measure and those on the attitude toward course structure subscale of the CSI (high scores indicating preference for high structure). A significant positive correlation was observed between scores on the dogmatism measure and the attitude toward course structure subscale. Finally, a significant negative correlation was found between scores on the dogmatism measure and those on the subscale representing attitude toward course difficulty (high scores indicating a preference for more difficult courses). Potential applications of the CSI are discussed.

The Berkeley student evaluation questionnaire by Hildebrand, Wilson, and Dienst was modified to reflect the Whitely-Doyle factorial structure of implicit theories students hold about teaching characteristics of their instructors. Evidence of factorial validity with respect to hierarchial order of faculty academic ranks was found for factors pertaining to the knowledge of the subject matter and expositional skills. With respect to these factors, student ratings appear to be valid indicators of teaching competency, admissible as evidence to be used for faculty promotion decisions.

In professional education, there is continuing interest in the use of student evaluations of instructional effectiveness. The intent of this study was to describe the dimensions of effective teaching behavior in a clinical setting and to determine the factorial validity of a scale used to assess clinical teaching behaviors. Medical students in their third year of training responded to a 40-item questionnaire in reference to their recently completed clinical experiences. Their responses were subjected to alpha factor analysis. Results indicated that the dimensionality of clinical teaching could be reduced and described in terms useful for specification of teaching competencies about which inservice education for clinical teachers could be designed.

Measures of academic job involvement, verbal ability, academic performance, and satisfaction with courses and with college in general, obtained for 155 students during the freshman year, were correlated with attrition on a two-year follow-up. The two significant predictors of attrition, grade point average during the fresh-man year and satisfaction with college in general, combined to yield a multiple correlation of .42 (P < .01).

The admissions process for a graduate Master of Business Administration (MBA) program was modeled for 406 applicants to a newly designed MBA program. The decision process model was then applied to a sample of students admitted to and enrolled in an old MBA program, where less stringent admissions procedures had been used. Results supported the adequacy of both Graduate Management Admission Test scores and undergraduate grades in the prediction of admissions officer decisions as well as the validity of such information in the prediction of graduate performance and turnover. The results are discussed in light of the recent criticisms of the use of standardized tests as admissions criteria for programs of higher education.

Results of a study undertaken to determine the predictive validity of admission variables used to screen applicants for graduate study in business at a medium sized university are reported. Stepwise linear regression analysis was utilized to identify the variables which explain significant amounts of variance in graduate grade point average (GGPA)—the criterion variable. The study revealed that the following variables explained slightly less than 21 percent of the total variance in GGPA: (1) junior/senior undergraduate grade point average (GPA), (2) Graduate Management Aptitude Test (GMAT) score, (3) full/part-time attendance in the graduate program, (4) number of credit hours required in the Master of Business Administration (MBA) program, (5) undergraduate major, (6) age and (7) pursuance of the MBA at the same institution where the student received the undergraduate degree. Junior/senior GPA, GMAT score, and full vs. part-time attendance in the MBA program were significant at the .01 level.

For a sample of 570 female nursing students in a three-year, hospital-based, nursing diploma program, the predictive validity coefficients were determined for the ACT Assessment composite score, for ten subscales of the Edwards Personal Preference Schedule (EPPS), and for three scales (Lie, F, and Pd) of the Minnesota Multiphasic Personality Inventory (MMPI) relative to a criterion measure of grade point average (GPA) through graduation or point of withdrawal. The validity coefficient of .35 for the ACT Assessment composite score was statistically significant (p < .01). One of the validity coefficients for any other predictive variables (which ranged from -.09 to .08) was significant. The multiple correlation coefficient of .377, associated with 14 predictor variables, was not significantly larger than the zero-order coefficient afforded by the ACT Assessment composite score alone. A stepwise discriminant analysis, relative to a dichotomous criterion variable of graduation versus elimination from the program, yielded essentially parallel results to those provided by the multiple regression analysis. On the basis of these findings, it appears that personality scales hold little promise as predictive variables of academic success in a three-year, diploma nursing program, relative to which criterion measures (such as GPA) confound academic and patient care activities. Further study is needed in nursing programs for which criteria involve only one of these two areas of performance.

At an upper division university, students entering the Bachelor of Science in Nursing (BSN) program were graduated from Associate Degree in Nursing (ADN) programs or graduated from hospital-based (diploma) programs. The admission criteria for the two groups of students were almost identical. The exception was that the diploma nurses were required to pass three ACT Proficiency Examination Program (PEP) tests with a standard score of 45 or more. The cumulative grade point averages (CGPAs) of the two groups of students at the upper division institution were compared. The CGPA of the diploma nurses was significantly higher than that of their ADN peers. Thus, the PEP credit-by-examination testing model was found to be an effective alternative to determine the advanced standing of students graduated from hospital-based nursing programs who were entering a BSN program at an upper division university.

Scores on three pretest forms of a content-validated multiple-choice test for entry level social workers were correlated with job performance ratings by social worker supervisors. Generally positive but mixed correlations were found between scores on the Social Work subtest and performance ratings. For social workers who deal mainly with urban clients, correlations between Reading-English subtest scores and performance ratings tended to be slightly positive but markedly lower than those for Social Work subtest scores. Correlations of Social Work subtest scores with ratings of empathy and creativeness were among the higher ones obtained. Women pretest examinees had significantly higher Social Work subtest scores than did men on two of the three pretest forms. Test scores predicted the job performance ratings of workers who were employed in urban settings much more accurately than those of workers who were located in rural settings. The existence of differences in knowledge held by supervisors of their workers was offered as a possible explanation of this finding of differential validity.

An 11-item questionnaire to ascertain the incidence and significance of major risk factors for atherosclerotic cardiovascular diseases was used to survey a nationwide television audience. Of 250,000 responses, 131,603 were complete and suitable for inclusion in the validity study of the questionnaire. Results of factor analysis and stepwise multiple regression analysis indicate that the questionnaire is a valid instrument for predicting the risk of developing atherosclerotic heart disease.

For each of two groups of 34 and 71 undergraduate college students respectively categorized as clinically test anxious (CTA sample) and as non-test anxious (NTA sample), evidence was sought regarding the discriminant validity and the concurrent validity of each of two measures—the 20-item self-report Test Anxiety Inventory (TAI) and the 8-item behavior analogue measure of test-taking performance entitled Anagram Solution Task. Normative information was also obtained for each measure to provide a basis for classifying students as being test anxious or as being non-test anxious. The statistical outcomes led to the conclusion that both the TAI and AST exhibit substantial discriminant validity, but relatively little concurrent validity. The creation of 90% confidence scoring intervals for each of the two measures shows considerable promise as a means for identifying students with differential levels of test anxiety. These instruments appear to afford a basis for both counseling and research purposes. Future directions in the assessment of test anxiety are discussed.

A modified multitrait multimethod validation matrix was generated to evaluate the construct validity of the newly revised Children's Manifest Anxiety Scale (RCMAS). For both males (N = 49) and females (N = 37) a pattern of results occurred that was generally consistent with the use of the RCMAS as a measure of chronic, manifest anxiety levels in children. Large correlations occurred with other trait measures of anxiety but not with state anxiety measures or IQ. Smaller but consistently positive correlations occurred between children's ratings of their anxiety and teacher observations of behavior problems.

The NachNaff scale is a 30-item questionnaire that requires examinees to choose between self-descriptions consistent with either need for achievement (n Ach) or need for affiliation (n Aff). This scale, which is scored in the n Ach direction, was administered to a sample of 108 female and 56 male undergraduates. Women's n Ach scores were found to be correlated with scores on Srong Vocational Interest Blank scales and on the California Psychological Inventory scales in ways that were generally consistent with n Ach theory. Statistically significant correlations for men were fewer and were less in magnitude than were those for women. The NachNaff scores were positively correlated with GPA for both sexes.

Evidence is reported for construct validation of the Good and Good measure of social-power motivation. Subjects (N = 121) completed the Good and Good social-power motivation scale. A questionnaire also developed for the present study requested certain biographical data as well as information believed by the present authors to be indicative of manifestations of the need to obtain social power. Of the 20 variables studied as a basis for construct validation of the Good and Good measure of social-power motivation, 15 demonstrated significant results in the expected direction. Of the two biographical variables, age emerged as a significant variable, but sex did not.

To establish further the validity of the Interpersonal Jealousy scale, the investigators correlated it with each of two behavioral measures of (a) threat when one is confronted by a rival and (b) expressions of possessiveness toward the partner. Significant correlations were found to provide evidence of validity. As the Interpersonal Jealousy Scale and the behavioral measures of threat and possessiveness did not share method variance, the obtained validity data were judged to be impressive.

The present study attempted to determine whether the 10 subscales of the Family Environment Scale do measure 10 dimensions (factors) of the family environment. In a factor analysis of data from high school freshmen and sophomores (n = 344), the 10 subscales did not emerge as 10 factors. Hence, seven new sub-scales were proposed: Conflict, Group Cohesion, Structure, Religion, Activities, Verbal-Intellectual Orientation, and Protestant Ethic. The responses of a second sample of students (n = 342) were then factor analyzed to determine the stability of the proposed subscales. The factor structure of the first four subscales was interpreted to replicate well, whereas that of the last three subscales was judged to replicate moderately well. Implications of the factor structure for conceptualization and measurement of family environment variables are discussed.

In two different investigations constructs of family orientation (including marital perception) and status perception were exposed to construct validation procedures involving a sample of 1,428 respondents randomly selected from two larger representative national samples of American adult men and women utilized in a 1957 study and in a replication of that study in 1976. Data were based upon a thematic apperceptive procedure in which projective stories were coded for perceptions of family and marital relations, interpersonal conflict, and status differences. With selected variables from the coding and from the questionnaire, these concepts were validated according to two independent procedures. Support for face validity was obtained only with demographic variables, and in construct validity the strongest relationships were observed with variables from the projective measure rather than from the questionnaire. Theoretical issues pertaining to the meaning of the Thematic Apperception Test and to the advantages and disadvantages of different methodological approaches to validity are discussed. Particular attention is given to the influence of picture stimuli in the interpretation of results.

The factorial structure of the California Achievement Test was examined for 3813 public school students (2266 fifth-grade subjects and 1547 seventh-grade participants) in the Tucson Unified School District. Intercorrelation matrices were analyzed by using principal factors solutions. The factorial structure consisted of one large common factor in each grade analysis. The findings of this study indicated that there was considerable redundancy among subtests. The use of subtest scores as measures of independent abilities is questioned.

Scores reflecting the attitudes of fourth, fifth, and sixth-grade Mexican-American and Anglo students about reading were compared to assess the relevance and validity of the Estes Reading Attitude Scale for each group. The validity of application of results from the Estes Reading Attitude Scale when administered to students of differing ethnic backgrounds was examined by considering the factorial complexity of the instrument overall for Anglo and Mexican American children, as well as at three grade levels. The comparability of the internal consistency indices and of factorial structure between ethnic groups within grade levels, and the consistency of item interpretation (or lack of item "bias") between ethnic groups within grade level suggested that scores on the scale afforded a valid means for interpretation of attitudes held by elementary school pupils towards reading irrespective of their ethnic status (Anglo or Mexican-American).

Guided by Public Law 94-142 in which it is mandated that assessment instruments be validated for specific use on exceptional populations, the intent of the present study was to examine the effectiveness of the McCarthy Scales of Children's Abilities in predicting school achievement (as measured by the Comprehensive Tests of Basic Skills) for 31 English-speaking, second-grade Mexican American children. It was found that the validity coefficients between McCarthy Scale scores and total Reading/Mathematics achievement scores ranged from .39 to .75. The observed r of .75 between the McCarthy General Cognitive Index and Reading achievement was of comparable magnitude to what has been found in the literature. The finding that the Verbal Scale was the least valid composite predictor of Reading Achievement was discussed in the context of psychometric and cultural considerations. It was concluded that in light of other McCarthy research on Mexican American children and the findings of the present study, the McCarthy appears to be a promising tool in the assessment of English-speaking Mexican American children.

Although several managerial job analysis questionnaires have been developed, they are not widely used in industrial situations. One reason for this lack of use may be that the available instruments tend to require one or more hours to complete. The current study reports the cross-validation of the Short Form of the Position Description Questionnaire (PDQ). A secondary analysis of data with the PDQ on a cross-validation sample of 2,280 managers indicates that the PDQ-Short Form has measurement properties similar to the PDQ and that it its stable across samples. Applications of the PDQ Short Form for individualized training, career pathing, and research are discussed.

A study of the criterion-related validity of the Crites' Career Maturity Inventory (CMI) was undertaken in a secondary school. Success and satisfaction in a work-study program and grade-point average were used as the criterion measures. Correlational analyses provided generally promising results on the validity of the Competence Test in the CMI.

Similar scales on the COPSystem Interest Inventory and the Kuder General Interest Survey were compared in terms of correlations between the scales. Congruence of results in relation to profile similarity in the high interest areas was determined by comparing the three highest interest areas on the two instruments. The magnitude of coefficients between conceptually similar scales ranged from low (r = .21) between the Kuder Persuasive scale and the COPS Business, Skilled scale to moderate (r = .49) between the Kuder Scientific scale and the COPS Science Professional and between the Kuder Literary scale and the COPS Communication scales. The pattern of correlations wherein the magnitude of positive correlation was higher between conceptually similar scales than between any other scales in the array provided evidence of construct validity for these scales. A comparison of the three highest areas of interest from both instruments revealed that 89% of the sample had at least one area of interest the same on both instruments. For 25% of the sample, results from the two instruments were the same in terms of the three highest areas: that is the top three areas of interest were the same on both instruments. These results are interpreted as supporting the construct validity of the instruments.

The Vocational Preference Inventory (VPI) and the Hand Test were administered to 120 male undergraduate students. Subjects classified into one of six Holland personality types on the basis of VPI high-point codes were found to differ significantly with respect to both the absolute and relative number of responses on certain Hand Test scoring variables. As these differences appeared to be reasonably consistent with Holland's personality theory, additional evidence was provided for the construct validity of the postulated vocational types.

For a sample of 116 first-grade pupils, scores from a reading attitude inventory were correlated .18 (p < .05) with the Reading composite scores on the Metropolitan Achievement Test. Coefficients of .48 (p < .01) and of .32 (p < .01) were also found for females and blacks, respectively. The low degree of correspondences, however, limited the practical significance of the results.