Educational and Psychological Measurement

Reformers have long argued that misuse of Null Hypothesis Significance Testing (NHST) is widespread and damaging. The authors analyzed 150 articles from the Journal of Applied Psychology (JAP) covering 1940 to 1999. They examined statistical reporting practices related to misconceptions about NHST, American Psychological Association (APA) guidelines, and reform recommendations. The analysis reveals (a) inconsistency in reporting alpha and p values, (b) the use of ambiguous language in describing NHST, (c) frequent acceptance of null hypotheses without consideration of power, (d) that power estimates are rarely reported, and (e) that confidence intervals were virtually never used. APA guidelines have been followed only selectively. Research methodology reported in JAP has increased greatly in sophistication over 60 years, but inference practices have shown remarkable stability. There is little sign that decades of cogent critiques by reformers had by 1999 led to changes in statistical reporting practices in JAP.

The importance of interpreting structure coefficients throughout the General Linear Model (GLM) is widely accepted. However, regression researchers too infrequently consult regression structure coefficients to augment their interpretations. The authors reviewed articles published in the Journal of Applied Psychology to determine how interpretations might have differed if standardized regression coefficients and structure coefficients (or else bivariate rs of predictors with the criterion) had been interpreted. Some dramatic misinterpretations or incomplete interpretations are summarized. It is suggested that beta weights and structure coefficients (or else bivariate rs of predictors with the criterion) ought to be interpreted when noteworthy regression results have been isolated.

Five procedures for estimating a common risk difference in a set of independent 2 × 2 tables were assessed via Monte Carlo simulation in terms of their bias, efficiency, confidence level adjustment, and statistical power. The maximum likelihood estimator showed the best performance, very closely followed by Cochran’s and Mantel-Haenszel’s procedures. The conditional weighted estimator, d CW, showed an irregular performace. The unweighted estimator, d U, showed less efficiency and statistical power than that of the other procedures. As a consequence, the use of the d CW and d U estimators is not recommended. The implications of the results in the practice of meta-analysis are discussed.

This article addresses the problem of accounting overall chance-corrected interobserver agreement among the multivariate ratings of several judges. Modifying an approach by Berry and Mielke, a generalization of Cohen’s kappa coefficient is proposed. The generalized statistic accounts agreement for multivariate interval or nominal observations among several judges. The statistic’s metric is conventional, and in the univariate case it is equivalent to existing extensions of the kappa coefficient to several observers, as well as to a certain intraclass correlation coefficient. Application is exemplified using hypothetical and real-life data sets.

Two issues regarding the factor structure of the Sixteen Personality Factor Questionnaire (16PF) Fifth Edition were investigated: (a) unidimensionality of the 16 noncognitive scales and (b) hierarchical factor structure of the inventory. The Schmid and Leiman orthogonalization procedure was applied to obtain a hierarchical factor solution in which multi-item composites were mapped directly onto first- and second-order orthogonal factors while simultaneously illustrating the relative positions of factors in the hierarchy. Unidimensionality was assessed using modified parallel analysis. The results indicated that the noncognitive multi-item composites could be factored into 16 first-order and 5 second-order orthogonal factors.

The psychometric properties of the Beck Hopelessness Scale (BHS) were examined among 544 university students. Internal consistency of scores was satisfactory (Cronbach’s alpha = .88). A principal-axis factor analysis with oblique rotation suggested four factors that were largely uninterpretable. Removal of items and further analysis suggested a two-factor structure of questionable interpretability. Convergent validity was demonstrated by high correlations with the Hope Scale and the Life Orientation Test (LOT). A correlation of .57 between the BHS and perceived stress reduced to .24 when trait negative affect was controlled for, indicating a degree of discriminant utility. It is concluded that for normal populations, measures such as the LOT and Hope Scale are more appropriate.

The present study investigated descriptive statistics for and factor validity of scores on the Revised Conflict Tactics Scale (CTS2) based on the responses of 295 high-risk postpartum women. The CTS2 yielded results for this sample similar to results obtained from a sample of college students in a previous study. Furthermore, confirmatory factor analytic results indicated that a five-factor model, specifying factors of Negotiation, Minor Psychological Aggression, Severe Psychological Aggression, Minor Physical Assault, and Severe Physical Assault, yielded better fit statistics than a three-factor model that combined the minor and severe scales within the domains of psychological aggression and physical assault.

The present study examined the use of the Graduate Record Examination (GRE-Verbal and GRE-Quantitative) and undergraduate grade point average (UGPA) to predict long-term performance in an MA program in forensic psychology. The criterion measures were graduate grade point average (GGPA) and time to completion (TTC). Data were available for 206 graduates. Regression analysis indicated that a linear combination of GRE-V, GRE-Q, and UGPA correlated 0.63 with GGPA. Predictive efficiency was reduced by only 2% of the variance when GRE subscores are combined into a total score. The correlation with TTC was smaller (R = 0.31) but nonetheless translated into meaningful differences in student performance. Most noteworthy, GRE scores and UGPA appear to predict better for forensic psychology than for social sciences in general.

The present study was designed to develop a brief instrument to measure empathy in health care providers in patient care situations. Three groups participated in the study: Group 1 consisted of 55 physicians, Group 2 was 41 internal medicine residents, and Group 3 was composed of 193 third-year medical students. A 90-item preliminary version of the Empathy scale was developed based on a review of the literature and distributed to Group 1 for feedback. After pilot testing, a revised and shortened 45-item version of the instrument was distributed to Groups 2 and 3. A final version of the Jefferson Scale of Physician Empathy containing 20 items based on statistical analyses was constructed. Psychometric findings provided support for the construct validity, criterion-related validity (convergent and discriminant), and internal consistency reliability (coefficient alpha) of the scale scores.