Scoring Method and the Detection of Person Misfit in a Personality Assessment Context

Abstract

The purpose of this research was to explore psycho metric issues pertinent to the application of an IRT based person-fit (response aberrancy) detection statistic in the personality measurement domain. Monte carlo data analyses were conducted to address issues regarding the l _z person-fit statistic. The major issues explored were characteristics of the null distribution of l _z and its power to identify nonfitting response patterns under different scoring strategies. There were two main results. First, the l _z index null distribution was not well standardized when item parameters of personality scales were used; the l _z null distribution variance was significantly less than the hypothesized value of 1.0 under several conditions. Second, the power of l _z to detect response misfit was affected by the scoring method. Detection power was optimal when a biweight estimator of θ was used. Recommendations are made regarding proper implementation of person-fit statistics in personality measurement.

Keywords

Index terms: appropri ateness measurement item response theory lz statistic,person fit personality assessment response aberrancy,scoring methods two-parameter model.

Get full access to this article

View all access options for this article.

References

Birenbaum, M. (1986). Effect of dissimulation motivation and anxiety on response pattern appropriateness measures. Applied Psychological Measurement, 10, 167-174.

Bock, R.D. , & Mislevy, R.J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.

Drasgow, F. (1982). Choice of test model for appropriateness measurement . Applied Psychological Measurement, 6, 297-308.

Drasgow, F. , Levine, M.V. , & McLaughlin, M.E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.

Drasgow, F. , Levine, M.V. , & McLaughlin, M.E. (1991). Appropriateness measurement for some multidimensional test batteries. Applied Psychological Measurement, 15, 171-191.

Drasgow, F. , Levine, M.V. , & Williams, E.A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.

Gafni, N. (1988). Detection of systematic and unsystematic aberrancy as a function of ability estimate by several person-fit indices. (Doctoral dissertation, University of Minnesota, 1987). Dissertation Abstracts International, 49, 943.

Hambleton, R.K. , Swaminathan, H. , & Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park CA: Sage.

Harnisch, D.L. , & Tatsuoka, K.K. (1983). A comparison of appropriateness indices based on item response theory. In R. Hambleton (Ed.), Applications of item response theory (pp. 104-122). Vancouver: Educational Research Institute of British Columbia.

10.

Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale NJ: Erlbaum.

11.

Mislevy, R.J. , & Bock, D.R. (1982). Biweight estimates of latent ability. Educational and Psychological Measurement, 42, 725-737.

12.

Reise, S.P. , & Due, A.M. (1991). Test characteristics and their influence on the detection of aberrant response patterns. Applied Psychological Measurement, 15, 217-226.

13.

Reise, S.P. , & Waller, N.G. (1990). Fitting the two-parameter model to personality data. Applied Psychological Measurement, 14, 45-58.

14.

Reise, S.P. , & Waller, N.G. (1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology , 65, 143-151.

15.

Schmitt, N. , Cortina, J.M. , & Whitney, D.J. (1993). Appropriateness fit and criterion-related validity . Applied Psychological Measurement, 17, 143-150.

16.

Tellegen, A. (1982). Briefmanual of the Multidimensional Personality Questionnaire. Unpublished manuscript.

17.

Tellegen, A. , & Waller, N.G. (in press). Exploring personality through test construction: Development of the Multidimensional Personality Questionnaire. In S. R. Briggs & J. M. Cheek (Eds.), Personality measures : Development and evaluation (Vol. 1). Greenwich CN: JAI Press.

18.

Thissen, D. (1986). MULTILOG [Computer program]. Mooresville IN: Scientific Software.

19.

Wainer, H. , & Thissen, D. (1987). Estimating ability with the wrong model. Journal of Educational Statistics, 12,339-368.

20.

Weiss, D.J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.