Rasch Analysis as a Technique to Examine the Psychometric Properties of a Career Ability Placement Survey Subtest

Abstract

Rasch analysis offers researchers an innovative method for developing and validating rehabilitation instruments. Despite the increased use of Rasch analysis by allied health researchers, this method continues to be underutilized by rehabilitation counseling professionals. In the rehabilitation counseling profession, vocational evaluators use various instruments, such as the Career Ability Placement Survey (CAPS), to determine client aptitudes and justify job placement recommendations. The purpose of this article is to introduce Rasch analysis as a method for analyzing vocational assessments by examining the psychometric properties of the Word Knowledge (WK) subtest of the CAPS. University undergraduate and graduate students (n = 227) completed the WK subtest as part of a classroom exercise. The analysis determined item hierarchy, item gaps and redundancies, item-fit statistics (unidimensionality), and person reliability and separation statistics. Although the WK construct demonstrates good psychometric characteristics relating to unidimensionality and to person reliability and separation statistics, several items deviated from the expected ordering pattern. The reordering of items revealed potential item gaps and redundancies as well as poor person—instrument match. Adding and removing items will improve the validity and reliability of the WK subtest psychometrics.

Keywords

rehabilitation research design methodology/statistics career/vocational assessment assessment measurement

Get full access to this article

View all access options for this article.

References

Barley, E. , & Jones, P. (2006). Repeatability of a Rasch model of the AQ20 over five assessments. Quality of Life Research, 15(5), 801-809.

Bond, T.G. , & Fox, C.M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Erlbaum.

Bouman, T.K. , & Kok, A.R. (1987). Homogeneity of Beck's Depression Inventory (BDI): Applying Rasch analysis in conceptual exploration. Acta Psychiatrica Scandinavica, 76(5), 568-573.

Chang, C. (1996). Finding two dimensions in MMPI-2 Depression. Structural Equation Modeling, 3(1), 41-49.

Conrad, K. , & Smith, E. (2004). International conference on objective measurement applications of Rasch analysis in health care. Medical Care , 42, I-1-I-6.

Duncan, P.W. , Bode, R.K. , Lai, S.M. , & Perera, S. (2003). Rasch analysis of a new stroke-specific outcome scale: The Stroke Impact Scale. Archives of Physical Medicine and Rehabilitation, 84, 950-963.

Haley, S.M. , McHorney, C.A. , & Ware, J.E., Jr. (1994). Evaluation of the MOS SF-36 physical functioning scale (PF-10): Unidimensionality and reproducibility of the Rasch item scale. Journal of Clinical Epidemiology, 47, 671-684.

Hambleton, R.K. (2000). Response to Hays et al. and McHorney and Cohen: Emergence of item response modeling in instrument development and data analysis . Medical Care, 38, 60-65.

Hays, R.D. , Morales, L.S. , & Reise, S.P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, 28-42.

10.

Knapp, L. , & Knapp, R.R. (1992). Career Ability Placement Survey (CAPS). San Diego, CA: EDITS.

11.

Knapp, L. , Knapp, R.R. , & Michael, W.B. (1977). Stability and concurrent validity of the Career Ability Placement Survey (CAPS) against the DAT and GATB. Educational and Psychological Measurement, 37, 1081-1085.

12.

Knapp, L. , Knapp, R.R. , Strnad, L. , & Michael, W.B. (1978). Comparative validity of the Career Ability Placement Survey (CAPS) and the General Aptitude Test Battery (GATB) for predicting high school marks. Educational and Psychological Measurement , 38, 1053-1056.

13.

Kornetti, D.L. , Fritz, S.L. , Chiu, Y.P. , Light, K.E. , & Velozo, C.A. (2004). Rating scale analysis of the Berg Balance Scale . Archives of Physical Medicine and Rehabilitation, 85(7), 1128-1135.

14.

Linacre, J.M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.

15.

Linacre, J.M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.

16.

Linacre, J.M. , Heinemann, A.W. , Wright, B.D. , Granger, C.V. , & Hamilton, B.B. (1994). The structure and stability of the Functional Independence Measure. Archives of Physical Medicine and Rehabilitation, 75(2), 127-132.

17.

Merbitz, C. , Morris, J. , & Grip, J.C. (1989). Ordinal scales and foundations of misinference . Archives of Physical Medicine and Rehabilitation, 70, 303-313.

18.

Ruch, F.L. , & Ruch, W.W. (1963). Employee Aptitude Survey technical report. Los Angeles: Psychological Services.

19.

Van der Ven, A.H.G.S. , & Ellis, J.L. (2000). A Rasch analysis of Raven's standard progressive matrices. Personality and Individual Differences, 29(1), 45-64.

20.

Velozo, C.A. , Kielhofner, G. , & Lai, J.-S. (1999). Quantitative research series-The use of Rasch analysis to produce scale-free measurement of functional ability . American Journal of Occupational Therapy, 53, 83-90.

21.

Velozo, C.A. , Magalhaes, L. , Pan, A. , & Leiter, P. (1995). Differences in functional scale discrimination at admission and discharge: Rasch analysis of the Level of Rehabilitation Scale-III (LORS-III). Archives of Physical Medicine and Rehabilitation , 76, 705-712.

22.

Velozo, C.A. , & Peterson, E.W. (2001). Developing meaningful fear of falling measures for community dwelling elderly. American Journal of Physical Medicine and Rehabilitation, 80, 662-673.

23.

Wright, B.D. , & Linacre, J.M. (1989). Observations are always ordinal: Measure, however, must be interval. Archives of Physical Medicine and Rehabilitation , 70, 857-860.

24.

Wright, B.D. , & Linacre, J.M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.

25.

Wright, B.D. , & Linacre, J.M. (2001). WINSTEPS: A Rasch model computer program (version 3.1) [Computer software]. Chicago: Mesa Press.

26.

Wright, B.D. , & Masters, G.N. (1982). Rating scale analysis. Chicago : Mesa Press.