A new method for evaluating the content representation of a test is illustrated. Item similari ty ratings were obtained from content domain ex perts in order to assess whether their ratings cor responded to item groupings specified in the test blueprint. Three expert judges rated the similarity of items on a 30-item multiple-choice test of study skills. The similarity data were analyzed using a multidimensional scaling (MDS) procedure followed by a hierarchical cluster analysis of the MDS stimulus coordinates. The results indicated a strong correspondence between the similarity data and the arrangement of items as prescribed in the test blueprint. The findings suggest that analyzing item similarity data with MDS and cluster analysis can provide substantive information pertaining to the content representation of a test. The advantages and disadvantages of using MDS and cluster analysis with item similarity data are discussed.
Arabie, P., Carroll, J.D., & DeSarbo, W.J. (1987). Three-way scaling and clustering. Newbury Park CA: Sage.
2.
Borg, I., & Lingoes, J.C. (1980). A model and algorithm for multidimensional scaling with external constraints on the distances. Psychometrika , 45, 25-38.
3.
Carroll, J.D., & Arabie, P. (1983). INDCLUS: An individual differences generalization of the ADCLUS model and the MAPCLUS algorithm. Psychometrika , 48, 157-169.
4.
Carroll, J.D., & Chang, J.J. (1970). An analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition. Psychometrika, 35, 238-319.
5.
Cattell, R.B. (1957). Personality and motivation structure and measurement . New York: Harcourt, Brace, and World.
6.
Crocker, L.M., Miller, D., & Franks E.A. (1989). Quantitative methods for assessing the fit between test and curriculum. Applied Measurement in Education, 2, 179-194.
7.
Davison, M.L. (1983). Multidimensional scaling. New York: Wiley.
8.
Davison, M.L. (1985). Multidimensional scaling versus components analysis of test intercorrelations. Psychological Bulletin, 97, 94-105.
9.
Fitzpatrick, A.R. (1983). The meaning of content validity. Applied Psychological Measurement, 7, 3-13.
10.
Green, S.B. (1983). Identifiability of spurious factors with linear factor analysis with binary items. Applied Psychological Measurement , 7, 139-147.
11.
Guion, R.M. (1977). Content validity: The source of my discontent . Applied Psychological Measurement, 1, 1-10.
12.
Guion, R.M. (1978). Scoring of content domain samples: The problem of fairness. Journal of Applied Psychology, 63, 499-506.
13.
Hambleton, R.K. (1980). Test score validity and standard setting methods . In R. A. Berk (Ed.), Criterion-referenced measurement: The state of the art (pp. 80-123). Baltimore : Johns Hopkins University Press.
14.
Hambleton, R.K. (1984). Validating the test score. In R. A. Berk (Ed.), A guide to criterion-referenced test construction (pp. 199-230). Baltimore: Johns Hopkins University Press.
15.
Heiser, W.J., & Meulman, J. (1983). Constrained multidimensional scaling, including confirmation. Applied Psychological Measurement, 7, 381-404.
Kruskal, J.B., & Wish, M. (1978). Multidimensional scaling. Newbury Park CA: Sage.
19.
Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563-575.
20.
Messick, S. (1958). The perception of social attitudes. Journal of Abnormal and Social Psychology, 52, 57-66.
21.
Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955-966.
22.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35, 1012-1027.
23.
Messick, S. (1989a). Meaning and values in test validation : The science and ethics of assessment. Educational Researcher, 18, 5-11.
24.
Messick, S. (1989b). Validity. In R. Linn (Ed.), Educational measurement (3rd ed.), (pp. 13-103). Washington DC: American Council on Education.
25.
Morris, L.L., & Fitz-Gibbon, C.T. (1978). How to measure achievement . Beverly Hills: Sage.
26.
Napior, D. (1972) Nonmetric multidimensional techniques for summated ratings. In R. N. Shepard, A. K. Romney, & S. B. Nerlove (Eds.), Multidimensional scaling: Theory and applications in the behavioral sciences. Volume 1: Theory (pp. 157-178). New York: Seminar Press.
27.
Oltman, P.K., Stricker, L.J., & Barrows, T.S. (1990). Analyzing test structure by multidimensional scaling . Journal of Applied Psychology, 75, 21-27.
28.
Osterlind, S.J. (1989). Constructing test items. Norwell MA: Academic Press.
29.
Ramsay, J.O. (1980). Some small sample results for maximum likelihood estimation in multidimensional scaling. Psychometrika, 45, 139-144.
30.
Ramsay, J.O. (1981). How to use MULTISCALE. In S. S. Schiffman, M. L. Reynolds, & F. W. Young (Eds.), Multidimensional scaling: Theory, methods, and applications (pp. 211-235). New York: Academic Press.
31.
Ramsay, J.O. (1986). MULTISCALE-II manual [Computer program manual] . Montreal Quebec: McGill University .
32.
Schiffman, S.S., Reynolds, M.L., & Young, F.W. (1981). Multidimensional scaling: Theory, methods, and applications. New York: Academic Press.
33.
Shepard, R.N. (1962). The analysis of proximities: Multidimensional scaling with an unknown distance function. Psychometrika, 27, 125-140.
34.
Sireci, S.G. (1988). The Sst: A test of study skills. Unpublished test, Fordham University, Bronx NY.
35.
Sokal, R., & Michener, C.D. (1958). A statistical method for evaluating systematic relationships. University of Kansas Scientific Bulletin, 38, 1409-1438.
36.
Spence, I. (1982). Incomplete experimental designs for multidimensional scaling. In R. G. Goledge & J. N. Rayner (Eds.), Proximity and preference: Problems in the multidimensional analysis of large data sets (pp. 29-46). Minneapolis : University of Minnesota Press.
37.
Spence, I. (1983). Monte carlo simulation studies. Applied Psychological Measurement, 7, 405-426.
38.
Takane, Y., Young, F.W., & de Leeuw, J. (1977). Nonmetric individual differences multidimensional scaling: An alternating least squares method with optimal scaling features . Psychometrika, 42, 7-67.
Thorndike, R.L. (1982). Applied psychometrics. Boston : Houghton Mifflin.
41.
Torgerson, W.S. (1958). Theory and methods of scaling. New York: Wiley.
42.
Tucker, L.T. (1961). Factor analysis of relevance judgements: An approach to content validity. Paper presented at the Invitational Conference on Testing Problems, Princeton NJ. [Reprinted in A. Anastasi (Ed.), Testing problems in perspective, (1966), (pp. 577-586) Washington D.C.: American Council on Education.]
43.
Wainer, H., Hurt, S., & Aiken, L. (1976). Rorschach revisited: A new look at an old test . Journal of Consulting and Clinical Psychology, 44, 390-399.
44.
Wainer, H., & Kaye, K. (1978). Multidimensional scaling of concept learning in an introductory course. Journal of Educational Psychology , 66, 591-598.
45.
Young, F.W., Takane, Y., & Lewyckyj, R. (1978). ALSCAL: A nonmetric multidimensional scaling program with several difference options. Behavioral Research Methods and Instrumentation, 10, 451-453.
