Traditional item response theory (IRT) measurement invariance approaches examine measurement equivalence (ME) between observed groups (e.g., race, gender, culture). By contrast, mixed-measurement item response theory (MM-IRT) ascertains ME among unobserved groups (i.e., latent classes [LC] of respondents distinguished by differences in scale use). Both approaches can be integrated by using the Mixed-Measurement Item Response Theory with Covariates (MM-IRT-C) model, in which covariates (i.e., observed characteristics) are modeled in conjunction with LCs, thereby elucidating if ME is attributable to observed and/or unobserved groupings. We first show how this technique can be used to ascertain ME over multiple observed characteristics (categorical and/or continuous) concomitantly, thereby advancing a more general approach to observed ME. Next, we illustrate how the full MM-IRT-C can be used to: (a) infer underlying latent measurement classes (LCs), (b) determine associations of LC membership with observed characteristics, and (c) determine if observed measurement nonequivalence occurs predominantly within a particular latent measurement class. This method is demonstrated using a measure of union citizenship behavior, with years of work experience and gender as covariates. The proposed framework extends organizational ME research from considering a single question (i.e., Is there ME between categorical observed groups?) to addressing eight, separate questions about observed and unobserved ME. The substantive and methodological contributions of this model for rethinking ME and its use in organizational research are discussed.
Akaike, H. ( 1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.
2.
Baker, F. ( 1995). EQUATE 2.1: Computer program for equating two metrics in item response theory. (Version 2.1). Madison: University of Wisconsin, Laboratory of Experimental Design.
3.
Baron, R.M., & Kenny, D.A. ( 1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations . Journal of Personality and Social Psychology, 51, 1173-1182.
4.
Bolt, D.M., Cohen, A.S., & Wollack, J.A. ( 2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39, 331-348.
5.
Bozdogan, H. ( 1993). Choosing the number of component clusters in the mixture-model using a new informational complexity criterion of the inverse-Fisher information matrix. In O. Opitz, B. Lausen & R. Klar (Eds.), Studies in classification, data analysis, and knowledge organization. Heidelberg, Germany: Springer-Verlag.
6.
Candell, G.L., & Drasgow, F. ( 1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement , 12, 253-260.
7.
Cohen, A.S., & Bolt, D.M. ( 2005). A mixture model analysis of differential item functioning . Journal of Educational Measurement, 42, 133-148.
8.
Collins, W.C., Raju, N.S., & Edwards, J.E. ( 2000). Assessing differential functioning in a satisfaction scale. Journal of Applied Psychology, 85, 451-461.
9.
De Ayala, R.J., Kim, S.-H., Stapleton, L.M., & Dayton, C.M. ( 2002). Differential item functioning: A mixture distribution conceptualization. International Journal of Testing, 2, 243-276.
10.
Drasgow, F. ( 1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables. Psychological Bulletin, 95, 134-135.
11.
Drasgow, F. ( 1987). Study of measurement bias of two standardized psychological tests. Journal of Applied Psychology, 72, 19-29.
12.
Drasgow, F., & Kanfer, R. ( 1985). Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70, 662-680.
13.
Eid, M., & Diener, E. ( 2001). Norms for experiencing emotions in different cultures: Inter- and intranational differences. Journal of Personality and Social Psychology, 81, 869-885.
14.
Eid, M., & Rauber, M. ( 2000). Detecting measurement invariance in organizational surveys . European Journal of Psychological Assessment, 16, 20-30.
15.
Eid, M., & Zickar, M.J. ( 2007). Detecting response styles and faking in personality and organizational assessments by mixed Rasch models. In M. von Davier & C. H. Carstensen (Eds.), Multivariate and mixture distribution Rasch models (pp. 255-270). New York, NY: Springer.
16.
Glas, C.A.W. ( 1998). Detection of differential item functioning using Lagrange Multiplier tests. Statistica Sinica, 8, 647-667.
17.
Hambleton, R.K., & Kanjee, A. ( 1995). Increasing the validity of cross-cultural assessments: Use of improved methods for test adaptations. European Journal of Psychological Assessment, 11, 147-157.
18.
Hernandez, A., Drasgow, F., & Gonzalez-Roma, V. (2004). Investigating the functioning of a middle category by means of a mixed-measurement model. Journal of Applied Psychology, 89, 687-699.
19.
Hofstede, G.H. ( 1984). Culture’s consequences: International differences in work-related values (abridged ed.). Beverly Hills, CA : Sage.
Kim, S.-H., Cohen, A.S., & Park, T.-H. ( 1995). Detection of differential item functioning in multiple groups. Journal of Educational Measurement, 32, 261-276.
23.
Kulas, J.T., & Finkelstein, L.M. (2007). Content and reliability of discrepancy-defined self-awareness in multisource feedback. Organizational Research Methods, 10, 502-522.
24.
Landis, R.S., Beal, D.J., & Tesluk, P.E. ( 2000). A comparison of approaches to forming composite measures in structural equation models. Organizational Research Methods , 3, 186-207.
25.
Li, F., Cohen, A.S., Kim, S.H., & Cho, S.J. ( 2009). Model selection methods for mixture dichotomous IRT models . Applied Psychological Measurement, 33, 353-373.
26.
Lord, F.M. ( 1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum .
27.
Lubke, G.H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods , 10, 21-39.
28.
Magidson, J., & Vermunt, J.K. ( 2004). Latent class models. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 175-198). Newbury Park, CA: SAGE.
29.
Maij-de Meij, A.M., Kelderman, H., & van der Flier, H. (2005). Latent-trait latent-class anlaysis of self-disclosure in the work environment. Multivariate Behavioral Research, 40, 435-459.
30.
Maij-de Meij, A.M., Kelderman, H., & van der Flier, H. (2008). Fitting a mixture item response theory model to personality questionnaire data: Characterizing latent classes and investigating possibilities for improving prediction. Applied Psychological Measurement, 32, 611-631.
31.
McShane, S.L. ( 1986). The multidimensionality of union participation. Journal of Occupational Psychology, 59, 177-187.
32.
Mislevy, R.J., & Verhelst, N. ( 1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195-215.
33.
Muthén, B., & Muthén, L.K. (2007). Mplus Version 5.2 [Computer Program]. Los Angeles, CA: Muthén & Muthén.
34.
Penfield, R.D. ( 2001). Assessing differential item functioning among multiple groups: A comparison of three Mantel-Haenszel procedures. Applied Measurement in Education, 2001, 235-259.
35.
Raju, N.S. ( 1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.
36.
Raju, N.S. ( 1999). DFIT5P: A Fortran Program for Calculating DIF/DTF [Computer software]. Chicago: Illinois Institute of Technology.
37.
Raju, N.S., Laffitte, L.J., & Byrne, B.M. ( 2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517-529.
38.
Raju, N.S., van der Linden, W.J., & Fleer, P.F. ( 1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.
39.
Reise, S.P., & Gomel, J.N. ( 1995). Modeling qualitative variation within latent trait dimensions: Application of mixed-measurement to personality assessment. Multivariate Behavioral Research, 30, 341-358.
40.
Reise, S.P., & Waller, N.G. ( 1990). Fitting the two-parameter model to personality data. Applied Psychological Measurement, 14, 45-58.
41.
Riordan, C.M., & Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20, 643-671.
42.
Robert, C., Lee, W.C., & Chan, K.-Y. ( 2006). An empirical analysis of measurement equivalence with the INDCOL measure of individualism and collectivism: Implications for valid cross-cultural inference. Personnel Psychology, 59, 65-99.
43.
Rost, J. ( 1990). Rasch model in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement , 14, 271-282.
44.
Rost, J. ( 1991). A logistic mixture distribution model for polychotomous item responses. British Journal of Mathematical and Statistical Psychology, 44, 75-92.
45.
Rost, J., Carstensen, C., & von Davier, M. (1997). Applying the mixed Rasch model to personality questionnaires. In J. Rost & R. Langeheine (Eds.), Applications of latent trait and latent class models in the social sciences . Munster, Germany: Waxman.
46.
Schaffer, B.S., & Riordan, C.M. ( 2003). A review of cross-cultural methodologies for organizational research: A best-practices approach. Organizational Research Methods, 6, 169-215.
47.
Schwarz, G.E. ( 1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
48.
Smit, J.A., Kelderman, H., & van der Flier, H. (1999). Collateral information and mixed Rasch models. Methods of Psychological Research , 4, 19-32.
49.
Smit, J.A., Kelderman, H., & Van der Flier, H. (2000). The mixed Birnbaum model: Estimation using collateral information. Methods of Psychological Research, 5, 1-13.
50.
Stark, S. ( 2002). ITERLINK: Iterative linking and pairwise DIF detection for the 3PL model using Lord’s chisquare [Computer program]. Department of Psychology, University of Illinois at Urbana-Champaign.
51.
Stark, S., Chernyshenko, O.S., Chan, K.-Y., Lee, W.C., & Drasgow, F. ( 2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943-953.
52.
Stark, S., Chernyshenko, O.S., & Drasgow, F. ( 2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important? Journal of Applied Psychology, 89, 497-508.
53.
Stark, S., Chernyshenko, O.S., & Drasgow, F. ( 2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292-1306.
Thissen, D., Steinberg, L., & Wainer, H. ( 1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ: Erlbaum.
56.
Triandis, H., & Gelfand, M.J. ( 1998). Converging measurement of horizontal and vertical individualism and collectivism. Journal of Personality and Social Psychology , 74, 118-128.
57.
Vandenberg, R.J. ( 2002). Toward a further understanding of and improvement in measurement invariance methods and procedures. Organizational Research Methods, 5, 139-158.
58.
Vandenberg, R.J., & Lance, C.E. ( 2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.
59.
Vermunt, J.K., & Magidson, J. ( 2000). Latent GOLD 4.0 user manual. Belmont, MA: Statistical Innovations Inc.
60.
Vermunt, J.K., & Magidson, J. ( 2005). Technical guide for Latent GOLD 4.0: Basic and advanced . Belmont, MA: Statistical Innovations Inc.
61.
Vermunt, J.K., & Magidson, J. ( 2008). Latent GOLD 4.5 [Computer program]. Belmont, MA: Statistical Innovations Inc.
62.
von Davier, M. ( 1997). Bootstrapping goodness-of-fit statistics for sparse categorical data. Results on a Monte Carlo study. Methods of Psychological Research Online, 2, 29-48.
63.
Wang, M. ( 2007). Profiling retirees in the retirement transition and adjustment process: Examining the longitudinal change patterns of retirees’ psychological well-being. Journal of Applied Psychology, 92, 455-474.
64.
Woods, C.M. ( 2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44, 1-27.
65.
Woods, C.M., Oltmanns, T.F., & Turkheimer, E. (2009). Illustration of MIMIC-model DIF testing with the Schedule for Nonadaptive and Adaptive Personality. Journal of Psychopathology and Behavioral Assessment, 31, 320-330.
66.
Zickar, M.J., Gibby, R.E., & Robie, C. ( 2004). Uncovering faking samples in applicant, incumbent, and experimental data sets: An application of mixed-model item response theory . Organizational Research Methods, 7, 168-190.
