Statistic l z -Based Person-Fit Methods for Noncognitive Multiscale Measures

Abstract

Most person-fit statistics require long tests to reliably detect aberrant item-score vectors and are not readily applicable to noncognitive measures that consist of multiple short subscales. The authors propose combining subscale person-fit information to detect aberrant item-score vectors on noncognitive multiscale measures. They used a simulation study and three empirical personality and psychopathology test datasets to assess five multiscale person-fit methods based on the $l_{z}$ person-fit statistic with respect to (a) identifying aberrant item-score vectors, (b) improving accuracy of research results, and (c) understanding causes of aberrant responding. Simulated data analysis showed that the person-fit methods had good detection rates for substantially misfitting item-score vectors. Real-data person-fit analyses identified 4% to 17% misfitting item-score vectors. Removal of these vectors little improved model fit and test-score validity. The person-fit methods helped to understand causes of aberrant responding after controlling for response style on the explanatory variables. More real-data analyses are needed to demonstrate the usefulness of multiscale person-fit methods for noncognitive multiscale measures.

Keywords

aberrant response behavior multiscale person-fit analysis personality measurement

Lack of motivation, misunderstanding of questions, untraitedness, stylistic responding, or social desirability may cause aberrant response behavior on self-report measures of typical performance (Ferrando, 2012; Tellegen, 1988). Consequently, trait measurement may be confounded and individual decision-making may be adversely affected, for example, in personnel selection (Christiansen, Goffin, Johnston, & Rothstein, 1994) and clinical treatment planning (Piedmont, McCrae, Riemann, & Angleitner, 2000). Also, psychometric properties derived from contaminated data may be invalid (Meijer, 1997; Woods, 2006).

Item response theory (IRT)–based person-fit analysis (PFA; Meijer & Sijtsma, 2001) can be used to detect aberrant item-score vectors. PFA has its roots in cognitive and educational measurement (Levine & Drasgow, 1982), but PFA may also be used in noncognitive or typical performance measurement (e.g., Emons, 2008; Ferrando, 2010, 2012; Reise, 1995; Reise & Flannery, 1996). Inventories for noncognitive measurement often assess a general trait by means of several short, unidimensional subscales containing no more than 15 items. However, person-fit (PF) statistics lack power to detect misfit on scales containing fewer than 20 items (Emons, 2008; Reise, 1995; Reise & Flannery, 1996). Hence, one needs PF statistics that combine the statistical information from the multiple subscales to gain power and allow drawing conclusions about the general trait. This study proposes and investigates multiscale PF statistics.

The core PF statistic used was statistic $l_{z}$ (Drasgow, Levine, & McLaughlin, 1987). Statistic $l_{z}$ is the standardized log-likelihood of an item-score vector given the estimated, unidimensional IRT model. Five l_z-based PF methods that combine person-fit information from multiple short scales into one PF statistic were compared with respect to the degree to which they (a) identified aberrant item-score vectors, (b) improved the accuracy of research results, and (c) enabled understanding the causes of aberrant responding. To address the first issue, a simulation study was used to determine the multiscale PF methods’ Type I error rate and detection rate. To address the other two issues, the multiscale PF methods were used to analyze panel data from the International Personality Item Pool 50-item questionnaire (IPIP-50; Goldberg et al., 2006) clinical data from the and the Brief Symptom Inventory (BSI; Derogatis, 1993). Based on the results, the authors discuss the usefulness of the $l_{z}$ -based multiscale PF methods for noncognitive assessment.

Multiscale PFA

Statistic $l_{z}$ for Polytomous Items

Noncognitive measurement commonly uses rating scale items; hence, statistic $l_{z}$ for polytomous items, denoted by $l_{z}^{p}$ , was used (Drasgow, Levine, & Williams, 1985). Emons (2008) found that among several polytomous-item PF statistics, statistic $l_{z}^{p}$ had a higher detection rate. The authors of this study defined $l_{z}^{p}$ under the graded response model (GRM; Samejima, 1997). Larger negative $l_{z}^{p}$ values indicate higher degree of misfit. See online Appendix A for the equations of the GRM and the $l_{z}^{p}$ statistic (available at http://apm.sagepub.com/supplemental).

Multiscale PF Approaches

Five multiscale PF methods based on statistic $l_{z}^{p}$ were assessed. Methods 2 to 5 were constructed for multiscale measures and assume unidimensionality for each subscale. Methods 4 and 5 solve problems that Methods 1 to 3 have.

Approach 1: The unidimensional approach

If a general, higher order trait drives the subscale traits producing positively correlating subscale scores, one may compute statistic $l_{z}^{p},$ here denoted $l_{z (uni)}^{p},$ for the complete set of items collected from the subscales (Conrad et al., 2010). This unidimensional approach identifies item-score patterns that are invalid measures of the general higher order trait but is useless if the subscales measure distinct uncorrelated traits (e.g., see the IPIP-50). Statistic $l_{z (uni)}^{p}$ uses many items; hence, it is powerful and may have high detection rate.

Approach 2: Subscale analysis

Statistic $l_{z}^{p}$ is computed for each subscale separately (e.g., Emons, 2008; Reise & Waller, 1993), and an item-score vector is classified as aberrant if at least one of the $l_{z}^{p} s$ shows evidence of person misfit. The subscale-analysis approach is denoted by $l_{z (sub)}^{p} .$ This approach suffers from low power, and requires control of the Type I error rate if used to obtain a conclusion about fit or misfit on the complete multiscale measure.

Approach 3: Multiscale extension

Based on the multitest extension of statistic $l_{z}$ for dichotomous item scores (Drasgow, Levine, & McLaughlin, 1991), statistic $l_{zm}^{p}$ is the sum of the $l_{z}^{p}$ values of S unidimensional subscales indexed s =1, …, S, such that $l_{zm}^{p} = \sum_{s = 1}^{S} l_{z}^{p (s)} .$ Statistic $l_{zm}^{p}$ collects PF information from multiple subscales into one measure but allows misfit on one scale to be compensated by fit on another scale. The method best identifies misfit that is consistent across subscales, for example, when a respondent lacks motivation or concentration throughout the whole test, but may lack power for identifying subscale-specific misfit (Conijn, Dolan, & Vorst, 2007; Schmitt, Chan, Sacco, McFarland, & Jennings, 1999).

Approaches 4 and 5: Combining $l_{z}^{p}$ and $l_{zm}^{p}$

Method $l_{z (com)}^{p}$ combines subscale $l_{z}^{p}$ values and statistic $l_{zm}^{p},$ and it is expected that $l_{z (com)}^{p}$ improves detection rates for persons consistently showing misfit across subscales compared with separate-subscale analyses. In contrast to statistic $l_{zm}^{p},$ statistic $l_{z (com)}^{p}$ uses subscale-specific information separately; thus, statistic $l_{z (com)}^{p}$ may also improve detection rates for persons that show misfit on a few but not on all subscales.

For statistic $l_{z (com)}^{p},$ for all possible subsets out of a total of S subscales, the $l_{zm}^{p}$ (or $l_{z}^{p}$ ) values are computed. For example, for $S = 3$ , $l_{zm}^{p}$ (or $l_{z}^{p}$ ) is computed for seven subsets ( $s_{1}$ , $s_{2}$ , $s_{3}$ ), ( $s_{1}$ , $s_{2}$ ), ( $s_{1}$ , $s_{3}$ ), ( $s_{2}$ , $s_{3}$ ), and ( $s_{1}$ ), ( $s_{2}$ ), and ( $s_{3}$ ). If at least one of the resulting statistics suggests significant misfit, the item-score vector is classified as misfitting. A variation of $l_{z (com)}^{p}$ is to only use the $l_{zm}^{p}$ statistic based on all subscales and the $l_{z}^{p} s$ for the single subscales, and denote the method $l_{z (sel)}^{p} .$ This means that for $S = 3,$ an item-score vector is classified as misfitting if $l_{zm}^{p}$ and l^p_z for at least one of the subsets ( $s_{1}$ , $s_{2}$ , $s_{3}$ ), ( $s_{1}$ ); ( $s_{2}$ ), or ( $s_{3}$ ) are significant.

Common Issues for $l_{z}^{p}$ -Multiscale Methods

Provided the IRT model is consistent with the item-score vector and true $θ$ values are available, statistic $l_{z}^{p}$ is standard normally distributed (Drasgow et al., 1985), but Nering (1995) showed that the use of an estimate of $θ$ invalidates standard normality. In this study, a parametric bootstrap procedure (De la Torre & Deng, 2008) was used to compute $l_{z}^{p}$ and $l_{zm}^{p},$ and the p values. To prevent inflated Type I error rates for statistics $l_{z (sub)}^{p},$ $l_{z (com)}^{p},$ and $l_{z (sel)}^{p},$ the false discovery rate (FDR) was controlled instead of the more traditional and less powerful family-wise error rate (e.g., Bonferroni correction). To this end, the Benjamini and Hochberg (BH) procedure (Benjamini & Hochberg, 1995) was used.

Study 1: Simulation Study

Research Questions

In this project, a simulation study was used to investigate whether the $l_{z}^{p}$ -multiscale methods are useful for detecting aberrant item-score vectors. More specifically, the following questions were addressed:

Do empirical Type I error rates adhere to the nominal Type I error rates?

What are the detection rates for realistic test length and realistic item properties?

Method

Data generation

The GRM was used to simulate for 10,000 simulees item scores with five ordered values, 0, …, 4, for a multiscale measure with five subscales. Item parameter estimates (online Appendix B, available at http://apm.sagepub.com/supplemental) were based on the empirical IPIP-50 data (Study 2, Method). The latent trait $θ$ values followed a standard normal multivariate distribution.

Lack of motivation or concentration, misunderstanding of questions, or low traitedness may cause random errors in the context of PFA known as random responding. Person misfit was therefore simulated using a response probability equal to .2, for each response option. Extreme response style (ERS) or agreement bias may result in more typical aberrant item-score vectors than random error, hence restrict generalization of results. Moreover, Emons (2008) found that statistic $l_{z}^{p}$ detects random responding twice as well as ERS. Hence, it was decided not to include response styles and to limit attention to random responding.

Simulated data including misfitting item-score vectors were used to estimate the item, and person parameter values used to compute the PF statistics. For parameter estimation, MULTILOG 7 (Thissen, Chen, & Bock, 2003) was used. The $l_{z}^{p} s$ of item-score vectors that contain only 0s or 4s are uninformative of person fit and were excluded from the analyses. To classify item-score vectors as misfitting, significance testing was one-tailed, $α = . 05 .$ For methods $l_{z (sub)}^{p},$ $l_{z (com)}^{p},$ and $l_{z (sel)}^{p},$ an item-score vector was classified as misfitting if at least one of the statistics $l_{zm}^{p}$ and $l_{z}^{p}$ was significant using the BH procedure to control the FDR at level $α .$

Design characteristics

A cross-factorial design with four factors and 60 conditions was defined: (a) percentage of misfitting item-score vectors was 10% or 30%; (b) the latent traits corresponding to the five subscales correlated .4, .6, or .8; (c) number of items per subscale was 6 (30 items in total) or 12 (60 items in total); and (d) methods $l_{z (uni)}^{p},$ $l_{z (sub)}^{p},$ $l_{zm}^{p},$ $l_{z (com)}^{p},$ and $l_{z (sel)}^{p} .$ For these conditions, the data followed a simple factor structure. In each cell, 50 replications were realized.

Real noncognitive data do not always show a simple factor structure. Hence, in a separate design, data consistent with the bifactor model (Reise, Morizot, & Hays, 2007) were generated. Each item loaded on a general trait factor, and items of three subscales loaded on one of three subscale-specific trait factors. Item discrimination parameters for the specific and general factors were in proportion of 5:6 and were chosen such that correlations between subscale scores and coefficient alphas equaled those generated under the simple structure condition and θs correlating .6. Having the same data-correlational structure for the bifactor model and the simple structure model enabled the authors to assess effects the bifactor structure had on PF statistic performance. The factors subscale length and PF method were combined with a fixed misfit of 10% to generate 10 bifactor conditions. In each cell, 50 data replications were simulated.

Random responding was implemented as “global misfit” or “subscale misfit,” and the percentage of random item scores in an item-score vector was varied. Global misfit was realized by randomly selecting 20%, 40%, 60%, or 80% of the items from all subscales. Subscale misfit was realized by first nominating subscales and then, for nominated subscales, randomly selecting 50% or 100% of the item scores, either from one subscale or from two subscales. Each of the eight kinds of misfitting item-score vectors was equally represented in the data.

Results

For the simple structure conditions, Table 1 shows the Type I error rate (rows corresponding to “no misfit”) and the detection rate for methods $l_{z (uni)}^{p},$ $l_{z (sub)}^{p},$ $l_{zm}^{p},$ $l_{z (com)}^{p},$ and $l_{z (sel)}^{p} .$ For $l_{z (uni)}^{p},$ the results for all three θ-correlation levels are reported. Variation in θ-correlation had little effect on performance of other methods; hence, only results for θ-correlation of .6 are discussed.

Table 1.

Mean Detection Rates for Item-Score Vectors Without Misfit (i.e., Type I error), With Global Misfit, or With Subscale Misfit for the Simple Structure Conditions.

Kind of misfit	% random	$l_{z (uni)}^{p}$			$l_{z (sub)}^{p}$	$l_{zm}^{p}$	$l_{z (com)}^{p}$	$l_{z (sel)}^{p}$	$l_{z (uni)}^{p}$			$l_{z (sub)}^{p}$	$l_{zm}^{p}$	$l_{z (com)}^{p}$	$l_{z (sel)}^{p}$
		r = .4	r = .6	r = .8	r = .6				r = .4	r = .6	r = .8	r = .6
		30-item scale, 10% misfitting item-score vectors							30-item scale, 30% misfitting item-score vectors
No misfit	0	.04	.03	.02	.03	.02	.01	.02	.00	.01	.00	.01	.00	.00	.01
Global
Detection rates	20	.31	.33	.40	.38	.42	.36	.39	.15	.16	.19	.20	.20	.17	.20
	40	.72	.77	.83	.70	.84	.76	.76	.52	.57	.62	.48	.64	.53	.55
	60	.94	.96	.97	.89	.97	.94	.94	.84	.87	.91	.73	.90	.82	.82
	80	.99	.99	1.00	.96	.99	.98	.99	.96	.97	.98	.88	.98	.95	.95
Subscale
1 scale: 50%	10	.14	.14	.15	.28	.15	.18	.26	.05	.05	.05	.17	.04	.08	.15
2 scales: 50%	20	.32	.35	.40	.47	.40	.40	.46	.15	.16	.19	.29	.18	.22	.27
1 scale: 100%	20	.31	.35	.39	.56	.32	.44	.55	.15	.16	.18	.44	.12	.28	.41
2 scales: 100%	40	.71	.77	.83	.81	.75	.77	.80	.51	.56	.62	.68	.50	.61	.66
Average		.55	.58	.62	.63	.60	.60	.64	.42	.44	.47	.48	.45	.46	.50
		60-item scale, 10% misfitting item-score vectors							60-item scale, 30% misfitting item-score vectors
No misfit	0	.05	.03	.02	.02	.01	.01	.02	.01	.01	.00	.00	.00	.00	.00
Global
Detection rates	20	.48	.53	.59	.54	.67	.58	.59	.23	.25	.29	.28	.32	.27	.30
	40	.93	.95	.97	.92	.98	.96	.96	.76	.81	.86	.73	.89	.82	.82
	60	1.00	1.00	1.00	.99	1.00	1.00	1.00	.98	.99	.99	.96	1.00	.99	.99
	80	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Subscale
1 scale: 50%	10	.19	.20	.22	.50	.22	.37	.48	.06	.06	.06	.32	.04	.17	.29
2 scales: 50%	20	.46	.53	.60	.75	.64	.71	.74	.23	.26	.29	.54	.29	.45	.51
1 scale: 100%	20	.46	.54	.59	.88	.54	.79	.87	.22	.25	.28	.77	.18	.60	.75
2 scales: 100%	40	.92	.96	.96	.99	.96	.98	.99	.74	.80	.85	.95	.80	.93	.94
Average		.68	.71	.74	.82	.75	.80	.83	.53	.55	.58	.69	.57	.65	.70

Note. Means were based on 50 replications; standard errors were <.01; due to missing $l_{z}^{p} s,$ on average 1.2% and 0.1% item-score vectors were excluded from the analyses for the 30-item condition and the 60-item condition, respectively.

Research Question 1: Adherence to nominal Type I error

For both the simple structure conditions (Table 1) and the bifactor conditions (not tabulated), empirical Type I error ranged from .01 to .05 in the 10% misfit condition. Type I error ranged from .00 to .01 in the 30% misfit condition. Low Type I error was due to random item scores that caused biased item discrimination ( $α_{j}$ ) estimates; on average, $α_{j} s$ were 0.13 and 0.32 units too low in the 10% misfit and 30% misfit conditions, respectively. As a result, the $l_{z}^{p}$ values were too high, and too few item-score vectors were classified as misfitting. Using true item parameters to compute $l_{z}^{p},$ Type I errors on average were .05, .05, .03, and .04 for $l_{z (sub)}^{p},$ $l_{zm}^{p},$ $l_{z (com)}^{p},$ and $l_{z (sel)}^{p},$ respectively. For $l_{z (uni)}^{p},$ using $α_{j}$ and $δ_{jm}$ values estimated in a dataset without person misfit resulted in Type I errors between .05 and .08, with higher values for a lower θ-correlation.

Research Question 2: Detection rates

All methods showed detection rates between .73 and 1.00 for the 60-item condition if at least 40% of the item scores were random, and for the 30-item condition if at least 60% of the item scores were random. All methods failed to have good detection rates in the other conditions. Detection rates decreased as percentage of misfitting item-score vectors increased, and increased as number of items and percentage of random item scores increased. Contrary to expectation, $l_{z (uni)}^{p}$ was rather insensitive to size of θ-correlation. Detection rate of PF statistics differed little between simple structure data and the bifactor data. The largest difference was .026 higher in bifactor data (condition: $l_{z (uni)}^{p},$ 30 items, θ-correlation .60).

For global misfit, method $l_{zm}^{p}$ had the highest detection rate. For 60 items and 30 items, including 10% misfit and at least 40% random item scores, $l_{zm}^{p}$ had a detection rate between .84 and 1.00. For 30 items including 30% misfit, detection rates of $l_{zm}^{p}$ were only good when at least 60% of the item scores were random. For subscale misfit, method $l_{z (sub)}^{p}$ had the highest detection rates. In the 60-item condition, detection rates (range : .77-.99) of $l_{z (sub)}^{p}$ were good for item-score vectors with 100% random scores on one or two subscales. Across all conditions, method $l_{z (sel)}^{p}$ had the best detection rate, almost as good as $l_{zm}^{p}$ for global misfit and $l_{z (sub)}^{p}$ for subscale misfit.

Conclusion From Study 1

Detection rates of different methods strongly depend on misfit type. Compared with the other methods, methods $l_{z (uni)}^{p}$ and $l_{zm}^{p}$ had higher detection rates for global misfit, and methods $l_{z (sub)}^{p}$ and $l_{z (sel)}^{p}$ had higher detection rates for subscale misfit. As expected, method $l_{z (sel)}^{p}$ had relatively high detection rates for both subscale and global misfit. If one does not have articulated expectations of the manifestation of misfit, method $l_{z (sel)}^{p}$ is a safe choice in terms of power.

Study 2: Real-Data Applications

Research Questions

The authors investigated whether $l_{z}^{p}$ -multiscale methods can be used to correct bias in research results due to person misfit, and to understand the causes of aberrant responding. Specifically,

Research Question 1: Does removal of misfitting item-score vectors identified by the $l_{z}^{p}$ -multiscale methods improve the fit of confirmatory factor analysis (CFA) models and provide more convincing evidence of discriminant and convergent validity?

Research Question 2: Does statistic $l_{zm}^{p}$ relate to explanatory variables for aberrant responding?

These questions were addressed using empirical data collected by means of three multiscale measures with short subscales: the IPIP-50 (Goldberg et al., 2006), the BSI (Derogatis, 1993), and the BSI-18 (Derogatis, 2001). As the authors only had access to relevant explanatory variables for the IPIP-50 data, they addressed the second question only for the IPIP-50 data.

Method

Participants

The IPIP-50 data came from the Longitudinal Internet Studies for the Social sciences (LISS) panel and were collected by CentERdata (Tilburg University, the Netherlands) in 2008. The panel completed a survey that included the IPIP-50 and several other personality, mood, and attitude scales. The study sample consisted of 6,791 participants (45.4% male). The BSI data were collected in a sample of 1,270 clinical outpatients (38.6% male) that completed the BSI at intake at four sites of a Dutch public mental health care institution.

Measures

The Dutch IPIP-50 (Hendriks, Hofstee, & De Raad, 1999) consists of five 10-item subscales, each measuring one of the Big-Five personality factors: extraversion, agreeableness, conscientiousness, neuroticism, and intellect. All items have a five-point rating scale. The Dutch BSI (De Beurs, 2004) consists of 53 five-point rating scale items, 49 of which are divided across nine subscales that include 4 to 7 items and measure different symptoms of psychopathology. The BSI short version, the BSI-18 (Derogatis, 2001), consists of three 6-item subscales measuring somatization, depression, and anxiety. In practice, for the BSI and BSI-18, subscale scores and a total score referred to as the global severity index are used.

Statistical analyses

Methods $l_{z (uni)}^{p},$ $l_{z (sub)}^{p},$ $l_{zm}^{p},$ $l_{z (com)}^{p},$ and $l_{z (sel)}^{p}$ were used for PFA, but method $l_{z (uni)}^{p}$ was not used for IPIP-50 analysis as the inventory measures five distinct traits. The methods assume a fitting GRM; a misfitting GRM may confound person misfit results. Therefore, prior to the PFA, GRM fit to subscale data was assessed. Except for the Agreeableness subscale, for the other subscales, fit results for the one-factor model suggested multidimensionality and local dependence. Except for the BSI-18 Depression subscale, fit for the other subscales of the BSI and the BSI-18 was sufficient.

To decide how to deal with the IPIP-50 model misfit, subscale PFA was used for the complete IPIP and for a shortened IPIP composed of subscales of 7 to 10 items that the GRM fitted well. The comparison of the correlations between subscale $l_{z}^{p} s$ resulting from the two analyses shed light on the usefulness of the PFA results; we assumed that higher correlations indicated more valid $l_{z}^{p} s$ because a confounding of model misfit and person misfit would lower the correlations. On average, the correlations between subscale $l_{z}^{p} s$ were .04 lower (range : –.01 to .10) for the shorter IPIP, suggesting that removing items sacrificed relevant PF information (Woods, 2006). Therefore, items were not excluded from the PFA.

Results for IPIP-50

The percentages of persons the five $l_{z}^{p}$ -multiscale methods detected ranged from 15.6% ( $l_{z (com)}^{p}$ ) to 17.3% ( $l_{z (sel)}^{p}$ ). Methods $l_{z (com)}^{p},$ $l_{z (sub)}^{p},$ and $l_{z (sel)}^{p}$ primarily detected the same persons; hence, for Research Question 1, the results for only methods $l_{zm}^{p}$ and $l_{z (sel)}^{p}$ are discussed.

Research Question 1

The authors determined whether removal of misfitting item-score vectors improved the fit of the theoretical five-factor model (Hendriks et al., 1999) and affected the correlations between the five subscales. Low correlations may be considered evidence supporting discriminant validity for each of the IPIP-50 subscales (John & Srivastava, 1999).

The nonlinear CFA model for categorical data (Muthén & Muthén, 2007) was fitted to the IPIP-50 data. A value for the root mean square error of approximation (RMSEA) of .08 or less suggests acceptable model fit, but appropriate cut-off values also depend on sample size, model size, and model specifications (Kenny, Kaniskan, & McCoach, 2011). Values for the Tucker–Lewis index (TLI) and the Comparative Fit index (CFI) in excess of .95 suggest not only good model fit (Hu & Bentler, 1999) but also depend on sample size (e.g., Bollen, 1990). Hence, the following procedure was conducted. First, the model-fit indices for the original data were determined. Second, the model-fit indices were determined for the original data in which persons classified as misfitting were replaced by a random sample of persons not classified as misfitting, and the second step was repeated 10 times using different random samples, comparing the mean model-fit indices to the model-fit indices for the original data. Table 2 shows the values of RMSEA, TLI, and CFI for the total sample, and the mean values for the samples excluding misfit using either $l_{zm}^{p}$ or $l_{z (sel)}^{p} .$

Table 2.

Model Fit Indices (With Standard Errors Within Brackets) Before and After Excluding Person Misfit.

	Total sample	Sample excluding misfit
		$l_{z (uni)}^{p}$	$l_{zm}^{p}$	$l_{z (sel)}^{p}$
IPIP-50^a
No. of detected persons		—	16.9%	17.3%
RMSEA	.115	—	.113 (.000)	.109 (.000)
TLI	.834	—	.866 (.000)	.860 (.000)
CFI	.642	—	.660 (.001)	.649 (.001)
BSI
No. of detected persons		10.8%	11.5%	15.4%
RMSEA	.119	.109 (.000)	.120 (.000)	.119 (.000)
TLI	.946	.961 (.000)	.950 (.000)	.945 (.000)
CFI	.588	.664 (.003)	.634 (.004)	.613 (.003)
BSI-18
No. of detected persons		7.3%	6.4%	4.5%
RMSEA	.142	.130 (.001)	.131 (.000)	.137 (.001)
TLI	.934	.953 (.000)	.949 (.000)	.943 (.000)
CFI	.818	.862 (.002)	.849 (.001)	.831 (.002)

Note. For the IPIP-50, n = 6,786; for the BSI, n = 1,268; for the BSI-18, n = 1,258. Due to item-score vectors including only 0s or 4s, < 1% persons were excluded from each data set. IPIP-50 = International Personality Item Pool 50-item questionnaire; RMSEA = root mean square error of approximation; TLI = Tucker–Lewis index; CFI = Comparative Fit index; BSI = Brief Symptom Inventory.

We allowed correlated trait factors (e.g., Borkenau & Ostendorf, 1990; Lim & Ployhart, 2006).

Removing misfitting item-score vectors improved model fit only little. Exclusion based on $l_{z (sel)}^{p}$ decreased RMSEA the most (.006), and removal based on $l_{zm}^{p}$ increased TLI (.032) and CFI (.018) the most. In the total sample, the correlations between the IPIP-50 scales ranged from –.27 to .34, and increased after misfitting item-score vectors had been removed. The absolute differences were the largest using $l_{zm}^{p},$ and ranged from .01 to .03. Hence, removal of misfitting item-score vectors weakened evidence of discriminant validity of the IPIP-50 subscales.

Research Question 2: Explaining person misfit

A multiple regression analysis was used to study whether $l_{zm}^{p}$ relates to explanatory variables (online Appendix C provides a description of the scales used as explanatory variables, available at http://apm.sagepub.com/supplemental). Table 3 gives the authors’ expectation of the effects of the explanatory variables.

Table 3.

Relationships Between Explanatory Variables (x) and $l_{zm}^{p}$ in the IPIP-50 Data.

Variable	Expectation	Reference(s) supporting expectation	$r (l_{zm}^{p}, x)$	B
				Model 1	Model 2
Intercept		—	—	0.02	−0.03*
Female	+	Pinsoneault, 2002; Schmitt, Chan, Sacco, McFarland, & Jennings, 1999	−0.02	0.08**	0.07***
Education	+	—	0.07***	0.05***	−0.01
Need to evaluate	+	Krosnick, Narayan, & Smith, 1996	−0.04**	−0.02	0.06***
Need for cognition	+	Krosnick et al., 1996	0.04***	0.13***	0.06***
Survey attitude	+	Krosnick et al., 1996	−0.12***	−0.06***	0.05***
Survey involvement	+	Krosnick et al., 1996	−0.10***	−0.03*	−0.02*
Survey understanding	+	Krosnick et al., 1996	−0.07***	−0.05***	0.05***
Negative affect	−	Reise & Waller, 1993	−0.02*	0.00	−0.20***
Agreeableness	+	Krosnick et al., 1996	−0.12***	−0.07***	0.12***
Conscientiousness	+	Ferrando, 2009; LaHuis & Copeland, 2009; Schmitt et al., 1999	−0.08***	−0.05***	0.11***
Neuroticism	−	LaHuis & Copeland, 2009	−0.07***	−0.09***	−0.15***
Intellect	+	Krosnick et al., 1996	−0.08***	−0.14***	0.03*
Extreme response style	−	—	−0.61***	—	−0.87***
R ²				.06	.52

Note.“+”/“−” indicates a positive/negative effect of the explanatory variable on $l_{zm}^{p}$ . n = 6,250. IPIP-50 = International Personality Item Pool 50-item questionnaire.* p < 0.10, ** p < 0.05, *** p < 0.01 (one tailed).

The multiple regression model explained 6% of the variance. Table 3 shows the correlations between $l_{zm}^{p}$ and the explanatory variables, and the regression coefficients (Model 1). Except for gender, regression coefficients and correlations had the same sign. Gender, education level, need for cognition, and neuroticism had significant effects in the expected direction. Other significant effects ran counter to expectation. Persons scoring higher on survey attitude, survey understanding, survey involvement, agreeableness, conscientiousness, and intellect showed poorer person fit. Effects were small, which is a consistent finding in explanatory PFA (Conijn, Emons, Van Assen, Pedersen, & Sijtsma, 2013).

Unexpected results may be due to a confounding effect of response styles. Response styles that relate to item content or item wording may lead to spuriously high or low scores on explanatory variables but they may also produce low $l_{zm}^{p}$ values. To study the effect of response style on the relationships between person fit and explanatory variables, measures for social desirability bias, agreement bias, and ERS were added to the model. ERS was quantified by the total number of most extreme scores, 0 and 4.

Results suggest that ERS probably confounded the effects survey attitude, survey understanding, survey involvement, agreeableness, conscientiousness, and intellect had on person fit. Table 3 (Model 2) shows the results for a regression model including ERS, which produced a 46% explained-variance increase. The effect of ERS on person fit was negative. After having accounted for ERS, all but one of the explanatory variables that initially had an unexpected effect on $l_{zm}^{p}$ now had an effect in the expected direction. The exception was the effect of survey involvement, which remained negative. Effect of education level changed from significant to not significant, which also ran counter to the authors’ expectations. To conclude, ERS related to social desirability (correlation .92) produced biased explanatory-variable scores and interfered with explanatory PFA.

Results for BSI and BSI-18

For the BSI, the percentage of detected persons ran from 10.8% ( $l_{z (uni)}^{p}$ ) to 15.4% ( $l_{z (sel)}^{p}$ ), and for the BSI-18, the percentage ran from 3.8% ( $l_{z (com)}^{p}$ ) to 7.3% ( $l_{z (uni)}^{p}$ ). The authors discuss the results for Research Question 1 for methods $l_{z (uni)}^{p},$ $l_{zm}^{p},$ and $l_{z (sel)}^{p} .$ As with the IPIP-50, they do not report results for $l_{z (sub)}^{p}$ and $l_{z (com)}^{p}$ .

Research Question 1

To evaluate the improvement of model fit by excluding misfitting item-score vectors, the theoretical nine-factor and three-factor models were first fitted to the BSI and the BSI-18 data, respectively (Derogatis, 2001). Overlap between subscale traits in both cases rendered the covariance matrix of the latent factors not positive definite. Hence, the models could not be used, but for the BSI, a second-order factor model was used instead of the nine-factor model (Hoe & Brekke, 2009), and for the BSI-18, a one-factor model (Meijer, De Vries, & Van Bruggen, 2011) was used. The authors also studied whether removing misfitting item-score vectors changed the correlations of the BSI and BSI-18 with the symptom distress subscale of the Dutch version of the Outcome Questionnaire-45 (OQ-45; De Jong et al., 2007; Lambert et al., 2001). These correlations provide support that the BSI and BSI-18 have convergent validity (De Jong et al., 2007).

Table 2 shows the effect of removing misfitting item-score vectors on model-fit indices for the BSI and the BSI-18. Results were similar for both inventories. Removing misfitting item-score vectors based on $l_{z (uni)}^{p}$ had the largest effects, for both scales causing RMSEA to decrease by .01. TLI increased by .017, and contrary to the total sample result, TLI suggested good model fit. CFI increased by .076 for the BSI and by .044 for the BSI-18. In the total sample, both the BSI and the BSI-18 correlated .76 with the OQ-45 symptom distress subscale. The correlation did not change after removing misfitting vectors using either $l_{z (uni)}^{p},$ $l_{zm}^{p},$ or $l_{z (sel)}^{p} .$

Conclusion From Study 2

Model fit improved modestly when misfitting item-score vectors were removed from the data. Exclusion of person misfit hardly affected correlations supporting either discriminant or convergent validity or had an effect contrary to theoretical expectations. Statistic $l_{zm}^{p}$ explained aberrant response behavior in the IPIP-50 data after accounting for ERS related to socially desirable responding. Low power, meaning many misfitting item-score vectors go undetected, may partly explain the small effect excluding person misfit has on model fit and validity measures. To investigate possible low power, the detection rates of the $l_{z}^{p}$ -multiscale methods were determined, given the properties of the IPIP-50, the BSI, and the BSI-18 data, by conducting simulations similar to those of Study 1. For the IPIP-50, the PF methods had good power for detecting substantial misfit, but for the BSI and the BSI-18, good detection rates were found only for item-score vectors with at least 60% random item scores (BSI) and item-score vectors with 100% aberrant item scores (BSI-18). As item-score vectors including more than 60% aberrant item scores may be rare in real data, low power is expected for the BSI and BSI-18.

Meijer (2003) recommended choosing a more liberal α level so as to increase power in PFA. Using $α = . 10,$ for the IPIP-50, $l_{z (sel)}^{p}$ was used, and for the BSI and the BSI-18, $l_{z (uni)}^{p}$ was used to remove misfitting item-score vectors. Little additional effect of removal on model fit and validity measures was found relative to $α = . 05 .$ An explanation other than low power is that the type of person misfit the $l_{z}^{p}$ -based methods detected hardly affected the item-score correlation matrix used as the input for CFA or correlations supporting validity. Statistic $l_{z}^{p}$ has relatively low power for detecting misfitting item-score vectors due to a systematic response style, for example, agreement bias or ERS (e.g., Emons, 2008). Statistic $l_{z}^{p}$ has reasonable power for detecting random error, but correcting data for random error may affect correlations only little (Drasgow & Kang, 1984).

General Discussion

The simulation study showed that $l_{z}^{p}$ -multiscale methods have good power for long tests (≥ 60 items), and for medium-length tests (30 items) provided the data include little person misfit. Combining information from different subscales increases power considerably. For example, Emons (2008) found that for a 12-item subscale with 50% random item scores, $l_{z}^{p}$ had a detection rate of .50. In contrast, the authors of this study found that for five 12-item scales with 40% random item scores in all subscales, the best-performing multiscale method, $l_{z (sel)}^{p},$ had a detection rate of .96. If only one subscale included 50% random item scores, $l_{z (sel)}^{p}$ had a detection rate of .47. Hence, using $l_{z}^{p}$ -multiscale methods when misfit is consistent across subscales may produce a substantial gain in power, whereas the use hardly reduces power when misfit is subscale specific. The authors advise using multiscale PF statistics when a test consists of several short subscales, and choosing a method dependent on whether misfit is expected to be a local or a global phenomenon.

The $l_{z}^{p}$ statistic performed relatively well when unidimensionality was violated, suggesting that PFA is robust against violations of GRM assumptions. This finding contradicts previous suggestions. Woods (2008) separated balanced scales with poor model fit into item subsets containing only positively worded or negatively worded items, and Emons (2008) suggested resorting to less powerful nonparametric PF statistics. However, if PFA is robust against model violations, following their suggestions may weaken PFA results. More research is needed on this topic.

Most PFA research on Type I error rates and detection rates used item parameters obtained in data without person misfit (e.g., Emons, 2008; Reise, 1995), but in noncognitive test data, lack of self-interest or faking good or bad is a realistic cause of aberrant responding, rendering misfitting item-score vectors and biased item parameter estimates unavoidable. Based on the findings of this study, the authors speculate that previous studies overestimated the performance of PFA in real-life settings.

The real-data applications suggest that a multiple regression model for statistic $l_{zm}^{p}$ was useful for exploring the causes of aberrant responding as they were predicted in the literature. An important finding was that one needs to account for response styles when the explanatory variables can be affected by aberrant response behavior. Furthermore, the results suggest that the usefulness of the $l_{z}^{p}$ -multiscale methods for correcting bias in research results may be limited.

Item-score vectors $l_{z}$ classified as misfitting are not necessarily influential with respect to misfit of the normative model (also see Reise & Widaman, 1999) and may have minor effects on criterion-related validity (also see Meijer, 1997; Schmitt, Cortina, & Whitney, 1993; Schmitt et al., 1999). Consistent with outlier analysis, it may therefore be useful in PFA to distinguish misfitting or “outlying” observations and influential observations (Barnett & Lewis, 1994, pp. 9, 317).

Real-data analysis in PF research is a useful addition to simulation studies: The simulation study provided information about PF statistics' performance, and the real-data analyses suggested that the methods may not detect misfit that negatively affects model fit or distorts validity indices. Methods’ detection rates were sufficient, so that future studies may demonstrate the usefulness of the $l_{z}^{p}$ -multiscale methods for improving individual decision-making. PF methods may also detect substantively interesting person misfit that requires follow-up analysis for individual respondents (Emons, Sijtsma, & Meijer, 2005; Ferrando, 2012). The authors conclude that more real-data studies are needed to demonstrate the usefulness of the $l_{z}^{p}$ -multiscale methods for noncognitive measurement.

Footnotes

Acknowledgements

We thank Kim de Jong for providing the data of the Brief Symptom Inventory and the Outcome Questionnaire-45.

Authors’ Note

The Longitudinal Internet Studies for the Social sciences (LISS) panel data were collected by CentERdata (Tilburg University, the Netherlands) through its MESS project funded by the Netherlands Organization for Scientific Research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research and/or authorship of this article.

References

Barnett

Lewis

(1994). Outliers in statistical data. New York, NY: Wiley.

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 289-300.

Bollen

K. A.

(1990). Overall fit in covariance structure models: Two types of sample size effects. Psychological Bulletin, 107, 256-259.

Borkenau

Ostendorf

(1990). Comparing exploratory and confirmatory factor analysis: A study on the 5-factor model of personality. Personality and Individual Differences, 11, 515-524.

Cacioppo

J. T.

Petty

R. E.

Kao

C. F.

(1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306-307.

Christiansen

N. D.

Goffin

R. D.

Johnston

N. G.

Rothstein

M. G.

(1994). Correcting the 16PF for faking: Effects on criterion-related validity and individual hiring decisions. Personnel Psychology, 47, 847-860.

Conijn

J. M.

Dolan

C. V.

Vorst

H. C. M.

(2007). Method effects due to item wording and their underlying causes (Unpublished master’s thesis). University of Amsterdam, Netherlands.

Conijn

J. M.

Emons

W. H. M.

Van Assen

M. A. L. M.

Pedersen

S. S.

Sijtsma

(2013). Explanatory, multilevel person-fit analysis of response consistency on the Spielberger State-Trait Anxiety Inventory. Multivariate Behavioral Research, 48, 692-718.

Conrad

K. J.

Bezruczko

Chan

Y. F.

Riley

Diamond

Dennis

M. L.

(2010). Screening for atypical suicide risk with person fit statistics among people presenting to alcohol and other drug treatment. Drug and Alcohol Dependence, 106, 92-100.

10.

De Beurs

(2004). De Brief Symptom Inventory: Handleiding [Manual]. Leiden, the Netherlands: Pits Publishers.

11.

De Jong

Nugter

M. A.

Polak

Wagenborg

J. E. A.

Spinhoven

Heiser

W. A.

(2007). The Outcome Questionnaire (OQ-45) in a Dutch population: A cross-cultural validation. Clinical Psychology & Psychotherapy, 14, 288-301.

12.

De la Torre

Deng

(2008). Improving person-fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45, 159-177.

13.

De Leeuw

(2010, May). Measuring and comparing survey attitude among new and repeat respondents cross-culturally. Paper presented at the 63rd Annual Conference World Association for Public Opinion Research (WAPOR), Chicago, IL.

14.

Derogatis

L. R.

(1993). BSI Brief Symptom Inventory: Administration, scoring, and procedures manual (4th ed.). Minneapolis, MN: National Computer Systems.

15.

Derogatis

L. R.

(2001). Brief Symptom Inventory (BSI)-18: Administration, scoring and procedures manual. Minneapolis, MN: NCS Pearson.

16.

Drasgow

Kang

(1984). Statistical power of differential validity and differential prediction analyses for detecting measurement nonequivalence. Journal of Applied Psychology, 69, 498-508.

17.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.

18.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1991). Appropriateness measurement for some multidimensional test batteries. Applied Psychological Measurement, 15, 171-191.

19.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.

20.

Emons

W. H. M.

(2008). Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement, 32, 224-247.

21.

Emons

W. H. M.

Sijtsma

Meijer

R. R.

(2005). Global, local, and graphical, person-fit analysis using person response functions. Psychological Methods, 10, 101-119.

22.

Ferrando

P. J.

(2009). A graded response model for measuring person reliability. British Journal of Mathematical and Statistical Psychology, 62, 641-662.

23.

Ferrando

P. J.

(2010). Some statistics for assessing person-fit based on continuous-response models. Applied Psychological Measurement, 34, 219-237.

24.

Ferrando

P. J.

(2012). Assessing inconsistent responding in E and N measures: An application of person-fit analysis in personality. Personality and Individual Differences, 52, 718-722.

25.

Goldberg

L. R.

Johnson

J. A.

Eber

H. W.

Hogan

Ashton

M. C.

Cloninger

C. R.

Gough

H. C.

(2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84-96.

26.

Hendriks

A. A. J.

Hofstee

W. K. B.

De Raad

(1999). The Five-Factor Personality Inventory (FFPI). Personality and Individual Differences, 27, 307-325.

27.

Hoe

Brekke

(2009). Testing the cross-ethnic construct validity of the Brief Symptom Inventory. Research on Social Work Practice, 19, 93-103.

28.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

29.

Jarvis

W. B. G.

Petty

R. E.

(1996). The need to evaluate. Journal of Personality and Social Psychology, 70, 172-194.

30.

John

O. P.

Srivastava

(1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In Pervin

L. A.

John

O. P.

(Eds.), Handbook of personality: Theory and research (pp. 102-138). New York, NY: Guilford.

31.

Kenny

D. A.

Kaniskan

McCoach

D. B.

(2011). The performance of RMSEA in models with small degrees of freedom (Unpublished paper). University of Connecticut, Storrs.

32.

Krosnick

J. A.

Narayan

Smith

W. R.

(1996). Satisficing in surveys: Initial evidence. In Braverman

M. T.

Slater

J. K.

(Eds.), Advances in survey research (Vol. 70, pp. 29-44). San Francisco, CA: Jossey-Bass.

33.

LaHuis

D. M.

Copeland

(2009). Investigating faking using a multilevel logistic regression approach to measuring person fit. Organizational Research Methods, 12, 296-319.

34.

Lambert

M. J.

Hansen

N. B.

Umpress

Lunnen

Okiishi

Burlingame

G. M.

Reisinger

C. W.

(2001). Administration and scoring manual for the OQ-45. Orem, UT: American Professional Credentialing Services.

35.

Levine

M. V.

Drasgow

(1982). Appropriateness measurement: Review, critique and validating studies. British Journal of Mathematical and Statistical Psychology, 35, 42-56.

36.

Lim

B. C.

Ployhart

R. E.

(2006). Assessing the convergent and discriminant validity of Goldberg’s International Personality Item Pool: A multitrait-multimethod examination. Organizational Research Methods, 9, 29-54.

37.

Meijer

R. R.

(1997). Appropriateness-fit and criterion related validity: An extension of the Schmitt, Cortina, and Whitney (1993) study. Applied Psychological Measurement, 21, 99-113.

38.

Meijer

R. R.

(2003). Diagnosing item score patterns on a test using item response theory based person-fit statistics. Psychological Methods, 8, 72-87.

39.

Meijer

R. R.

De Vries

R. M.

Van Bruggen

(2011). An evaluation of the Brief Symptom Inventory–18 using item response theory: Which items are most strongly related to psychological distress?Psychological Assessment, 23, 193-202.

40.

Meijer

R. R.

Sijtsma

(2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.

41.

Muthén

B. O.

Muthén

L. K.

(2007). Mplus: Statistical analysis with latent variables (Version 5.0). Los Angeles, CA: Statmodel.

42.

Nering

M. L.

(1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121-129.

43.

Piedmont

R. L.

McCrae

R. R.

Riemann

Angleitner

(2000). On the invalidity of validity scales: Evidence from self-reports and observer ratings in volunteer samples. Journal of Personality and Social Psychology, 78, 582-593.

44.

Pinsoneault

T. B.

(2002). A variable response inconsistency scale and a true response inconsistency scale for the Millon Adolescent Clinical Inventory. Psychological Assessment, 14, 320-328.

45.

Reise

S. P.

(1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213-229.

46.

Reise

S. P.

Flannery

W. P.

(1996). Assessing person-fit on measures of typical performance. Applied Measurement in Education, 9, 9-26.

47.

Reise

S. P.

Morizot

Hays

R. D.

(2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16, 19-31.

48.

Reise

S. P.

Waller

N. G.

(1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65, 143-151.

49.

Reise

S. P.

Waller

N. G.

(2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 27-48.

50.

Reise

S. P.

Widaman

K. F.

(1999). Assessing the fit of measurement models at the individual level: A comparison of item response theory and covariance structure approaches. Psychological Methods, 4, 3-21.

51.

Samejima

(1997). Graded response model. In Van der Linden

W. J.

Hambleton

(Eds.), Handbook of modern item response theory (pp. 85-100). New York, NY: Springer.

52.

Schmitt

Chan

Sacco

J. M.

McFarland

L. A.

Jennings

(1999). Correlates of person-fit and effect of person-fit on test validity. Applied Psychological Measurement, 23, 41-53.

53.

Schmitt

Cortina

J. M.

Whitney

D. J.

(1993). Appropriateness fit and criterion-related validity. Applied Psychological Measurement, 17, 143-150.

54.

Tellegen

(1988). The analysis of consistency in personality assessment. Journal of Personality, 56, 621-663.

55.

Thissen

Chen

W. H.

Bock

R. D.

(2003). MULTILOG for Windows (Version 7). Lincolnwood, IL: Scientific Software International.

56.

Watson

Clark

L. A.

Tellegen

(1988). Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 54, 1063-1070.

57.

Woods

C. M.

(2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 3, 189-194.

58.

Woods

C. M.

(2008). Monte Carlo evaluation of two-level logistic regression for assessing person-fit. Multivariate Behavioral Research, 43, 50-76.

Statistic l z -Based Person-Fit Methods for Noncognitive Multiscale Measures

Abstract

Keywords

Multiscale PFA

Statistic l z for Polytomous Items

Multiscale PF Approaches

Approach 1: The unidimensional approach

Approach 2: Subscale analysis

Approach 3: Multiscale extension

Approaches 4 and 5: Combining l z p and l zm p

Common Issues for l z p -Multiscale Methods

Study 1: Simulation Study

Research Questions

Method

Data generation

Design characteristics

Results

Research Question 1: Adherence to nominal Type I error

Research Question 2: Detection rates

Conclusion From Study 1

Study 2: Real-Data Applications

Research Questions

Method

Participants

Measures

Statistical analyses

Results for IPIP-50

Research Question 1

Research Question 2: Explaining person misfit

Results for BSI and BSI-18

Research Question 1

Conclusion From Study 2

General Discussion

Footnotes

Acknowledgements

Authors’ Note

Declaration of Conflicting Interests

Funding

References

Statistic $l_{z}$ for Polytomous Items

Approaches 4 and 5: Combining $l_{z}^{p}$ and $l_{zm}^{p}$

Common Issues for $l_{z}^{p}$ -Multiscale Methods