Detecting Aberrant Responding on Unidimensional Pairwise Preference Tests

Abstract

This study investigated the efficacy of the l_z person fit statistic for detecting aberrant responding with unidimensional pairwise preference (UPP) measures, constructed and scored based on the Zinnes–Griggs item response theory (IRT) model, which has been used for a variety of recent noncognitive testing applications. Because UPP measures are used to collect both “self-” and “other” reports, the capability of l_z to detect two of the most common and potentially detrimental response sets, namely fake good and random responding, was explored. The effectiveness of l_z was studied using empirical and theoretical critical values for classification, along with test length, test information, the type of statement parameters, and the percentage of items answered aberrantly (20%, 50%, 100%). It was found that l_z was ineffective in detecting fake good responding, with power approaching zero in the 100% aberrance conditions. However, l_z was highly effective in detecting random responding, with power approaching 1.0 in long-test, high information conditions, and there was no diminution in efficacy when using marginal maximum likelihood estimates of statement parameters in place of the true values. Although using empirical critical values for classification provided slightly higher power and more accurate Type I error rates, theoretical critical values, corresponding to a standard normal distribution, provided nearly as good results.

Keywords

person fit appropriateness measurement pairwise preference ideal point item response theory (IRT)noncognitive assessment aberrance faking random responding

In the fields of psychology and education, there are long histories of research on noncognitive constructs, such as personality, vocational interests, self-efficacy, and values. Measures administered in research settings for developmental and diagnostic purposes were shown early on to predict important outcomes and those successes raised intriguing possibilities about the use of noncognitive tests in the workplace. One notable line of research involved the behaviorally anchored rating scales (BARS; Smith & Kendall, 1963), which were originally designed to reduce the leniency, severity, central tendency, and halo errors often associated with Likert-type rating scales.

In 1998, Borman et al. proposed a “next-generation” version of BARS, called Computerized Adaptive Rating Scales (CARS; Borman et al., 2001), which integrated research on observer ratings, forced choice assessment, and modern psychometric theory. Specifically, Borman et al. assessed contextual (i.e., citizenship) performance (Borman & Motowidlo, 1993) using computerized adaptive unidimensional pairwise preference (UPP) measures composed of pairs of statements that represented different levels of employee effectiveness. A rater’s task was to choose the statement in each pair that better characterized the behavior of the ratee. By making repeated pairwise preference judgments across items chosen dynamically via computerized adaptive testing (CAT) principles (Stark & Chernyshenko, 2011; Stark & Drasgow, 1998), measurement error was reduced relative to BARS and Likert-type graphical rating scales (Borman et al., 2001).

Since the Borman et al. (2001) study, the suitability of UPP measures has been explored for other organizational applications. For example, Borman and colleagues implemented adaptive UPP measurement in the Navy Computerized Adaptive Personality Scales assessment (NCAPS; Houston, Borman, Farmer, & Bearden, 2005), which was designed to accommodate large volume testing applications. In addition, Chernyshenko, Stark, and Williams (2009) described how to construct nonadaptive UPP measures of person-organization fit using a relatively small pool of statements in the context of a university research study.

Although evidence suggests that UPP scales mitigate errors occur when raters evaluate other rating targets (Borman et al., 2001), research was needed to examine their resistance to response biases associated with self-report data, such as socially desirable (fake good) and careless or random responding. A simulation study examining the power and Type I error to detect fake good and random responding using the model-based standardized log likelihood statistic, known as $l_{z}$ , was therefore conducted (Drasgow, Levine, & McLaughlin, 1987; Drasgow, Levine, & Williams, 1985). Over the years, $l_{z}$ has been used to detect aberrant responding in connection with dichotomously and polytomously scored cognitive ability tests, as well as Likert-type noncognitive measures, but there has been little to no research on applications involving forced choice assessments. Consequently, a key purpose of this research was to provide benchmarks against which other person fit methods can be evaluated under similar simulation conditions.

$l_{z}$ is a model-based index that evaluates the standardized log likelihood of a respondent’s answer pattern relative to critical values derived from statistical theory or empirical methods. If a respondent’s observed $l_{z}$ is less than the critical value, the response pattern is classified as aberrant; otherwise the pattern is classified as “normal.” In this context, normal means that a respondent answered items in accordance with the predictions of an underlying item response theory (IRT) model, and aberrant means that the response pattern was inconsistent with model predictions.

The model chosen to represent normal UPP responding was the Zinnes and Griggs (ZG; 1974) ideal point IRT model. The ZG model assumes that when a rater is presented with a pair of statements describing different levels of, for example, effectiveness, conscientiousness, or autonomy, the rater carefully considers the statements and chooses the one in each pair that better describes the ratee. Formally, if s and t represent the first and second statements in a performance appraisal item, the probability of choosing or preferring statement s to statement t is given by

P_{s t} (θ) = 1 - Φ (a_{s t}) - Φ (b_{s t}) + 2 Φ (a_{s t}) Φ (b_{s t}),

a_{s t} = (2 θ - μ_{s} - μ_{t}) / \sqrt{3},

and

b_{s t} = μ_{s} - μ_{t},

where $θ$ represents the ratee’s perceived location on the performance continuum, $μ_{s}$ and $μ_{t}$ represent the perceived locations of the respective effectiveness statements, and Φ $(a_{st})$ and Φ $(b_{st})$ are cumulative standard normal density functions evaluated at $a_{st}$ and $b_{st}$ , respectively. Note that each statement is characterized by a single parameter, but to compute UPP item response probabilities, three parameters $(μ_{s}, μ_{t}, and θ)$ , are needed.

In contrast to normal responding, aberrant responding, such as faking good and random responding, presumes a different psychological process. With UPP assessments, fake good responding implies a rater chooses the more positive or socially desirable statement in a UPP item, regardless of whether it accurately depicts the ratee. In work settings, fake good responding can occur when job applicants want to increase their scores to get hired, when raters wants to give positive impressions of well-liked coworkers in 360° appraisals, and when supervisors want to enhance their own reputations for employee development by manipulating the ratings of subordinates under their tutelage. Alternatively, random responding might occur when busy employees are surveyed too frequently without compensation, when respondents do not understand the context or meaning of questionnaire items, or when supervisors have many subordinates to evaluate and are familiar with only a few. Finally, random responding might also occur when respondents answer an Internet-based self-report survey in an anonymous way (Meade & Craig, 2012).

Adopting the l_z Person Fit Index for ZG UPP Model

In general, person fit statistics examine either residuals (i.e., differences between observed and expected responses patterns) or the likelihood of response patterns assuming a formal model of item responding (Nering & Meijer, 1998). IRT methods generally use the latter. The likelihood of a response pattern is calculated using item and person parameter estimates for a designated item response model, and aberrant or atypical patterns are signaled by low (or, in the log metric, negative) likelihoods. The advantage of IRT methods is that they readily permit the assessment of overall model-data fit unlike classical test theory methods.

One of the most widely used and researched IRT-based person fit statistics is $l_{z}$ (Drasgow et al., 1987, 1985). For Zinnes–Griggs (Zinnes & Griggs, 1974) UPP model applications, $l_{z}$ is computed as follows. First, the log likelihood of a rater’s response pattern is given by

l_{0} = \sum_{i = 1}^{n} u_{i} \log P_{s t_{i}} (u_{i} = 1 | \hat{θ}) + (1 - u_{i}) \log (1 - P_{s t_{i}} (u_{i} = 1 | \hat{θ})),

where $\hat{θ}$ is an estimate of θ, the latent trait representing a ratee’s trait or performance level, i is an index for items, i = 1, . . ., n, $u_{i}$ represents an item response coded 1 if statement s is preferred to statement t and 0 otherwise, $P_{s t_{i}}$ is the probability of preferring statement s to statement t in the ith item, and log represents the natural logarithm function. The approximate expectation of this log likelihood is

E (l_{0}) \approx \sum_{i = 1}^{n} P_{s t_{i}} (u_{i} = 1 | \hat{θ}) \log P_{s t_{i}} (u_{i} = 1 | \hat{θ}) + [1 - P_{s t_{i}} (u_{i} = 1 | \hat{θ})] \log [1 - P_{s t_{i}} (u_{i} = 1 | \hat{θ})]

The approximate variance is

Var (l_{0}) \approx \sum_{i = 1}^{n} P_{s t_{i}} (u_{i} = 1 | \hat{θ}) [1 - P_{s t_{i}} (u_{i} = 1 | \hat{θ})] {log \frac{P_{s t_{i}} (u_{i} = 1 | \hat{θ})}{[1 - P_{s t_{i}} (u_{i} = 1 | \hat{θ})]}}^{2} .

Finally, the approximately standardized person fit statistic is

l_{z} = \frac{l_{0} - E (l_{0})}{\sqrt{Var (l_{0})}} .

The standardization step is important because it eliminates the dependence of the resulting person fit statistic on test length and θ, which was a concern with the $l_{0}$ statistic that was proposed originally by Levine and Rubin (1979).

The use of $l_{z}$ for ZG UPP data screening thus requires observed item responses and estimates of ZG item parameters $(μ_{s_{i}}, μ_{t_{i}})$ and trait scores $(\hat{θ})$ . By substituting the values into the appropriate equations from earlier, $l_{z}$ can be computed for each ratee’s response pattern and normal versus aberrant classification decisions can be made by comparing each observed $l_{z}$ to a critical value. When an observed $l_{z}$ is less than the critical $l_{z}$ , a response pattern is classified as aberrant because the data are inconsistent with the predictions of the ZG model. Patterns that are highly inconsistent with model predictions will have large negative $l_{z}$ values (e.g., −2), whereas patterns that are consistent with model predictions will have positive $l_{z}$ values (e.g., >+1) values on a roughly standard normal scale.

Factors Influencing l_z Efficacy

One of the most widely studied issues associated with $l_{z}$ is standardization. The original research by Drasgow et al. (1985) as well as subsequent examinations (e.g., Molenaar & Hoijtink, 1990) found that $l_{z}$ is approximately, but not exactly, standardized. That is, the empirical distribution of $l_{z}$ departs somewhat from normality when $\hat{θ}$ is used instead of $θ$ for the calculations (e.g., Molenaar & Hoijtink, 1990; Nering, 1997; Snijders, 2001). Therefore, using a lower one-tailed critical value of, say, −1.64 for classification decisions could lead to Type I error rates that differ from the expected (.05) level.

To address this limitation, some authors have explored the use of empirical critical values as alternatives to those based on normality assumptions (Stark, Chernyshenko, & Drasgow, 2012; see also Nering, 1997). Essentially, one must simulate large numbers of normal response patterns based on actual exam or scale characteristics, compute $l_{z}$ for each pattern, find the $l_{z}$ value corresponding to the 5th percentile, for example, and use that value as a lower-bound critical value for screening the real response data. Although this method has proved useful in other contexts, such as differential item functioning detection (e.g., Flowers, Oshima, & Raju, 1999; Meade, Lautenschlager, & Johnson, 2007; Seybert & Stark, 2012), research is needed with $l_{z}$ to determine whether it provides better classification accuracy normal distribution theory critical values, especially when considering the computational complexity it introduces.

Past studies involving applications of $l_{z}$ and other person fit statistics have identified several other important factors affecting the detection of aberrant response patterns. First, certain types of aberrant responding appear to be easier to detect than others (Drasgow, 1982; Meijer, Molenaar, & Sijtsma, 1994). Intuition suggests that faking might be difficult to detect, especially if the degree of distortion is consistent across items. If there are too few apparent inconsistencies between observed and predicted item response probabilities, then it would be virtually impossible to distinguish between truly high or low performance and spuriously high or low performance.

A second factor affecting the power of person fit statistics is the proportion of items answered aberrantly. Several studies involving dominance IRT models have shown that higher proportions of aberrant responding are associated with higher detection rates (Drasgow et al., 1987; Levine & Rubin, 1979).

Test composition has also been found to influence detection rates. Given the same type and relative proportions of aberrant responding, detection rates are consistently higher with longer tests (Emons, Sijtsma, & Meijer, 2004; Nering & Meijer, 1998), perhaps because trait scores are more accurately estimated and there are more opportunities to observe inconsistencies with model predictions. Second, higher power and lower Type I error are typically observed with tests having more discriminating items (Emons et al., 2004; Meijer, 1997; Meijer et al., 1994) and more variation in item extremity (Reise, 1995). This makes intuitive sense because higher discrimination leads to higher test information and, thus, more accurate trait estimation. And, variations in extremity highlight inconsistencies between predicted and observed responses given one’s estimated trait score.

Finally, some recent studies have examined the effects of parameter estimation error on the power and Type I error of person fit indexes. In accordance with the statistical principle of consistency, large samples are always desirable for item/statement parameter estimation. The more these parameter estimates differ from their true values, the more error there will be in the estimated trait scores and, thus, the lower the power to detect aberrance. Fortunately, person fit research with single-statement measures has shown only small detrimental effects for parameter estimation error on power (Hendrawan, Glas, & Meijer, 2005). However, research is needed to see whether this finding generalizes to ZG-based UPP measures calibrated via marginal maximum likelihood (MML) estimation (Stark & Drasgow, 2002).

Method

Study Design

This research investigated the power and Type I error rates for $l_{z}$ aberrance detection using a Monte Carlo simulation involving four primary factors: (a) UPP test length (10 items, 20 items), (b) percent of items answered aberrantly (20%, 50%, 100%), (c) response style (normal, random, fake good), and (d) test information (medium, high). In addition, to examine how UPP statement parameter estimation accuracy in a pretesting scenario would affect subsequent operational $l_{z}$ screening decisions, classification accuracy was studied with empirical critical values using true and MML statement parameter estimates, based on samples of 1,000 and 500 examinees, respectively (TRUE, MML1000, MML500), as well as critical values based on normal distribution theory. For this aspect of the research, lower one-tailed critical values corresponding to nominal alpha levels of .01, .05, .10, and .20 were used.

Test Characteristics

In preparation for this simulation, four tests were created to satisfy the test information and test length considerations mentioned above. First, in accordance with the recommendation by Stark and Drasgow (2002), a 10-item high information UPP test was assembled by pairing statement parameters that differed by about 2.5 units along different parts of the trait continuum. The result was a test information function that had an amplitude of approximately 5 near $θ = 0.8$ . Next, a 10-item medium information test was created by pairing statement parameters that differed by about 1.5 units along different parts of the trait continuum. Finally, 20-item medium and high information tests were created by adding replicas of the respective 10-item tests.

Data Generation and l_z Analyses

Power and Type I error rates for $l_{z}$ classification decisions were computed more than 100 replications in each of the 28 experimental conditions. A C++ program was developed to perform the following sequence of steps for data generation and analysis in the simulation study (this program can be obtained by contacting the first author):

One thousand trait scores (thetas) were obtained by sampling from a standard normal distribution.

“Normal” responses to each item of each test were simulated by computing the probability of preferring statement s to statement t in item i given a simulee’s true trait score (see Equation 1) and comparing the value to a random uniform number.

Three sets of statement parameters for each of the four tests were used to investigate the effects of MML estimation error on $l_{z}$ classification accuracy. Specifically, the response data generated in Step 2 were calibrated using the ZG MML computer program (Stark & Drasgow, 2002) using the full sample of 1,000 simulees (MML1000) and a randomly selected subsample of 500 (MML500). The TRUE statement parameters served as a baseline for comparison.

Three sets of $l_{z}$ values were computed for each simulee using the true and estimated ZG statement parameters from Step 3 in conjunction with the respective expected a posteriori (EAP) trait score estimates provided by the ZG_EAP computer program (Stark, 2006).

Lower one-tailed empirical critical values for “operational” $l_{z}$ classification decisions were obtained by sorting the respective sets of observed $l_{z}$ values from Step 4 in ascending order and identifying the values corresponding to the 1st, 5th, 10th, and 20th percentiles. Theoretical critical values for those same percentiles under normality assumptions were obtained by using an inverse normal probability table.

New response data reflecting varying degrees of aberrance (0%, 20%, 50%, 100%) were generated to under an operational testing scenario. For the 0% (no aberrance or normal) conditions, 1,000 new trait scores were sampled from a standard normal distribution and used to generate UPP responses to each of the four tests using the TRUE statement parameters. For the 20% and 50% aberrance conditions, item responses from those same data sets were randomly designated for replacement with fake good or random responses. In the 100% condition, all of the responses were replaced with fake good or random responses. Random responses were generated by sampling a random number from a uniform distribution and comparing it to 0.5. If the result exceeded 0.5, the response was scored as 1; otherwise, the response was scored as 0. Fake good responses were simulated by adding 1.5 to a simulee’s trait score when computing $P_{s t_{i}} (θ)$ for the designated items. If the result exceeded a randomly sampled uniform number, then the response was coded 1; otherwise, 0. Simulating faking in this way has been referred to as the theta-shift method (Zickar & Drasgow, 1996).

As in Step 4, three sets of $l_{z}$ values were computed for each response pattern generated in the previous step using the true and MML statement parameter estimates and the resulting EAP trait scores. Each response pattern was then classified as normal or aberrant using each of the observed $l_{z}$ values in conjunction with each of the empirical and theoretical critical values from Step 5.

Type I error was computed for each of the critical values by calculating the proportion of response patterns in the 0% conditions that were misclassified as aberrant. Power was computed in the 20%, 50%, and 100% conditions by computing the proportion of response patterns that were correctly identified as aberrant.

Steps 1 through 8 were repeated until 100 replications were performed.

Power and Type I error results were tabulated, and ANOVA was used to test for the statistical significance of main effects and interactions involving up to three variables.

Results

Tables 1 through 3 show the average Type I error and power rates across the 100 replications in each simulation condition. In particular, Table 1 presents detailed results for Type I error under conditions of test length (10 and 20 items), test information (medium and high information), type of statement parameters (TRUE, MML1000, MML500), and type of critical values (empirical, theoretical).

Table 1.

Type I Error Rates for Empirically and Theoretically Driven Critical Values.

			Empirical critical values				Theoretical critical values
			Nominal alpha				Nominal alpha
Length	Information	Parameter	.01	.05	.10	.20	.01	.05	.10	.20
10	Medium	TRUE	.01	.05	.10	.20	.01	.04	.07	.13
		MML1000	.01	.05	.10	.20	.02	.04	.07	.13
		MML500	.01	.05	.10	.20	.01	.04	.08	.13
	High	TRUE	.01	.05	.10	.20	.02	.04	.06	.11
		MML1000	.01	.05	.10	.20	.02	.04	.06	.10
		MML500	.01	.05	.10	.20	.01	.03	.05	.09
20	Medium	TRUE	.01	.05	.10	.20	.01	.04	.08	.14
		MML1000	.01	.05	.10	.20	.01	.04	.07	.14
		MML500	.01	.05	.10	.20	.01	.04	.07	.13
	High	TRUE	.01	.05	.10	.20	.02	.04	.07	.13
		MML1000	.01	.05	.10	.20	.01	.04	.06	.12
		MML500	.01	.05	.10	.20	.01	.03	.06	.11

Table 2.

Power to Detect Random Responding With Empirically and Theoretically Driven Critical Values.

				Empirical critical values				Theoretical critical values
				Nominal alpha				Nominal alpha
Length	Information	Aberrancy	Parameter	.01	.05	.10	.20	.01	.05	.10	.20
10	Medium	20%	TRUE	.03	.13	.23	.38	.04	.11	.16	.26
			MML1000	.03	.13	.23	.38	.04	.11	.17	.26
			MML500	.03	.13	.23	.38	.04	.11	.16	.26
		50%	TRUE	.15	.40	.53	.69	.19	.37	.47	.58
			MML1000	.16	.40	.54	.69	.20	.37	.47	.59
			MML500	.16	.40	.54	.69	.21	.37	.47	.59
		100%	TRUE	.47	.71	.81	.89	.52	.69	.77	.84
			MML500	.47	.71	.81	.89	.52	.68	.76	.84
			MML1000	.47	.71	.81	.89	.52	.69	.77	.84
	High	20%	TRUE	.08	.22	.34	.51	.11	.20	.29	.36
			MML1000	.08	.23	.35	.51	.09	.18	.25	.34
			MML500	.08	.24	.35	.51	.08	.17	.24	.33
		50%	TRUE	.43	.62	.71	.80	.52	.60	.66	.73
			MML1000	.44	.63	.71	.81	.50	.59	.64	.72
			MML500	.43	.63	.71	.80	.48	.58	.63	.71
		100%	TRUE	.76	.89	.92	.96	.82	.88	.90	.93
			MML1000	.77	.89	.92	.96	.80	.87	.90	.92
			MML500	.76	.89	.92	.96	.78	.86	.89	.92
20	Medium	20%	TRUE	.05	.17	.27	.43	.06	.14	.22	.34
			MML1000	.05	.17	.27	.43	.06	.14	.22	.33
			MML500	.05	.17	.27	.43	.06	.14	.21	.33
		50%	TRUE	.37	.63	.75	.86	.40	.60	.70	.81
			MML1000	.37	.63	.75	.86	.39	.59	.69	.80
			MML500	.37	.63	.75	.86	.39	.59	.69	.79
		100%	TRUE	.81	.93	.96	.98	.83	.92	.95	.97
			MML1000	.81	.93	.96	.98	.82	.91	.95	.97
			MML500	.81	.93	.96	.98	.82	.91	.95	.97
	High	20%	TRUE	.12	.27	.38	.55	.15	.25	.33	.44
			MML1000	.12	.28	.40	.56	.14	.23	.31	.42
			MML500	.12	.28	.40	.56	.13	.22	.29	.41
		50%	TRUE	.70	.86	.91	.95	.75	.85	.89	.93
			MML1000	.71	.86	.91	.95	.74	.84	.88	.92
			MML500	.71	.86	.91	.95	.72	.82	.87	.92
		100%	TRUE	.96	.99	.99	1.00	.97	.99	.99	1.00
			MML1000	.96	.99	.99	1.00	.97	.98	.99	.99
			MML500	.96	.99	.99	1.00	.96	.98	.99	.99

Table 3.

Power to Detect Faking Good With Empirically and Theoretically Driven Critical Values.

				Empirical critical values				Theoretical critical values
				Nominal alpha				Nominal alpha
Length	Information	Aberrancy	Parameter	.01	.05	.10	.20	.01	.05	.10	.20
10	Medium	20%	TRUE	.02	.10	.19	.33	.03	.09	.13	.23
			MML1000	.02	.10	.19	.33	.03	.09	.13	.23
			MML500	.02	.10	.19	.33	.03	.09	.13	.23
		50%	TRUE	.03	.11	.19	.32	.04	.10	.14	.23
			MML1000	.03	.11	.20	.34	.04	.10	.15	.23
			MML500	.03	.11	.19	.34	.04	.09	.15	.23
		100%	TRUE	.01	.05	.09	.18	.01	.04	.07	.12
			MML1000	.01	.05	.10	.20	.01	.04	.07	.12
			MML500	.01	.05	.10	.19	.02	.04	.07	.12
	High	20%	TRUE	.04	.16	.25	.45	.06	.15	.22	.27
			MML1000	.04	.17	.26	.45	.05	.13	.19	.25
			MML500	.04	.17	.26	.44	.05	.12	.18	.25
		50%	TRUE	.05	.17	.31	.44	.08	.15	.20	.33
			MML1000	.05	.17	.29	.44	.07	.13	.19	.30
			MML500	.06	.17	.29	.43	.06	.12	.18	.28
		100%	TRUE	.01	.04	.08	.16	.02	.03	.05	.09
			MML1000	.01	.04	.08	.16	.01	.03	.04	.07
			MML500	.01	.04	.08	.16	.01	.03	.04	.07
20	Medium	20%	TRUE	.03	.11	.20	.34	.04	.10	.16	.26
			MML1000	.03	.12	.20	.34	.04	.10	.16	.26
			MML500	.03	.11	.20	.34	.04	.10	.15	.25
		50%	TRUE	.03	.11	.19	.33	.03	.09	.15	.25
			MML1000	.03	.11	.19	.33	.03	.09	.14	.24
			MML500	.03	.11	.19	.33	.03	.09	.14	.24
		100%	TRUE	.01	.04	.09	.18	.01	.04	.06	.12
			MML1000	.01	.05	.09	.18	.01	.04	.07	.12
			MML500	.01	.05	.10	.19	.01	.04	.06	.12
	High	20%	TRUE	.05	.17	.29	.45	.07	.15	.22	.32
			MML1000	.06	.18	.29	.45	.07	.14	.20	.31
			MML500	.06	.18	.29	.45	.06	.13	.19	.29
		50%	TRUE	.04	.15	.26	.41	.06	.13	.20	.30
			MML1000	.05	.16	.27	.41	.06	.12	.18	.29
			MML500	.05	.16	.26	.41	.05	.11	.17	.27
		100%	TRUE	.01	.04	.08	.16	.01	.03	.06	.10
			MML1000	.01	.04	.08	.15	.01	.03	.05	.09
			MML500	.01	.04	.08	.15	.01	.02	.04	.08

As can be seen, the Type I error rates for the empirical critical values matched perfectly with the respective nominal alpha levels. Specifically, Type I errors of .01, .05, .10, and .20 were found for the nominal alphas of .01, .05, .10, and .20, respectively, regardless of test length, test information, and the type of statement parameters. In contrast, with the exception of the .01 nominal alpha level, the theoretical critical values resulted in consistently lower than expected Type I errors and the negative bias increased as the nominal alpha increased from .05 to .20. Importantly, however, there were no marked differences in Type I error as a function of the type of statement parameters, test information, or test length.

Table 2 presents the power results for random response detection using empirical and theoretical $l_{z}$ critical values. First, with the exception of the .01 nominal alpha, power was slightly higher when using the empirical $l_{z}$ critical values, which is consistent with the findings of lower than expected Type I error for the theoretical critical values in Table 1. Second, power to detect random responding increased somewhat with test length and test information, and there was a sharp improvement in overall power as the percentage of aberrant items increased. Importantly, these results clearly show that there is ample power to detect 100% random responding with informative tests, regardless of the type of critical value or statement parameters used for the $l_{z}$ computations.

Table 3 shows the power results for detecting faking good, which was operationalized as a consistent upward shift in trait scores on items that were designated as aberrant. As can be seen in the table, power to detect faking was poor in every case. In what were optimal conditions for detecting random responding (20 items, high information, 100% aberrance), power for detecting faking was only .16 with empirical critical values and a nominal alpha of .20, and the results were even worse with stricter alphas. Neither test length nor test information had a beneficial effect on power, nor did the use of true statement parameters or empirical critical values. The only interesting finding is that power was lowest, as expected, in the respective 100% aberrance conditions due to the inability to distinguish an “across-the-board” faker from a truly high-trait responder.

To buttress the interpretation of the power results in Tables 1 through 3 and address the specific hypotheses that were proposed above, an ANOVA and planned comparisons were conducted. Table 4 shows the ANOVA results for main effects and interactions that accounted for at least 1% of the variance in power. All of the factors manipulated were statistically significant (p < .05), with the largest effect observed for the type of aberrance. Power was markedly higher for detecting random responding than for fake good responding (p < .0001; $ω^{2}$ = .478). In addition, power to detect random responding increased with test length (p < .0001; $ω^{2}$ = .012), test information (p < .0001; $ω^{2}$ = .017), and the percentage of aberrant items (p < .0001; $ω^{2}$ = .122). As expected, the highest power was observed in the 20-item, high information conditions and the lowest power was found in the 10-item, medium information conditions. Also as expected, the power to detect fake good responding was low, especially in the 100% aberrance conditions.

Table 4.

Main Effects and Interactions for Independent Variables on Power Rates.

Source	df _B	F	$ω^{2}$
Response style (R)	1	77698.7	.478
Percentages of aberrance (A)	2	9945.61	.122
Nominal alpha (N)	3	469.65	.087
Test information (I)	1	284.17	.017
Test length (L)	1	1961.02	.012
Type of critical values (C)	1	528.7	.003
Type of statement parameters (S)	2	3.19	.000
R×A	2	18474.4	.227
R×L	1	1859.85	.011

Note. All effects shown were significant at p < .05. Only interaction effects that accounted for at least 1% of the variance in power are included. $ω^{2}$ = proportion of variance accounted for by the independent variables. df_B = degrees of freedom between; for all effects; degrees of freedom within = 392.

Discussion

The primary goal of this study was to investigate the efficacy of the $l_{z}$ person fit statistic for detecting aberrant responding with UPP measures constructed and scored based on the Zinnes–Griggs (Zinnes & Griggs, 1974) IRT model, which has proven useful for a variety of noncognitive testing applications in organizational settings (e.g., Borman et al., 2001; Chernyshenko et al., 2009; Houston et al., 2005). Because UPP measures are now being used to collect both “self-” and “other” reports, the capability of $l_{z}$ to detect two of the most common and potentially detrimental response sets, namely, fake good and random responding, was explored. The effectiveness of $l_{z}$ was studied using empirical as well as theoretical critical values for classification, along with test length, test information, the type of statement parameters, and the percentage of items answered aberrantly (20%, 50%, 100%).

In short, $l_{z}$ was ineffective in detecting fake good responding, with power approaching zero in the 100% aberrance conditions. However, $l_{z}$ was highly effective in detecting random responding, with power approaching 1.0 in the long-test, high information conditions, and there was no diminution in efficacy when using MML estimates of statement parameters in place of the true values. Furthermore, although using empirical critical values for classification provided slightly higher power, theoretical critical values, corresponding to a standard normal distribution provided nearly as good results.

Finding that faking good is difficult to detect was not surprising. If a respondent fakes on a large proportion of items, there will be few apparent inconsistencies in the response pattern, making it difficult to distinguish a spuriously high from a truly high-trait score. Similarly, if just a small percentage of items are faked, the likelihood of the response pattern would be very similar to that of a normal responder, which would also reduce hit rates. These results are consistent with the optimal appropriateness measurement findings and conclusions of Zickar and Drasgow (1996), who examined fake good response detection with Likert-type personality scales in an experiment involving coached and ad-lib faking conditions, and thus confirm existing evidence that faking is difficult to detect.

Another interesting and important finding of the study was that using MML statement parameter estimates, based on samples of 500 yielded power and Type I error rates, were nearly identical to those for the true parameter values. This is consistent with the findings of small effects in research involving single-statement IRT models (Hendrawan et al., 2005). It is also good news for practitioners because the true parameters are never known and, for obvious reasons, pretest samples of 500 and smaller are preferred. In the future, it would be interesting to explore whether subject matter expert (SME) ratings of statement location would be as effective for $l_{z}$ aberrance detection as the MML500 estimates, given that recent research with a ZG-based computer adaptive test showed little differences between trait scores based on true statement parameters, MML estimates, and SME ratings of statement location (Stark et al., 2011).

Finally, although previous $l_{z}$ research has raised concern about the use of critical values based on normality assumptions (Nering, 1995; Reise, 1995), it was found that a computationally intensive method of obtaining empirical critical values provided relatively small improvements. In this study, using theoretical critical values (i.e., those corresponding to a standard normal distribution) resulted in lower than expected Type I error rates, which, in turn, reduced power somewhat. In practice, using empirical critical values should enhance classification accuracy, but a reasonable simple alternative might be just to choose a slightly higher theoretical critical value for flagging examinees.

It is also important to note that significant $l_{z}$ values do not provide information about the source of aberrance. Unlike optimal appropriateness measurement approaches, where an aberrant model has to be specified a priori based on hypotheses about the nature of aberrant responding, $l_{z}$ just addresses the “typicality” of response patterns. Consequently, post hoc analyses involving other background variables are essential for exploring the psychological meaning of misfit (Meijer, Egberink, Emons, & Sijtsma, 2008).

Limitations and Future Research

This study has some limitations that might be explored in future investigations. First, the data structures were very clean; that is, normal data were generated according to the ZG model and aberrant responses were generated based on the operationalizations of random responding and faking. However, in the real world, some degree of model-data misfit is always present, so it would be interesting to explore the power to detect random responding when model assumptions are violated. It might be also beneficial to compare $l_{z}$ efficacy with empirical and theoretical critical values using trait scores sampled, for example, from a negatively skewed distribution, which might better reflect the distribution of performance ratings among students in accelerated learning programs or experienced job incumbents.

Another limitation of this study is that it focused on one index for detecting aberrance. In future work, it would be interesting to compare the performance of ZG-based $l_{z}$ with optimal appropriateness measurement methods (Levine & Drasgow, 1988), as well as simpler statistical approaches that might be easier for practitioners to implement. It might also be useful to investigate how $l_{z}$ can be adapted for use with more complex forced choice formats, such as multidimensional pairs or tetrads, and whether the findings for the key variables examined here will generalize.

Finally, no real data were examined in this study, so the researchers are encouraged to conduct investigations aimed at explaining aberrance using auxiliary information, such as response latencies, conscientiousness scores, and perhaps level of education. In addition, researchers might explore the possibility of using $l_{z}$ values as predictors of criteria such as citizenship and counterproductive performance in academic and work settings.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Borman

W. C.

Buck

Hanson

M. A.

Motowidlo

S. J.

Stark

Drasgow

(2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using Computerized Adaptive Rating Scales. Journal of Applied Psychology, 86, 965-973.

Borman

W. C.

Hanson

M. A.

Motowidlo

S. J.

Drasgow

Foster

Kubisiak

U. C.

(1998, April). Computerized Adaptive Rating Scales that measure contextual performance. Paper presented at the 13th Annual Conference for the Society of Industrial and Organizational Psychology, Dallas, TX.

Borman

W. C.

Motowidlo

S. J.

(1993). Expanding the criterion domain to include elements of contextual performance. In Schmitt

Borman

W. C.

(Eds.), Personnel selection in organizations (pp. 71-98). San Francisco, CA: Jossey-Bass.

Chernyshenko

O. S.

Stark

Williams

(2009). Latent trait theory approach to measuring person-organization fit: Conceptual rationale and empirical evaluation. International Journal of Testing, 9, 358-380.

Drasgow

(1982). Choice of test models for appropriateness measurement. Applied Psychological Measurement, 6, 297-308.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.

Emons

W. H. M.

Sijtsma

Meijer

R. R.

(2004). Testing hypotheses about the person response function in person-fit analysis. Multivariate Behavioral Research, 39, 1-35.

Flowers

C. P.

Oshima

T. C.

Raju

N. S.

(1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326.

10.

Hendrawan

Glas

Meijer

(2005). The effect of person misfit on classification decisions. Applied Psychological Measurement, 29, 26-44.

11.

Houston

J. S.

Borman

W. C.

Farmer

W. L.

Bearden

R. M.

(2005). Development of the Enlisted Computer Adaptive Personality Scales (ENCAPS) for the United States Navy, Phase 2 (Institute Report No. 503). Minneapolis, MN: Personnel Decisions Research Institute.

12.

Levine

M. V.

Drasgow

(1988). Optimal appropriateness measurement. Psychometrika, 53, 161-176.

13.

Levine

M. V.

Rubin

D. B.

(1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269-289.

14.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455.

15.

Meade

A. W.

Lautenschlager

G. J.

Johnson

E. C.

(2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430-455.

16.

Meijer

R. R.

(1997). Person fit and criterion-related validity: An extension of the Schmitt, Cortina, and Whitney study. Applied Psychological Measurement, 21, 99-113.

17.

Meijer

R. R.

Egberink

I. J. L.

Emons

W. H. M.

Sijtsma

(2008). Detection and validation of unscalable item score patterns using item response theory: An illustration with Harter’s Self-Perception Profile for Children. Journal of Personality Assessment, 90, 227-238.

18.

Meijer

R. R.

Molenaar

I. W.

Sijtsma

(1994). Influence of test and person characteristics on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.

19.

Molenaar

I. W.

Hoijtink

(1990). The many null distributions of person fit indices. Psychometrika, 55, 75-106.

20.

Nering

M. L.

(1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121-129.

21.

Nering

M. L.

(1997). The distribution of indexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115-127.

22.

Nering

M. L.

Meijer

R. R.

(1998). A comparison of the person response function and the l_z person-fit statistic. Applied Psychological Measurement, 22, 53-69.

23.

Reise

S. P.

(1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213-229.

24.

Seybert

J. M.

Stark

(2012). Iterative linking with the differential functioning of items and test (DFIT) method: Comparison of testwide and item parameter replication (IPR) critical values. Applied Psychological Measurement, 36, 494-515.

25.

Smith

Kendall

(1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.

26.

Snijders

T. A. B.

(2001). Asymptotic distribution of person fit statistics with estimated person parameters. Psychometrika, 66, 331-342.

27.

Stark

(2006). ZG-EAP: A computer program for EAP scoring with the Zinnes-Griggs model. Unpublished manuscript, University of South Florida, Tampa.

28.

Stark

Chernyshenko

O. S.

(2011). Computerized adaptive testing with the Zinnes and Griggs pairwise preference ideal point model. International Journal of Testing, 11, 231-247.

29.

Stark

Chernyshenko

O. S.

Drasgow

(July, 2012). Development of a person-fit index for multidimensional pairwise preference tests. Invited presentation at the 8th Conference of the International Test Commission, Amsterdam, the Netherlands.

30.

Stark

Chernyshenko

O. S.

Lee

W. C.

Drasgow

White

L. A.

Young

M. C.

(2011). Optimizing prediction of attrition with the U.S. Army’s Assessment of Individual Motivation (AIM). Military Psychology, 23, 180-201.

31.

Stark

Drasgow

(1998, April). Application of an item response theory ideal point model to computer adaptive assessment of job performance. Paper presented at the 13th Annual Conference for the Society of Industrial and Organizational Psychology, Dallas, TX.

32.

Stark

Drasgow

(2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model. Applied Psychological Measurement, 26, 208-227.

33.

Zickar

M. J.

Drasgow

(1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71-87.

34.

Zinnes

J. L.

Griggs

R. A.

(1974). Probabilistic, multidimensional unfolding analysis. Psychometrika, 39, 327-350.

Detecting Aberrant Responding on Unidimensional Pairwise Preference Tests

Abstract

Keywords

Adopting the lz Person Fit Index for ZG UPP Model

Factors Influencing lz Efficacy

Method

Study Design

Test Characteristics

Data Generation and lz Analyses

Results

Discussion

Limitations and Future Research

Footnotes

Declaration of Conflicting Interests

Funding

References

Adopting the l_z Person Fit Index for ZG UPP Model

Factors Influencing l_z Efficacy

Data Generation and l_z Analyses