Though Forced,Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Abstract

Forced-choice (FC) measures are gaining popularity as an alternative assessment format to single-statement (SS) measures. However, a fundamental question remains to be answered: Do FC and SS instruments measure the same underlying constructs? In addition, FC measures are theorized to be more cognitively challenging, so how would this feature influence respondents’ reactions to FC measures compared to SS? We used both between- and within-subjects designs to examine the equivalence of the FC format and the SS format. As the results illustrate, FC measures scored by the multi-unidimensional pairwise preference (MUPP) model and SS measures scored with the generalized graded unfolding model (GGUM) showed strong equivalence. Specifically, both formats demonstrated similar marginal reliabilities and test-retest reliabilities, high convergent validities, good discriminant validities, and similar criterion-related validities with theoretically relevant criteria. In addition, the formats had little differential impact on respondents’ general emotional and cognitive reactions except that the FC format was perceived to be slightly more difficult and more time-saving.

Keywords

forced-choice single-statement equivalence MUPP GGUM

The scientific study of personality, job attitudes, job engagement, vocational interests, and many other psychological attributes heavily relies on the use of self-reported measures. Since its introduction by Likert in 1932, the single-statement (SS) rating scale format has no doubt been the most widely used form of measurement (Brown & Maydeu-Olivares, 2013). SS measurement is, however, not without issues. Faking and rating scale errors, such as halo, acquiescent responding, extreme response style, and midpoint response style, lead to serious concerns (e.g., Donovan, Dwight, & Schneider, 2014; Landy & Farr, 1980; Pavlov, Maydeu-Olivares, & Fairchild, 2018). An alternative that was originally developed to overcome problems with SS measures was the forced-choice (FC) scale (Rundquist, 1946). It has the advantage of deeming some response styles impossible (e.g., midpoint/extreme response style), and some have found improved measurement with this format (Bartram, 2007; Brown, Inceoglu, & Lin, 2017; Cao & Drasgow, in press; Guenole, Brown, & Cooper, 2018).

However, psychometric models make different assumptions about response processes for FC and SS measures (Sass, Frick, Reips, & Wetzel, 2018). These theoretical differences raise a fundamental question: Are scales using FC and SS formats measuring the same construct? In addition, respondents’ reactions to the FC format compared to their reactions to the SS format is another important but largely ignored topic. The present study is designed to examine the comparability of SS and FC measures in the context of response processes modeled by ideal point models. An abridged static version of the Tailored Adaptive Personality Assessment System (TAPAS; Drasgow et al., 2012), which measures narrow facets of personality factors and has been widely used for military selection and classification (Drasgow et al., 2012), is used to answer our research questions. To the extent that SS and FC measures of the same construct are equivalent under ideal conditions (e.g., low stakes, homogeneous sample), we can build on SS findings with a more robust measurement method and psychometric model that may provide meaningful results under much wider circumstances.

The Single-Statement Format

Likert’s (1932) SS approach avoids items reflecting intermediate trait levels and requires respondents to rate their degree of agreement with each statement on an n-point scale. Individual trait scores are then computed as the sum of all responses after reverse-coding negative items. Likert’s methods for item selection and scoring are consistent with a dominance response process which assumes that the probability of endorsing an item will monotonically increase as the trait level increases (Drasgow, Chernyshenko, & Stark, 2010). For example, when responding to the statement “I am very conscientious,” more conscientious respondents are more likely to endorse this item, thus obtaining higher scores. Most popular personality scales were constructed and scored following the dominance assumption.

Recent research suggests that ideal point models, rather than a dominance model, should be used for personality measures. These models posit that each statement has a location parameter describing its extremity that is on the same metric as the latent trait. It is assumed that a respondent is more likely to endorse an item when the statement’s location is closer to the individual’s latent trait level. The more a person’s trail level departs from the statement’s location, either higher or lower, the less likely he or she is to endorse the statement. If we plot the endorsement probability against the latent trait level, we will see an inverse U-shaped curve (Cao, Drasgow, & Cho, 2015). For instance, when responding to the statement “I am about average in regards to conscientiousness,” moderately conscientious respondents have the greatest probability of endorsement because the statement’s location matches their latent trait levels. However, those very high or low on conscientiousness are less likely to endorse this item because there is a mismatch between the statement’s location and their own standings on the latent trait continuum.

Evidence in support of ideal point models has been found for personality assessment (e.g., Cao et al., 2015; Chernyshenko, Stark, Drasgow, & Roberts, 2007; Stark, Chernyshenko, Drasgow, & Williams, 2006), vocational interests (Tay, Drasgow, Rounds, & Williams, 2009), job satisfaction (Carter & Dalal, 2010), emotional intelligence (Cho, Drasgow, & Cao, 2015), and attitude measures (Roberts & Laughlin, 1996). Using a dominance model when an ideal point process applies may impair diagnostic, selection, and classification decisions due to reduced measurement precision (e.g., Chernyshenko et al., 2007; Roberts, Laughlin, & Wedell, 1999; Sun, 2018; Tay et al., 2009) and hinder the detection of interaction effects (Cao, Song, & Tay, 2018; Carter, Dalal, Guan, LoPilato, & Withrow, 2017). Therefore, the current study used ideal point models for items in the SS format.

The Forced-Choice Format

In an FC item, respondents are presented with at least two single statements in a block. They are instructed to choose the statement that is “more like me,” to choose one statement that is “more like me” and one that is “the least like me,” or to rank all statements within a block (Brown, 2016a). Recently, some other variations of the FC format are also proposed (Brown, 2016b; Brown & Maydeu-Olivares, 2017). Statements within a block are usually balanced on social desirability. A typical example of an FC item is given as follows:

Choose the statement that is the more like you:

I keep my room tidy.

I like talking to people.

These statements assess the conscientiousness and extraversion dimensions of personality. If a respondent chooses the first statement as “the more like me,” the traditional scoring approach would allocate 1 point to conscientiousness and 0 points to extraversion. Trait scores would be computed by adding up scores for all of the items assessing each construct. This scoring of FC responses, however, leads to ipsativity, which means that “the sum of the scores obtained over the attributes measured for each respondent is a constant” (Hicks, 1970, p. 169). Consequently, ipsative scoring allows only intraindividual comparisons, and between-person comparisons are technically not appropriate. As a result of the concerns about ipsativity, the popularity of the FC format appears to have declined substantially from its high point in the 1950s.

Recently, however, the tide appears to have turned due to the development of several psychometric models that enable researchers to obtain normative trait estimates from FC measures. Building upon the finding that ideal point models may better describe the underlying response process for some constructs, Stark, Chernyshenko, and Drasgow (2005) developed the multi-unidimensional pairwise preference (MUPP) model. This model assumes that when a respondent is asked to indicate a preference for one of the two statements in a block, he or she first evaluates each statement separately and independently and then decides whether or not to endorse each statement. If he or she initially decides to endorse both or neither of the statements, the respondent has to reevaluate the statements until a single preference is reached. The MUPP model has solved the ipsativity problem and produces normative scores (Stark, Chernyshenko, Drasgow, & White, 2012; Stark et al., 2005; see also Brown & Maydeu-Olivares, 2011, and McCloy, Heggestad, & Reeve, 2005, for alternative approaches that also overcome ipsativity).

Psychometric Equivalence Between FC and SS

The use of the FC format seems to be on the upswing. However, psychometric models make different assumptions about response processes for FC and SS measures. According to Tourangeau and Rasinski (1988), when answering an SS measure, a respondent first interprets the statement by understanding the content of the statement and estimating the location of the statement on the trait continuum. Then he or she searches memories for relevant information to locate himself or herself along the same trait continuum. At the information integration stage, he or she calculates the distance between his or her location and the item’s location and uses the distance to form a judgment. The inverse of the distance is similar to Thurstone’s perceived utility (Thurstone, 1927), which then determines choice. When answering an FC measure, however, respondents are assumed to make relative judgments among two or more options. They first evaluate each statement separately, perhaps following the process described above. However, what determines the final choice is the perceived utility differences among alternatives (Thurstone, 1927). When the utility difference is small, this process may require deep contemplation to arrive at a finer differentiation (Meglino & Ravlin, 1998). These response process differences lead to a fundamental question: Do FC and SS scales measure the same construct?

Concerns about the construct interpretation of FC-derived scores have been long recognized (Tenopyr, 1988; Usami, Sakamoto, Naito, & Abe, 2016), but little addressed. Although simulation studies have shown that modern item response theory models can recover normative person parameters (e.g., Brown & Maydeu-Olivares, 2011; Stark et al., 2005), these models have specific assumptions about the underlying response process. If these assumptions do not hold empirically, treating FC estimates and SS estimates as equivalent may be problematic. Empirical data are needed to examine their degree of equivalence.

To ensure that a lack of convergence is not due to content differences, researchers need to compare FC and SS measures that are constructed from the same statement pool. To our knowledge, however, only a few studies have done this. Among these studies, most have examined the equivalence of dominance-model-based SS and FC formats and found mixed evidence. For example, statement factor loadings were found to differ substantially between FC and SS formats (Ackerman, Donnellan, Roberts, & Fraley, 2016; Dueber, Love, Toland, & Turner, 2019; Guenole et al., 2018; Wetzel, Roberts, Fraley, & Brown, 2016), suggesting that respondents may interpret them differently (Bürkner, Schulte, & Holling, in press). While some have reported moderate to high convergent validity (Brown & Maydeu-Olivares, 2011, 2013; Lee, Joo, & Lee, 2019; Lee, Lee, & Stark, 2018; Usami et al., 2016), others have observed very low convergent validity (Anguiano-Carrasco, MacCann, Geiger, Seybert, & Roberts, 2015; Dueber et al., 2019; Seybert, 2013). Moreover, as mentioned above, evidence has been accumulating that shows ideal point models more accurately capture the response processes underlying various psychological measures. Thus, it is largely unknown whether ideal-point-based SS and FC formats produce scores that are equivalent. To date, only one study has compared ideal-point-based FC and SS formats, and found supportive evidence of equivalence (Chernyshenko et al., 2009).

These earlier studies have limitations that need to be addressed. First, the single ideal-point-based study that has been conducted only tested format equivalence with three constructs in a homogenous college sample. The generalizability of its results is unknown. Second, earlier studies have generally adopted a within-subjects design in which respondents finished both FC and SS measures in the same session. Such a design may artificially increase their equivalence due to single-subject response consistency error (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Third, previous studies have not counterbalanced the order in which FC and SS were presented to respondents, which may further inflate the consistency between FC and SS responses. Fourth, little empirical evidence on the temporal stability of FC measures has been reported. As hypothesized by Meglino and Ravlin (1998), the cognitive processes underlying FC may be very intricate in that respondents need to make fine differentiations among statements to arrive at their final decision. Such fine differentiations may be much less consistent across time.

Research Questions

The first research question we address is the psychometric equivalence of FC scales and SS scales that are constructed from the same statement pool. If FC scores and SS scores are equivalent, we would expect them to display the following psychometric properties: (a) both formats have similar reliability; (b) scores on the same facet are highly correlated across formats; (c) scores of both formats show similar discriminant validity; (d) scores on the same facet show similar high correlations with a third external measure of the same construct; and (e) both formats show very similar criterion-related validity. We will examine these specific questions.

Regarding criterion-related validity, we selected four theoretically relevant criteria that have differential and robust correlations with the Big Five personality factors. The first criterion is subjective well-being (SWB). Steel, Schmidt, and Shultz (2008) meta-analytically showed that neuroticism has the largest correlation with SWB, followed by extraversion, conscientiousness, and agreeableness. The second criterion is core self-evaluation (CSE), which has been consistently found to be highly correlated with neuroticism, followed by extraversion, conscientiousness, and agreeableness (e.g., Judge, Erez, Bono, & Thoresen, 2003; Judge, Van Vianen, & De Pater, 2004). The third criterion is subjective health (SH). Many studies have found that SH is strongly related to neuroticism, extraversion, and conscientiousness (e.g., Löckenhoff, Sutin, Ferrucci, & Costa, 2008; Stephan, Demulier, & Terracciano, 2012). The fourth criterion is job satisfaction (JS). Meta-analytic evidence shows that workers low in neuroticism and high in conscientiousness, extraversion, and agreeableness are more likely to be satisfied with their jobs (Judge, Heller, & Mount, 2002). If FC scores and SS scores are psychometrically equivalent, they are expected to demonstrate similar correlations with these criteria.

Another important question concerns the comparability of respondent reactions to the FC format and SS format. Previous studies have found that applicants’ reactions to test formats and procedures can affect test validity, test motivation, adverse impact, and various other behavioral intentions (e.g., Hausknecht, Day, & Thomas, 2004; Smither, Reilly, Millsap, Pearlman, & Stoffey, 1993). Clearly, respondent reactions should be studied, and this seems particularly important for the two-alternative forced-choice format, which may require respondents to make very difficult choices (Bartram & Brown, 2004). We are particularly interested in whether respondents react differently to FC and SS in both emotional and cognitive aspects.

The Present Study

Two samples were recruited to examine the degree of equivalence between the FC and SS formats and ascertain respondents’ reactions. Sample 1 employed a between-subjects design where a group of respondents completed the FC version and another group completed the SS measure. A between-subjects design allowed us to study score equivalence and circumvent issues related to single subject response consistency. Sample 2 adopted a within-subjects design where all respondents completed both FC and SS measures, but with an interval of two days between assessments. The order of administration was counterbalanced. This within-subjects design allowed us to directly study the convergent validity between the two formats and the two-day interval should minimize memory effects without running the risk of personality change.

Method

Sample

Between-subjects sample

Two subsamples were recruited from the Amazon Mechanical Turk worker pool. The FC subsample completed TAPAS_FC (see below) and other criterion measures. The SS subsample rated the same statements that make up the forced-choice pairs using a 5-point Likert format and completed the same criterion measures. Six quality control items were embedded in the FC condition and five in the SS condition. The quality control items instructed the respondent to select a particular response option (e.g., strongly agree). In this study, we deleted data from respondents who responded incorrectly to more than one quality control item. To assess test-retest reliability, respondents were asked to complete exactly the same instruments twice with a 10-day interval. Respondents were allowed to enroll in only one sample. All the measures were administered twice. Independent groups t tests on the four demographic variables showed that the two samples were demographically similar (t = 0.77 ∼ 1.42, p = .16 ∼ .44). We note that the same set of analyses reported below were also performed on Time 2 data and the results were almost identical to those of Time 1. Therefore, we report only results from Time 1 data because the sample size is larger.

Within-subjects sample

About half of this sample completed the FC format first and the SS format two days later (FS group), and the other half completed the SS format first and the FC format two days later (SF group). Six quality control items were embedded and we allowed one quality control item to be incorrect. Independent groups t tests on the four demographic variables revealed no significant demographic difference between the two groups (t = –0.92 ∼ 0.39, p = .36 ∼ .70). Also, no order effect was found for the FC and SS scores after Bonferroni correction (t = –2.22 ∼ 2.33, p = .03 ∼ .93). Thus, data from the two groups were combined for further analysis.

All respondents received monetary compensation. Details about sample size and demographic composition of each subsample at each time are shown in Table 1.

Table 1.

Sample Demographic Information.

Time	Variable	Sample 1: Between-Subjects				Sample 2: Within-Subjects
		FC Group		SS Group		FS Group		SF Group
		M	SD	M	SD	M	SD	M	SD
T1	Sample size	806 (781)		811 (793)		251 (245)		260 (257)
	Age	39.15	13.01	40.40	13.77	37.91	12.51	37.66	12.73
	Gender	0.70	0.46	0.68	0.47	0.63	0.48	0.67	0.47
	Education	3.65	0.96	3.61	0.98	3.68	0.88	3.63	0.92
	Income	5.03	2.10	4.88	2.12	4.89	2.23	5.00	2.13
T2	Sample size	580 (562)		576 (553)		193 (193)		210 (204)
	Age	40.12	12.99	40.61	13.64	38.15	12.66	38.67	12.63
	Gender	0.70	0.46	0.70	0.46	0.60	0.49	0.65	0.48
	Education	3.71	0.96	3.64	1.00	3.70	0.87	3.64	0.92
	Income	5.04	2.06	4.94	2.11	4.98	2.24	5.11	2.19

Note: FC = forced choice; SS = single statement; TAPAS = Tailored Adaptive Personality Assessment System. FS group = The group that completed TAPAS-FC first and TAPAS-SS two days later. SF group = The group that completed TAPAS-SS first and TAPAS-FC two days later. For cells that contain sample sizes, the first number is the raw sample size and the number in parentheses indicates the valid sample size after excluding these who failed the quality control test. Gender: male is coded as 0 and female is coded as 1. Education is measured using a 6-point scale: 1 = primary school; 2 = high school or equivalent; 3 = some college or vocational school; 4 = a bachelor’s degree or equivalent; 5 = a master’s degree or equivalent; 6 = a doctoral or professional degree. Income is measured in dollars using an 8-point scale. 1 = under 10,000; 2 = 10,000-19,000; 3 = 20,000-29,000; 4 = 30,000-39,000; 5 = 40,000-49,000; 5 = 50,000-74,999; 6 = 75,000-99,999; 7 = 100,000-150,000; 8 = over 150,000.

Measures

TAPAS

TAPAS statements were specifically developed for ideal point measurement, and therefore the statement pools for each facet include both extreme (positive and negative) and intermediate items that cover the entire range of the facet. This abridged version used in the current study included 10 facets: intellectual efficiency (IE) and tolerance (TO) for openness, achievement (AC) and order (OR) for conscientiousness, dominance (DO), sociability (SO) and physical conditioning (PC) for extraversion, selflessness (SE) for agreeableness, and even tempered (ET) and optimism (OP) for emotional stability. FC items were created by matching statements on extremity and social desirability based on preexisting ratings collected before (Stark et al., 2014). A brief description of each facet is shown in Table 2.

Table 2.

Brief Descriptions of TAPAS Facets (Nye et al., 2012).

Big Five Domain	TAPAS Facet (No. of Items)	Brief Description
Openness	Intellectual efficiency (16) Tolerance (17)	Process information and make decisions quickly Interested in other cultures and opinions different from their own
Conscientiousness	Achievement (17) Order (16)	Hard working, ambitious, confident, and resourceful Organize tasks and activities and desire to maintain neat surroundings
Extraversion	Dominance (14) Sociability (18) Physical conditioning (17)	Domineering, take charge, natural leaders Tend to seek out and initiate social interactions Tend to engage in activities to maintain one’s physical fitness
Agreeableness	Selflessness (15)	Generous with one’s time and resources
Emotional stability	Even tempered (18) Optimism (18)	Tend to be calm and stable Have a positive outlook on life and tend to experience a sense of well-being

TAPAS_SS

There were 16 statements for IE, 17 items for TO, 17 items for AC, 16 items for OR, 14 items for DO, 17 items for PC, 18 items for SO, 15 items for SE, 18 items for ET, and 18 items for OP.

TAPAS_FC

Statements that made up the FC pairs were the same as those in TAPAS_SS. There were 10% unidimensional pairs and 90% multidimensional pairs.

Criterion measures

Throughout the two samples, we used the Subjective Well-Being Scale (SWBS; Diener, Emmons, Larsen, & Griffin, 1985), the Core Self-Evaluation Scale (CSES; Judge et al., 2003), and a single-item measure of SH (Pinquart, 2001). The Big Five Inventory (BFI; John & Srivastava, 1999) was used in the between-subject sample, and the Big Five Inventory-2 (BFI-2; Soto & John, 2017) was used in the within-subject sample as external measures of the Big Five factors. Job satisfaction was measured using the Abridged Job Descriptive Index (AJDI; Stanton et al., 2002) in the within-subject sample.

Respondent reactions

We also measured respondents’ immediate affect and vitality level right before and right after the administration of either format of TAPAS using the 10-item version of the Positive and Negative Affect Schedule (PANAS; Thompson, 2007) and three items from the Subjective Vitality Scale (SVS; Bostic, Rubio, & Hood, 2000). Respondents were also asked to rate perceived difficulty of responding, preference, how much effort they had expended, and the degree of concentration during the process of responding.

Details about the criterion measures and respondents’ reactions can be found in Table 3. The criterion measures and respondent reactions were all scored by computing the mean after reverse coding because there were no intermediate items. Details about scoring the two formats of TAPAS can be found in the online supplemental material.

Table 3.

Details About Criterion Measures and Respondent Reaction Measures.

Scale Name	Sample Item	Rating Scale	No. of Items	Cronbach’s α
SWBS	In most ways my life is close to my ideal	1 = strongly disagree…5 = strongly agree	5	.90-.91
CSES	I am confident I get the success I deserve in life	1 = strongly disagree…5 = strongly agree	12	.86-.90
SH	In general, how would you describe your health	1 = excellent…5 = bad	1	NA
BFI-E	Is talkative	1 = strongly disagree…5 = strongly agree	8	.86-.87
BFI-A	Has a forgiving nature	1 = strongly disagree…5 = strongly agree	9	.81-.81
BFI-C	Is a reliable worker	1 = strongly disagree…5 = strongly agree	9	.83-.83
BFI-N	Can be moody	1 = strongly disagree…5 = strongly agree	8	.88-.88
BFI-O	Is inventive	1 = strongly disagree…5 = strongly agree	10	.84-.84
BFI2-E	Is outgoing, sociable	1 = strongly disagree…5 = strongly agree	12	.88
BFI2-A	Is compassionate, has a soft heart	1 = strongly disagree…5 = strongly agree	12	.83
BFI2-C	Is dependable, steady	1 = strongly disagree…5 = strongly agree	12	.85
BFI2-N	Is relaxed, handles stress well	1 = strongly disagree…5 = strongly agree	12	.93
BFI2-O	Is complex, a deeper thinker	1 = strongly disagree…5 = strongly agree	12	.89
AJDI-General	What was your job in general like most of the time? Good	0 = no, 1 = cannot decide, 3 = yes	8	.89
AJDI-People	Do the following words/phrases describe the people you work with? Smart	0 = no, 1 = cannot decide, 3 = yes	6	.84
AJDI-Pay	…the pay you get now? Comfortable	0 = no, 1 = cannot decide, 3 = yes	6	.86
AJDI-Promotion	…the promotion opportunities you get now? Dead-end job	0 = no, 1 = cannot decide, 3 = yes	6	.90
AJDI-Work	…the work you do now? Exciting	0 = no, 1 = cannot decide, 3 = yes	6	.88
AJDI-Supervision	…the supervision that you get on your job. Tactful	0 = no, 1 = cannot decide, 3 = yes	6	.86
PANAS-P	Active, Determined, Attentive, Inspired, Alert	1 = not at all…5 = extremely	5	.84
PANAS-N	Afraid, Nervous, Upset, Hostile, Ashamed	1 = not at all…5 = extremely	5	.87
SVS	I feel energized right now	1 = not at all…5 = extremely	3	.85
Perceived Difficulty	How hard do you feel it is to respond to the previous survey questions?	1 = not difficult at all…5 = very difficult	1	NA
Preference	How much did you like to respond to the previous survey questions?	1 = not at all…5 = a lot	2	.62
Effort	How much effort do you put into responding to the previous survey?	1 = no effort at all…5 = a lot of effort	1	NA
Concentration	It was hard to keep my mind on the previous survey questions	1 = strongly disagree…5 = strongly agree	2	.63

Note: SWBS = Subjective Well-Being Scale; CSES = Core Self-Evaluation Scale; SH = subjective health; BFI = Big Five Inventory; E = extraversion; A = agreeableness; C = conscientiousness; N = neuroticism; O = openness; AJDI = Abridged Job Descriptive Index; PANAS = Positive and Negative Affect Schedule; SVS = Subjective Vitality Scale. We were not able to calculate Cronbach’s α for single item measures.

Response time

Qualtrics recorded response time for the whole test battery. The difference between subgroups (for the between-subject sample) or the difference for each person between his/her response times for the two tests (for the within-subject sample) can be used as a proxy to indicate response time differences between the FC and SS measures. This is because the only difference was the TAPAS format (FC or SS), and all other measures were exactly the same.

Results

Psychometric Equivalence

Reliability

Because it is inappropriate to calculate Cronbach’s α for the FC format and the SS format with intermediate items, we calculated item response theory model-based marginal reliability instead (Green, Bock, Humphreys, Linn, & Reckase, 1984). As reported in Table 4, there were two replicable patterns. First, the rank order of reliabilities across formats was similar in both samples (r = 78 and .79). Second, TAPAS_SS facet scores had relatively higher marginal reliability than TAPAS_FC facet scores in both samples. The average marginal reliability was .76 and .77 for TAPAS_FC in the two samples (ranging from .66 to .86), and .89 for TAPAS_SS in both samples (ranging from .79 to .94). Achievement and selflessness assessed by the FC format had reliability lower than .70 in both samples (Rel_AC = .66 and .69, Rel_SE = .67 and .67). The reliability of other facets was above .70. The same two patterns were observed for test-retest reliability: (a) the rank order of test-retest reliabilities of the 10 facets was similar across formats (r = .75); and (b) TAPAS_SS facet scores had relatively higher test-retest reliabilities than TAPAS_FC facet scores. The average was .77 (ranging from .69 to .83) for TAPAS_FC and .89 (ranging from .85 to .93) for TAPAS_SS. The achievement facet again showed the lowest reliability in the FC format (r = .69). Overall, TAPAS_FC showed marginal and test-retest reliability that were generally comparable to the TAPAS_SS.

Table 4.

Marginal Reliability and Test-Retest Reliability.

Facets	Marginal Reliability				Test-Retest
	Between-Subjects		Within-Subjects		Between-Subjects
	FC	SS	FC	SS	FC	SS
DO	.80	.94	.81	.94	.80	.90
SO	.77	.94	.78	.93	.83	.89
PH	.86	.92	.86	.94	.79	.92
SE	.67	.79	.67	.79	.73	.85
AC	.69	.86	.66	.87	.69	.87
OR	.84	.93	.85	.93	.83	.93
EV	.70	.91	.73	.89	.77	.88
OP	.80	.92	.82	.92	.77	.90
IE	.69	.83	.74	.82	.73	.86
TO	.75	.87	.77	.87	.77	.90
Rank order correlation	.78		.79		.75

Note. FC = forced choice; SS = single statement; DO = dominance; SO = sociability; PH = physical conditioning; SE = selflessness; AC = achievement; OR = order; EV = even tempered; OP = optimism; IE = intellectual efficiency; TO = tolerance.

In addition to overall reliability, we also examined conditional standard errors (SEs), which are inversely related to reliability. Specifically, we divided the trait continuum into three ranges ([–4, –1], [–1, 1], and [1, 4]) and computed the average SEs of trait estimates within each range. Results are shown in Table 5. Consistent with overall reliability, trait scores based on the SS format had slightly smaller SEs. Generally, individuals in the middle of the trait continuum (i.e., with scores between –1 and 1) were measured more reliably than those at the ends when the SS format was used. However, the FC format seemed to measure individuals equally well regardless of their standings on the trait continuum. In some cases, the FC format measured individuals at the ends even more reliably than the SS format.

Table 5.

Mean Standard Errors Conditioning on Different Theta Values.

Facets	Between-Subjects						Within-Subjects
	FC			SS			FC			SS
	[–4, –1]	[–1, 1]	[1, 4]	[–4, –1]	[–1, 1]	[1, 4]	[–4, –1]	[–1, 1]	[1, 4]	[–4, –1]	[–1, 1]	[1, 4]
DO	.30	.31	.33	.29	.20	.30	.30	.31	.33	.28	.20	.29
SO	.38	.38	.35	.31	.24	.35	.38	.37	.34	.32	.24	.34
PH	.36	.29	.35	.31	.20	.27	.35	.29	.35	.32	.20	.27
SE	.32	.34	.33	.49	.40	.38	.32	.34	.33	.49	.39	.39
AC	.34	.36	.33	.26	.33	.45	.35	.36	.33	.26	.33	.43
OR	.29	.29	.32	.29	.20	.38	.29	.29	.32	.28	.20	.37
EV	.32	.35	.32	.40	.27	.31	.32	.36	.32	.41	.27	.30
OP	.41	.37	.34	.36	.25	.27	.41	.37	.34	.36	.25	.27
IE	.36	.39	.38	.49	.35	.34	.35	.39	.37	.48	.35	.34
TO	.37	.35	.33	.44	.31	.28	.37	.35	.33	.43	.32	.28

Note: FC = forced choice; SS = single statement; DO = dominance; SO = sociability; PH = physical conditioning; SE = selflessness; AC = achievement motivation; OR = order; EV = even tempered; OP = optimism; IE = intellectual efficiency; TO = tolerance.

Convergent and discriminant validity

As shown in Table 6, the average correlation between scores on the same dimensions assessed using the different formats was .73 and ranged from .58 ∼ .78. A closer inspection revealed that achievement and selflessness had relatively low convergent validities (r = .58 and .65, respectively); those two facets also had lower reliabilities compared to the other personality dimensions. When we corrected the raw convergent validities for unreliability using Spearman’s formula (Spearman, 1904), the average convergent validity reached .89 (ranging from .77 ∼ .96). Overall, there appears to be strong evidence that the FC and SS versions are measuring the same underlying constructs.

Table 6.

Convergent and Discriminant Validity (Within-Subjects Sample).

Facets	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20
1. FC.DO											.87
2. FC.SO	.33											.93
3. FC.PH	.06	.20											.86
4. FC.SE	–.04	.07	.09											.89
5. FC.AC	.16	.03	.09	.28											.77
6. FC.OR	–.05	.05	.08	–.11	.25											.87
7. FC.EV	–.08	.21	.08	.22	.14	.00											.96
8. FC.OP	.20	.39	.30	.07	.21	.12	.36											.90
9. FC.IE	.31	.03	–.13	.13	.26	–.03	.09	.06											.87
10. FC.TO	.02	.10	.07	.43	.07	–.10	.10	.06	.12											.95
11. SS.DO	.76	.31	.09	.00	.23	–.04	.00	.26	.26	.08
12. SS.SO	.32	.79	.25	.13	.07	.04	.20	.42	.01	.17	.43
13. SS.PH	.12	.27	.77	.15	.18	.10	.22	.42	–.09	.13	.26	.36
14. SS.SE	.00	.15	.04	.65	.16	–.13	.21	.10	.04	.42	.08	.31	.12
15. SS.AC	.32	.11	.04	.30	.58	.13	.16	.30	.34	.16	.51	.32	.29	.40
16. SS.OR	–.07	.13	.18	–.07	.22	.77	.06	.26	–.05	–.08	.07	.18	.24	–.04	.29
17. SS.EV	–.08	.15	.05	.22	.13	.01	.77	.37	.10	.09	.03	.25	.19	.28	.30	.17
18. SS.OP	.16	.38	.27	.11	.21	.11	.33	.78	.05	.04	.33	.54	.47	.20	.44	.32	.44
19. SS.IE	.36	.11	–.03	.17	.30	–.02	.21	.24	.68	.22	.42	.28	.14	.29	.61	.11	.32	.31
20. SS.TO	.00	.12	.02	.42	.04	–.11	.16	.06	.15	.78	.07	.21	.09	.57	.34	–.03	.22	.09	.34

Note: FC = forced choice; SS = single statement; DO = dominance; SO = sociability; PH = physical conditioning; SE = selflessness; AC = achievement; OR = order; EV = even tempered; OP = optimism; IE = intellectual efficiency; TO = tolerance. Shaded correlations are uncorrected convergent validities. Correlations in the upper triangle are convergent validities corrected for unreliability.

The mean absolute correlations between different facets within TAPAS_FC and TAPAS_SS were .14 (|r| = .03 ∼ .39) and .27 (|r| = .03 ∼ .57), respectively. These two sets of discriminant correlations had very similar rank orders (r = .85). Similarly, the mean absolute interfacet correlations across formats was .16 (|r| = .00 ∼ .42). In sum, positive results for discriminant validity were found.

Correlations with external measure of the Big Five factors

In the next analysis, TAPAS facet scores were correlated with scores obtained from the BFI (between-subjects sample) and the BFI2 (within-subjects sample). Results are presented in Table 7. The shaded cells represent the convergent validities with external measures. Two patterns were apparent. First, both TAPAS_FC scores and TAPAS_SS scores showed moderate to high correlations with external measures of the same factors (M_FC.Between = .47, M_SS.Between = .56, M_FC.Within = .52, M_SS.Within = .64; the rank order correlations of the validity of the 10 facets across formats were .96 and .98 in the two samples), providing construct validity evidence for both TAPAS formats. Physical conditioning was an exception because it was only weakly correlated with extraversion scores in the between-subject sample. Second, TAPAS_FC facet scores showed slightly lower correlations with the single-statement BFI and BFI2 than TAPAS scores. There were also several moderate correlations among different personality domains that replicated across formats and samples. For example, optimism was moderately correlated with extraversion (r = .35 ∼ .62), agreeableness (r = .31 ∼ .44), and conscientiousness (r = .35 ∼ .44). Even tempered was also moderately correlated with agreeableness (r = .47 ∼ .57). In sum, these findings also support the psychometric equivalence of TAPAS_FC and TAPAS_SS.

Table 7.

Correlations With External Measures of the Big Five Factors.

Sample	Facet	E		A		C		N-		O
Sample	Facet	FC	SS	FC	SS	FC	SS	FC	SS	FC	SS
Between-subjects sample (BFI)	DO	.48	.49	–.06	.02	.12	.24	.16	.24	.18	.29
	SO	.72	.80	.28	.34	.18	.26	.35	.43	.10	.18
	PH	.17	.31	.03	.17	.27	.32	.16	.33	.04	.12
	SE	.01	.16	.38	.47	.08	.14	.01	.05	.25	.32
	AC	.09	.28	.15	.33	.47	.64	.21	.24	.20	.41
	OR	.07	.17	.10	.21	.58	.66	.21	.35	-.06	.00
	EV	–.02	.08	.47	.56	.25	.28	.51	.60	.07	.08
	OP	.35	.43	.31	.41	.42	.49	.62	.71	.01	.10
	IE	.07	.20	–.03	.16	.13	.34	.11	.21	.37	.47
	TO	.06	.11	.11	.25	–.06	.07	.08	.03	.37	.43
	Rank order correlation	.98		.97		.96		.96		.95
Within-subjects sample (BFI2)	DO	.55	.67	–.12	.03	.02	.16	.14	.27	.15	.27
	SO	.67	.80	.28	.39	.16	.24	.38	.47	.10	.23
	PH	.36	.48	.13	.22	.19	.31	.22	.41	.04	.10
	SE	.05	.20	.47	.58	.11	.14	.07	.14	.33	.35
	AC	.17	.45	.18	.43	.43	.54	.17	.37	.17	.46
	OR	.06	.23	.07	.19	.63	.75	.10	.30	-.02	.06
	EV	.09	.15	.48	.57	.21	.32	.54	.65	.16	.18
	OP	.49	.62	.31	.44	.35	.44	.69	.80	.09	.15
	IE	.15	.36	–.01	.26	.08	.25	.11	.34	.41	.58
	TO	.07	.13	.20	.35	–.10	.04	.03	.04	.43	.52
	Rank order correlation	.96		.95		.98		.95		.90

Note: FC = forced choice; SS = single statement; DO = dominance; SO = sociability; PH = physical conditioning; SE = selfishness; AC = achievement; OR = order; EV = even tempered; OP = optimism; IE = intellectual efficiency; TO = tolerance; E = extraversion; A = agreeableness; C = conscientiousness; N- = neuroticism (scored in the direction of emotional stability); O = openness. Shaded correlations are convergent validities.

Criterion-related validity

Correlations with criterion variables are shown in Table 8. Two similar patterns emerged. First, the criterion-related validities of TAPAS_FC and TAPAS_SS scores showed highly similar rank orders across criteria (the rank order correlations of the criterion-related validity of the 10 facets across formats ranged from .90 to .98 in the two samples). Second, TAPAS_SS scores had relatively higher criterion-related validity in most cases. These validity estimates were largely consistent with previous findings reviewed in the introduction, thus providing another piece of evidence supporting the psychometric equivalence of TAPAS_FC and TAPAS_SS. Next, we discuss each criterion separately.

Table 8.

Criterion-Related Validity.

Sample	Facet	SWB		CSE		SH		JS
Sample	Facet	FC	SS	FC	SS	FC	SS	FC	SS
Between-subjects sample	DO	.02	.16	.14	.25	.14	.15	—	—
	SO	.15	.25	.26	.40	.19	.22	—	—
	PH	.17	.30	.21	.32	.38	.49	—	—
	SE	.04	.09	–.01	.10	.02	.02	—	—
	AC	.12	.22	.27	.39	.14	.20	—	—
	OR	.14	.25	.25	.40	.16	.25	—	—
	EV	.21	.30	.31	.42	.12	.22	—	—
	OP	.58	.72	.74	.83	.32	.39	—	—
	IE	–.02	.08	.08	.26	.02	.11	—	—
	TO	–.07	.08	–.10	.06	.02	.09	—	—
	Rank order correlation	.98		.99		.97
Within-subjects sample	DO	.12	.21	.20	.35	.09	.17	.16	.29
	SO	.24	.31	.35	.49	.21	.28	.29	.37
	PH	.22	.33	.24	.45	.39	.48	.15	.31
	SE	.03	.11	.05	.13	–.03	.06	.10	.17
	AC	.09	.28	.24	.44	–.03	.16	.23	.37
	OR	.10	.24	.16	.35	–.01	.14	.06	.20
	EV	.17	.25	.32	.42	.16	.18	.26	.26
	OP	.64	.76	.73	.86	.40	.46	.44	.55
	IE	.01	.17	.14	.33	–.09	.08	.04	.22
	TO	–.04	.01	–.02	.03	–.03	.06	.06	.11
	Rank order correlation	.98		.97		.96		.90

Note: FC = forced choice; SS = single statement; DO = dominance; SO = sociability; PH = physical conditioning; SE = selflessness; AC = achievement; OR = order; EV = even tempered; OP = optimism; IE = intellectual efficiency; TO = tolerance; SWB = subjective well-being; CSE = core self-evaluation; SH = subjective health; JS = job satisfaction.

Subjective well-being

Neuroticism facets had the highest correlations with SWB (M_FC = .40, M_SS = .51), followed by extraversion facets (M_FC = .15, M_SS = .26), and conscientiousness facets (M_FC = .11, M_SS = .25). The agreeableness facet and openness facets were had very low correlation with SWB.

Core self-evaluation

Neuroticism facets again showed the highest correlations (M_FC = .53, M_SS = .63), followed by conscientiousness facets (M_FC = .23, M_SS = .40), and extraversion facets (M_FC = .23, M_SS = .38). The openness facet intellectual efficiency was also correlated with CSE (M_FC = .11, M_SS = .30). Selflessness and tolerance did not show substantial correlations with CSE.

Subjective health

Neuroticism facets (M_FC = .27, M_SS = .33) and extraversion facets (M_FC = .24, M_SS = .30) showed comparable correlations with SH. The rest of the facets had generally low correlations with SH.

Job satisfaction

We only report results for overall job satisfaction. Detailed results for each job satisfaction facet can be found in the online supplemental material. Neuroticism facets again had the highest correlation with JS (M_FC = .35, M_SS = .41), followed by the facets of extraversion (M_FC = .20, M_SS = .32) and achievement (M_FC = .23, M_SS = .37). Order, selflessness, and the two openness facets did not show consistent correlations with job satisfaction.

Respondent Reaction

As shown in Table 9, even though respondents found the FC format more difficult to answer than the SS format (d = .59) and showed a slight preference for the SS format (d = –.15), they generally did not show differential emotional or cognitive reactions. For example, respondents in both conditions had similar levels of positive affect, negative affect, and vitality before and after responding to TAPAS. They also devoted similarly high levels of effort and concentration.

Table 9.

Respondent Reactions.

Respondent reactions	FC		SS		r	t	p	Cohen’s d
Respondent reactions	M	SD	M	SD	r	t	p	Cohen’s d
Positive affect 1	3.15	0.87	3.15	0.92	.80	–0.41	.97	.00
Positive affect 2	3.08	0.92	3.04	0.99	.80	1.18	.24	.04
Negative affect 1	1.26	0.51	1.26	0.51	.75	0.14	.99	.00
Negative affect 2	1.20	0.46	1.19	0.44	.71	0.62	.54	.02
Vitality 1	3.65	0.72	3.60	0.79	.61	1.38	.17	.06
Vitality 2	3.60	0.83	3.61	0.98	.56	–0.18	.86	–.01
Perceived difficulty	2.50	1.11	1.85	1.07	.39	1.56	<.01	.59
Effort	4.38	0.70	4.43	0.67	.67	–1.68	.10	.07
Concentration	3.07	0.90	3.03	0.93	.60	0.99	.32	.04
Preference	3.87	0.71	3.99	0.78	.45	–2.88	<.01	–.15
Response time-between_raw	19.72	12.76	24.56	12.47	—	–7.30	<.01	–.37
Response time-within_raw	24.88	15.12	31.31	27.41	.24	–4.47	<.01	–.25
Response time-between_log	2.85	0.51	3.08	0.47	—	–9.47	<.01	–.48
Response time-within_log	3.08	0.52	3.29	0.50	.40	–8.26	<.01	–.46

Note: FC = forced choice; SS = single statement. It was not possible to calculate correlation between response time from two independent samples. Dashes are placed in these two cells. Raw = raw response time; log = log-transformed response time. Formula described in Dunlap, Cortina, Vaslow, and Burke (1996, S.171) was used to calculate effect sizes for dependent groups.

Because the distribution of raw response time was skewed (skewness ranged from 1.94 to 7.8 and kurtosis ranged from 5.61 to 82.79),¹ t test results based on both raw data and log-transformed data are reported. Whether transformed or not, both between-subject data and within-subject data indicated that responding to TAPAS_FC required significantly less time than responding to TAPAS_SS (ps < .001, d_raw = –.37 ∼ –.25, d_log = –.48 ∼ –.46).

Discussion

The present study used both between- and within-subjects designs to examine the equivalence of FC and SS versions of TAPAS. As the results illustrate, substantial evidence was found for equivalence of the two formats. Specifically, both formats demonstrated similarly rank-ordered marginal reliabilities and test-retest reliabilities, high convergent validities, good discriminant validities, and similar criterion-related validities with theoretically relevant criteria. In addition, the formats had little differential impact on respondents’ general emotional and cognitive reactions except that the FC format was perceived to be a bit more difficult and more time-saving.

Psychometric Equivalence Between Formats

Both marginal reliability and temporary stability were obtained for 10 facets of TAPAS administered in FC and SS formats. All 10 facets assessed via the FC format showed moderate to high levels of reliabilities. More importantly, FC marginal and test-retest reliabilities resembled those of their SS counterparts in terms of rank order despite being relatively lower. We see two possible explanations for FC’s lower reliability. First, scoring of each FC pair was dichotomous: one statement was selected as “most like me” and the other statement was not. SS scoring was polytomous in nature, which appears to offer more psychometric information. For example, respondents can differentiate whether they disagree or strongly disagree. It is therefore not surprising that the SS scale scores were more reliable. A second, more pernicious explanation is that responses to SS scales may be influenced by respondents’ focal trait level as well as other stable but irrelevant traits, such as response styles. Previous studies have found evidence that response styles are stable even across multiple years (e.g., Weijters, Geuens, & Schillewaert, 2010; Wetzel, Lüdtke, Zettler, & Böhnke, 2016). Such stable but irrelevant traits could artificially increase the reliability of SS scales. From this perspective, the seemingly higher reliability of the SS format would not necessarily mean that it is more precisely measuring the intended construct. More studies are needed to test these two explanations.

It is also interesting to see that the FC format provided roughly equal measurement precision across the entire trait continuum. Moreover, the FC format provided slightly more reliable measurement at the two ends of the trait continuum for some traits. This is relevant for personnel selection as it is often those who are high/low on certain traits that are selected/deselected. Accurate measurement of those individuals can facilitate effective decision making. A post hoc explanation is that the test information (inversely related to SE) of the FC format is more spread out than for the SS format due to the difference in the assumed response process. More future studies are needed to examine the generalizability of this finding.

Scores on the same facet measured by different formats were highly correlated. Compared to the FC format, the SS format also consistently demonstrated similar rank-orders but slightly higher correlations with external measures of the Big Five factors, which may be a result of common method bias (Podsakoff et al., 2003). Our findings provide evidence for the construct validity of both formats. A reviewer noted that some facets (e.g., sociability, even tempered) seemed to be slightly more equivalent than other facets (e.g., selflessness, achievement). We suspect that this may be partially due to reliability differences because scores of the less consistent facets had relatively lower reliabilities. Indeed, the discrepancies shrank after correcting for unreliability. However, we still observed some differential equivalence. We speculate that trait desirability may also play a role. Specifically, perhaps more neutral traits (e.g., sociability) can be measured by both formats equally well, thus displaying higher equivalence across formats. More desirable traits (e.g., selflessness) may have more construct irrelevant variance in the SS format (i.e., socially desirable responding), which leads to lower equivalence across formats. However, our current study design and data are insufficient to examine this explanation. Future researchers are encouraged to investigate it by manipulating assessment contexts as trait desirability may change in different contexts (Krumpal, 2013).

Apart from the high convergent validity, both formats also displayed good discriminant validity (i.e., correlations between facets were generally low). It is particularly interesting to see that the FC format had greater discriminant validity than the SS format (M_FC = .14, M_SS = .27), given that the Big Five factors were originally theorized to be orthogonal (Goldberg, 1990; Saucier, 2002). We see this finding as consistent with one of the purported advantages of the FC format: reducing the influence of response styles (Brown & Maydeu-Olivares, 2011). As discussed above, response styles introduce domain-general but irrelevant variance, and thus can artificially increase interfacet correlations. Importantly, the FC format can reduce the impact of response styles.

Although we observed a few high cross-domain correlations (e.g., optimism and extraversion), they were not a big concern for the present study because they were also well-replicated across formats and samples, which again supported the equivalence between the FC and SS formats. In fact, these high cross-domain correlations were consistent with previous findings (e.g., Duijsens & Diekstra, 1995; Milligan, 2003; Sharpe, Martin, & Roth, 2011). Both formats also showed similar correlations with relevant outcomes. They are largely in line with previous studies (e.g., Judge et al., 2002; Judge et al., 2003; Steel et al., 2008; Stephan et al., 2012).

Respondent Reaction Comparability

It is not surprising that the FC measure was considered harder to complete than the SS measure. This finding is in line with what was found with think-aloud techniques (Sass et al., 2018). Despite the perceived difficulty, respondents reported the same degree of concentration and an equal (and high) level of effort when answering both formats. The two formats also elicited almost identical levels of affective states and subjective vitality after their administrations. Although they reported a statistically significant preference for the SS format, the effect size was small (d = –.15). A surprising finding is that respondents spent less time answering the FC measures than the SS measures, which is in direct contrast with previous findings showing that FC measurement is more time consuming (Bowen, Martin, & Hunt, 2002). One potential explanation is that previous studies asked respondents to choose “the most like me” and “the least like me” statements from a tetrad. To respond to such a tetrad, three pairwise comparisons per block are needed. However, in our design, only one pairwise comparison is involved in each FC block (pair). Therefore, it is expected that a tetrad design is more time-consuming than a two-alternative design. Such a design difference may also explain why previous studies found more negative reactions to FC while we did not. More pairwise comparisons per block would induce more cognitive load, which might result in negative reactions (Fraser, Ma, Teteris, Baxter, Wright, & McLaughlin, 2012).

Limitations and Future Directions

There are several limitations to our studies that need to be addressed in future work. First, we included only low-stakes samples and do not know the generalizability of current results to high-stakes contexts where faking may occur. Faking has been found to result in score inflation (Viswesvaran & Ones, 1999) for SS formats and may even impair the construct validity of SS scales (Schmit & Ryan, 1993). As one of the main feature FC is its faking-resistance potential (Cao & Drasgow, in press), it would be a great contribution if we could empirically show that FC scales can prevent score inflation and retain their construct validity in high-stakes contexts. Future research should ask participants to answer FC scales in both high- and low-stakes contexts to compare their psychometric equivalence under these conditions. However, our focus was on the equivalence of the FC and SS formats. This is the most fundamental question that needs to be explicitly answered before researchers and practitioners embrace the FC format. If these formats are not equivalent even in low-stakes situations, it may be pointless to explore the performance of FC in high-stakes situations because researchers cannot be confident that they are measuring the same thing. Fortunately, we found strong evidence supporting equivalence across the two formats. Second, the test-retest intervals used in our studies were relatively short. It would be informative to administer FC to the same people at multiple times across longer intervals. In this way, we would be able to see how the test-retest reliability of FC measures changes across time and compare the change trajectory to that of SS measures. The third limitation is that total response time was used instead of response time for TAPAS. Total response time may introduce more noise and affect the validity of the current conclusion. Readers should take caution when interpreting this result. Fourth, the current study exclusively relied on self-report data, which limits the generalizability of our findings. It is not known whether we would observe the same degree of equivalence if the FC and SS measures were used for third-party ratings. Future researchers are encouraged to explore this direction. Fifth, the external measures used in the present study did not differentiate between facets and the Big Five factors. Again, research is needed with more diverse criteria.

Conclusion

Based on both between-subjects and within-subjects samples with a total of approximately 2,000 participants, we demonstrated that the FC and SS formats are psychometrically equivalent in terms of reliability, convergent validity, discriminant validity, and criterion-related validity. In addition, respondents did not report substantially negative reactions to the FC format. Therefore, we encourage future research examining the performance of FC scales in high-stakes contexts.

Supplemental Material

Supplemental Material, ORM-18-0089_Supplementary_materials - Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Supplemental Material, ORM-18-0089_Supplementary_materials for Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures by Bo Zhang, Tianjun Sun, Fritz Drasgow, Oleksandr S. Chernyshenko, Christopher D. Nye, Stephen Stark and Leonard A. White in Organizational Research Methods

Footnotes

Authors’ Note

Part of the results was presented on April 20, 2018, at the 33rd Annual Conference of the Society for Industrial and Organizational Psychology.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Bo Zhang

Supplemental Material

Supplemental material for this article is available online.

Note

References

Ackerman

R. A.

Donnellan

M. B.

Roberts

B. W.

Fraley

R. C.

(2016). The effect of response format on the psychometric properties of the Narcissistic Personality Inventory: Consequences for item meaning and factor structure. Assessment, 23(2), 203–220.

Anguiano-Carrasco

MacCann

Geiger

Seybert

J. M.

Roberts

R. D.

(2015). Development of a forced-choice measure of typical-performance emotional intelligence. Journal of Psychoeducational Assessment, 33(1), 83–97.

Bartram

(2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15(3), 263–272.

Bartram

Brown

(2004). Online testing: Mode of administration and the stability of OPQ 32i scores. International Journal of Selection and Assessment, 12(3), 278–284.

Bostic

T. J.

Rubio

D. M.

Hood

(2000). A validation of the subjective vitality scale using structural equation modeling. Social Indicators Research, 52(3), 313–324.

Bowen

C. C.

Martin

B. A.

Hunt

S. T.

(2002). A comparison of ipsative and normative approaches for ability to control faking in personality questionnaires. International Journal of Organizational Analysis, 10(3), 240–259.

Brown

(2016a). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1), 135–160.

Brown

(2016b). Thurstonian scaling of compositional questionnaire data. Multivariate Behavioral Research, 51(2-3), 345–356.

Brown

Inceoglu

Lin

(2017). Preventing rater biases in 360-degree feedback by forcing choice. Organizational Research Methods, 20(1), 121–148.

10.

Brown

Maydeu-Olivares

(2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502.

11.

Brown

Maydeu-Olivares

(2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52.

12.

Brown

Maydeu-Olivares

(2017). Ordinal factor analysis of graded-preference questionnaire data. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 516–529.

13.

Bürkner

P. C.

Schulte

Holling

(in press). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement. Advanced online publication. doi:0013164419832063

14.

Cao

Drasgow

(in press). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology. doi:10.1037/apl0000414

15.

Cao

Drasgow

Cho

(2015). Developing ideal intermediate personality items for the ideal point model. Organizational Research Methods, 18(2), 252–275.

16.

Cao

Song

Q. C.

Tay

(2018). Detecting curvilinear relationships: A comparison of scoring approaches based on different item response models. International Journal of Testing, 18(2), 178–205.

17.

Carter

N. T.

Dalal

D. K.

(2010). An ideal point account of the JDI Work satisfaction scale. Personality and Individual Differences, 49(7), 743–748.

18.

Carter

N. T.

Dalal

D. K.

Guan

LoPilato

A. C.

Withrow

S. A.

(2017). Item response theory scoring and the detection of curvilinear relationships. Psychological Methods, 22(1), 191–203.

19.

Chernyshenko

O. S.

Stark

Drasgow

Roberts

B. W.

(2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19(1), 88–106.

20.

Chernyshenko

O. S.

Stark

Prewett

M. S.

Gray

A. A.

Stilson

F. R.

Tuttle

M. D.

(2009). Normative scoring of multidimensional pairwise preference personality scales using IRT: Empirical comparisons with other formats. Human Performance, 22(2), 105–127.

21.

Cho

Drasgow

Cao

(2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27(4), 1241–1252.

22.

Diener

E. D.

Emmons

R. A.

Larsen

R. J.

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75.

23.

Donovan

J. J.

Dwight

S. A.

Schneider

(2014). The impact of applicant faking on selection measures, hiring decisions, and employee performance. Journal of Business and Psychology, 29(3), 479–493.

24.

Drasgow

Chernyshenko

O. S.

Stark

(2010). 75 years after Likert: Thurstone was right! Industrial and Organizational Psychology, 3(4), 465–476.

25.

Drasgow

Stark

Chernyshenko

O. S.

Nye

C. D.

Hulin

C. L.

White

L. A.

(2012). Development of the Tailored Adaptive Personality Assessment System (TAPAS) to support army personnel selection and classification decisions. Urbana, IL: Drasgow Consulting Group.

26.

Dueber

D. M.

Love

A. M.

Toland

M. D.

Turner

T. A.

(2019). Comparison of single-response format and forced-choice format instruments using Thurstonian item response theory. Educational and Psychological Measurement, 79, 108–128.

27.

Duijsens

I. J.

Diekstra

R. W.

(1995). The 23bb5: a new bipolar big five questionnaire. Personality and Individual Differences, 19(5), 753–755.

28.

Dunlap

W. P.

Cortina

J. M.

Vaslow

J. B.

Burke

M. J.

(1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1(2), 170–177.

29.

Fraser

Teteris

Baxter

Wright

McLaughlin

(2012). Emotion, cognitive load and learning outcomes during simulation training. Medical Education, 46(11), 1055–1062.

30.

Goldberg

L. R.

(1990). An alternative “description of personality”: The big-five factor structure. Journal of Personality and Social Psychology, 59(6), 1216–1229.

31.

Green

B. F.

Bock

R. D.

Humphreys

L. G.

Linn

R. L.

Reckase

M. D.

(1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21(4), 347–360.

32.

Guenole

Brown

A. A.

Cooper

A. J.

(2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513–526.

33.

Hausknecht

J. P.

Day

D. V.

Thomas

S. C.

(2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57(3), 639–683.

34.

Hicks

L. E.

(1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167–184.

35.

Joanes

D. N.

Gill

C. A.

(1998). Comparing measures of sample skewness and kurtosis. Journal of the Royal Statistical Society: Series D (The Statistician), 47(1), 183–189.

36.

John

O. P.

Srivastava

(1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In Pervin

L. A.

John

O. P.

(Eds.), Handbook of personality: Theory and research (pp. 102–138). New York, NY: Guilford.

37.

Judge

T. A.

Erez

Bono

J. E.

Thoresen

C. J.

(2003). The Core Self-Evaluations Scale: Development of a measure. Personnel Psychology, 56(2), 303–331.

38.

Judge

T. A.

Heller

Mount

M. K.

(2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87(3), 530–541.

39.

Judge

T. A.

Van Vianen

A. E.

De Pater

I. E.

(2004). Emotional stability, core self-evaluations, and job outcomes: A review of the evidence and an agenda for future research. Human Performance, 17(3), 325–346.

40.

Krumpal

(2013). Determinants of social desirability bias in sensitive surveys: A literature review. Quality & Quantity, 47(4), 2025–2047.

41.

Landy

F. J.

Farr

J. L.

(1980). Performance rating. Psychological Bulletin, 87(1), 72–107.

42.

Lee

Joo

S. H.

Lee

(2019). Examining stability of personality profile solutions between Likert-type and multidimensional forced choice measure. Personality and Individual Differences, 142, 13–20.

43.

Lee

Joo

S. H.

Stark

Chernyshenko

O. S.

(2018). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement. Advance online publication.

44.

Lee

Stark

(2018). Examining validity evidence for multidimensional forced choice measures with different scoring approaches. Personality and Individual Differences, 123, 229–235.

45.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–55.

46.

Löckenhoff

C. E.

Sutin

A. R.

Ferrucci

Costa

P. T.

Jr (2008). Personality traits and subjective health in the later years: The association between NEO-PI-R and SF-36 in advanced age is influenced by health status. Journal of Research in Personality, 42(5), 1334–1346.

47.

McCloy

R. A.

Heggestad

E. D.

Reeve

C. L.

(2005). A silk purse from the sow’s ear: Retrieving normative information from multidimensional forced-choice items. Organizational Research Methods, 8(2), 222–248.

48.

Meglino

B. M.

Ravlin

E. C.

(1998). Individual values in organizations: Concepts, controversies, and research. Journal of Management, 24(3), 351–389.

49.

Milligan

(2003). Optimism and the five-factor model of personality, coping, and health behavior. Dissertation Abstracts International, 64(11-B), 5830. (UMI 3112405)

50.

Nye

C. D.

Drasgow

Chernyshenko

O. S.

Stark

Kubisiak

U. C.

White

L. A.

Jose

(2012), Assessing the Tailored Adaptive Personality Assessment System (TAPAS) as an MOS qualification instrument (Tech. Rep. No. 1312). Ft. Belvoir, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

51.

Pavlov

Maydeu-Olivares

Fairchild

A. J.

(2018). Effects of applicant faking on forced-choice and Likert scores. Organizational Research Methods. Advance online publication.

52.

Pinquart

(2001). Correlates of subjective health in older adults: A meta-analysis. Psychology and Aging, 16(3), 414–426.

53.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J. Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.

54.

Revelle

(2016). How to: Use the psych package for factor analysis and data reduction. Evanston, IL: Northwestern University, Department of Psychology.

55.

Roberts

J. S.

Laughlin

J. E.

(1996). A unidimensional item response model for unfolding responses from a graded disagree-agree response scale. Applied Psychological Measurement, 20(3), 231–255.

56.

Roberts

J. S.

Laughlin

J. E.

Wedell

D. H.

(1999). Validity issues in the Likert and Thurstone approaches to attitude measurement. Educational and Psychological Measurement, 59(2), 211–233.

57.

Rundquist

E. A.

(1946, September). The forced-choice technique and rating scales. Paper presented at the American Psychological Association, Philadelphia.

58.

Sass

Frick

Reips

U. D.

Wetzel

(2018). Taking the test taker’s perspective: Response process and test motivation in multidimensional forced-choice versus rating scale instruments. Assessment. Advance online publication.

59.

Saucier

(2002). Orthogonal markers for orthogonal factors: The case of the Big Five. Journal of Research in Personality, 36(1), 1–31.

60.

Schmit

M. J.

Ryan

A. M.

(1993). The big five in personnel selection: Factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78(6), 966–974.

61.

Seybert

(2013). A new item response theory model for estimating person ability and item parameters for multidimensional rank order responses (Unpublished doctoral dissertation). University of South Florida, Tampa.

62.

Sharpe

J. P.

Martin

N. R.

Roth

K. A.

(2011). Optimism and the big five factors of personality: Beyond neuroticism and extraversion. Personality and Individual Differences, 51(8), 946–951.

63.

Smither

J. W.

Reilly

R. R.

Millsap

R. E.

Pearlman

Stoffey

R. W.

(1993). Applicant reactions to selection procedures. Personnel Psychology, 46(1), 49–76.

64.

Soto

C. J.

John

O. P.

(2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117–143.

65.

Spearman

(1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101.

66.

Stanton

J. M.

Sinar

E. F.

Balzer

W. K.

Julian

A. L.

Thoresen

Aziz

… Smith

P. C.

(2002). Development of a compact measure of job satisfaction: The abridged Job Descriptive Index. Educational and Psychological Measurement, 62(1), 173–191.

67.

Stark

Chernyshenko

O. S.

Drasgow

(2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203.

68.

Stark

Chernyshenko

O. S.

Drasgow

Nye

C. D.

White

L. A.

Heffner

Farmer

W. L.

(2014). From ABLE to TAPAS: A new generation of personality tests to support military selection and classification decisions. Military Psychology, 26(3), 153–164.

69.

Stark

Chernyshenko

O. S.

Drasgow

White

L. A.

(2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15(3), 463–487.

70.

Stark

Chernyshenko

O. S.

Drasgow

Williams

B. A.

(2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology, 91(1), 25–39.

71.

Steel

Schmidt

Shultz

(2008). Refining the relationship between personality and subjective well-being. Psychological Bulletin, 134(1), 138–161.

72.

Stephan

Demulier

Terracciano

(2012). Personality, self-rated health, and subjective age in a life-span sample: The moderating role of chronological age. Psychology and Aging, 27(4), 875–880.

73.

Sun

(2017). When matches are ideal: Fitting measurement models to adult attachment data (Unpublished master’s thesis). University of Illinois at Urbana-Champaign.

74.

Tay

Drasgow

Rounds

Williams

B. A.

(2009). Fitting measurement models to vocational interest data: Are dominance models ideal? Journal of Applied Psychology, 94(5), 1287–1304.

75.

Tenopyr

M. L.

(1988). Artifactual reliability of forced-choice scales. Journal of Applied Psychology, 73(4), 749–751.

76.

Thompson

E. R.

(2007). Development and validation of an internationally reliable short-form of the Positive and Negative Affect Schedule (PANAS). Journal of Cross-Cultural Psychology, 38(2), 227–242.

77.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 79, 281–299.

78.

Tourangeau

Rasinski

K. A.

(1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299–314.

79.

Usami

Sakamoto

Naito

Abe

(2016). Developing pairwise preference-based personality test and experimental investigation of its resistance to faking effect by item response model. International Journal of Testing, 16(4), 288–309.

80.

Viswesvaran

Ones

D. S.

(1999). Meta-analyses of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59(2), 197–210.

81.

Weijters

Geuens

Schillewaert

(2010). The stability of individual response styles. Psychological Methods, 15(1), 96–110.

82.

Wetzel

Böhnke

J. R.

Brown

(2016). Response biases. In Leong

F. T. L.

Bartram

Cheung

Geisinger

K. F.

Iliescu

(Eds.), The ITC international handbook of testing and assessment (pp. 349–363). New York, NY: Oxford University Press.

83.

Wetzel

Lüdtke

Zettler

Böhnke

J. R.

(2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23(3), 279–291.

84.

Wetzel

Roberts

B. W.

Fraley

R. C.

Brown

(2016). Equivalence of Narcissistic Personality Inventory constructs and correlates across scoring approaches and response formats. Journal of Research in Personality, 61, 87–98.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB