A Psychometric Model for Discrete-Option Multiple-Choice Items

Abstract

Discrete-option multiple-choice (DOMC) items differ from traditional multiple-choice (MC) items in the sequential administration of response options (up to display of the correct option). DOMC can be appealing in computer-based test administrations due to its protection of item security and its potential to reduce testwiseness effects. A psychometric model for DOMC items that attends to the random positioning of key location across different administrations of the same item is proposed, a feature that has been shown to affect DOMC item difficulty. Using two empirical data sets having items administered in both DOMC and MC formats, the variability in key location effects across both items and persons is considered. The proposed model exploits the capacity of the DOMC format to isolate both (a) distinct sources of item difficulty (i.e., related to the identification of keyed responses versus the ruling out of distractor options) and (b) distinct person proficiencies related to the same two components. Practical implications in terms of the randomized process applied to schedule item key location in DOMC test administrations are considered.

Keywords

discrete-option multiple-choice items multiple-choice items item response theory

The multiple-choice (MC) item has been a mainstay of educational measurement despite its known limitations. The transition to computer-based measurement has made it possible to address some of these limitations through alterations of the MC format. One potentially appealing alternative is the discrete-option multiple-choice (DOMC) item (Foster & Miller, 2009). Unlike traditional MC items, where the item stem and response options are presented simultaneously, under the DOMC format, examinees are shown the stem and then are presented and respond to response options one at a time. An incorrect response to a DOMC item occurs if the examinee either (a) selects a presented distractor option or (b) fails to select a presented keyed response, regardless of how many overall options the examinee is administered. A correct response to a DOMC item only occurs when the examinee avoids selecting any presented distractor option and ultimately selects the keyed response(s). (Commonly a subsequent additional response option will be administered with a certain specified probability so that the respondent will not be aware whether the final given response was correct or incorrect.) Important to the current application, the sequence in which DOMC response options are presented is randomly determined and for a given item will vary from one administration to the next.

Although the DOMC format is related to other format types (e.g., multiple-correct MC items, true-false items), its differences from the traditional MC format seems worthy of psychometric attention. Empirical results suggest that DOMC items tend to be more difficult than MC items (Eckerly, Smith, & Sowles, 2018). There are at least a couple of reasons that have been given for why such psychometric changes may reflect increases in validity. First, by having examinees evaluate one response option at a time, the DOMC format is believed to reduce the influence of testwiseness on test performance (Kingston, Tiemann, Miller, & Foster, 2012; Papenburg, Willing, & Musch, 2017; Willing, Ostapczuk, & Musch, 2015). Specifically, respondents are less able to arrive at a correct answer only through ruling out distractor response options, as they also need to recognize the keyed option when presented. Second, by varying the presentation of the item across examinees and reducing the exposure of distractor options, the format also helps protect item security and thus potentially reduces concerns over test compromise.

The goal of this article is to develop a psychometric model that can be used to better understand measurement differences under the DOMC format. One of the unique psychometric advantages of DOMC is the opportunity to evaluate the distractibility of incorrect response options independent of the correct (keyed) responses. In traditional MC items, only the relative attractiveness of distractors against the correct response can be determined. Thus, it is always challenging to determine whether a difficult MC item is difficult because of attractive distractors, a low attractiveness of the correct response, or some combination. This feature is also reflected in item response models typically applied to MC items (specifically, “divide-by-total” models, see Thissen & Steinberg, 1986). By contrast, under DOMC, it becomes theoretically possible to distinguish conditions where an item is difficult due to attractive distractors as opposed to an inability to identify the correct response.

At the same time, there are aspects of the DOMC format that need further exploration. Eckerly et al. (2018) highlight the role that key location can play in defining the difficulty of DOMC items. Naturally, when the keyed response occurs later in the sequence of scheduled options, more work is required of the examinee in achieving a correct response. Such differences may raise equity concerns to the extent that pure randomization of key location across items is not successful in balancing average key location across administrations. In addition, it is unclear how the nature of the proficiencies measured by MC and DOMC items may be altered by the change in format. In this paper, the proposed model is examined in application to two empirical data sets. Each data set permits the examination of the relationship between the same items administered under both DOMC and MC formats. The first data set comes from an information technology (IT) certification test currently administered using the DOMC format for which data from a separate population are available under the MC format. The second data set was collected from online respondents to a test of Harry Potter trivia in which each examinee was administered some subset of the items under the DOMC format and other items under the MC format.

Psychometric Model for DOMC Items

As noted above, a previously highlighted feature of DOMC items concerns the anticipated effect of key location. For DOMC items with just one keyed response, key location refers to the position in the sequence of presented options where the keyed option is scheduled to appear, irrespective of whether the key is ever actually administered to the candidate. For an item with four response options, this value could be 1, 2, 3, or 4. As one of the empirical data sets also contains items with multiple keyed responses (where all keys must be selected as correct by an examinee to obtain a correct response for the item), the last key location was more generally defined as the location in which the last of the keyed options is scheduled to appear. Although it has been observed that difficulty of the DOMC item generally increases as the last key location becomes later, it is conceivable that the effect of this feature may vary both across items and examinees. For example, a reduced effect of key location might be expected for items in which the distractors are not attractive. Examinees may also demonstrate a similar source of variability with respect to the key location effect. It might be anticipated that the ability to rule out distractor options could at some level be distinct from the ability to select a keyed response, in which case, a key location effect may show variability across examinees.

To account for each of these potential sources of variability, a psychometric model that is sensitive to both item and examinee variability in key location effects was considered. The model can be viewed as implying that each presented response option in a DOMC item is another “step” in a sequence of steps that must be followed to arrive at a correct response. The number of such steps is determined by the scheduled location of the last keyed response, denoted as $lo c_{ij}$ , which is randomly determined. To this end, responses were modeled to a DOMC item as follows:

P (U_{ij} = 1 | θ_{i}, κ_{i}) = {[φ_{j} (θ_{i})]}^{1 + [κ_{i} \times s_{j} \times (lo c_{ij} - 1)]},

(1)

where $U_{ij} = 1$ implies examinee i answers item j correctly; $φ_{j} (θ_{i})$ defines the probability of successfully executing a single item step (i.e., correctly identifying a correct response when presented as the first presented option) and is written as follows:

φ_{j} (θ_{i}) = \frac{\exp (a_{j} θ_{i} + b_{j})}{1 + \exp (a_{j} θ_{i} + b_{j})},

where $a_{j}$ and $b_{j}$ are item slope and intercept parameters, respectively; $θ_{i}$ represents examinee proficiency on the skill of interest ( $a_{j}$ estimates are positive for keyed options); $κ_{i}$ is an examinee parameter defining the effect of key location across items for the examinee (0 < $κ_{i}$ <∞); larger $κ_{i}$ implies a stronger effect of key location; $s_{j}$ is an item parameter defining the effect of key location for the item across examinees (0 < $s_{j}$ < ∞); larger $s_{j}$ implies a stronger effect of key location; and $lo c_{ij}$ is the (last) key location for item j when presented to examinee i (1 ≤ $lo c_{ij}$ ≤ 5).

The model in Equation 1 will be referred to as the key location (KL) model for DOMC items. Special cases of the KL model can be considered where $κ_{i}$ = 1 for all i (key location effect is constant across examinees), $s_{j}$ = 1 for all j (key location effect is constant across items), or both $κ_{i}$ and $s_{j}$ are collectively replaced by a constant ( $c_{\exp} = κ_{i} \times s_{j}$ ) for all examinees and all items (implying a constant key location effect across items and examinees). Note that the $a_{j}$ and $b_{j}$ parameters have traditional interpretations as slope and threshold parameters, although technically define the discrimination and easiness of the item when the keyed option is administered as the first response option. The actual discrimination and difficulty level of an item will also be affected by $lo c_{ij}$ (and by the accompanying $κ_{i}$ and $s_{j}$ parameters).

The use of exponent parameters to account for the effect of key location emphasizes the fact that a correct score on a DOMC item can be viewed as the outcome of a sequence of conjunctively interacting processes, specifically, a sequence of correct responses (i.e., correct rejections of distractor options, and correct selections of keyed options) up through the last keyed response. Prior psychometric models (see, for example, Samejima, 2000) have similarly accounted for conjunctively interacting subprocesses through an exponent parameter. Examining key location effects may also be informative in regard to how students solve MC items. Items with smaller $s_{j}$ are items for which key location has a weaker effect and thus less influential distractors. By contrast, an item with a larger $s_{j}$ has more potent distractors. As noted, one clear advantage of the DOMC format is the potential to evaluate the distracting power of distractors in a way that is independent of the presence of the correct response option. This feature is manifested in the KL model by separate parameters related to correct-option and distractor-related difficulty ( $b_{j}$ and $s_{j}$ ). On the person side, we can think of $κ_{i}$ as reflecting an examinee’s ability in not selecting distractor options. Examinees with smaller $κ_{i}$ are examinees for which key location has little effect, indicating a high proficiency in ruling out distractor options, a proficiency that may be distinct from $θ$ which is related to the correct selection of keyed responses. Even if distinct, these proficiencies are likely related. In particular, because higher $κ_{i}$ generally corresponds with reduced performance, $κ_{i}$ was expected to correlate negatively with $θ_{i}$ .

Interestingly, while both $θ_{i}$ and $κ_{i}$ can be estimated as distinct proficiencies under DOMC, both are also relevant to how examinees perform on MC items. As examinees can correctly respond to an MC item either (a) by identifying the correct response or (b) by ruling out all distractors, we might view these skills as interacting disjunctively under an MC item, but conjunctively under DOMC. The difference under MC is that such proficiencies are not distinguishable, as only an overall response to the item is observed. It is anticipated that the distinction between $θ_{i}$ and $κ_{i}$ may also play a role in understanding differences in the underlying proficiencies measured by MC versus DOMC items.

Model Estimation

The authors focus here on the most general KL model, as the special cases can also be estimated within the same general framework. The estimation approach is fully Bayesian, with priors specified so as to allow the data to strongly inform both examinee and item parameters. The general model includes two examinee parameters ( $θ, κ$ ) and three item parameters (a, b, s). Item responses are modeled as a function of these person and item parameters, along with information regarding where the key was located ( $lo c_{ij}$ ) when item j was administered to examinee i. For the item parameters, it was assumed that $a ~ LogNormal (0, 1)$ , $b ~ Normal (0, 1)$ , and $s ~ Gamma (5, 5)$ . For the examinee parameters, it was assumed that $θ ~ Normal (0, 1)$ and $κ ~ Gamma (5, 5)$ . This choice of priors was informed by earlier analyses involving related models (i.e., Patz & Junker, 1999; Sinharay, 2004 Bolfarine & Bazán, 2010). To the extent that the primary goal is to illustrate both between-person and between-item heterogeneity in key location effects, a critical element of these priors is that they are assumed the same across all items and across all examinees; consequently, differences seen in the estimates are differences that can be more confidently assumed to be differences determined by the data.

The authors implement an MCMC algorithm for the model using WINBUGS 1.4. The software implements a Metropolis-Hastings sampling algorithm in which 4,000 adaptive iterations are used to tune proposal distributions, and a subsequent 6,000 iterations are used to determine a multivariate posterior distribution of the model parameters. The authors apply the Gelman–Rubin criterion using five simulated chains to evaluate chain convergence. In all cases, the posterior means of the model parameters and their associated posterior standard deviations define the estimates and their precisions.

Real Data Illustrations

Two real test data sets are considered involving DOMC items. Importantly, each of these data sets involves the same items administered in both DOMC and MC formats.

IT Certification Exam

The first data set comes from an operational IT certification test. Item response data for two 59-item forms administered under the DOMC format are considered. Due to an overlap of items across forms, a total of 83 unique DOMC items were analyzed. Most (54) of the items are single-keyed items, 24 had two keyed responses, and five had three keyed responses. All items with two or fewer keyed responses had four response options; the items with three keyed responses each had five response options. As a result, the latest key location varies from one to four for single-keyed items, from two to four for two-keyed items, and from three to five for three-keyed items. In estimating the KL model, a concurrent calibration strategy was applied, with a total of 648 examinees providing item response data. Thirty-five of the 83 items had responses from all 648 examinees; the remaining 48 items were administered on just one of the two forms and thus have response data from approximately 324 examinees each.

In fitting the KL model in Equation 1, the authors simulated five chains out to 10,000 iterations. The Gelman–Rubin R² was effectively 1 for each of the three item parameters (a, b, and s) across all items, suggesting successful convergence. An overall comparison of the general model against the three restricted models—that is, the constant exponent model ( $κ_{i}$ = 1 for all i; $s_{j}$ = c for all j), the variable item only model ( $κ_{i}$ = 1 for all i), and the variable examinee only model ( $s_{j}$ = 1 for all j)—was performed using the Deviance Information Criterion (DIC; Spiegelhalter, Best, Carlin, & van der Linde, 2002). As seen in Table 1, the DIC criterion indicated the general KL model as the preferred model, implying detectable variability in the key location effect both across items and persons. Somewhat stronger evidence emerges for variability in key location effects across items than for persons.

Table 1.

Deviance Information Criterion Results for General Model and Comparison Models, IT Certification DOMC data.

Model	$\bar{D}$	$\hat{D}$	p_D	DIC
KL: Constant exponent	41,384.7	40,666.1	718.55	42,103.2
KL: $s_{j}$ only	40,976.9	40,211.5	765.47	41,742.4
KL: $κ_{i}$ only	41,076.6	40,145.1	931.48	42,008.0
KL: Both $s_{j}$ and $κ_{i}$	40,691.1	39,743.3	947.79	41,638.9

Note. Value in bold identifies lowest DIC. IT = information technology; DOMC = discrete-option multiple-choice; KL = key location; DIC = Deviance Information Criterion.

IT: Interpreting Item Parameter Estimates

Supplemental Appendix A reports the DOMC item parameter estimates for the 83 DOMC items from the IT certification test. Table 2 shows the corresponding descriptive statistics of the item parameter estimates. It is apparent that all three parameters vary substantially and capture unique aspects of the items; the $s$ estimates correlate with the a and b estimates at .233 (p = .034) and .630 (p < .001), respectively, while the a and b estimates correlate at .735 (p < .001).

Table 2.

Descriptive Statistics, DOMC Item Parameter Estimates, IT Certification Test Data (N = 83 Items).

	Minimum	Maximum	M	SD
$\hat{a}$	.09	1.29	.59	.31
$\hat{b}$	–.75	2.07	.59	.64
$\hat{s}$	.26	2.63	.92	.47

Note. DOMC = discrete-option multiple–choice; IT = information technology; M = Mean; SD = Standard Deviation.

The authors focus here on the $\hat{s}$ values, as these parameters are unique to the proposed model, and likely play a central role in informing how DOMC items differ from MC items. Figure 1 illustrates the variability implied by $\hat{s}$ across items. Figure 1a illustrates for each item the change in probability from the earliest to latest key location. (Note the effect of s on the probability is actually nonlinear; in the figure, lines are drawn for the model-based probabilities from the earliest to latest possible key location to facilitate a comparison across items.) Items having multiple keyed responses naturally have their earliest possible last key location defined by the number of keyed responses. The probabilities in Figure 1 are defined for a hypothetical respondent having $θ_{i}$ =.5, $κ_{i}$ =1. The variability induced by the s parameter across items is apparent from the variability in the slopes. Items with steeper slopes reflect items with larger s parameters. Figure 1b highlights two single-keyed items from Figure 1b (Item 1, $\hat{s} = . 35$ , and Item 63, $\hat{s} = 1.55$ ) that despite having very similar probabilities of correct response when the key is presented as the first response option show very different probabilities when the key is scheduled as the fourth response option. As noted earlier, such effects are suggestive of Item 63 having more attractive distractors.

Figure 1.

Illustration of change in predicted item probabilities from earliest to latest last key location for a hypothetical respondent ( $θ_{i}$ = .5, $κ_{i}$ = 1) as a function of item and last key location, key location model, information technology certification data.

To further validate the item parameter estimates of the KL model, a regression model in which the logit of the DOMC p value is regressed against all three of the KL item parameter estimates was considered. Table 3 displays the results. As anticipated, both $\hat{b}$ and $\hat{s}$ emerge as significant predictors of difficulty and in the anticipated directions. Despite the positive correlation between these item parameter estimates, the effects on the DOMC p value function in opposing directions. The results are consistent with the two different ways in which items become more difficult under a DOMC format: (a) a reduced attractiveness of the correct response option (lower $\hat{b}$ ) and (b) a higher attractiveness of distractor options (higher $\hat{s}$ ).

Table 3.

Regression of Logit of DOMC Item Difficulty (p Value) on DOMC Item Parameter Estimates, 83 DOMC Items, IT Certification Data ( $R^{2} = . 772$ ).

	b _reg	SE	β	t	p value
Intercept	–.941	.126		–7.484	<.001
$\hat{a}$	.544	.181	.265	3.009	.004
$\hat{b}$	.797	.110	.795	7.222	<.001
$\hat{s}$	–.391	.106	–.284	–3.695	<.001

Note. DOMC = discrete-option multiple-choice; IT = information technology.

IT: Interpreting Person Parameter Estimates

Similar to the item parameter estimates, the two examinee parameter estimates ( $\hat{θ}, \hat{κ}$ ) can be examined in validating the KL model. As seen in Table 4, for the 648 examinees administered the IT DOMC items, the $\hat{θ} s$ range from −1.88 to 2.75 (M = .08, SD = .84) and $\hat{κ} s$ from .37 to 2.22 (M = .99, SD = .27) with a correlation between estimates of –.37 (p < .001).

Table 4.

Descriptive Statistics, DOMC Person Parameter Estimates, IT Certification Test Data.

	N	Minimum	Maximum	M	SD
$\hat{θ}$	648	–1.88	2.75	.08	.84
$\hat{κ}$	648	0.37	2.22	.99	.27

Note. DOMC = discrete-option multiple-choice; IT = information technology; M = Mean; SD = Standard Deviation.

Figure 2 provides an illustration of the key location effect for two examinees of different $\hat{κ}$ levels. The chosen κ levels reflect examinees close to the two extremes on the κ dimension in the IT data.

Figure 2.

Illustration of change in predicted item probabilities from earliest to latest last key location for two hypothetical examinees with $θ_{i}$ = .5 but different κ levels on a hypothetical single-keyed item (a = 1, b = 1, s = 1).

In a similar way to how $\hat{s}$ can be viewed as increasing the effect of key location on item difficulty, larger κ estimates imply a greater decline in performance when the scheduled key location is later. Figure 3 displays a scatterplot of $\hat{θ}$ by $\hat{κ}$ for the 648 examinees administered the IT DOMC test. It is apparent from the figure that the variability in $\hat{κ}$ is greater at lower levels of $\hat{θ}$ . High levels of $\hat{θ}$ generally imply low levels of $\hat{κ}$ , reflecting that high proficiency examinees also tend to show both a greater likelihood of selecting keyed responses when presented as well as a diminished key location effect. These results are consistent with the claim that the two aspects of proficiency are distinguishable under DOMC, a result also consistent with the fact that these proficiencies should conjunctively interact in affecting test performance.

Figure 3.

Scatterplot of $\hat{θ}$ by $\hat{k}$ , 648 examinees from information technology certification test using discrete-option multiple-choice format (r = −.37).

The relevance of both person parameters to performance on the IT certification test in terms of observed sum score can be examined through a multiple regression analysis in which the person parameter estimates function as predictors. Table 5 displays the results of a multiple regression analysis in which the logit of the examinee proportion correct score on the DOMC test is the studied outcome. As seen in the table, both $\hat{θ}$ and $\hat{κ}$ are relevant to performance on the test. As anticipated, overall test performance is enhanced by a higher more positive $\hat{θ}$ and a lower $\hat{κ}$ , implying a weaker key location effect.

Table 5.

Regression of Logit of DOMC Proportion Correct Score on $\hat{θ}$ and $\hat{κ}$ From DOMC Model, IT Certification Data ( $R^{2} = . 923$ ).

	b _reg	SE	β	T	p value
Intercept	–.061	.027		–2.286	.023
$\hat{θ}$	.599	.008	.831	70.584	<.001
$\hat{κ}$	–.581	.026	–.264	–22.407	<.001

Note. DOMC = discrete-option multiple-choice; IT = information technology.

The relevance of key location to item performance, as well as the variability of its effects both across items and examinees, highlights a unique feature of DOMC items relative to MC items. The distinguishability of these proficiencies can also be related to concerns regarding testwiseness in MC items. Specifically, testwiseness effects on MC items might be viewed as an examinee using $\hat{κ}$ proficiency predominantly in solving an item. Although $\hat{κ}$ clearly affects performance on DOMC items, it cannot be relied on exclusively for good performance in DOMC.

These results also have some practical implications regarding the DOMC format. From a test equity perspective, the results raise questions about the implications of purely random scheduling of key location. Eckerly et al. (2018) have already demonstrated the potential for such a process to render an overall test that varies in difficulty across examinees; however, the likelihood that key location effects may also vary across persons and items (as reflected by $\hat{κ}$ and $\hat{s}$ ) further compound equity concerns. This issue is examined in a later section of the article.

Harry Potter Trivia Data

A second DOMC data set is based on participants administered MC Harry Potter trivia items, for which an underlying latent trait might be viewed as Harry Potter knowledge. Individuals were recruited for participation online. For this test, participants were also administered a mixture of MC and DOMC items; a total of 21 items are analyzed, each item being administered to a respondent in either the MC or DOMC format. All 21 items consisted of a stem followed by five response options, and all were single-keyed items. Ultimately, the test administration design entailed two forms, one for which 10 of the items were DOMC, the other 11 MC; the second form administered the same items but with a complete reversal of format such that the 10 DOMC were administered as MC, and the 11 MC as DOMC. The form administration was randomly determined. The analysis considers responses provided by 2,488 respondents.

The same models considered for the IT data were applied to the DOMC responses of the HP data. Table 6 displays the model comparison criteria. Although the KL model (including variability in both $s_{j}$ and $κ_{i}$ ) again emerges as best, its relative superiority in fit (in comparison to the constant exponent model) is clearly less than was observed for the IT data. These results are also confirmed from inspection of parameter estimates, which show much less variability in each of the $s_{j}$ and $κ_{i}$ parameters than was seen for the IT data, as documented in the next section. Finally, as with the IT data, slightly stronger evidence supporting item variability in key location effects than person variability was seen.

Table 6.

Deviance Information Criterion Results for General Model and Three Comparison Models, Harry Potter Trivia DOMC Data.

Model	$\bar{D}$	$\hat{D}$	p_D	DIC
Constant exponent	23,797.0	22,306.5	1,490.52	25,287.5
$s_{j}$ only	23,674.7	22,155.2	1,519.49	25,194.2
$κ_{i}$ only	23,932.0	22,281.4	1,650.57	25,582.6
$s_{j}$ and $κ_{i}$	23,552.0	21,926.3	1,625.69	25,177.7

Note. Value in bold identifies lowest DIC. HP: Item Parameter Estimates. DOMC = discrete-option multiple-choice; DIC = Deviance Information Criterion.

Table 7 displays descriptive statistics for the KL item parameter estimates. Consistent with the DIC results, the authors notice not only a much lower mean $\hat{s}$ compared with the IT data but also much less variance. It would appear that the key location effect is generally weaker in the HP data.

Table 7.

Descriptive Statistics, DOMC Item Parameter Estimates, Harry Potter Trivia Data (N = 21 Items).

	N	Minimum	Maximum	M	SD
$\hat{a}$	21	0.16	1.57	0.94	0.43
$\hat{b}$	21	–1.83	3.81	1.54	1.45
$\hat{s}$	21	0.12	1.21	0.41	0.25

Note. The $s$ estimates correlate with the a and b estimates at .027 (p = .908) and .378 (p = .091), respectively, while the a and b estimates correlate at .816 (p < .001). DOMC = discrete-option multiple-choice.

Similar results for the HP data were observed as were seen for the IT data when regressing the logit of the DOMC item p values on the corresponding item parameter estimates, as shown in Table 8. Importantly, as with the IT data, the authors see the DOMC difficulty is statistically affected by both the $\hat{b}$ and $\hat{s}$ and in opposing directions, implying overall item difficulty is again determined both by difficulty in recognizing the correct response as well as by difficulty in ruling out distractor options. Although the coefficient estimate on $\hat{s}$ is comparable to that observed with the IT data in Table 3, it should be recalled that the mean and variance of $\hat{s}$ is considerably lower in the HP data, ultimately implying less of an overall effect than for the IT data.

Table 8.

Regression of Logit of DOMC Item Difficulty (p Value) on DOMC Item Parameter Estimates, 21 DOMC Items, Harry Potter Trivia Data ( $R^{2} = . 976$ ).

	b _reg	SE	β	T	p value
Intercept	0.019	.178		0.105	.918
$\hat{a}$	–0.429	.206	–.159	–2.076	.053
$\hat{b}$	0.954	.066	1.185	14.345	<.001
$\hat{s}$	–1.036	.226	–.220	–4.595	<.001

Note. DOMC = discrete-option multiple-choice.

To further illustrate interpretation of item parameter estimates and the type of heterogeneity captured, two example items were considered: Items 15 ( $\hat{a} = . 54$ ; $\hat{b} = - . 06$ ; $\hat{s} = . 24$ ) and 19 ( $\hat{a} = . 18$ ; $\hat{b} = 1.61$ ; $\hat{s} = 1.21$ ). The item stems and response options for these items are shown in Supplemental Appendix C. In regard to their classical item difficulty, both items are relatively difficult in comparison to the other items, with Item 15 having a p value of .37 and Item 19 having a p value of .58. Based on the parameter estimates of the model, however, the causes of difficulty for these items are quite different. For Item 19, which has the highest $\hat{s}$ of all of the Harry Potter DOMC items, the difficulty of the item is strongly affected by an appealing distractor (specifically Option 4), which under DOMC format is selected by 52% of respondents when presented, despite the fact that most respondents will also select the correct option (Option 1) when presented. Hence, it seems the difficulty of this item comes from the attractiveness of this distractor, which has an increasing likelihood of being presented the later the scheduled key location for the item. By contrast, the difficulty of Item 15 appears to follow from a general difficulty in knowing the correct answer (which requires recalling information across all of the books). This is apparent from the item’s low $\hat{b}$ estimate (the correct option is selected less than 50% of the time even when shown as the first option); its corresponding low $\hat{s}$ estimate suggests this difficulty does not increase substantially when the key comes later in sequence.

HP: Person Parameter Estimates

Table 9 displays descriptive statistics for the person parameter estimates from the HP analysis. Consistent with the reduced variability seen in the $\hat{s}$ estimates, reduced variance in the $\hat{κ}$ estimates across respondents were also observed. These results would again suggest that performance on the HP items is less affected by an ability to rule out distractors, and more likely a function of respondents’ ability in identifying the correct response when presented.

Table 9.

Descriptive Statistics, DOMC Person Parameter Estimates, Harry Potter Trivia Data (N = 2,488 Persons).

	N	Minimum	Maximum	M	SD
$\hat{θ}$	2,488	–2.51	1.60	0.01	.76
$\hat{κ}$	2,488	0.66	1.40	1.0	.11

Note. DOMC = discrete-option multiple-choice.

Consistent with this interpretation, the scatterplot of $\hat{θ}$ by $\hat{κ}$ shown in Figure 4 (r = −.60), while again demonstrating a negative relationship, also shows a substantially lower range of $\hat{κ}$ relative to the IT data.

Figure 4.

Scatterplot of $\hat{θ}$ by $\hat{k}$ —2,488 examinees from Harry Potter trivia test using DOMC format (r = −.60).

As noted above, the design of the HP data permit a more formal comparison of ability estimates for DOMC in comparison to MC. To further explore this issue, the authors fit the traditional two-parameter logistic model (2PL) to the binary item scores from each of the MC and DOMC tests. The resulting correlation between ${\hat{θ}}_{DOMC}$ and ${\hat{θ}}_{MC}$ was (as expected) positive, but moderate (r = .606, p < .001), suggesting some differences in what is measured by the MC and DOMC tests (although this also reflects measurement error to some extent). To better understand such differences in relation to the general model, the authors regressed ${\hat{θ}}_{DOMC}$ from the 2PL model on ${\hat{θ}}_{MC}$ from the 2PL model as well as the $\hat{κ}$ of the KL model in Equation 1. The results of Table 10 suggest the $\hat{κ} s$ of the KL model play a role in understanding why the latent proficiency estimates may differ under the two formats. As expected, a larger $\hat{κ}$ implies a greater decline in the ${\hat{θ}}_{DOMC}$ relative to the ${\hat{θ}}_{MC}$ .

Table 10.

Regression of Latent Proficiency in DOMC ( ${\hat{θ}}_{DOMC}$ ) on Latent Proficiency in MC ( ${\hat{θ}}_{MC}$ ), Harry Potter Trivia Data (N = 2,488).

	B	SE	β	t	p value
Intercept	2.934	.103		28.498	<.001
${\hat{θ}}_{MC}$	0.428	.015	.435	28.860	<.001
$\hat{κ}$	–2.945	.103	–.432	–28.634	<.001

Note. DOMC = discrete-option multiple-choice; MC = multiple-choice.

Potential Implications for Administration of DOMC Items

As implied earlier, one of the implications of randomizing key location across items is the potential for variability in overall test difficulty across administrations. Quite clearly, examinees administered a DOMC test with later key locations will be at a disadvantage to those administered the test with earlier key locations. Following the KL model in Equation 1, such effects are further complicated by the presence of both person- and item-related factors, as reflected by $κ$ and s, that influence the effects of key location.

One advantage of the KL model is that the authors can explore the expected test score implications of the randomized key schedule as a function of both the item and person parameter estimates. Specifically, based on the item parameter estimates of each analysis, the implications of randomized key location can be studied for a hypothetical respondent at a given level of $θ$ and $κ$ by quantifying the variability in true scores across simulated readministrations of the same test. Note that each administration implies a randomly determined scheduled (last) key location for each item. The scheduled key locations were denoted as Ω_k for a hypothetical administration k. A given Ω_k defines $lo c_{ij}$ for each item with respect to the KL model. The authors can then use the KL model to define an expected (true) score for an examinee at ability levels $θ$ and $κ$ for that schedule Ω_k as follows:

T (θ_{i}, κ_{i}; Ω_{k}) = \sum_{j = 1}^{J} {[φ_{j} (θ_{i})]}^{1 + [κ_{i} \times s_{j} \times (lo c_{ij} - 1)]} .

(2)

The true scores vary across Ω_k for a fixed respondent and test. Consequently, a true score variance can be quantified for a hypothetical respondent at $θ$ and $κ$ , as

σ_{θ, κ}^{2} = Va r_{Ω_{k}} [T (θ_{i}, κ_{i}; Ω_{k})],

(3)

where variance is calculated across Ω_k and can be approximated by a sufficiently large number of simulations k = 1, . . ., K. A large $σ_{θ, κ}^{2}$ implies a high degree of sensitivity to the randomly determined key location schedule. Such variance should reduce both as the number of items increases and as the s_j parameters for a given test and/or the $κ_{i}$ for a given examinee approach 0. In terms of person parameters, greater variability in true scores was anticipated for respondents with larger κ, as the performances of such respondents are more affected by key location. As the IT test is naturally the test for which equity concerns would be more relevant, the two 59-item IT forms were considered for illustration. The administration of a relatively large number of DOMC items on each form should reduce variability related to $Ω_{k}$ , as variance in the mean key location should be reduced the larger the number of items.

One of the advantages of this approach is that it can also be used to quantify the implications of alternative key scheduling approaches. A constrained randomization schedule that seeks to balance the distribution of key location across hypothetical administrations was considered here. As noted, each form contains 38 single-key, 17 double-key, and four triple-key items in which the key locations are 1 through 4, 2 through 4, and 3 through 5, respectively. A constraint is considered in which for the single-keyed items, 9, 10, 10, and 9 items have key locations 1, 2, 3, and 4, respectively; for the double-keyed items, 5, 6, and 6 items have last key locations of 2, 3, and 4, respectively; and for the triple-keyed items, 1, 2, and 1 items have final key locations of 3, 4, and 5, respectively. Even subject to these constraints, a large number of permutations of items to key locations is possible. Equations 2 and 3 were evaluated under these constraints and compared against a completely randomized schedule.

Based on 1,000 simulated administrations, a true score, a true score variance, and a 95% interval of true scores can be defined at a level of $θ$ and $κ$ for a hypothetical respondent. For example, as seen in Figure 5a, for a hypothetical respondent at $θ$ = −1, $κ$ = 1.5, under complete randomization mean true scores of 10.55 and 10.75 were observed for Forms 1 and 2 and 95% true score intervals of [8.39, 12.89], and [8.76, 12.98], respectively. Hence, there can be more than a 4-score point difference in true scores for such an examinee depending on the schedule rendered by complete randomization. Under the constrained randomization approach, means of 10.30 and 10.55, respectively, but most importantly, much narrower 95% true score intervals of [9.60, 10.97] and [9.95, 11.17] were observed. It is thus clear that the constrained randomization approach substantially reduces the variability related to key location effects. Figure 5 illustrates the mean true score and 95% true score intervals for the two forms and hypothetical respondents having $κ$ = 1.5 but $θ$ varying at levels of −1, 0, 1, and 2, under both forms of scheduling. The 95% true score intervals, which are of primary interest here, are seen to be consistently reduced under the constrained randomization scheduling.

Figure 5.

Illustration of 95% true score intervals under randomized and constrained randomization schedules, Forms 1 and 2, information technology certification test, kappa = 1.5.

Discussion

The DOMC item is a promising item type for computer-based measurement. The article sought to better understand psychometric characteristics of the DOMC versus MC formats as observed in two real DOMC administrations. One advantage of the DOMC format is that by administering response options individually, it becomes possible to more effectively evaluate two distinguishable sources of item difficulty that are otherwise inseparable under the MC format, specifically, difficulty related to identification of keyed options as opposed to difficulty in not selecting distractor options. This separation can also be studied on the person side, where proficiencies related to the identification of correct responses ( $θ$ ) and in successfully ruling out distractors ( $κ$ ), while correlated, also emerge as distinct.

To the extent that incorrect responses on DOMC items can occur in either of two ways (i.e., not selecting a keyed response OR selecting a distractor), the correctness of a DOMC item score can be viewed as the outcome of conjunctively interacting processes, namely, successes in ruling out distractors AND successes in selecting keyed responses. This feature of DOMC items is emphasized in the proposed model. By contrast, under the MC format these processes are often believed to interact disjunctively, including the possibility that a correct response on an MC item may be arrived at solely by ruling out distractors (and selecting the one remaining response). This feature of MC items is frequently viewed as undesirable and a contributing factor to “testwiseness.” From another perspective, it becomes possible to perform well on an MC test even with a low $θ$ , but provided a sufficiently high ability exists in ruling out distractors.

The emphasis on the conjunctive interactions of processes related to the sequential administration of response options motivated the consideration of key location effects in the form of exponent parameters. However, it would also be possible to consider key location as a covariate, possibly with varying effects by item and examinee, in predicting parameters in traditional item response theory (IRT) models, such as the 2PL and 3PL models. Attempts to fit such models to the IT data are reported in the supplemental appendix, but the general KL model described in the article still emerged as statistically superior.

Some additional issues regarding the KL model seem important to consider. Alternative estimation methods beyond MCMC might be considered. The authors chose an MCMC approach following in part from work on related models involving the estimation of exponent parameters (e.g., Bolfarine & Bazán, 2010). Within the MCMC approach, a better understanding of the implications of alternative priors, and the relative strengths of different priors, could be studied. Some simulation analyses related to the current estimation approach are reported in the online supplemental appendix.

Other issues relate more to applications and interpretation of KL parameter estimates. The person proficiency $κ$ need not be viewed as a nuisance dimension. Depending on the measurement context, some combination of the proficiencies $θ$ and $κ$ may be desirable to measure. On a medical licensure test, for example, it may be equally important to know not only what actions to take in treating a patient but also what actions NOT to take. One potential use of the proposed model comes from its ability to evaluate the extent to which each of the $θ$ and $κ$ proficiencies is relevant to performance on the test. In this respect, variability in $κ$ is found to be much lower for the HP data than the IT data, suggesting that the variability of scores on the HP test is largely driven by respondents’ knowledge of the correct response. By contrast, for the IT data, $κ$ is found both to be much more variable across persons and thus of greater importance in comparing performances on the test.

A second application of the proposed model demonstrated in this article concerns its use in understanding the implications of a purely randomized key location schedule. Although randomization of key location should yield comparability in the distribution of key locations across a large number of administered items, the analyses in this paper suggest that pure randomization has the potential to yield inequities for tests of moderate length, inequities that become more consequential for respondents of high $κ$ levels, and tests having items with high s values. At the same time, it would appear that such inequities largely can be addressed by using a constrained randomization process. The authors suggest that such constraints can seemingly be imposed with little practical consequence (in terms of compromised items).

A third application of the model relates to insight into differences between the DOMC and MC formats. The authors find that effects related to key location significantly explain both variability in difficulty changes (when the same item is modified from MC to DOMC format) as well as variability in examinee proficiency changes (when the same examinee is administered both MC and DOMC tests). Specifically, DOMC items with a large s tend to be disproportionately more difficult under DOMC, while examinees with a large $κ$ appear to perform disproportionately worse under DOMC (see also Kim et al., 2018, for another individual difference that may have an effect). Other observations that emerge with respect to KL model parameter estimates under the DOMC format require further attention. For example, the strong positive correlation between a and b estimates and the generally low values of a estimates, especially for the IT test, likely require a better understanding. It is conceivable that both of these effects are due to the combination of effects related to the generally high difficulty of the IT DOMC items and the unmodeled effects of guessing. As can be inferred from Lee and Bolt (2018), for example, the omission of a lower asymptote when random guessing is present should lead to lower discrimination estimates as well as a negative relationship between discrimination and difficulty (as seen in this article). How best to model the effects of guessing in DOMC items remains an open question.

While this article has focused on key location as a factor to attend to in protecting test equity, other factors could also be examined. For example, while the analyses in this paper assumed a random equivalence of the difficulty of distractors, the distractor options for a fixed item will generally vary to some extent in terms of their distracting ability. This may well contribute an additional source of item administration variability independent of the key location effect. Further study (and alternative models) may be needed to effectively account for such occurrences where present. For example, as noted in the article, the distractor difficulty of Harry Potter trivia Item 19 appears largely driven by the difficulty of one particular distractor (as opposed to all of them), a characteristic that may well be important to address. Finally, different perspectives on measurement that might disagree with the notion of equity as discussed under the proposed model were recognized. It might be contended that to the extent that the same completely randomized key location schedule is applied for all examinees, differences in the actual scheduled key locations should not be viewed as source of inequity. Clearly, more study of the DOMC format would appear useful.

Supplemental Material

supplementary – Supplemental material for A Psychometric Model for Discrete-Option Multiple-Choice Items

Supplemental material, supplementary for A Psychometric Model for Discrete-Option Multiple-Choice Items by Daniel M. Bolt, Nana Kim, James Wollack, Yiqin Pan, Carol Eckerly and John Sowles in Applied Psychological Measurement

Footnotes

Acknowledgements

The authors gratefully acknowledge David Foster and Sarah Toton for access to the Harry Potter trivia data and comments on an earlier draft of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Daniel M. Bolt

Supplemental Material

Supplemental material for this article is available online.

References

Bolfarine

Bazán

J. L.

(2010). Bayesian estimation of the logistic positive exponent IRT model. Journal of Educational and Behavioral Statistics, 35, 693-713.

Eckerly

Smith

Sowles

(2018). Fairness concerns of discrete option multiple choice items. Practical Assessment, Research & Evaluation, 23(16), Article 2.

Foster

D. F.

Miller

H. L.

(2009). A new format for multiple-choice testing: Discrete option multiple-choice. Results from early studies. Psychology Science Quarterly, 51, 355-369.

Kim

Bolt

D. M.

Wollack

Pan

Eckerly

Sowles

(2018). Modeling examinee heterogeneity in discrete option multiple choice items. In: Wiberg

Culpepper

Janssen

González

Molenaar

(Eds.), Quantitative Psychology: 83rd Annual Meeting of the Psychometric Society, New York, NY, 2018. New York: Springer.

Kingston

N. M.

Tiemann

G. C.

Miller

H. L.

Foster

(2012). An analysis of the discrete-option multiple-choice item type. Psychological Test and Assessment Modeling, 54, 3-19.

Lee

Bolt

D. M.

(2018). Asymmetric item characteristic curves and item complexity: Insights from simulation and real data analyses. Psychometrika, 83, 453-475.

Papenburg

Willing

Musch

(2017). Sequentially presented response options prevent the use of testwiseness cues in multiple-choice testing. Psychological Test and Assessment Modeling, 59, 245-266.

Patz

R. J.

Junker

B. W.

(1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146-178.

Samejima

(2000). Logistic positive exponent family of models: Virtue of asymmetric item characteristic curves. Psychometrika, 65, 319-335.

10.

Sinharay

(2004). Experiences with Markov chain Monte Carlo convergence assessment in two psychometric examples. Journal of Educational and Behavioral Statistics, 29, 461-488.

11.

Spiegelhalter

D. J.

Best

N. G.

Carlin

B. P.

van der Linde

(2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society, Series B, 64(4), 583-639.

12.

Thissen

Steinberg

(1986). A taxonomy of item response models. Psychometrika, 51, 567-577.

13.

Willing

Ostapczuk

Musch

(2015). Do sequentially-presented answer options prevent the use of testwiseness cues on continuing medical education tests? Advances in Health Science Education, 20, 247-263. doi:10.1007/s10459-014-9528-2

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.53 MB