Minimum Sample Size Requirements for Mokken Scale Analysis

Abstract

An automated item selection procedure in Mokken scale analysis partitions a set of items into one or more Mokken scales, if the data allow. Two algorithms are available that pursue the same goal of selecting Mokken scales of maximum length: Mokken’s original automated item selection procedure (AISP) and a genetic algorithm (GA). Minimum sample size requirements for the two algorithms to obtain stable, replicable results have not yet been established. In practical scale construction reported in the literature, we found that researchers used sample sizes ranging from 133 to 15,022 respondents. We investigated the effect of sample size on the assignment of items to the correct scales. Using a misclassification of 5% as a criterion, we found that the AISP and the GA algorithms minimally required 250 to 500 respondents when item quality was high and 1,250 to 1,750 respondents when item quality was low.

Keywords

automated item selection procedure Mokken scale analysis genetic algorithm for item selection sample size requirements

Introduction

Mokken scale analysis (MSA; Mokken, 1971; Sijtsma & Molenaar, 2002; Van Schuur, 2011) is a popular scaling method in many research areas. Google Scholar reports approximately 1,900 hits for the search terms “Mokken scale”, “Mokken scaling”, “Mokken model”, and “monotone homogeneity”. Key references using MSA can be found in marketing (Paas & Molenaar, 2005), medicine and health (Paap et al., 2011), political science (Van Schuur, 2003), psychology (Alterman, Cacciola, Habing, & Lynch, 2011), and sociology (Gow, Watson, Whiteman, & Deary, 2011). An intriguing question for researchers using complex multivariate methods such as MSA is which minimum sample size they need to obtain stable, replicable results. For MSA, in particular its automated item selection algorithm, minimum sample size is unknown but in this article we provide guidelines.

The automated item selection algorithm of MSA has two versions pursuing the same goal using different algorithms (Straat, Van der Ark, & Sijtsma, 2013a). The goal is to produce a division of an item set into one or more Mokken scales, if the data allow. Like most multivariate procedures, the item selection procedure optimizes a particular formal criterion to arrive at a result, and in doing so threatens to capitalize on chance. The danger of chance capitalization raises the question which minimum sample size is needed to produce results that are replicable in a new sample of data. A literature search of recent MSA applications revealed sample sizes ranging from 133 (Adler & Brodin, 2011) to 15,022 observations (Prince et al., 2010). For the latter sample size, we expect stable results but for the former sample size this is not evident, and knowledge of minimally required sample size would be highly convenient for assessing the stability of results obtained in smaller samples. Researchers have a limited amount of time and finances to collect data (Hedeker, Gibbons, & Waternaux, 1999) but wish to be able to replicate their findings in future studies (Jackson, 2003) and to have adequate statistical power to find the effects they pursue (Hedeker et al., 1999). These requirements call for knowledge of minimum sample size needed to construct scales.

Many studies investigated the minimally required sample size for other statistical methods such as regression analysis (e.g., Cohen, 1988; Green, 1991), factor analysis (e.g., Guadagnoli & Velicer, 1988; MacCallum, Widaman, Preacher, & Hong, 2002; Mundfrom, Shaw, & Ke, 2005; Velicer & Fava, 1998), multilevel analysis (e.g., Cohen, 2005; Hedeker et al., 1999; Snijders & Bosker, 1993), structural equation modeling (e.g., Bentler & Yuan, 1999; Jackson, 2003; Muthén & Muthén, 2002), and item response theory (e.g., Chuah, Drasgow, & Leucht, 2006; Hambleton & Jones, 1994; Hulin, Lissak, & Drasgow, 1982; Reise & Yu, 1990). Thus far, for MSA such studies have not been done. Minimum sample-size studies investigated the sample size that is minimally required to have precise parameter estimates. For MSA, the question does not concern parameter estimation but the correct partitioning of the items into scales.

This article is organized as follows. First, we discuss the monotone homogeneity model (MHM; Mokken, 1971; Sijtsma & Molenaar, 2002). Second, we discuss the two automated item selection methods for MSA. Third, we study the minimally required sample size to find the correct partitioning of the items. Fourth, we provide recommendations about minimum sample sizes required for item selection in MSA.

Monotone Homogeneity Model

The MHM (Mokken, 1971, chap. 4; also see Sijtsma & Molenaar, 2002; Van Schuur, 2011) is a nonparametric item response theory (IRT) model that is at the basis of MSA. The MHM for dichotomous items, scored (0, 1), is defined by three assumptions: The latent variable $θ$ is unidimensional, the $J$ item-score variables $X_{j}$ $(j = 1, . . ., J)$ are locally independent given $θ$ , and each expected item score is a monotone nondecreasing function of $θ$ . These functions are called item response functions. Grayson (1988) proved that for a set of dichotomously scored items the MHM implies that the sum score on the $J$ items, denoted $X_{+} = \sum_{j} X_{j}$ , stochastically orders people on $θ$ , and thus can be used for ordinal person measurement. Van der Ark (2005) used a simulation study to demonstrate that a set of polytomously scored items consistent with the MHM can also be used for ordering persons.

Like the MHM, many parametric IRT models assume unidimensionality and local independence but unlike the MHM, they require parametric restrictions on the item response functions. IRT models for dichotomous item scores such as the Rasch (1960) model and the 2-parameter logistic model (Birnbaum, 1968) are special cases of the MHM. Hemker, Van der Ark, and Sijtsma (2001) showed that the polytomous-item MHM encompasses well-known parametric IRT models such as the partial credit model (Masters, 1982) and the graded response model (Samejima, 1969). Next, we discuss two automated item selection methods in MSA that can be used to partition items into clusters that satisfy the definition of a Mokken scale and approximate the requirements of the MHM.

Mokken Scale Analysis

Let $Cov (X_{j}, X_{k})$ denote the covariance between two items $j$ and $k$ , let $Co v_{\max} (X_{j}, X_{k})$ denote the maximum covariance between these items given their marginal item-score distributions, and let rest score $R_{(j)}$ denote the total score on $J - 1$ items excluding item $j$ ; that is, $R_{(j)} = X_{+} - X_{j} .$ Then, the scalability coefficient for an item pair $(j, k)$ is defined as

H_{jk} = \frac{Cov (X_{j}, X_{k})}{Co v_{max} (X_{j}, X_{k})};

the scalability coefficient for item $j$ is defined as

H_{j} = \frac{Cov (X_{j}, R_{(j)})}{Co v_{max} (X_{j}, R_{(j)})};

and the scalability coefficient for the total scale score is defined as

H = \frac{\sum_{j = 1}^{J} Cov (X_{j}, R_{(j)})}{\sum_{j = 1}^{J} Co v_{max} (X_{j}, R_{(j)})} .

A set of items forms a Mokken scale (Sijtsma & Molenaar, 2002, pp. 67-69) if (a) all interitem correlations are positive and (b) all coefficients $H_{j}$ exceed a user-specified, positive lower bound $c$ . Items that do not satisfy the criteria are defined to be unscalable. The requirements of the MHM and the definition of a Mokken scale do not coincide. The MHM implies the first criterion (Holland & Rosenbaum, 1986) and the second criterion for $c = 0$ but not for other $c$ values (Sijtsma & Molenaar, 2002, pp. 58-59). In practice, because higher values of coefficient $H_{j}$ imply better item discrimination one requires a higher positive lower bound $c$ . Mokken (1971, p. 184) proposed to choose $c = . 3$ . This value has remained the default ever since, but various authors have suggested to also try other $c$ values (e.g., Hemker, Sijtsma, & Molenaar, 1995; Straat, Van der Ark, & Sijtsma, 2013b). Automated item selection methods are used to partition $J$ items into one or more Mokken scales, and possibly identify one or more items that are unscalable (Mokken, 1971; Straat et al., 2013a). Because the requirements of the MHM and the definition of a Mokken scale do not coincide, researchers are recommended to check afterwards whether the item response functions of selected items are monotone. Experience has shown that the discrepancy between Mokken scales and MHM requirements is often small in real-data analysis (e.g., Sijtsma, Emons, Bouwmeester, Nykliček, & Roorda, 2008; Straat, Van der Ark, & Sijtsma, 2012; Wismeijer, Sijtsma, Van Assen, & Vingerhoets, 2008).

The objective of MSA’s automated item selection methods is to select a first Mokken scale containing as many items as possible, then from the unselected items, if any remain, to select a second Mokken scale containing as many items as possible, and so on until there are no items left or until items remain that are unscalable (Mokken, 1971; Straat et al., 2013a). The R package mokken (Van der Ark, 2012) contains two item selection algorithms that pursue this objective. One algorithm is the automated item selection procedure (AISP; Sijtsma & Molenaar, 2002) and the other is the genetic algorithm (GA; Straat et al., 2013a). We briefly describe the two item selection procedures. For a more extensive description, see Straat et al. (2013a).

Automated Item Selection Procedure

AISP is a bottom-up item selection procedure. AISP starts with selecting from all ½ $J (J - 1)$ item pairs the item pair with the highest $H_{jk}$ value that is significantly higher than 0 and exceeds lower bound $c$ . Subsequently, AISP adds a third item to the scale that (a) correlates positively with the selected items $j$ and $k$ , (b) has an $H_{j}$ coefficient with respect to the already selected items that is significantly higher than 0 and exceeds lower bound $c$ , and (c) produces the highest $H$ coefficient with the already selected items $j$ and $k$ among all unselected items that satisfy criteria (a) and (b). This procedure is repeated for a fourth item, a fifth item, and so on until there are no items left that satisfy the criteria (a) and (b). If items remain unselected, AISP tries to construct a second scale from the unselected items, then a third scale, and so on, until there are no items left or the items left are unscalable.

Genetic Algorithm

Procedure GA uses a genetic algorithm to pursue the same goal as AISP. GA mimics an evolutionary process to search among all possible partitionings the partitioning that satisfies MSA’s scaling objective (Straat et al., 2013a). First, GA generates random partitionings and evaluates each partitioning with respect to the scaling objective. Second, the better a partitioning represents the scaling objective, the more likely it is that the partitioning is selected in a new, second population of partitionings that is drawn with replacement from the first population. The partitionings in the second population are randomly assorted and subjected to crossovers and mutations to accomplish that some of these partitionings become different from the partitionings in the first population. Next, GA evaluates the partitionings in the second population and produces a third population following the same rules that were used to produce the second population. After the formation of each population, GA records which partitioning was the best partitioning until the most recent population. If the best partitioning remains the same after having produced a pre-specified number of populations, this partitioning is reported as final.

Method

Due to lack of an analytical method for deriving the minimally required sample size for procedures AISP and GA, we used a simulation study to investigate the minimally required sample size. This took two stages. At the first stage, we studied the effect of sample size (16 levels, ranging from 50 to 3,500) on the assignment of items to scales. At the second stage, we searched for the minimally required sample sizes to obtain at least 80%, 90%, 95%, and 99% correct item assignment.

We also included independent variables in our design that may interact with the effect of sample size on the assignment of items. In exploratory factor analysis, Hogarty, Hines, Kromrey, Ferron, and Mumford (2005) found that in addition to sample size, size of the factor loadings, test length, and correlation between factors may have an effect on assigning items to scales based on the outcome of the exploratory factor analysis. We used the same design characteristics, replacing size of factor loadings by size of $H_{j}$ values.

Simulation Model

For the data simulation, we assumed a test consisting of $J$ items each with five ordered answer categories, scored $x = 0, . . ., 4 .$ A two-dimensional version of the graded response model (De Ayala, 1994) was used for data simulation. Let $θ = (θ_{1}, θ_{2})$ be the vector containing two latent variables. Let $δ_{jx}$ $(j = 1, . . ., J; x = 1, . . ., 4)$ be the location parameter of item $j$ and item score $x$ , and let $α_{j} = (α_{j 1}, α_{j 2})$ be the vector of discrimination parameters for item $j$ . The two-dimensional graded response model expresses the probability of obtaining a score of at least $x$ on item $j$ as a function of $θ$ ,

P (X_{j} \geq x | θ) = \frac{\exp [α_{j 1} (θ_{1} - δ_{jx}) + α_{j 2} (θ_{2} - δ_{jx})]}{1 + \exp [α_{j 1} (θ_{1} - δ_{jx}) + α_{j 2} (θ_{2} - δ_{jx})]} .

Design

In the design of the simulation study, six characteristics were fixed: (a) the distribution of the latent variables was bivariate standard normal; (b) the number of latent variables equaled 2; (c) the number of answer categories equaled 5; (d) lower bound $c$ equaled the default value of $. 3$ ; (e) the number of replications in each design cell equaled 100; and (f) the location parameters of the $J$ items were spaced equidistantly, such that the location parameters of item $j$ equaled $(- 1.5 + \frac{j - 1}{J - 1}, - 1.0 + \frac{j - 1}{J - 1}, - 0.5 + \frac{j - 1}{J - 1}, 0.0 + \frac{j - 1}{J - 1}) .$ Hence, for different items we defined different sets of location parameters. The independent variables were the following.

Sample Size

We investigated 16 different sample sizes: N = 50, 100, 250, 500, 750, 1,000, 1,250, 1,500, 1,750, 2,000, 2,250, 2,500, 2,750, 3,000, 3,250, 3,500. A pilot study showed that for sample sizes larger than 3,500 AISP and GA produced stable partitionings. This rendered studying larger sample sizes superfluous.

Test Length

We investigated short tests ( $J = 10$ ) and long tests ( $J = 20$ ).

Correlation Between Latent Variables

The correlation between the latent variables $θ_{1}$ and $θ_{2}$ had three values: $r = . 3$ (weak), $r = . 6$ (strong), and $r = 1$ (perfect) resulting in one effective latent variable. For weakly and strongly correlating latent variables, we simulated data assuming a simple structure with $α_{j 1} > 0$ and $α_{j 2} = 0$ for the odd-numbered items, and $α_{j 1} = 0$ and $α_{j} 2 > 0$ for the even-numbered items.

$H_{j}$ Value

A higher item discrimination causes a higher $H_{j}$ value (De Koning, Sijtsma, & Hamers, 2002). Table 1 shows the $H_{j}$ values for item discrimination parameters of 1, 1.3, and 1.6, and tests containing 5 items (not in the design), 10, and 20 items. We included the range of $H_{j}$ values for $J = 5$ in Table 1 because our experiment also includes conditions in which the $J$ (either 10 or 20) items are divided over two dimensions such that half (either 5 or 10) of the items load on the first dimension and the other half load on the second dimension. We chose the discrimination parameters such that we obtained conditions with $H_{j}$ values exceeding .20, .30, or .40 (Table 1). For condition $α = 1$ all $H_{j}$ s were approximately equal to .22, for condition $α = 1.3$ all $H_{j}$ s were approximately equal to .32, and for condition $α = 1.6$ all $H_{j}$ s were approximately equal to .42.

Table 1.

Range of $H_{j}$ Coefficients.

	Test length
α	5	10	20
1	.219-.230	.218-.229	.216-.227
1.3	.320-.335	.319-.333	.316-.330
1.6	.414-.430	.412-.430	.411-.428

Item Selection Procedure

We used procedures AISP and GA to analyze each data set. We used the R package mokken (Van der Ark, 2012) to run AISP and GA.

Dependent Variable

The Adjusted Rand Index (Hubert & Arabie, 1985; also see, e.g., Steinley, 2004) is the most commonly used index expressing the similarity of two partitionings. It can be used for evaluating a partitioning of an item set into one or more Mokken scales, and does this by comparing the obtained partitioning with a baseline partitioning. However, if all items are correctly selected into the same scale, the denominator of the Adjusted Rand Index equals 0; hence, the index is not defined. Especially when a single scale is the correct partitioning, we expect this situation to occur frequently. Thus, we decided not to use this popular index.

The partitioning of an item set into subsets can also be assessed by means of the Per Element Accuracy ( $PEA$ ; Hogarty et al., 2005) and the minimum number of items to be moved to another scale for two partitionings to be equal ( $MIN$ ; Van der Ark & Sijtsma, 2005). Both $PEA$ and $MIN$ compare the obtained partitioning with a baseline partitioning. $PEA$ is the proportion of items that is classified in agreement with the baseline partitioning; hence, $PEA$ is the proportion of correctly classified items. $MIN$ is the number of items to be moved from one scale to another scale to retain the baseline partitioning; hence, $MIN$ counts the number of misclassified items. Ratio MIN/J yields the proportion of misclassified items; hence, $PEA = 1 -$ MIN/J Because of their dependence, in the first stage of the study we used $PEA$ as the dependent variable.

To obtain baseline partitionings, in each design cell we used 1,000,000 observations to determine the “true” item partitioning. The baseline partitionings were the following. For $H_{j} \approx . 22$ , all items were unscalable; and for $H_{j} \approx . 32$ and $H_{j} \approx . 42$ , the “true” partitioning depended on the correlation between the latent variables. If $r = . 3$ and $r = . 6$ , the AISP and GA algorithms assigned the odd-numbered items to one scale and the even-numbered items to another scale. If $r = 1$ , the AISP and GA algorithms assigned all items to one scale.

In the first stage, in each design cell $PEA$ was the proportion of items in agreement with the “true” partitioning. $PEA$ values were based on the product of the number of items and the number of replications. Given 100 replications per cell, for $J = 10$ $PEA$ was based on 1,000 item classifications and for $J = 20$ $PEA$ was based on 2,000 item classifications. These numbers produced stable $PEA$ values. In the second stage, the minimally required sample size for different $PEA$ values was obtained. Because the literature does not provide guidelines for the interpretation of $PEA$ , values were based on typical Type I error rates in hypothesis testing (.20, .10, .05, and .01). Type I error rate refers to wrong decisions, whereas $PEA$ refers to correct decisions; hence, we used values equal to (1 − Type I error rate) (i.e., .80, .90, .95, and .99). We evaluated the minimally required sample size for mediocre $PEA$ (at least 80% of the items correctly classified), adequate $PEA$ (at least 90%), good $PEA$ (at least 95%), and excellent $PEA$ (at least 99%). Although arbitrary, the labels facilitate interpretation of results.

Results

The minimally required sample sizes for the four pre-specified levels of $PEA$ (Table 2) were derived from the $PEA$ s for each combination of test length, sample size, $H_{j}$ value, and latent-variable correlation (Table 3). The results for AISP and GA were almost equal except for the condition with strongly correlated latent variables (i.e., $r = . 6$ ) and highly discriminating items (i.e., $H_{j} \approx . 42$ ). In these conditions, GA obtained one scale containing all items instead of two scales (i.e., correct solution), each containing J/2 items. Hence, because half of the items were assigned to the wrong scale for procedure GA $PEA \approx . 5$ . In Table 2, we only report the minimally required sample sizes for AISP.

Table 2.

Minimum Sample Size Requirements for AISP and GA Procedures for Four Different Levels of PEA (Per Element Accuracy).

$H_{j}$	$r (θ_{1}, θ_{2})$	J	PEA
			Mediocre	Adequate	Good	Excellent
.22	.3	10	750	1,000	1,250	2,500
		20	750	1,250	1,750	3,000
	.6	10	500	1,000	1,250	2,750
		20	750	1,250	1,500	2,750
	1.0	10	1,000	1,250	2,000	3,250
		20	1,000	1,750	2,000	3,500
.32	.3	10	250	750	1,500	3,500
		20	250	750	1,500	3,500
	.6	10	500	750	1,500	3,000
		20	250	750	1,250	3,000
	1.0	10	250	750	1,250	3,000
		20	250	750	1,500	2,750
.42	.3	10	50	50	250	250
		20	50	50	250	250
	.6	10	250	250	500	750
		20	250	250	500	750
	1.0	10	50	50	250	250
		20	50	50	250	250

Note. .80 < PEA ≤ .90: mediocre; .90 < PEA ≤ .95: adequate; .95 < PEA ≤ .99: good; PEA > .99: excellent.

Table 3.

Per Element Accuracy Results for AISP.

		Approximate item scalability $H_{j}$
		.22			.32			.42
		Correlation latent variables
J	N	.3	.6	1.0	.3	.6	1.0	.3	.6	1.0
10	50	.22	.23	.37	.62	.48	.72	.90	.65	.93
	100	.49	.45	.34	.76	.71	.77	.94	.78	.94
	250	.66	.67	.50	.82	.79	.81	.99	.92	1.00
	500	.79	.81	.70	.85	.87	.88	1.00	.97	1.00
	750	.87	.87	.79	.91	.90	.91	1.00	.99	1.00
	1,000	.93	.92	.87	.92	.90	.93	1.00	1.00	1.00
	1,250	.96	.95	.90	.94	.94	.95	1.00	1.00	1.00
	1,500	.96	.97	.93	.95	.95	.96	1.00	1.00	1.00
	1,750	.97	.98	.94	.96	.95	.96	1.00	1.00	1.00
	2,000	.98	.98	.96	.96	.95	.95	1.00	1.00	1.00
	2,250	.98	.98	.96	.96	.96	.98	1.00	1.00	1.00
	2,500	.99	.98	.97	.97	.96	.98	1.00	1.00	1.00
	2,750	1.00	.99	.98	.97	.98	.98	1.00	1.00	1.00
	3,000	1.00	1.00	.98	.97	.98	.99	1.00	1.00	1.00
	3,250	1.00	1.00	.99	.98	.98	.99	1.00	1.00	1.00
	3,500	1.00	1.00	.99	.99	.99	.99	1.00	1.00	1.00
20	50	.24	.22	.36	.69	.57	.77	.91	.66	.94
	100	.49	.45	.34	.76	.71	.77	.94	.78	.94
	250	.53	.49	.40	.81	.80	.84	.99	.93	1.00
	500	.70	.69	.60	.85	.86	.88	1.00	.98	1.00
	750	.80	.80	.72	.91	.90	.93	1.00	.99	1.00
	1,000	.88	.87	.81	.93	.92	.93	1.00	1.00	1.00
	1,250	.91	.93	.86	.93	.95	.94	1.00	1.00	1.00
	1,500	.94	.95	.89	.95	.94	.96	1.00	1.00	1.00
	1,750	.96	.96	.92	.97	.95	.96	1.00	1.00	1.00
	2,000	.97	.97	.95	.97	.97	.97	1.00	1.00	1.00
	2,250	.97	.97	.96	.98	.97	.98	1.00	1.00	1.00
	2,500	.98	.98	.96	.97	.97	.98	1.00	1.00	1.00
	2,750	.98	.99	.97	.98	.98	.99	1.00	1.00	1.00
	3,000	.99	.99	.98	.98	.99	.99	1.00	1.00	1.00
	3,250	.99	.99	.98	.98	.99	.99	1.00	1.00	1.00
	3,500	.99	.99	.99	.99	.99	.99	1.00	1.00	1.00

Note. N = sample size; J = number of items.

The minimally required sample size for different $PEA$ levels mainly depended on the $H_{j}$ values. If $H_{j} \approx . 22$ and $H_{j} \approx . 32$ , larger sample sizes were needed to obtain at least an adequate $PEA$ than if $H_{j} \approx . 42$ . For $H_{j} \approx . 22$ , $N$ should be 750 to 1,000 to produce at least mediocre $PEA$ ; 1,000 to 1,250 to produce at least adequate $PEA$ ; 1,250 to 2,000 to produce at least good $PEA$ ; and 2,750 to 3,500 for excellent $PEA$ . For $H_{j} \approx . 32$ , $N$ should be 250 to produce at least mediocre $PEA$ ; 750 to produce at least adequate $PEA$ ; 1,250 to 1,500 to produce at least good $PEA$ ; and 3,000 to 3,500 for excellent $PEA$ . For $H_{j} \approx . 42$ , $N$ should be 50 to 250 to produce at least mediocre to adequate $PEA$ ; 250 to 500 to produce at least good $PEA$ ; and 250 to 750 for excellent $PEA$ .

The minimally required sample size was larger if the $H_{j}$ values were close to lower bound .3 (i.e., $H_{j} \approx . 32$ ). We investigated whether this result could be generalized to the other $H_{j}$ values (i.e., $H_{j} \approx . 22$ and $H_{j} \approx . 42$ ) being close to alternative lower bounds .2 and .4, respectively. The results for $H_{j} \approx . 22$ with $c = . 2$ and $H_{j} \approx . 42$ with $c = . 4$ were similar to the results for $H_{j} \approx . 32$ with $c = . 3$ (Table 2).

Discussion

The minimum sample size the AISP and GA procedures require to obtain stable results depends mainly on the positive difference between the $H_{j}$ sample values and lower bound $c$ . This result should come as no surprise because as $(H_{j} - c)$ becomes smaller and approaches 0, for a fixed sample size the power of detecting whether $H_{j} > c$ decreases, and larger sample sizes are needed to maintain a particular power. The result implies that the required sample size can be estimated only after the $H_{j}$ sample values have been computed; that is, after the sample has been drawn and the data have been collected.

Due to this rather awkward situation, detailed sample-size guidelines cannot be provided prior to data collection. This situation is not typical for MSA and is also found with other methods. For example, for factor analysis the minimally required sample size depends on a complex interplay of many aspects, including the estimated factor loadings (Hogarty et al., 2005; also see Muthén & Muthén, 2002). For structural equation models, Muthén and Muthén (2002) suggested the minimally required sample size should be estimated by means of Monte Carlo studies in which for different sample sizes one simulates data sets using the hypothesized parameter values. If the model parameters are recovered well from the data, then the sample size is probably sufficient. For MSA, the situation is simpler because the minimally required sample size predominantly depends on one factor only, which is the distance between the observed and criterion values of the scalability coefficients. Because we have studied this factor extensively, we believe researchers need not resort to Monte Carlo studies to obtain the minimally required sample size for the AISP and GA procedures; the following three guidelines are sufficient.

First, one should always choose $N > 1, 500 .$ This is meaningful advice in two situations: (a) The researcher investigates the structure of the data for different lower bounds $c$ , say, $c = . 00, . 05, . 10, . 15, . . ., . 60$ . This strategy, proposed by Hemker et al. (1995) and recently discussed by Straat et al. (2013b), allows the researcher to investigate the dimensionality of the data at an increasingly refined level. When using this strategy, one always encounters at least one lower bound $c$ that is close to a sample value of $H_{j}$ and that requires $N = 1, 500 .$ (b) The researcher has no prior information whatsoever. In this situation, requiring $N > 1, 500$ is always safe.

Second, if possible test constructors should use a well-founded theory of the attribute of interest to develop tests, and carefully choose items that are well-discriminating indicators of the attribute, expressed by relatively high $H_{j}$ values. The absence of a good theory may hamper the realization of this goal and, moreover, experience shows-that constructing items that have particular $H_{j}$ levels is difficult, also because $H_{j}$ values depend on the other items in the test. These hurdles do not diminish the need for well-discriminating items having high $H_{j}$ s, and pursuing construction of such items, however difficult to realize, is believed to increase the probability that $H_{j} \geq c$ . For high-quality items with $H_{j} \geq c,$ MSA rarely misclassifies items if $N > 250$ .

Third, one should use prior information. In some cases, previous studies or pilot studies can be used to predict the $H_{j}$ s in the main study. For a choice of $c$ , the predicted $H_{j}$ s may then be used to determine the sample sizes for the main study by means of the guidelines provided here. Confidence intervals (Kuijpers, Van der Ark, & Croon, 2013) may be used to determine the range of plausible values for the population $H_{j}$ s.

The guidelines in Table 2 can also be used as a posterior check to evaluate previous research. For example, Watson, Wang, Ski, and Thompson (2012) used MSA to analyze data obtained by administering a Chinese version of the MIDAS to a sample of 180 respondents, and reported a unidimensional scale with $. 39 \leq H_{j} \leq . 60$ . Although $N = 180$ may seem rather small, the guidelines in Table 2 suggest that the sample size may be qualified adequate ( $N = 50$ ) to good ( $N = 250$ ).

Minimally required sample sizes have already been established for exploratory factor analysis and depend on a complex interplay between the item loadings on a factor and the number of items per factor. The effects of test length and number of latent variables on the minimally required sample size were negligible for the AISP and GA procedures. The decision rule for the minimally required sample size for the AISP and GA procedures is easier: For high-quality items, the AISP and GA procedures perform well for small sample sizes.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Adler

Brodin

(2011). An IRT validation of the Affective Self Rating Scale. Nordic Journal of Psychiatry, 65, 396-402.

Alterman

A. I.

Cacciola

J. S.

Habing

Lynch

K. G.

(2011). ASI recent and lifetime summary indices based on nonparametric IRT models. Psychological Assessment, 19, 119-132.

Bentler

P. M.

Yuan

K. H.

(1999). Structural equation modeling with small samples: Test statistics. Multivariate Behavioral Research, 34, 181-197.

Birnbaum

(1968). Some latent variable models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.

Chuah

S. C.

Drasgow

Leucht

(2006). How big is big enough? Sample size requirements for CAST item parameter estimation. Applied Measurement in Education, 19, 241-255.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Cohen

M. P.

(2005). Sample size considerations for multilevel surveys. International Statistical Review, 73, 279-287.

De Ayala

R. J.

(1994). The influence of multidimensionality on the graded response model. Applied Psychological Measurement, 18, 155-170.

De Koning

Sijtsma

Hamers

J. H. M.

(2002). Comparison of four IRT models when analyzing two tests for inductive reasoning. Applied Psychological Measurement, 26, 302-320.

10.

Gow

A. J.

Watson

Whiteman

Deary

I. J.

(2011). A stairway to heaven? Structure of the Religious Involvement Inventory and Spiritual Well-Being Scale. Journal of Religion & Health, 50, 5-19.

11.

Grayson

D. A.

(1988). Two-group classification in latent trait theory: scores with monotone likelihood ratio. Psychometrika, 53, 383-392.

12.

Green

S. B.

(1991). How many subjects does it take to do a regression analysis? Multivariate Behavioral Research, 26, 499-510.

13.

Guadagnoli

Velicer

W. F.

(1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103, 265-275.

14.

Hambleton

R. K.

Jones

R. W.

(1994). Item parameter estimation errors and their influence on test information functions. Applied Measurement in Education, 7, 171-186.

15.

Hedeker

Gibbons

R. D.

Waternaux

(1999). Sample size estimation for longitudinal designs with attrition: Comparing time-related contrasts between groups. Journal of Educational and Behavioral Statistics, 24, 70-93.

16.

Hemker

B. T.

Sijtsma

Molenaar

I. W.

(1995). Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model. Applied Psychological Measurement, 19, 337-352.

17.

Hemker

B. T.

Van der Ark

L. A.

Sijtsma

(2001). On measurement properties of continuation ratio models. Psychometrika, 66, 487-506.

18.

Hogarty

K. Y.

Hines

C. V.

Kromrey

J. D.

Ferron

J. M.

Mumford

K. R.

(2005). The quality of factor solutions in exploratory factor analysis: The influence of sample size, communality, and overdetermination. Educational and Psychological Measurement, 65, 202-226.

19.

Holland

P. W.

Rosenbaum

P. R.

(1986). Conditional association and unidimensionality in monotone latent variable models. Annals of Statistics, 14, 1523-1543.

20.

Hubert

L. J.

Arabie

(1985). Comparing partitions. Journal of Classification, 2, 193-218.

21.

Hulin

C. L.

Lissak

R. I.

Drasgow

(1982). Recovery of two- and three-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6, 249-260.

22.

Jackson

D. L.

(2003). Revisiting sample size and number of parameter estimates: Some support for the N:q hypothesis. Structural Equation Modeling, 10, 128-141.

23.

Kuijpers

R. E.

Van der Ark

L. A.

Croon

M. A.

(2013). Standard errors and confidence intervals for scalability coefficients in Mokken scale analysis using marginal models (with discussion). Sociological Methodology, 43, 42-69.

24.

MacCallum

R. C.

Widaman

K. F.

Preacher

K. J.

Hong

(2002). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 11, 611-637.

25.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

26.

Mokken

R. J.

(1971). A theory and procedure of scale analysis. The Hague, the Netherlands: Mouton / Berlin: De Gruyter.

27.

Mundfrom

D. J.

Shaw

D. G.

T. L.

(2005). Minimum sample size recommendations for conducting factor analyses. International Journal of Testing, 5, 159-168.

28.

Muthén

L. K.

Muthén

(2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9, 599-620.

29.

Paap

M. C. S.

Meijer

R. R.

Van Bebber

Pedersen

Karterud

Hellem

F. M.

Haraldsen

I. R.

(2011). A study of the dimensionality and measurement precision of the SCL-90-R using item response theory. International Journal of Methods in Psychiatric Research, 20, E39-E55.

30.

Paas

L. J.

Molenaar

I. W.

(2005). Analysis of acquisition patterns: A theoretical and empirical evaluation of alternative methods. International Journal of Research in Marketing, 22, 87-100.

31.

Prince

Acosta

Ferri

C. P.

Guerra

Huang

Jacob

K. S.

. . . Hall

K. S.

(2010). A brief dementia screener suitable for use by non-specialists in resource poor settings—The cross-cultural derivation and validation of the brief Community Screening Instrument for Dementia. International Journal of Geriatric Psychiatry, 26, 899-907.

32.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Chicago, IL: University of Chicago Press.

33.

Reise

S. P.

(1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144.

34.

Samejima

(1969). Estimation of latent trait ability using a response pattern of graded scores (Psychometric Monograph No. 17). Richmond, VA: Psychometric Society.

35.

Sijtsma

Emons

W. H. M.

Bouwmeester

Nykliček

Roorda

L. D.

(2008). Nonparametric IRT analysis of quality of life scales and its application to the World Health Organization Quality of Life scale (WHOQOL-Bref). Quality of Life Research, 17, 275-290.

36.

Sijtsma

Molenaar

I. W.

(2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.

37.

Snijders

T. A. B.

Bosker

R. J.

(1993). Standard errors and sample sizes for two-level research. Journal of Educational Statistics, 18, 237-259.

38.

Steinley

(2004). Properties of the Hubert-Arabie adjusted Rand index. Psychological Methods, 9, 386-396.

39.

Straat

J. H.

Van der Ark

L. A.

Sijtsma

(2012). Multi-method analysis of the internal structure of the Type D Scale-14 (DS14). Journal of Psychosomatic Research, 72, 258-265.

40.

Straat

J. H.

Van der Ark

L. A.

Sijtsma

(2013a). Comparing optimization algorithms for item selection in Mokken scale analysis. Journal of Classification, 30, 72-99.

41.

Straat

J. H.

Van der Ark

L. A.

Sijtsma

(2013b). Methodological artifacts in dimensionality assessment of the Hospital Anxiety and Depression Scale (HADS). Journal of Psychosomatic Research, 74, 116-121.

42.

Van der Ark

L. A.

(2005). Stochastic ordering of the latent trait by the sum score under various polytomous IRT models. Psychometrika, 70, 283-304.

43.

Van der Ark

L. A.

(2012). New developments in Mokken scale analysis in R. Journal of Statistical Software, 48(5), 1-27.

44.

Van der Ark

L. A.

Sijtsma

(2005). The effect of missing data imputation on Mokken scale analysis. In Van der Ark

L. A.

Croon

M. A.

Sijtsma

(Eds.), New developments in categorical data analysis for the social and behavioral sciences (pp. 147-166). Mahwah, NJ: Erlbaum.

45.

Van Schuur

W. H.

(2003). Mokken scale analysis: Between the Guttman scale and parametric item response theory. Political Analysis, 11, 139-163.

46.

Van Schuur

W. H.

(2011). Ordinal item response theory: Mokken scale analysis. Thousand Oaks, CA: Sage.

47.

Velicer

W. F.

Fava

J. L.

(1998). Effects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3, 231-251.

48.

Watson

Wang

Ski

Thompson

(2012), The Chinese version of the Myocardial Infarction Dimensional Assessment Scale (MIDAS): Mokken scaling. Health and Quality of Life Outcomes, 10(2). Retrieved from http://www.hqlo.com/content/10/1/2

49.

Wismeijer

A. A. J.

Sijtsma

Van Assen

M. A. L. M.

Vingerhoets

A. J. J. M.

(2008). A comparative study of the dimensionality of the self-concealment scale using principal components analysis and Mokken scale analysis. Journal of Personality Assessment, 90, 323-334.

Minimum Sample Size Requirements for Mokken Scale Analysis

Abstract

Keywords

Introduction

Monotone Homogeneity Model

Mokken Scale Analysis

Automated Item Selection Procedure

Genetic Algorithm

Method

Simulation Model

Design

Sample Size

Test Length

Correlation Between Latent Variables

H j Value

Item Selection Procedure

Dependent Variable

Results

Discussion

Footnotes

Declaration of Conflicting Interests

Funding

References

$H_{j}$ Value