Exploratory Mokken Scale Analysis as a Dimensionality Assessment Tool

Abstract

The assessment of the number of dimensions and the dimensionality structure of questionnaire data is important in scale evaluation. In this study, the authors evaluate two dimensionality assessment procedures in the context of Mokken scale analysis (MSA), using a so-called fixed lowerbound. The comparative simulation study, covering various theoretically and empirically relevant conditions, indicates that the MSA procedures may result in scales that are inconsistent with the dimensionality of the data set at hand. That is, a single Mokken scale can be multidimensional, and two Mokken scales can pertain to a single dimension. In an illustrative evaluation, MSA using a range of lowerbounds, rather than a fixed lowerbound, was shown to have some benefits, but not to solve all limitations. The results of this study imply that MSA is perfectly suitable to create Mokken scales. However, MSA appears of limited value as a dimensionality assessment method.

Keywords

nonparametric item response theory unidimensional item response data multidimensional item response data factor structure bifactor model

An essential part of the psychometric analyses of questionnaire data is the assessment of the dimensionality. The dimensionality is defined as the minimum number of latent traits that is needed to describe the statistical dependency in the data (Zhang & Stout, 1999a). Because the dimensionality refers to the data, it is dependent on the characteristics of the items and the respondents (Reckase, 2009). Many researchers are interested in the dimensionality of a questionnaire in a population. By applying a specific dimensionality assessment procedure on sample data they try to get an indication of the dimensionality in the population of interest. When the data are described by more than a single dimension, researchers are usually also interested in the dimensionality structure of their data (Zhang & Stout, 1999b), that is, which items are related to which trait(s). In what follows, the general term dimensionality will be used to indicate the number of latent traits involved and the dimensionality structure.

To assess the dimensionality of questionnaire data, multiple methods have been developed. In recent years, there has been an increasing interest in the use of nonparametric item response theory (IRT; see Embretson & Reise, 2000) methods, such as exploratory Mokken scale analysis (MSA; e.g., Sijtsma, Meijer, & van der Ark, 2011). MSA (Mokken, 1971) has been used as a dimensionality assessment method in different research areas in the social sciences, political sciences, psychiatry, and marketing (Zijlstra, van der Ark, & Sijtsma, 2011). For instance, MSA was used to assess the dimensionality of the Social Participation Questionnaire (SPQ; Koster, Timmerman, Nakken, Pijl, & van Houten, 2009), the Brief Symptom Inventory–18 (BSI-18; Meijer, de Vries, & van Bruggen, 2011), the Hospital Anxiety and Depression Scale (HADS; Emons, Sijtsma, & Pedersen, 2010), and the SERVQUAL instrument (Paas & Sijtsma, 2008).

The popularity of nonparametric methods, such as MSA, in empirical practice is not surprising. First, nonparametric methods are generally based on less restrictive assumptions than on parametric methods (Junker & Sijtsma, 2001), and therefore, they are more likely to fit empirical data than parametric methods. Furthermore, nonparametric methods are less sensitive to small sample sizes than parametric methods (Stout, 2001). For a recent overview about MSA and its assumptions, see Sijtsma et al. (2011).

The comparative performances of different nonparametric dimensionality assessment methods have been investigated in various simulation studies. Those studies suggest that MSA is inferior in relation to alternative nonparametric methods in recovering the correct dimensionality structure (Mroch & Bolt, 2006; van Abswoude, van der Ark, & Sijtsma, 2004). In particular, those studies revealed that MSA does not function well in conditions in which the traits correlate (Mroch & Bolt, 2006; van Abswoude et al., 2004), or in which the items load on more than one trait (van Abswoude et al., 2004). Although those test conditions are the rule rather than the exception in empirical practice (van Abswoude, Vermunt, & Hemker, 2007), after publication of these articles, numerous researchers have still used MSA as a dimensionality assessment tool (e.g., Bech, Fava, Trivedi, Wisniewski, & Rush, 2011; Chen, Tseng, Hu, & Koh, 2010; Doyle, Conroy, McGee, & Delaney, 2010; Emons et al., 2010; Koster et al., 2009; Meijer et al., 2011; Ordoñez, Ponsoda, Abad, & Romero, 2009; Roorda, Houwink, Smits, Molenaar, & Geurts, 2011; Sousa et al., 2010). Furthermore, MSA is still recommended to assess unidimensionality (Sijtsma et al., 2011). The authors suspect that one of the reasons for this continuous use and recommendation of MSA as a dimensionality assessment method is that previous studies have mainly focused on the comparison of different nonparametric IRT methods, and to a lesser degree on the mechanisms behind MSA that cause MSA to be insufficient for dimensionality assessment. In particular, the authors conjecture that the continued use of MSA as a dimensionality assessment method might be due to a misunderstanding about what a Mokken scale reflects, namely, that it would necessarily consist of items that measure a distinct single underlying trait. The latter would be necessary to function well as a dimensionality assessment method. Therefore, in this study, they focus on the mechanisms of MSA that underlie the underperformance of MSA as a dimensionality assessment tool.

Overview

The aim of the present article is to examine and explain the performances of MSA as a dimensionality assessment method, in a range of conditions that are relevant from a theoretical and/or empirical perspective. For the latter, the authors focus on data structures that have been shown to be particularly relevant in the personality field. They extend the literature on the performance of MSA by discussing the mechanisms of MSA that underlie the underperformance of MSA as a dimensionality assessment tool and by including a recently proposed method that is based on a genetic algorithm (GA; Straat, van der Ark, & Sijtsma, in press).

The remainder of this article is organized as follows. First, the Monotone Homogeneity Model (MHM) and the scalability coefficient H on which MSA is based are introduced. Second, the conditions that must be satisfied for a set of items to form a Mokken scale are presented, and the most commonly used item selection methods to create Mokken scales are discussed. Third, the meaning of a Mokken scale is explained. Fourth, the simulation study is presented, in which the performance of MSA as a dimensionality assessment method is examined, using a fixed lowerbound. A range of conditions is included in which it is speculated that MSA has difficulties in recovering the dimensionality structure. Fifth, the value of MSA using a range of lowerbounds to assess the dimensionality is considered. For illustrative purposes, an empirical example is presented in which the use of MSA, with a fixed lowerbound and a range of lowerbounds, as a dimensionality assessment method appears less suitable than the use of an alternative method, namely, factor modeling. Finally, the implications of the findings are discussed.

MSA

MHM

MSA is based on the MHM. The MHM is based on the assumptions of unidimensionality, local independence, and monotonicity of the item response function (IRF). An IRF describes the probability of an affirmative response to (an answer category of) an item, as a function of the latent trait scores. If the MHM holds, persons can be ordered on the latent trait using their ordering on the total score of the questionnaire.

Scalability Coefficients

To explore whether the MHM fits for a given set of items, the scalability coefficient H is commonly used. There are three versions of the scalability coefficient, namely, between two items X_i and X_j (H_ij), between an item X_i and the remaining items of the item set (H_i), and for the item set (H).

The scalability coefficient H_ij of items X_i and X_j is defined as

H_{ij} = \frac{Cov (X_{i}, X_{j})}{Cov {(X_{i}, X_{j})}_{max}},

where Cov(X_i,X_j) is the covariance between items X_i and X_j; Cov(X_i,X_j)_max is the maximum covariance between items X_i and X_j, given the marginal distributions of the items X_i and X_j; and hence, H_ij is the ratio between these two.

The scalability coefficient H_i of a single item X_i with respect to the other items in the item set is defined as

H_{i} = \frac{Cov (X_{i}, R_{(i)})}{Cov {(X_{i}, R_{(i)})}_{max}},

where R_(i) is the total rest score (the unweighted total score minus the score of item X_i), Cov(X_i,R_(i)) is the covariance between item X_i and the total rest score R_(i), Cov(X_i,R_(i))_max is the maximum covariance between item X_i and the total rest score R_(i), and hence, H_i is the ratio between these two.

The scalability coefficient H is defined as the aggregated item scalability coefficients over the item set, namely, as

H = \frac{\sum_{i} Cov (X_{i}, R_{(i)})}{\sum_{i} Cov {(X_{i}, R_{(i)})}_{max}} .

Mokken Scale Conditions

MSA aims at finding Mokken scales. Mokken (1971) defines a set of items as a Mokken scale if it meets two requirements:

The covariances between the items are nonnegative: $Cov (X_{i}, X_{j}) > 0$ .

The scalability coefficients for all items are larger than or equal to a positive lowerbound c: $H_{i} \geq c > 0$ , for all items, where c is a positive lowerbound of H_i.

The rationale behind those two requirements is as follows: The first Mokken scale condition stems from the requirement that items must comply with the MHM. As under the MHM the covariance of each item pair is nonnegative in the population, negative covariances are in conflict with MHM (Sijtsma & Molenaar, 2002). The second Mokken scale condition stems from the desire to construct scales with items that discriminate sufficiently between persons. As items with positive but small H_i values contribute little to a reliable person ordering, H_i values are required to be larger than or equal to a specified lowerbound c (Sijtsma & Molenaar, 2002).

The Aim of MSA

The aim of MSA is to find from an item set, one or more subset(s) of items that each meet the Mokken scale conditions. Such subset(s) of items are commonly denoted as Mokken scale(s). The Mokken scale conditions are necessary but insufficient conditions to create unique Mokken scales. That is, there may be multiple subsets of items that meet the Mokken scale conditions. Therefore, further conditions must be specified to ensure the unicity of the partitioning of items in subscales. One of such an additional condition is the scale length. Mokken (1971) proposed to select as many items as possible in a scale that satisfy the scale conditions. Nonetheless, the condition of scale length is still insufficient to obtain a unique partitioning. To address this issue, different so-called item selection methods have been proposed.

Item Selection Methods

The available item selection methods can be divided into two main classes, namely, heuristic procedures and procedures based on a specific objective function. The latter may differ with respect to the algorithm used to optimize the objective function.

The most popular item selection method is the automated item selection procedure, here referred to as MSA-AISP. MSA-AISP is implemented in MSP5.0 (Molenaar & Sijtsma, 2000) and in R (Straat et al., in press; van der Ark, 2007). The MSA-AISP is a sequential item selection method. The procedure consists of the following steps:

The item pair with the highest positive H_ij value is selected.

From the pool of the remaining items, the item is selected that

covaries positively with the items selected in Step 1 (Scale Condition 1)

has an H_i value with respect to the already selected items that is significantly larger than zero and is equal or larger than the predefined lowerbound c (Scale Condition 2)

maximizes the total H, considering the items already selected.

Subsequently, Step 2 is repeated until there are no items left that fulfill the conditions for being selected. From the remaining items, if any, a new scale is formed, following the same steps as described above. This forming of additional scales continues until there are no items left that meet the Conditions 2a through 2c. Any possible remaining items are denoted as being unscalable (Sijtsma & Molenaar, 2002).

A disadvantage of the heuristic MSA-AISP is that it does not consider all possible combinations of item pairs and, therefore, it relies heavily on the starting set of items. As a consequence, the MSA-AISP can yield suboptimal results in the sense that it can form scales which do not have the maximum possible H coefficient and/or is not of maximal length, given all possible partitions of the item set (van Abswoude et al., 2004). As an alternative, item selection methods based on an objective function can be used.

A recently proposed item selection method based on an objective function uses a GA to find the optimal scales (Straat et al., in press; van der Ark, 2007), to be denoted here as MSA-GA. The aim of the MSA-GA is to select from all possible partitionings of items, the partitioning for which the largest scale contains the maximum possible number of items, the second largest scale contains the second maximum possible number of items, and so on.

To explain MSA-GA further, suppose there are F possible partitions that meet the Mokken scale conditions. Let $v_{f} (f = 1, \dots, F)$ be a scale indicator vector, which is a vector that indicates for an item set which items belong to which scale(s), and which item(s) is (are) unscalable. For example, for an item set of six items, one could have v_f = [1 2 1 1 u 2]′, with 1 and 2 the serial numbers of the scales and u indicating an unscalable item. Now, the objective function that is maximized in MSA-GA (Straat et al., in press) can be written as

O (v_{f}) = \sum_{k = 1}^{K_{f}} J^{- k} \times J_{fk},

where $O (v_{f})$ is the objective function value associated with the fth partitioning, k is the serial number of the scale, K_f is the number of scales in partitioning f, J is the total number of items in the item set, and J_fk is the number of items in scale k in partitioning f. Thus, the scale with serial number 1 gets by far the largest weight in the objective function, and in this way, the length of the scales with a low serial number is heavily rewarded. For a more detailed description of the item selection method, see Straat et al. (in press).

The Meaning of a Mokken Scale

Irrespective of the item selection method applied, a MSA is performed with the aim to create Mokken scales. As discussed earlier, a Mokken scale has to satisfy the two scale conditions and is furthermore required to be of maximum scale length. In contrast to those clear properties of a Mokken scale, its meaning does not always seem to be clear.

A correct interpretation of a Mokken scale is that it consists of items that “a) measure a common trait . . . with b) reasonable discriminative power determined by lowerbound c” (Sijtsma & Molenaar, 2002, p. 68). From those two properties, it is sometimes deduced that MSA “reveals the dimensionality structure of the data by selecting one or more subsets of items” (Sijtsma & Molenaar, 2002, p. 71). The authors conjecture that this deduction is not necessarily true. First, note that MSA only reveals the dimensionality if MSA would reveal either a single unidimensional scale, or if MSA would reveal various scales that are each unidimensional and mutually distinct. They conjecture that MSA may reveal scale(s) that do not satisfy those requirements, because MSA could reveal a Mokken scale that is multidimensional, or MSA could reveal different unidimensional Mokken scales that are not distinct. A simulation study was performed to assess under which conditions this may occur.

Simulation Study

Generation of Simulated Data

To mimic a realistic data structure, each data set was simulated to have item scores on 30 items, each on a 5-point Likert-type scale. The simulated data were generated according to a common factor model. If y_i* denotes the p-dimensional vector of continuous latent scores of subject i on p items, the following linear factor model was used for y_i*:

{y_{i}}^{*} = ν + Λ η_{i} + ε_{i},

where ν is a p-dimensional vector of intercepts, Λ is a (p×q) matrix of factor loadings, η_i is a q-dimensional vector of common factor scores, and ε_i is a p-dimensional vector of residuals. The following usual assumptions are made: E(η_i) = E(ε_i) = 0, η_i and ε_i are independent, and ε_i are mutually independent. To partly identify the solution, the variance of η_i was fixed at 1. The dimensionality structure of the items is expressed in the factor loading matrix Λ, where a loading equal to zero indicates that the item and the factor concerned are unrelated, and in the correlation matrix of the factor $Φ = η_{i} η'_{i}$ . To ease the discussion, each continuous latent item y_j* had a variance of 1; this implies that the discriminative power of each item j can be expressed via $({λ^{'}}_{j} η_{i} {η^{'}}_{i} λ_{j})$ , with $λ'_{j}$ the jth row of Λ.

To relate the continuous latent score of subject i on item j (y_ij*) to the observed ordered polytomous data, prespecified thresholds were used for variable $j (j = 1, \dots, p) τ_{cj}$ ( $τ_{cj} = τ_{0 j}, \dots, τ_{Cj}$ , with C the number of categories),

y_{ij} = c, if τ_{jc - 1} < y_{ij} * < τ_{jc} .

For all items in the simulation study, the thresholds were specified such that one third (i.e., 10) is symmetrically distributed, and two thirds (i.e., 20) are skewed, of which half (i.e., 10) in the opposite direction, so as to mimic differences in item difficulty across variables. The thresholds were chosen such that the expected proportion of observations in each of the five categories ( $c = 1, \dots, 5$ ) were [.15 .20 .30 .20 .15] for the symmetric items, [.42 .25 .21 .08 .03] for the right-skewed items, and [.03 .08 .21 .25 .42] for the left-skewed items.

Conditions Simulation Study

The data were simulated under a number of conditions in which it is speculated that MSA may create Mokken scales that are not necessarily distinct and unidimensional; furthermore, the sensitivity to sampling fluctuations was assessed. In particular, the following three factors were manipulated:

Sample size (at N = 250 and 1,000)

Percentage common variance ( $σ_{c}^{2}$ at 25.0, 37.5, 50.0, 62.5, and 75.0)

Dimensionality structure as manipulated via the percentage of common variance attributed to the general factor g ( $σ_{g}^{2}$ at 0.0, 25.0, 75.0, and 100.0)

The dimensionality structure was manipulated via a loading matrix that complied with a bifactor model, while considering uncorrelated factors (i.e., taking Φ = I). To ease the interpretation of the simulation results, the loadings (and hence the discriminative power per item) were kept as similar as possible. Before discussing the implications of the authors’ choices for the variance of the general factor, they first motivate their choice for the bifactor model.

The bifactor model (Holzinger & Swineford, 1937) is a model, in which each item may load on a general factor, and on one or more specific factors, while the factors are uncorrelated. Commonly, an item loads on one specific factor at maximum. The bifactor model is very attractive in the personality and psychopathology domain, because in this context, one typically aims at measuring complex constructs, with an interest in the general trait and in specific aspects of the trait (e.g., Reise, Morizot, & Hays, 2007).

In this simulation study, a bifactor model with three specific factors was used. This bifactor loading matrix can be depicted as

Λ_{b i f a c t o r} = {[\begin{matrix} a & a & a \\ b & 0 & 0 \\ 0 & b & 0 \\ 0 & 0 & b \end{matrix}]}^{'}

with Λ_bifactor of size (30 × 4), where a and b are vectors of length 10, with each element equal to a and b, respectively, with a the loading on the general factor and b the loading on a specific factor. The relative sizes of a and b determine the proportion of variance of the general and specific factors, respectively. In particular, the percentage of the common variance that is attributed to the general factor can be expressed as $σ_{g}^{2} = (a^{2} / (a^{2} + b^{2}) \times 100 %)$ .

The four levels of the dimensional structure were defined in terms of the percentage of common variance attributed to the general factor g. The four levels lead to structures with the following properties:

The condition with g = 100.0 was achieved by taking in Λ_bifactora > 0 and b = 0. This is equivalent to a unidimensional structure.

The condition with g = 0.0 was achieved by taking in Λ_bifactora = 0 and b > 0; this yields a three-factor loading matrix with simple structure and uncorrelated factors.

In the conditions with $σ_{g}^{2} = 25.0$ and 75.0, a and b must be larger than zero. For $σ_{g}^{2} = 25.0$ , a < b, implying a weak general factor. For $σ_{g}^{2} = 75.0$ , a > b, implying a strong general factor. Thus, one has a bifactor loading matrix, with each item loading on the general factor and on a single specific factor.

The specific values for a and b were chosen such that they are in accordance with the required percentage common variances.

To relate the conditions with a weak and a strong general factor (i.e., $σ_{g}^{2} = 25.0$ and 75.0, respectively) to earlier studies (e.g., Mroch & Bolt, 2006; van Abswoude et al., 2004), in which the cross-loadings and interfactor correlations were manipulated, it is instructive to note some properties of the bifactor loading matrix. First, because equal nonzero loadings per factor were considered, the matrix Λ_bifactor is of Rank 3 and hence is not of full rank. This implies that the four-factor loading matrix Λ_bifactor can be rotated to a three-factor loading matrix without loss of fit. The following two rotations are of particular interest. Λ_bifactor can be orthogonally rotated toward a three-factor loading matrix with cross-loadings (i.e., each item has a nonzero loading on all three factors). Alternatively, Λ_bifactor can be obliquely rotated toward a three-factor loading matrix with simple structure (i.e., a loading matrix that is proportional to columns 2 through 4 of Λ_bifactor), but at the cost of the introduction of correlated factors, with a correlation equal to the proportion of common variance attributed to the general factor (Schmid & Leiman, 1957).

As an example, consider the condition with the common variance equal to 75.0%, and the weak general factor. This is achieved by taking a = .433 and b = .750 in Λ_bifactor. The resulting loading matrix can be orthogonally rotated to a three-factor loading matrix as

Λ_{rot} = {[\begin{matrix} c & d & d \\ d & c & d \\ d & d & c \end{matrix}]}^{'}

where c and d are vectors of length 10, with each element equal to c and d, respectively, with values of c = .854 and d = .104. Alternatively, one could obliquely rotate to a loading matrix Λ_rot, with values of c = .866 and d = .000, and with each interfactor correlation equal to .250.

Now, the four levels of the dimensional structure are explained, and what the true dimensionality is can be indicated. The number of dimensions involved is one for $σ_{g}^{2} = 100.0$ , and three for $σ_{g}^{2} = 0.0$ , 25.0, and 75.0. Note that the number of dimensions of the latter two (involving a general and three specific factors) equals three only because in the simulations, the loading matrix was of Rank 3. For the conditions with three dimensions, the true dimensionality structure involved is the one where items 1 to 10, 11 to 20, and 21 to 30 are associated with the three respective factors.

In what follows, the four dimensionality structures will be referred to as a unidimensional structure ( $σ_{g}^{2} = 100.0$ ), a three-dimensional structure with uncorrelated factors ( $σ_{g}^{2} = 0.00$ ), a three-dimensional structure with weakly correlated factors ( $σ_{g}^{2} = 25.0$ ), and a three-dimensional structure with strongly correlated factors ( $σ_{g}^{2} = 75.0$ ).

A fully crossed design with 500 replications per condition was used, yielding 2 (sample size) × 5 (percentage common variance) × 4 (dimensionality structure) × 500 (replications) = 20,000 generated data sets.

Analysis of Simulated Data

Each of the 20,000 simulated data sets was analyzed with MSA in the statistical program R (R Development Core Team, 2011; Mokken package in R, see van der Ark, 2007). The simulated data sets were analyzed using the MSA-AISP and the MSA-GA (Straat et al., in press; van der Ark, 2010). Because most researchers use the default setting for a lowerbound for the scalability coefficient of c = .3 (Zijlstra et al., 2011), all simulated data sets were analyzed with c = .3.

Recovery Measures

A MSA results in a percentage of unscalable items and a partitioning vector that indicates which items are allocated to which subscales. The MSA results were compared with the simulated data structures with respect to the number of dimensions, the dimensionality structure, and the percentage of unscalable items.

Number of dimensions

To assess to what extent the partitioning of MSA deviates from the correct partitioning, the absolute deviation was evaluated between the number of subscales revealed by MSA and the true number of dimensions (i.e., one for $σ_{g}^{2} = 1$ and three for $σ_{g}^{2} < 1$ ). Furthermore, to assess the direction in case of an incorrect numbers of dimensions, the percentages over-, correct, and underestimation of the number of dimensions in each condition were considered.

Dimensionality structure

To assess to what extent the partitioning of MSA corresponds to the simulated data structure, the Rand Index (Rand, 1971) was considered between the partitioning of the scalable items revealed by MSA and the true partitioning (i.e., the unidimensional structure for $σ_{g}^{2} = 1$ and the three-dimensional structure for $σ_{g}^{2} < 1$ ). The Rand Index is a measure of similarity between two partitionings, where a Rand Index of 1 indicates perfect similarity and a Rand Index of 0 indicates no similarity.

Percentage of unscalable items

To assess how many items MSA indicate as unscalable and, thus, how many items are included in the partitioning of MSA, the percentage unscalable items was considered.

Results

Performance of MSA to indicate the number of dimensions

Figures 1a to 1d depict the mean absolute deviation (MAD) between the number of subscales revealed by MSA and the true number of dimensions, as a function of the dimensionality structure and the common variance, for each combination of sample size and item selection method. Note that a MAD of zero indicates that MSA revealed the correct number of dimensions across all replicates in this condition. In Figures 1a to 1d, one can distinguish four general patterns in performances:

Figure 1.

MAD from number of correct dimensions based on the scalable items

If the percentage common variance is low (i.e., for $σ_{c}^{2} = 25.0$ ), MSA-AISP and MSA-GA fail to correctly indicate the number of dimensions, across all dimensionality structures and sample sizes considered.

If the percentage common variance is moderate to large (i.e., for $σ_{c}^{2} \geq 37.5$ ) and the data are unidimensional or three dimensional with uncorrelated factors (i.e., for $σ_{g}^{2} = 100.0$ or for $σ_{g}^{2} = 0.0$ ), MSA-AISP and MSA-GA indicate the number of dimensions relatively well; only in the condition with high common variance ( $σ_{c}^{2} = 75.0$ ), MSA-GA does not always reveal the correct number of dimensions.

MSA-AISP indicates the number of dimensions well if the percentage common variance is moderate to large (i.e., for $σ_{c}^{2} \geq 37.5$ ) and the data are three-dimensional with weakly correlated factors (i.e., $σ_{g}^{2} = 25.0$ ). In contrast, for strongly correlated factors ( $σ_{g}^{2} = 75.0$ ), MSA-AISP fail to indicate the correct number of dimensions.

MSA-GA fails to correctly indicate the number of dimensions if the percentage common variance is moderate to large (i.e., for $σ_{c}^{2} \geq 37.5$ ) and the data are three-dimensional with weakly or strongly correlated factors (i.e., $σ_{g}^{2} = 25.0$ or $σ_{g}^{2} = 75.0$ ).

To obtain further insight into the performance of MSA in indicating the number of dimensions, Table 1 presents the percentage under-, over-, and correct estimation of the number of dimensions by MSA-AISP and MSA-GA, as a function of the dimensionality structure and the common variance, at a sample size of 1,000. The results at a sample size of 250 are not presented because they appeared to be rather similar to the ones in Table 1. As can be seen in Table 1, MSA-AISP tends to overestimate the number of dimensions if the percentage common variance is low (i.e., for $σ_{c}^{2} = 25.0$ ). For three-dimensional data with strongly correlated factors, MSA-AISP appears to underestimate the number of dimensions if the percentage common variance is moderate to large (i.e., for $σ_{c}^{2} \geq 37.5$ ). MSA-GA shows a similar pattern as MSA-AISP, except for the finding that MSA-GA also tends to underestimate the number of dimensions for three-dimensional data with weakly correlated factors, and this tendency becomes more severe with increasing common variances in the data. Moreover, surprisingly, MSA-GA also tends to underestimate the dimensionality for three-dimensional data with uncorrelated factors, in conditions with high common variance ( $σ_{c}^{2} = 75.0$ ).

Table 1.

The Percentages of Under-, Over-, and Correct Estimation of the Number of Dimensions by MSA for the Data Sets With a Sample Size of 1,000 per Condition Considered

Method	1-dim			3-dim-sc			3-dim-wc			3-dim-unc
	Under	Correct	Over	Under	Correct	Over	Under	Correct	Over	Under	Correct	Over
MSA-AISP
$σ_{c}^{2} = 25.0$	0.0	0.0	100.0	10.0	11.8	78.2	5.2	14.1	80.7	6.4	13.7	79.9
$σ_{c}^{2} = 37.5$	0.0	100.0	0.0	30.8	69.2	0.0	0.0	100.0	0.0	0.0	99.8	0.2
$σ_{c}^{2} = 50.0$	0.0	100.0	0.0	100.0	0.0	0.0	0.0	100.0	0.0	0.0	100.0	0.0
$σ_{c}^{2} = 62.5$	0.0	100.0	0.0	100.0	0.0	0.0	0.0	100.0	0.0	0.0	100.0	0.0
$σ_{c}^{2} = 75.0$	0.0	100.0	0.0	100.0	0.0	0.0	0.0	100.0	0.0	0.0	100.0	0.0
MSA-GA
$σ_{c}^{2} = 25.0$	0.0	0.0	100.0	9.4	10.8	79.8	4.2	8.6	87.2	3.6	8.4	88.0
$σ_{c}^{2} = 37.5$	0.0	100.0	0.0	46.4	37.8	15.8	0.0	100.0	0.0	0.0	99.8	0.2
$σ_{c}^{2} = 50.0$	0.0	100.0	0.0	100.0	0.0	0.0	6.2	70.8	23.0	0.0	100	0.0
$σ_{c}^{2} = 62.5$	0.0	100.0	0.0	100.0	0.0	0.0	99.8	0.2	0.0	1.8	93.0	5.2
$σ_{c}^{2} = 75.0$	0.0	100.0	0.0	100.0	0.0	0.0	100.0	0.0	0.0	51.2	32.8	16.0

Note: MSA = Mokken scale analysis; 1-dim = one dimensional; 3-dim-sc = three-dimensional strongly correlated; 3-dim-wc = three-dimensional weakly correlated; 3-dim-unc = three-dimensional uncorrelated; AISP = automated item selection procedure; GA = genetic algorithm. Percentage correct estimation is underlined. Cells representing 50% or more of the data sets are printed in boldface.

Performance of MSA to indicate the dimensionality structure

Figure 2 depicts the Rand Index between the MSA partitioning of the scalable items and the true dimensionality structure (of the scalable items), as a function of the dimensionality structure and the common variance, for each combination of sample size and item selection method. Note that a higher Rand Index indicates a better recovery, with a value equal to 1 indicating that MSA revealed the correct partitioning.

Figure 2.

MRI over scalable items between Mokken partition and correct partition

Figure 2 shows that the performances of MSA in indicating the dimensionality structure follow a highly comparable pattern with the performances of MSA in indicating the number of dimensions (Figure 1). That is, the four general patterns in performances as discussed previously also holds for the recovery of the dimensionality structure.

Percentage of unscalable items of MSA-AISP and MSA-GA

Figures 3 and 4 depict boxplots of the percentages of unscalable items indicated by MSA-AISP and MSA-GA, as a function of the dimensionality structure, the common variance, and the sample size. For MSA-AISP (see Figure 3), the percentages of unscalable items decrease with increasing percentages of common variance. This is not surprising, because an unscalable item indicates a weak item, and the percentage of common variance expresses the strength of an item. For MSA-GA, the percentages of unscalable items also decrease with increasing percentages of common variance, but it does so only up to a common variance of 50.0%, and for common variances larger than 62.5%, the percentages unscalable items appear to increase.

Figure 3.

Percentage unscalable items, method MSA-AISP

Figure 4.

Percentage unscalable items, method MSA-GA

For MSA-AISP and MSA-GA, in the low common variance condition (i.e., for $σ_{c}^{2} = 25.0$ ), the percentages of unscalable items appear to be lower for a sample size of N = 250 than for N = 1,000. For MSA-AISP, in the moderate to large common variance conditions (i.e., for $σ_{c}^{2} \geq 37.5$ ), the reverse occurs, thus lower percentages of unscalable items for N = 1,000 than for N = 250. Those effects can be explained from the idea that the items in the low common variance condition are so weak that they do not meet the strength criterion of a Mokken scale in the population. In the remaining conditions, the items are apparently strong enough at the population level. Hence, with increasing sample size, the items should be indicated as weak items with lower probabilities, as has been observed. In MSA-GA, a similar pattern occurs, albeit in the condition of three-dimensional data with uncorrelated factors with high common variance ( $σ_{c}^{2} = 75.0$ ), MSA-GA shows a larger percentage of unscalable items for N = 1,000 than for N = 250.

Discussion

The results of the simulation study showed that the performance of MSA as a dimensionality assessment method appears to be modest. That is, in this study, MSA-AISP indicates the correct dimensionality for unidimensional and three-dimensional data with uncorrelated or lowly correlated factors, in conditions with reasonable common variance (i.e., for $σ_{c}^{2} \geq 37.5$ ). In the condition with low common variance (i.e., for $σ_{c}^{2} \geq 25.0$ ), overestimation was clearly observed, implying that items that do belong to a single scale are divided into multiple scales. In contrast, underestimation was observed for three-dimensional data with strongly correlated factors, implying that items that do belong to separate scales are allocated to the same scale. MSP-GA appeared to show a similar pattern as MSA-AISP, albeit in a more extreme way. That is, MSA-GA already shows a tendency to underestimate the dimensionality for three-dimensional data with weakly correlated factors and even for three-dimensional data with uncorrelated factors in conditions with high common variance ( $σ_{c}^{2} = 75.0$ ). This phenomenon can be explained from the fact that MSA-GA explicitly maximizes the length of the first scale, thereby promoting the allocation of items from separate (but nonzero correlated) factors to a single scale. The authors conjecture that MSA-GA shows the tendency to underestimate the dimensionality even for three-dimensional data with uncorrelated factors at the population level, because the sample factors may correlate due to sampling fluctuations. In conditions with high common variance ( $σ_{c}^{2} = 75.0$ ), these nonzero correlations between factors at the sample level can apparently be large enough to allocate items from separate (but in the sample nonzero correlated) factors to a single scale.

In the conditions in which MSA (MSA-AISP or MSA-GA) creates Mokken scales that do not correspond to the true dimensionality of the data, MSA may have created a single Mokken scale with a truly multidimensional structure (yielding underestimation of the number of dimensions), and/or several Mokken scales that are truly unidimensional, but nondistinct (yielding overestimation of the number of dimensions). Below, these two situations are elaborated further.

One Mokken scale with a multidimensional structure

MSA can create a single Mokken scale with a truly multidimensional structure only in case the subscales do have sufficient “in common.” In the simulation experiment, this is particularly shown in the condition with correlated factors (i.e., for $σ_{g}^{2} = 25.0$ and 75.0), especially in the underestimation of the true number of dimensions.

The reason that MSA can create one multidimensional Mokken scale stems from the aim of MSA to select the maximum number of items that indicate a common trait and that discriminate well between persons. That is, if items discriminate well enough (i.e., $σ_{c}^{2}$ is relatively high) and the items have enough in common (i.e., $σ_{g}^{2}$ is relatively high), those items indicate a common trait with sufficient discriminative power. Thus, the goal of MSA to achieve a scale of maximal length—given a minimal strength of each item in the scale—precludes the construction of multiple smaller scales with a higher discriminative power.

Several nondistinct unidimensional Mokken scales

In the case of a truly unidimensional structure, MSA may create various unidimensional, but nondistinct Mokken scales. In this simulation study, this is illustrated in the condition with a unidimensional structure with a low common variance. This phenomenon can be understood as follows. MSA requires the items in a subscale to measure a common trait, with sufficient discriminative power. In the case of a low common variance, the unique parts of the items may happen to correlate, just because of sampling fluctuations. Those correlations may be just strong enough for certain subsets of items to let MSA define different subscales. This phenomenon becomes more likely with decreasing sample sizes, as is also shown in this simulation study, where the overestimation appeared to more severe in the conditions with a sample size of 250 than with 1,000.

MSA With a Range of Lowerbounds

In the simulation study, the performance of MSA with a fixed lowerbound in assessing the dimensionality was considered. Alternatively, one could apply MSA using a range of lowerbounds, as proposed by Hemker, Sijtsma, and Molenaar (1995). The pattern of Mokken scales over increasing lowerbounds would then reveal the dimensionality. In particular, a multidimensional scale would be associated with the following typical pattern of Mokken scales with increasing lowerbounds: “(1) most or all items are in one scale; (2) two or more unidimensional scales are formed; and (3) two or more smaller scales are formed and several items are rejected” (Hemker et al., 1995, p. 350). The scales in the second stage reflect the final solution of the so-called separate unidimensional scales. A unidimensional scale would be typically associated with the following pattern with increasing lowerbounds: “(1) most or all items are in one scale; (2) one smaller scale is found; and (3) one or a few small scales are found and several items are rejected” (Hemker et al., 1995, p. 350). The scale in the first stage reflects the unidimensional scale.

The fact that the lowerbound c plays an important role in the partitioning of items into Mokken scales is not surprising. As discussed, Mokken scales are scales that are of maximal length, given a minimal strength of each item in the scale. The lowerbound c tips the balance between item strength and item length, and thus affects the item partitioning. According to Hemker et al. (1995), this range of item partitionings over different c could be indicative of the dimensionality of the data. Because an extensive evaluation of the use of MSA with a range of lowerbounds is beyond the scope of this article, the “range-of-lowerbounds procedure” is examined here briefly.

To illustrate the possible benefits and limitations of the range-of-lowerbounds procedure, the authors have applied it on a number of selected data sets. Note that the procedure requires a visual inspection of the course of item partitionings over increasing lowerbounds, which precludes its implementation in a simulation study. “Easy” and “difficult” conditions are selected, that is, conditions in which the range-of-lowerbounds procedure is expected to perform better than a fixed threshold, and conditions in which the range-of-lowerbounds procedure may give results from which the dimensionality of the data is hard to disentangle.

The results of the range-of-lowerbounds procedure were presented using the MSA-AISP. In a preliminary analysis, this procedure was applied also on MSA-GA. Similar results were observed, albeit MSA-GA has a stronger tendency to join items into the first scale(s). This capacity is completely in line with the objective function of MSA-GA, in which the length of the first scales is heavily rewarded to be long. The authors will now discuss how the MSA-AISP range-of-lowerbounds procedure performs.

First, the condition that appeared relatively difficult for MSA with a fixed boundary in the simulation study, namely, the condition with a low common variance ( $σ_{c}^{2} = 25.0$ ), was considered. For each dimensionality structure, a single data set, with sample size N = 1,000, was drawn. For each of the four example data sets, MSA-AISP was run 6 times, with boundaries ranging from .0 through .5 with steps of .1.

For Examples 1 to 4, the MSA scales for each boundary are presented in Table 2. The judgment of the number of dimensions involved on the basis of the typical patterns for uni- and multidimensionality described by Hemker et al. (1995) are printed in boldface. As can be seen in Table 2, the range-of-lowerbounds procedure indicates the correct structure of the data in Examples 1, 3, and 4, where the data have a unidimensional structure, a three-dimensional structure with uncorrelated factors, and a three-dimensional structure with weakly correlated factors. However, if the factors are strongly correlated, as in Example 2, the range-of-lowerbounds procedure does not seem to offer a solution because it incorrectly indicates the three-dimensional data structure as being two dimensional.

Table 2.

Six Illustrative Examples (Ex.) of the Range-of-Lowerbounds Procedure Using MSA-AISP

		No. of scales	Item numbers per indicated Mokken scale(s) for each boundary value c										Unscalable items
Condition	c	No. of scales	1	2	3	4	5	6	7	8	9	10
Equal item strength
Ex. 1: $σ_{c}^{2} = 25.0$ (1-dim)	0.0	1	1-30
	0.1	1	1-30
	0.2	1	1-30
	0.3	7	28, 30	13, 14	12, 25	21, 22	11, 2	16, 1	10, 15				1-9, 17, 19, 20, 23, 27, 29
	0.4	0											1-30
	0.5	0											1-30
Ex. 2: $σ_{c}^{2} = 25.0$ (3-dim-sc)	0.0	1	1-30
	0.1	1	1-30
	0.2	2	1-10, 22-26, 28-30	11-20									21, 27
	0.3	7	7, 9	15, 19	22, 30	1, 3	23, 2	8, 10	4, 6				2, 5, 11-14, 16-18, 20, 21, 24-26, 28, 29
	0.4	0											1-30
	0.5	0											1-30
Ex. 3: $σ_{c}^{2} = 25.0$ (3-dim-wc)	0.0	2	1-18, 21	19-20, 22-30
	0.1	3	1-18	21-30	19-20
	0.2	3	1-10	11-20	21-30
	0.3	4	3, 10	7, 9	16, 18	4, 6							1, 2, 5, 8, 11-15, 17, 19-30
	0.4	0											1-30
	0.5	0											1-30
Ex. 4: $σ_{c}^{2} = 25.0$ (3-dim-unc)	0.0	3	1-2, 3-8	3, 10, 21-30	11-20
	0.1	3	1-10	11-20	21-30
	0.2	3	1-10	11-20	21-30
	0.3	10	28, 30	7, 9	12, 14	15, 16	23, 2	5-6	1-2	18, 1	3, 10	13, 1	4, 8, 11, 20-22, 24, 26, 27, 29
	0.4	0											1-30
	0.5	0											1-30
Unequal item strength
Ex. 5: $σ_{c}^{2} = 25.0$ (3-dim-unc)	0.0	3	1, 3, 10, 11-20	2, 4-9	26, 28, 30								21-25, 27, 29
	0.1	4	11-20	1-10	28, 30	25, 26							21-24, 27, 29
	0.2	2	11-20	1-10									21-30
	0.3	2	11-20	1-10									21-30
	0.4	2	11-20	1-10									21-30
	0.5	4	11-20	1, 3, 5, 8	4, 9	6, 7							2, 10, 21-30
Ex. 6: $σ_{c}^{2} = 25.0$ (3-dim-wc)	0.0	3	2-5, 9-20, 22-25, 27-30	1, 6-8	21, 26
	0.1	3	1-5, 9-20, 22, 25	6-8	21, 23, 24, 26-30
	0.2	3	11-20	1-10	21-22, 24-25								23, 26-30
	0.3	2	11-20	1-7, 9									8, 10, 21-30
	0.4	3	13, 15	11, 12	1, 3								2, 4-10, 14, 16-30
	0.5	0											1-30

Note: MSA = Mokken scale analysis; AISP = automated item selection procedure; 1-dim = one dimensional; 3-dim-sc = three-dimensional strongly correlated; 3-dim-wc = three-dimensional weakly correlated; 3-dim-unc = three-dimensional uncorrelated. For the conditions with three dimensions, the true partitioning is Items 1 to 10, 11 to 20, and 21 to 30. The numbers in boldface indicate the judgment of the number of dimensions involved on the basis of the range-of-lowerbounds procedure.

Next, the performance of the range-of-lowerbounds procedure in two conditions in which the item strengths are allowed to differ across and within the scales (rather than be equal as in the previous Examples) is illustrated. In Example 5, the condition of a three-dimensional uncorrelated dimensionality structure was considered, with a moderately high overall common variance ( $σ_{c}^{2} = 41.8$ ) in which the items of the first and second scale are relatively strong ( $σ_{c 1}^{2} = 67.5$ and $σ_{c 2}^{2} = 55.0$ , respectively), and the items of the third scale are relatively weak ( $σ_{c 3}^{2} = 3.0$ ). The item strength within a scale also differed, with half of the items loading high and half loading low on the factors: The items in the first scale have communalities of .75 and .60, the items in the second scale have communalities of .60 and .50, and the items in the third scale have communalities of .01 and .05. In Example 6, the condition of a three-dimensional weakly correlated dimensionality structure was considered, with a moderately low overall common variance ( $σ_{c}^{2} = 28.3$ ) in which the items of the first and second scale are moderately strong ( $σ_{c 1}^{2} = σ_{c 2}^{2} = 35.0$ ) and the items of the third scale are moderately weak ( $σ_{c 3}^{2} = 15.0$ ). The item strength within each scale was manipulated, as in Example 5, with items in the first scale having communalities of .4 and .3, with items in the second scale having communalities of .4 and .3, and items in the third scale having communalities of .2 and .1.

In Example 5 (see Table 2), MSA creates only two coherent Mokken scales, corresponding to the first two subscales, of which the items discriminate well. Thus, the range-of-lowerbounds procedure incorrectly indicates a three-dimensional data structure as being two dimensional. Whether the unscalable items do belong to one scale or to multiple scales is not deducible from this result. In Example 6, the range-of-lowerbounds procedure provides a solution that is difficult to interpret. Given the pattern of Mokken scales over increasing c, researchers will probably denote the solution of c = .1 as the final solution and, thus, will incorrectly ascribe too many items of two separate scales as belonging to the same scale. If the solution of c = .2 is interpreted as the final solution, the third scale is very small, and, given the different partitionings of Items 21 to 30 over increasing c, this third scale would be difficult to interpret. In either case, the correct dimensionality structure of the data is hard to deduce from the course of partitionings of Mokken scales over different c.

In sum, the application of MSA with a range of lowerbounds on a limited number of data sets has illustrated that MSA with a range of lowerbounds seems to have some benefits over the use of a fixed lowerbound c. However, MSA with a range of lowerbounds does not solve all limitations of MSA as a dimensionality assessment method, because the general mechanisms of MSA apply also to this procedure. That is, MSA creates Mokken scales, scales that are of maximal length, given a minimal strength of each item in the scale. If, as in real-data applications, the strength of the items differ within and between scales, then the dimensionality structure of the data can be difficult to disentangle from MSA, whether a fixed or a range of lowerbounds is used.

Empirical Example

To illustrate the difficulties in disentangling the dimensionality of a data set on the basis of MSA, MSA was applied on an empirical data set. The data from a study by Koster et al. (2009), who studied the psychometric qualities of the SPQ, were used. The SPQ was designed to assess four key themes of social participation of pupils with special needs in regular primary education. Koster et al. (2009) hypothesized that the four key themes would be represented by four subscales of the SPQ. In addition, the total SPQ scale was hypothesized to reflect social participation in general.

The data of all items of the SPQ were reanalyzed with MSA-AISP and MSA-GA, using a fixed lowerbound of c = .3 and a range of lowerbounds from .0 to .5 with steps of .1. The number of Mokken scales indicated, and the structure and strength of the items were evaluated. Furthermore, a bifactor model (a full hierarchical model; see Chen, West, & Sousa, 2006) was fitted, where each item loads on a general factor and a single specific factor, according to the subscale the items were hypothesized to belong to. The bifactor model was fitted using weighted least squares means and variance adjusted (WLSMV) estimation (suitable for polytomous data) in the Mplus program (Muthén & Muthén, 1998-2007). To determine the model fit of the bifactor model, the root mean square error of approximation (RMSEA; Browne & Cudeck, 1993; Steiger, 1990) and the comparative fit index (CFI; Bentler, 1990) were considered. The prevailing convention is that a RMSEA value of .08 or smaller and a CFI value of .95 or larger indicate an acceptable fit (e.g., Schermelleh-Engel, Moosbrugger, & Müller, 2003).

The MSA-AISP and the MSA-GA with a fixed lowerbound of c = .3 indicated a single Mokken scale with no unscalable items. The H value of this scale was .46, indicating a moderately strong scale. As can be seen in Table 3, the strength of the individual items range from .33 to .59.

Table 3.

The SPQ Items, With Loadings on the General (Λ_g) and Specific Factors (Λ_s) of the Bifactor Model, the H_i Coefficients of the MSA at c = .3, and Partitioning of Items Into Mokken Scales at Different Lowerbounds c.

					Partitioning for each boundary value c
Subscale	Statement in shortened form	Λ _g	Λ _s	H_i at c = .3	c = .0	c = .1	c = .2	c = .3	c = .4	c = .5
Friendships/Relationships	Membership of a group of friends	.84	.32	.55	1	1	1	1	1	1
	Friends in the classroom	.73	.50	.50	1	1	1	1	1	1
	Engagements with classmates to play after school	.72	.61	.51	1	1	1	1	1	1
	Invitations to play during holidays	.71	.52	.51	1	1	1	1	1	1
	Invitations to birthday parties	.79	.33	.53	1	1	1	1	1	1
Contacts/Interactions	Having fun with classmates	.81	−.13	.51	1	1	1	1	1	1
	Exclusion from activities by classmates	.80	.18	.49	1	1	1	1	1	1
	Provocation by classmates	.55	.65	.33	1	1	1	1	3	3
	Being asked to play by classmates	.92	−.06	.59	1	1	1	1	1	1
	Being teased by classmates	.70	.55	.41	1	1	1	1	1	3
	Having a part in a game	.84	−.26	.51	1	1	1	1	1	1
	Classmates laughing at the pupil	.62	.50	.37	1	1	1	1	3	3
	Classmates taking a seat next to the pupil	.79	.07	.50	1	1	1	1	1	1
	Working together on tasks	.77	−.20	.48	1	1	1	1	1	1
Pupil’s Social Self-Perception	Feelings of belonging to the group	.77	.35	.51	1	1	1	1	1	1
	Feelings of loneliness	.70	.48	.45	1	1	1	1	1	4
	Feelings of being able to be oneself	.54	.47	.36	1	1	1	1	4	0
	Feelings of being called names	.69	.27	.42	1	1	1	1	1	3
	Taking pleasure in going to school	.55	.44	.37	1	1	1	1	4	4
Acceptance by Classmates	Willingness of classmates to adapt rules of a game	.59	.46	.42	1	1	1	1	1	2
	Classmates take account of the pupil’s (im)possibilities regarding learning	.47	.64	.35	1	1	1	1	2	2
	Classmates take account of the pupil’s (im)possibilities in choosing games	.52	.73	.39	1	1	1	1	2	2
	Classmates stand up for the pupil	.65	.37	.45	1	1	1	1	1	2
	Willingness of classmates to assist the pupil	.63	.32	.42	1	1	1	1	1	0
				No. of scales:	1	1	1	1	4	4

Note: SPQ = Social Participation Questionnaire; MSA = Mokken scale analysis. Unscalable items are indicated with 0. Loadings >.30 in absolute value are indicated in boldface.

The range-of-lowerbounds procedure indicated a partitioning to which a unidimensional pattern and a multidimensional pattern appear to apply. As can be seen in Table 3, with increasing c, (a) all items are in one scale (up to c = .3), (b) four scales are formed (of which one larger one; with c = .4), and (c) four smaller scales are formed, and several items are rejected (with c = .5). This fit the description of multidimensionality, with the partitioning associated to the one of c = .4 as the final solution.

Confusingly, this partitioning also fits with the description of unidimensionality, as “(1) most or all items are in one scale (up to c = .3); (2) one smaller scale is found (with c = .4); and (3) one or a few small scales are found and several items are rejected (with c = .5)” (Hemker et al., 1995, p. 350). This would indicate one dimension, with the partitioning associated to the one of c = .0, hence including all items, as the final solution. Because the partitioning fits a unidimensional pattern and a multidimensional pattern, the authors conclude that the dimensionality structure of the data cannot be deduced from the range-of-lowerbounds procedure.

The bifactor analysis revealed that the bifactor model considered fitted adequately (RMSEA = .069, CFI = .974). Table 3 shows the items of the SPQ with their factor loadings on the general and specific factors of the bifactor model. As can be seen in Table 3, all items of the SPQ load moderately high on the general factor (loadings ranging from .47 to .92). This suggests that the items measure a common trait, albeit to a different extent. Furthermore, the loadings of the items on their specific factor range—in absolute value—from .06 to .73, suggesting that the items differ widely in their capability to measure a specific key theme. As can be seen in Table 3, the loadings on the specific factor for the key theme “Contacts/Interactions” are rather low (i.e., <.30 in absolute value) for six out of nine items, implying that those six items do not measure anything substantial beyond the common trait. For the remaining three key themes, the loadings on their specific factors are moderate to large (>.27), suggesting that those key themes can be sensibly distinguished from the common trait. On the basis of the bifactor analysis, the authors conclude that the SPQ–Revised measures one general trait, social participation, and, in addition, that the four key themes of social participation can be sensibly distinguished.

General Discussion

The aim of this study was to evaluate how well MSA performs as a dimensionality assessment method. To this end, the authors performed a simulation study in which they examined for a range of conditions how well MSA reveals the correct number of dimensions and the correct dimensionality structure. Their results indicate that MSA does not necessarily create scales that reveal the dimensionality of a data set. Rather, MSA creates Mokken scales: scales that are of maximal length, given a minimal strength of each item in the scale. This implies that a single Mokken scale can reflect unidimensional data and multidimensional data with strong items. Analogously, multiple Mokken scales can reflect multidimensional data and unidimensional data with weak items. As one does not know whether empirical data complies with the conditions in which MSA generally performs well, the empirical value of MSA as a dimensionality assessment method appears to be limited.

The alternative application of MSA with a range of lowerbounds seems to be a good alternative if the factors are not strongly correlated and the items do not differ substantially in item strength. If the items considerably differ in item strength, this application does not seem to be beneficial. The authors did consider only a limited number of data sets, and further research could reveal in more detail the benefits and limitations of the range-of-lowerbounds procedure.

In this study, the authors considered only a constrained variant of the bifactor model, namely, one in which the factor loadings of the bifactor model could be rotated to a model without a general factor, but with correlated factors (here a correlated three-factor model). This was done to ease the interpretation of the results and to make a connection to earlier studies (e.g., Mroch & Bolt, 2006; van Abswoude et al., 2004). However, the current findings do generalize to more complicated bifactor models, as the authors verified in a small simulation study.

This study indicated that MSA is not generally applicable as a dimensionality assessment method. This result gives rise to the question what alternative dimensionality assessment methods are. If one wishes to stay with nonparametric methods, possible alternatives are DIMTEST (Nandakumar & Stout, 1993; Stout, 1987; Stout, Goodwin Froelich, & Gao, 2001), DETECT (Zhang & Stout, 1999b), or HCA/CCPROX (Roussos, Stout, & Marden, 1998). More research should be conducted on these different methods to indicate which of these nonparametric alternatives are most viable for dimensionality assessment.

If empirical data would appear to comply to a reasonable extent with a parametric model, the authors would advocate the use a parametric model for dimensionality assessment. The use of parametric models has the advantage that it gives a sparse representation of the dimensionality structure of the data. In addition, the use of a parametric model has the advantage that it gives an estimation of the uncertainty of the model estimates. An example of a useful parametric model is a bifactor model, which is currently very popular in the personality and psychopathology domain (e.g., Reise et al., 2007). Because of the intimate relationship between factor models and IRT-based models (see, for example, Takane & de Leeuw, 1987), factor models such as the bifactor structure can be used in the general multidimensional item response theory (MIRT)–based models as well.

Footnotes

Acknowledgements

The authors would like to thank Marloes Koster, Sip Jan Pijl, and Els J. van Houten for sharing their data.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bech

Fava

Trivedi

M. H.

Wisniewski

S. R.

Rush

A. J.

(2011). Factor structure and dimensionality of the two depression scales in STAR*D using level 1 datasets. Journal of Affective Disorders, 132, 396-400. doi:10.1016/j.jad.2011.03.01110.1016/j.jad.2011.03.011

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. doi:10.1037/0033-2909.107.2.23810.1037/0033-2909.107.2.238

Browne

M. W.

Cudeck

(1993). Alternative ways of assessing model fit. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.

Chen

Tseng

Koh

(2010). Pediatric evaluation of disability inventory: A cross-cultural comparison of daily function between Taiwanese and American children. Research in Developmental Disabilities, 31, 1590-1600. doi:10.1016/j.ridd.2010.05.00210.1016/j.ridd.2010.05.002

Chen

F. F.

West

S. G.

Sousa

K. H.

(2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189-225. doi:10.1207/s15327906mbr4102_510.1207/s15327906mbr4102_5

Doyle

Conroy

McGee

Delaney

(2010). Depressive symptoms in persons with acute coronary syndrome: Specific symptom scales and prognosis. Journal of Psychosomatic Research, 68, 121-130. doi:10.1016/j.jpsychores.2009.07.01310.1016/j.jpsychores.2009.07.013

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.

Emons

W. H. M.

Sijtsma

Pedersen

S. S.

(2010). Dimensionality of the Hospital Anxiety and Depression Scale (HADS) in cardiac patients: Comparison of Mokken scale analysis and factor analysis. Assessment. Advance online publication. doi:10.1177/107319111038495110.1177/1073191110384951

Hemker

B. T.

Sijtsma

Molenaar

I. W.

(1995). Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model. Applied Psychological Measurement, 19, 337-352. doi:10.1177/01466216950190040410.1177/014662169501900404

10.

Holzinger

K. J.

Swineford

(1937). The bi-factor method. Psychometrika, 2, 41-54. doi:10.1007/BF0228796510.1007/BF02287965

11.

Junker

B. W.

Sijtsma

(2001). Nonparametric item response theory in action: An overview of the special issue. Applied Psychological Measurement, 25, 211-220. doi:10.1177/0146621012203202810.1177/01466210122032028

12.

Koster

Timmerman

M. E.

Nakken

Pijl

S. J.

van Houten

E. J.

(2009). Evaluating social participation of pupils with special needs in regular primary schools: Examination of a teacher questionnaire. European Journal of Psychological Assessment, 25, 213-222. doi:10.1027/1015-5759.25.4.21310.1027/1015-5759.25.4.213

13.

Meijer

R. R.

de Vries

R. M.

van Bruggen

(2011). An evaluation of the Brief Symptom Inventory–18 using item response theory: Which items are most strongly related to psychological distress? Psychological Assessment, 23, 193-202. doi:10.1037/a002129210.1037/a0021292

14.

Mokken

R. J.

(Ed.). (1971). A theory and procedure of scale analysis. The Hague, Netherlands: Mouton.

15.

Molenaar

I. W.

Sijtsma

(2000). User’s manual. MSP5 for Windows: A program for Mokken scale analysis for polytomous items. Groningen, Netherlands: iec ProGAMMA.

16.

Mroch

A. A.

Bolt

D. M.

(2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19, 67-91. doi:10.1207/s15324818ame1901_410.1207/s15324818ame1901_4

17.

Muthén

L. K.

Muthén

B. O.

(1998-2007). Mplus user’s guide (5th ed.). Los Angeles, CA: Author.

18.

Nandakumar

Stout

W. F.

(1993). Refinements of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18, 41-68. doi:10.2307/116518210.2307/1165182

19.

Ordoñez

X. G.

Ponsoda

Abad

F. J.

Romero

S. J.

(2009). Measurement of epistemological beliefs: Psychometric properties of the EQEBI test scores. Educational and Psychological Measurement, 69, 287-302. doi:10.1177/001316440832322610.1177/0013164408323226

20.

Paas

L. J.

Sijtsma

(2008). Nonparametric item response theory for investigating dimensionality of marketing scales: A SERVQUAL application. Marketing Letters, 19, 157-170. doi:10.1007/s11002-007-9031-010.1007/s11002-007-9031-0

21.

Rand

W. M.

(1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66, 846-850.

22.

R Development Core Team. (2011). A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

23.

Reckase

M. D.

(Ed.). (2009). Multidimensional item response theory. New York, NY: Springer.

24.

Reise

S. P.

Morizot

Hays

R. D.

(2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 16, 19-31. doi:10.1007/s11136-007-9183-710.1007/s11136-007-9183-7

25.

Roorda

L. D.

Houwink

Smits

Molenaar

I. W.

Geurts

A. C.

(2011). Measuring upper limb capacity in poststroke patients: Development, fit of the Monotone Homogeneity Model, unidimensionality, fit of the Double Monotonicity Model, Differential Item Functioning, internal consistency, and feasibility of the Stroke Upper Limb Capacity Scale, SULCS. Archives of Physical Medicine and Rehabilitation, 92, 214-227. doi:10.1016/j.apmr.2010.10.03410.1016/j.apmr.2010.10.034

26.

Roussos

L. A.

Stout

W. F.

Marden

J. I.

(1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1-30. doi:10.1111/j.1745-3984.1998.tb00525.x10.1111/j.1745-3984.1998.tb00525.x

27.

Schermelleh-Engel

Moosbrugger

Müller

(2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research, 8(2), 23-74.

28.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF0228920910.1007/BF02289209

29.

Sijtsma

Meijer

R. R.

van der Ark

L. A.

(2011). Mokken scale analysis as time goes by: An update for scaling practitioners. Personality and Individual Differences, 50, 31-37. doi:10.1016/j.paid.2010.08.01610.1016/j.paid.2010.08.016

30.

Sijtsma

Molenaar

I. W.

(2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.

31.

Sousa

R. M.

Dewey

M. E.

Acosta

Jotheeswaran

A. T.

Castro-Costa

Ferri

C. P.

. . . Prince

M. J.

(2010). Measuring disability across cultures—The psychometric properties of the WHODAS II in older people from seven low- and middle-income countries. The 10/66 Dementia Research Group population-based survey. International Journal of Methods in Psychiatric Research, 19, 1-17.

32.

Steiger

J. H.

(1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180. doi:173-180.1991-03242-001173-180.1991-03242-001

33.

Stout

W. F.

(1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 293-325.

34.

Stout

W. F.

(2001). Nonparametric item response theory: A maturing and applicable measurement modeling approach. Applied Psychological Measurement, 25, 300-306. doi:10.1177/0146621012203210910.1177/01466210122032109

35.

Stout

W. F.

Goodwin Froelich

Gao

(2001). Using resampling methods to produce an improved DIMTEST procedure. In Boomsma

van Duijn

M. A. J.

Snijders

T. A. B.

(Eds.), Essays on item response theory (pp. 357-375). New York, NY: Springer.

36.

Straat

J. H.

van der Ark

L. A.

Sijtsma

(in press). Comparing optimization algorithms for item selection in Mokken scale analysis. Journal of Classification. Retrieved from: https://spitswww.uvt.nl/~avdrark/research/research.htm

37.

Takane

de Leeuw

(1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393-408.

38.

van Abswoude

A. A. H.

van der Ark

L. A.

Sijtsma

(2004). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28, 3-24. doi:10.1177/014662160325927710.1177/0146621603259277

39.

van Abswoude

A. A. H.

Vermunt

J. K.

Hemker

B. T.

(2007). Assessing dimensionality by maximizing H coefficient-based objective functions. Applied Psychological Measurement, 31, 308-330. doi:10.1177/014662160629519610.1177/0146621606295196

40.

van der Ark

L. A.

(2007). Mokken scale analysis in R. Journal of Statistical Software, 20, 1-19.

41.

van der Ark

L. A.

(2010). Getting started with Mokken scale analysis in R. Retrieved from: http://spitswww.uvt.nl~avdrark/research/mokkenstart.pdf

42.

Zhang

Stout

W. F.

(1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152. doi:10.1007/BF0229453210.1007/BF02294532

43.

Zhang

Stout

W. F.

(1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.

44.

Zijlstra

W. P.

van der Ark

L. A.

Sijtsma

(2011). Outliers in questionnaire data: Can they be detected and should they be removed? Journal of Educational and Behavioral Statistics, 36, 186-212. doi:10.3102/107699861036626310.3102/1076998610366263