An SEM Algorithm for Scale Reduction Incorporating Evaluation of Multiple Psychometric Criteria

SEM and Scale Development

Across a variety of disciplines, SEM and confirmatory factor analysis (CFA) have long played an important role in scale development and evaluation. SEM-based criteria can be used to assess classical psychometric ideas, including criterion validity (MacCallum and Austin 2000), discriminant validity (Henseler, Ringle, and Sarstedt 2014), reliability (Miller 1995), and dimensionality (McGartland Rubio, Berg-Weger, and Tebb 2001), in addition to proposed structural relationships between related constructs. Noar (2003) provides a useful overview of the process of scale development through the lens of SEM, describing several stages: defining latent constructs and a theoretical framework, determining the scale purpose and length, developing an initial item pool, locating a relevant sample, selecting items based on statistical criteria and construct validity, and finally evaluating the properties of the final scale. The demonstration of SEM in scale development provided by Noar (2003) is typical of the literature in emphasizing the role of SEM in the final stage (scale evaluation) but neglecting the potential use of SEM for the second to last step (item selection). For example, in the example provided by Noar (2003), item selection was done using a classical approach, that is, manually and on the basis of a number of criteria: mean score, skewness, kurtosis, factor loadings, and item content. Eliminating items manually, while also attempting to simultaneously optimize these multiple criteria, is necessarily an ad hoc process. Graduate texts in psychometrics, such as Furr (2011), also tend to present SEM exclusively as a means to assess a scale once the candidate items have been determined and describe no role for SEM in the item selection process itself.

Automated Algorithms in SEM and Item Selection

Although the use of automated algorithms for specifying SEMs is not widespread, some researchers have applied machine learning or data mining methods for the task of exploring a large space of possible models. Marcoulides, Ing, and Hoyle (2012) provide an overview of the application of several algorithmic optimization methods for SEM. These include simulated annealing and genetic algorithms (GAs) as well as related procedures such as ruin and recreate, Tabu search, and ant colony optimization (Marcoulides and Drezner 2003). While algorithms vary, they share the common feature of using nongradient-based methods to search a discrete space of possible models, with the aim of minimizing a given criterion, such as the noncentrality parameter. No studies could be located which used automated SEM algorithms for scale development or item selection. However, Yarkoni (2010) presents a promising GA-based search approach to short-form development, based on optimizing the ability of the short form to capture variability of the full-length scale. Lastly, Mokken (1971) analysis, an item response theoretic (IRT)-based approach to item evaluation, incorporates an automated bottom-up stepwise item selection procedure, which has since been augmented by a GA-based search methods (Straat, van der Ark, and Sijtsma 2013).

Applying SEM for Item Selection

SEM provides a flexible and unified methodology for a researcher to specify the expected relationships between observed measurements, including both measurement of constructs and structural relationships between constructs. Diagnostic information from a fitted model provides complete information regarding discrepancies between the model and the data. Although this information is usually used to decide among alternative models (using fixed data), how it can be used to decide among alternative indicators or items (using a fixed model) will be described. Finally, the convergence between SEM and IRT (Bentler 2008; Cai 2008; Glockner-Rist and Hoijtink 2003; Lu, Thomas, and Zumbo 2005) arguably not only removes any impediment to using SEM to evaluate binary or polytomous self-report items but also offers better estimates of psychometric constructs than classical methods (e.g., α; see Yang and Green 2011). We will consider whether scale items might be selected purely based on terms derived SEM misfit, in place of one or more specific measures of reliability or validity. To explore this, data from a concrete case study will be used to relate different psychometric criteria to the residual variance–covariance matrix.

To illustrate this application, the “4Es” questionnaire will be shortened and generalized. The 4Es questionnaire was originally designed to assess psychological trait risk for development of problem gambling (Rockloff and Dyer 2006, 2007) and consists of four related components of psychological risk: a desire to escape—capturing a desire to avoid or escape life problems and responsibilities, lack of self-esteem—feelings of worthlessness, shame, and self-loathing, excess—primarily a lack of caution and deliberation, and the need for excitement—a tendency toward boredom and restlessness. The original 40-item Likert-type scale (10 items per subscale, 5 points per item) was generated from an initial 240-item pool and was developed in an extensive documented process, resulting in a scale with good construct validity, internal consistency, and criterion validity for predicting gambling problems (Rockloff and Dyer 2006, 2007). Although not an investigative focus of the original studies, the scale was also useful in predicting other forms of addiction: particularly substance use, alcohol, and nicotine. Figure 1 summarizes the SEM corresponding to the theoretical model for this scale. For ease of presentation, only the first 3 items of each subscale are shown, and correlated error terms between the addiction criterions are omitted. This relatively simple application nevertheless provides a useful means to summarize how SEM misfit corresponds with several classical psychometric properties. In the following discussion, it is asserted that (a) parameters shown in Figure 1 are freely estimated using a sample data set; (b) methods appropriate to ordinal data are applied: for example, polychoric correlations or robust weighted least squares estimation (Flora and Curran 2004; Lei 2007; Muthén, Du Toit, and Spisic 1997; Schumacker and Beyerlein 2000), and (c) candidate scale items have been screened for possessing adequate construct validity.

OPEN IN VIEWER

Figure 1.

Measurement and structural model of the 4Es scale with criterion behavioral measures.

SEM Residuals, Homogeneity, and Discriminant Validity

We shall first describe how SEM misfit attributable to an item, in terms of residuals between the observed and fitted covariance matrix, is an indication of poor item fit functioning. Reasons for poor item fit have been well explored and include similar item wording, logical dependencies between items, overlapping content, and unspecified hierarchical or multidimensional structure (Heene et al. 2012). Regardless of the cause, residual covariances tend to fall into several categories. First, residuals between indicators of the same scale, shown by the link “A” in Figure 1, are an indication of factor heterogeneity or unwarranted logical, conceptual, or methodological overlap between items. Ideally, covariance between items would be completely explained by their shared contribution to the latent construct, errors would be independent, and this residual term would be zero. Second, residuals of the type indicated by “B” indicate unexpected relationships between items from separate constructs, which correspond to a violation of discriminant validity. Ideally, items should measure only the construct to which they are attached, and covariance between items in different subscales would be fully explained by intermediate structural model links—in the present example, their shared contribution to psychological risk. Third, residuals of type “C” indicate another form of discriminant validity violation: in terms of unexpected relationships between scale items with the criterion or other measures. To illustrate a situation in which this might occur using our case study, a hypothetical excess item, “I never think about the morning-after when partying with friends,” might display positive residual covariance with the criterion alcohol due to implicit connotations of alcohol consumption contained in the item. This is one example of how standardized SEM covariance residuals would appropriately penalize a positive correlation in the data that less sophisticated methods would be incorrectly treated as contributing positively to criterion validity. Each of the residual terms discussed above corresponds to discrepancies between model reproduced σ ^ i j and observed s_ij covariance: these form the basis of the SEM omnibus fit measure standardized root mean square residual (SRMR) given p observed variables in the model.

SRMR = { 2 ∑ i = 1 p ∑ j = 1 i [ ( s i j − σ ^ i j ) / ( s i i s j j ) ] 2 } / p ( p + 1 ) .

SEM Error Terms, Reliability, and Criterion Validity

The points made above illustrate how the SEM diagnostics capture divergence between the observed and expected (i.e., model specified) relationships with other measures, capturing factor heterogeneity and discriminant validity in a broad sense. However, like factor analysis, an SEM fit is based on the common variance of observed variables: Unique item variance is treated in a separate model term and do not contribute to the residual correlation matrix. This corresponds to a basic distinction between error-based and model-based fit criteria. If item performance based on the expected covariance were the only criteria of interest, then we might select items based on optimizing the SRMR or similar fit measure. However, we likely also wish to take into account the correlation of items with the latent factor (classical reliability), and the degree to which their inclusion leads to the estimation of a latent factor that explain greater variation in the criterion measures. With a fitted SEM, one way to do this is to also take into account the unique error variance terms of scale items δ and the criterion measures ε. The argument for this follows from the definitions: δ are a negative indicator of the proportion of variance shared between an item and the latent variable, and ε are a negative indicator of the variance explained in the criterion measures. We note that for any given item, the SRMR, ε, and δ all depend on the mix of other items in the scale. For example, the magnitude of the unique item variance depends on variance it shares with other items in the scale. This leads to a combinatorial problems and local minima for any attempt to optimize these quantities. For such multivariate applications, this is inevitable: Stepwise multiple regression or stepwise item removal based on “alpha if item removed” also faces this issue. In considering both the residual correlation matrix and the unique diagonal matrix, both classical and common factor approaches play a role. The present article makes no attempt to engage in the long-standing theoretical debate between classical and common factor approaches to evaluating scales. Rather, it aims to provide a practical approach for a pragmatic scale developer who wishes to select items that both possess covariance that conform to model-implied expectations and contribute to classical reliability of measured constructs and predictability of external measures.

Toward SEM-based Item Evaluation

In order to take into account each component of reliability and validity accessible from a fitted SEM, we would tend to prefer items that:

are homogenous and possess good discriminant validity, corresponding to a covariance structure that conforms to the model: that is, minimize SRMR;

Each of these diagnostics is accessible from a fitted SEM that includes the item considered for deletion. While the manner of combining these three criteria into a single loss function is bound to be somewhat arbitrary, we suggest that a scale-invariant aggregation—described below—is suitable for most applications. We can imagine a set of fitted models with fixed structure and fixed number of indicators N − 1 from a set of N indicators. Importantly, the only difference between models in this set is the choice of item that is dropped. Assuming δ and ε are standardized to unit total variance, the model with the best diagnostics corresponds to a dropped indicator with the worst performance. This represents a special case in which violating normal SEM practice—involving fixed data and alternative models—is meaningful, since we implement a comparison of fixed models, where one column of data differs in each pairwise comparison.

To conclude with some cautionary comments, the proposed method is designed specifically for item-set reduction. It is not proposed to substitute for normal practices in development of the initial item pool, including checking items for construct validity. A poor initial item pool will almost certainly lead to a poor solution. Next, a stepwise approach to item-set reduction does not explore the complete space of possible item combinations and is not guaranteed to find the optimal item subset. Finally, it does not substitute for the standard practice of detailed testing and reporting regarding objective performance of the reduced scale. Given the model flexibility inherent in choosing the best N from M > N indicators, it should be assumed that diagnostics will be inflated, and validation of the final items should be done using a separate test data set before publication of the new scale.

Loss Function for Evaluating Item Subsets

Where data comprising the mth indicator are selected from a candidate pool of items, we suggest an SEM-based loss function for evaluating substitutions of indicator variables when comparing models of fixed typology. As per standard SEM methodology, it requires a theoretical model that incorporates a measurement component, and (optionally) structural, indicators of external convergent or discriminant validity. We desire a loss function that simultaneously measures both errors in the covariance structure and the variance structure: that is, unexplained variance in the indicators and criterion measures. A reasonable—though admittedly arbitrary—loss function is given by the product of three terms:

L = ( ∑ k δ k / K ) q ⋅ ( ∑ m ε m / m ) r ⋅ SRMR,

assuming δ and ε are standardized such that items possess unity variance. If q = r = 1, this treats all three aspects of item performance as equally important, such that a proportional improvement in each term contributes equally to the loss function. Researchers may vary the power terms to suit the priorities of a given scale development application. In situations where criterion variables are not available, the ε term may be removed. In special situations where the SEM is not identified, the SRMR will zero, and the term should be removed to permit comparison on the other two criteria.

Algorithm for Item-set Reduction

We define the goal of the analysis to reduce the number of items in each subscale from the number in the original pool of M items to N ^min. Given that the model architecture and candidate indicator data are fixed, the loss is a function only of the choice of indicator variables:

L = f ( k ) , k = [ k 1 k 2 . . . k N ] ,

where k is an integer index to the pool of candidate items, with 3 ≤ N ^min < N < M, and no replicated items k_i ≠ k_j for all i ≠ j. Given a reasonably sized item pool, an exhaustive search of item combinations is impractical. We propose a backward-stepwise algorithm that progressively eliminates items based on preserving the loss function. At the first step, k is the index to the full set of candidate items. At each subsequent step, the algorithm updates k such that:

k * ← k = arg min ( L ( k i ) ) , k i = k \ k i .

This means removing the item that minimizes the loss function from the current pool of items. Importantly, all pairwise model comparisons are between models with an identical architecture, using standardized error variance estimates, and possess an identical number of normalized indicators. The only pairwise differentiating feature is in the choice of k_i —the candidate item to be removed. For applications involving simultaneous reduction of multiple subscales, such as the present one, we suggest a round-robin approach such that each subscale is reduced in size at the same rate. The algorithm should be stopped at a minimum of 3 items, ¹ to satisfy fundamental SEM requirements for latent variable estimation. However, model fit criteria can be monitored during the item-set reduction process, allowing them to guide the researcher as to how many items to retain.

Miniature Simulations

We tested the sensitivity of the SRMR-based metric in capturing three classical measures of model misspecification: factor heterogeneity, discriminant validity, and factor reliability. The simulation results presented were based on a single framework for generating small artificial correlation matrices to demonstrate the relationship between SEM fit measures and classical psychometric constructs. Correlation matrices for six hypothetical indicators were simulated, varying the factor structure on a continuum between a unidimensional (mixing parameter u = 1) and two-factorial (u = 0) structure, with half of the items loadings on each factor. For u = 1, correlations between items 1–3 and 4–6 were similar to those within each 3-item subset. For u = 0, correlations across each split-half were zero. For each simulation reported below, we independently varied item heterogeneity: that is, variation in factor loadings, with the following options for differences between loadings: h ↔ = { 0 , 0.1 , 0.2 } . Given a fixed mean factor loading of h = 0.7, the three loadings for each subset of items were none h_i = {0.7, 0.7, 0.7}, low h_i = {0.6, 0.7, 0.8}, and medium h_i = {0.5, 0.7, 0.9}. Two illustrative examples of the resulting heterogeneous and homogenous correlation matrices are given below: R ₁ corresponds to a clear two-factor structure with moderate item loading variation, whereas R ₂ corresponds to a more homogenous unifactorial structure.

R 1 ( u = 0.1 , h = 0.7 , h ↔ = 0.2 ) = [ 1 .35 1 .45 .64 1 .05 .07 .09 1 .07 .10 .13 .35 1 .09 .13 .16 .45 .64 1 ]

R 2 ( u = 0.7 , h = 0.7 , h ↔ = 0.1 ) = [ 1 .63 1 .72 .83 1 .50 .59 .67 1 .59 .69 .78 .63 1 .67 .78 .90 .72 .83 1 ] .

Factor Heterogeneity

The first simulation analysis addressed SRMR for the purpose of capturing factor heterogeneity; that is, a nonsingleton factor structure among a set of items purportedly representing a single construct. The mean item factor loading h was held constant at 0.7, while u and h ↔ were allowed to vary. We evaluated the set of correlation matrices using Revelle’s β (Revelle 1979), a classical measure of scale heterogeneity. A simple SEM was fitted to each correlation matrix, with all items loading on a single factor. In this toy example, the SRMR is entirely comprised of residuals between items specified to indicate the same model-specified construct and therefore are all of type “A” (refer Figure 1). A perfectly homogenous model is one in which all covariance between the indicators is captured by the latent factor. Figure 2 (a) shows a very similar and approximately linear relationship of SRMR to β for varying degrees of item heterogeneity.

OPEN IN VIEWER

Figure 2.

SEM residual metrics SRMR and mean indicator error variance compared to classical psychometric indicators: factor heterogeneity (a), discriminant validity (b), and factor reliability (c).

Discriminant Validity

We used the same simulation scheme to demonstrate the relationship between SRMR and discriminant validity. The model in this case specified items 1–3 and items 4–6 loading on separate factors, with zero-correlation between factors; thus corresponding to a measurement model of unrelated constructs. In this scheme, u is a direct measure of discriminant validity, with u = 0 corresponding to ideal and u = 1 corresponding to very poor discriminant validity. Figure 2 (b) shows a near linear relationship of SRMR with diverging item-construct correlations. This result corresponds to residuals of type “B” and “C” from Figure 1.

Factor Reliability

In the final simulation analysis, we again fit a unifactorial model. However, factor heterogeneity was fixed at u = 0.7, while mean item factor loadings h was allowed to vary between 0.3 and 0.6. In an SEM framework, reliability is indicated by indicator error variance, while among classical techniques, α is still commonly used. Figure 2 (c) compares the mean indicator error variance with α, again showing a near linear relationship between the classical and model-based measure.

Algorithm and Loss Function Efficiency

The goal of the algorithm is to detect and select better quality scale items from a larger set. In order to test the overall efficiency of the algorithm, and the loss function on which it operates, we simulated data using items of varying item quality. For convenience of exposition, we generated data using the same scenario as for the real data and model described in Figure 1: four subscales (Labeled A, B, C, D) with 10 items each and four measures (V1, V2, V3, and V4) of external validity (N = 2,107). Items 1–3 were ideal, in the sense of possessing loadings of unity on the latent constructs, while items 4–7 possessed varying degrees of misfit—varying in terms of reliability, factor heterogeneity, split loadings, and unexpected correlations (both positive and negative) with the validity measures. The population data generation model and the misfit characteristics of each item are most succinctly summarized using the commented model specification code using the R-package lavaan (Rosseel 2012), provided in Online Appendix 1. Two hundred sample polychoric correlation matrices and algorithm outputs were generated, with the final item selections (3/10 per subscale) recorded. The number of times each item was selected by the algorithm is shown in Figure 3. We also recorded the final value of the sample loss function L at algorithm termination (L′) as well as the sample loss function, given the population ideal solution (L*); that is, selection of items 1–3 for each subscale. One hundred ninety-seven times of 200, L′ was less than L*. From this, we may conclude that variation in item selection, at least for this simulated scenario, was overwhelmingly due to sampling variability, rather than algorithmic variability.

OPEN IN VIEWER

Figure 3.

Frequency of item selection by the stepwise backward algorithm on 200 simulated data sets. Refer Online Appendix 1: quality of simulated indicator decreases with item index.

Application: 4Es Data Set

The conceptual background to the 4Es scale has already been described. Uptake of the full 40-item scale has been limited due to its length, and a candidate short form was desired that could serve as a brief screen for psychological risk for addiction, which nevertheless still tapped each of the four constructs in the original scale. The data set of 2,107 respondents originally used to develop and validate the full-length instrument was analyzed; full methodological details of data collection can be found in Rockloff and Dyer (2006). The original instrument was intended for use in diagnosing risk for gambling problems, assessed using the Canadian Problem Gambling Index of Severity (Ferris and Wynne 2001) and validation by the authors focused on gambling as the primary measure of criterion validity. However, data were also gathered regarding three other forms of addictions: nicotine (Heatherton et al. 1991), illicit drugs (Gossop et al. 1995), and alcohol (Saunders et al. 1993). Each of the four outcome measures is well validated and accepted in the addiction literature. An evaluation of the 4Es as a measure of general psychological risk for addiction, as opposed to gambling-related risk, had not been done prior to the present study. Therefore, the goals of the analysis were twofold:

To assess the original instrument in terms of the structural model as an indicator of general psychological risk for addiction.

The algorithm was applied to the original 40-item (10 per subscale) instrument using the SEM specification shown in Figure 1 and stopped when 3 items per subscale remained. Table 1 compares psychometric criteria for evaluating the performance of the instrument. As α is sensitive to the number of items in the scale, Spearman–Brown (Schmitt 1996) corrected versions are given; this standardizes the reliability estimates so as to infer α given a fixed number (10 in the present case) of items. Thus, the first section of Table 1 indicates that the item-level reliabilities of the reduced scale were generally higher than the original scale. However, the scale-level reliabilities are still somewhat poor. This suggests we may be well advised to consider increasing the length of the target subscales to 4 or 5 items. Table 1 also shows the ratio of the smallest to the largest factor loading: A τ-equivalent set of items would possess a ratio close to one. The reduced scale yielded latent factor score estimates that were slightly less correlated with one another than the original scale, which better conforms to the SEM specification of separate constructs, and the goal of assessing four distinct psychological constructs. Regressions on the criterion measures suggest that the original scale was a valid indicator of risk for addiction generally, predicting drug and alcohol dependence equally well as gambling problems, and nicotine somewhat less so. The reduced scale possessed slightly smaller, but very similar, correlations to each of the addiction screens. Finally, the reduced scale yielded slightly improved overall SEM fit measures—again these should be interpreted with caution, keeping in mind the marked reduction in degrees of freedom caused by dropping items.

OPEN IN VIEWER

Table 1.

Comparison of Original (10 Items per Subscale) and Reduced (3 Items per Subscale) Model Parameters and Summary Statistics.

		Original (10 Item)	Reduced (3 Item)
α
	Escape	.83	.83 (.89)a
	Esteem	.73	.71 (.82)a
	Excess	.79	.72 (.90)a
	Excite	.76	.64 (.86)a
ω (McDonald’s ω t )
	Escape	.85	.79
	Esteem	.78	.67
	Excess	.84	.81
	Excite	.75	.72
Min/max factor loading
	Escape	.69	.89
	Esteem	.73	.76
	Excess	.56	.77
	Excite	.52	.69
Mean subscale covarianceb		.54	.51
Regressions on criterionc
	Drugs	.46	.43
	Gambling	.45	.43
	Nicotine	.26	.25
	Alcohol	.43	.40
SEM fit indices
	df	895	97
	SRMR	.052	.050
	IFI	.791	.944

Note: IFI = incremental fit index; SEM = structural equation modeling; SRMR = standardized root mean square residual.

^aSpearman–Brown correction to make α invariant to test length: standardized to 10 items to match original scale.

^bCalculated as the mean square correlation of the predicted latent scores of each 4E subscale.

^cStandardized regression coefficients. All significant at p < .001 criterion.

Table 2 shows the final 12-item set selected by the algorithm. The items selected had good construct validity with respect to the underlying constructs. The items selected correspond to purely psychological concepts and have no construct overlap with the behavioral addiction measures.

OPEN IN VIEWER

Table 2.

Reduced Item Set Selected by the Algorithm.

Subscale	#	Item
Escape	4	I feel like I’m already living in a prison
	5	I wish I could take the next flight or bus out of town
	3	I wish that nobody knew who I was
Esteem	5	I feel completely worthless
	2	I am miserable to be around
	1	The things I say and do are foolish
Excess	1	I am careful in my decision making (R)
	3	I carefully think about all my options before acting (R)
	4	Before deciding to do something important, I will thoroughly think it through (R)
Excite	1	There are times I get bored with day-to-day life
	5	Often times I find myself feeling restless
	7	I am nearly always looking for things to do that keep me from being bored

Note: R = reverse scored.

Limitations and Future Work

We have consciously avoided several issues in proposing the current method. The first limitation is the use of stepwise item-set reduction algorithm. This algorithm is “greedy,” in the sense of optimizing the cost function at each step, but not being guaranteed to converge to a global minima. Other legitimate choices of algorithm are possible. One alternative we tested involves drawing a very large number of random subsets of items (of fixed size) from the initial set. This implements a random uniform probe of the space of possible combinations. Also tested was a simple stochastic steepest descent algorithm. Other options include GAs or simulated annealing. Future work might profitably explore alternative optimization methods. While we do not report a systematic test of alternative algorithms, our intuition is the stepwise item-elimination approach, which incorporates information from the initial full item set for as long as possible, is more likely to find an item subset that well covers the domain of the original construct.

A more fundamental concern is that the item removal algorithm described progressively discards potentially useful information from eliminated items. The risk is that, as the retained item set grows smaller, the latent factor may become misspecified. An alternative perspective is to assume the latent variable derived from the full set represents the ground truth, and the objective becomes approximating this using a smaller item set. This may be a better option when the researcher is confident that the full item set well describes the construct of interest, and other sources of validity are not available.

As has been emphasized, the proposed loss function is somewhat arbitrary in that it does not provide guidance on the relative weighting of the three components of psychometric performance. It relies on the assumption that it is practical and meaningful to simultaneously optimize model fit, predictive utility, and error variance (one-step modeling). Some might argue that it is more meaningful to first optimize measurement error and then optimize other criteria. However, an equally valid view is that this unreasonably prioritizes reliability over other aspects of item functioning. The benefit of the suggested loss function is primarily in integrating various psychometric criteria into a single measure, thus permitting an omnibus measure of item performance, itself a prerequisite for simultaneous algorithmic optimization. Nevertheless, we recommend further work by theoretical statisticians to develop alternative loss functions with a stronger analytical basis.

We have deliberately avoided the question of whether to treat ordinal Likert-type data in SEM via an IRT-related graded response model (GRM; Samejima 1997) or via a polytomous correlation matrix (Lei 2007). For ordinal data, IRT has the additional feature of directing attention toward item intercepts as well as item discrimination/factor loadings. Polychoric correlations have been employed in the present article for computational stability and efficiency. Although they are theoretically an indirect indication of the correlations between the underlying hypothesized continuous variables, empirical evidence suggests they perform at least as well as theoretically appropriate multidimensional item response models (Knol and Berger 1991). One convenient approach for ordinal (Likert-type) or binary data would be to use polychoric correlations for the optimizing process, reserving an IRT/GRM-based SEM (Glockner-Rist and Hoijtink 2003) for evaluating the final model.

The algorithm is based on psychometric criteria alone and does not take into account content validity of the items. It presupposes that the original pool of items has been selected with care to represent the desired construct. As with any exercise in scale reduction, care must be taken to ensure that construct underrepresentation has not been introduced in the shortened scale. Therefore, the content of the reduced scale should be inspected carefully with this in mind. However, the loss function proposed here would certainly be less susceptible to this problem than other methods (e.g., relying on reliability alone). First, item subconstruct clusters would be characterized by excessive positive residual covariance, which is detected by SEM residuals and penalized by the loss function. Second, diminished construct representation should reduce predictive validity and therefore also be penalized accordingly. Nevertheless, consideration of item content remains a meaningful aspect of item evaluation for which there is no substitute for manual intervention.

Finally, an important characteristic of the algorithm detailed here is that the item-factor loadings are freely estimated. This might create issues of model misspecification when the item set is small (e.g., 3–5 items), and a certain item(s) might unduly influence the estimation of the latent factor. An alternative implementation of the method could involve constraining item-factor loadings to unity: entailing a τ-equivalent model constraint and item evaluation more in line with classical test theory. This more highly constrained model specification would restrict the relative influence of individual items on the latent factor and probably perform better when the item set is small. Future work needs to consider which approach is most effective for the practical purpose of item selection.

The limitations mentioned above are mostly specific to the current implementation. There would appear to be many opportunities to test potentially better alternatives; most notably in the choice of loss function, factor item link functions (e.g., IRT based), and in whether or not to employ a τ-equivalent item-factor model. The goal of this article has been to suggest the concept of a stepwise item selection algorithm, using an explicit prior measurement and structural model for evaluating item functioning. We have attempted to demonstrate that such an algorithm is practically feasible.

In terms of future work, we do not propose the described methodology is yet ready for general use. Rather, our goal is to open a dialogue on the potential for SEM to provide a framework for more explicit and integrated item evaluation. More theoretical and simulation work is required before any such method could be recommended for general use. If this is completed with positive results, the opportunity exists for the creation of add-on toolboxes to standard SEM software. These would deliver robust and user-friendly automated item selection and scale reduction procedures for general users, requiring only the ability to correctly specify a measurement and structural model within the standard SEM framework. The increased facility to more easily reduce scales should be combined with objective diagnostics that prevent misuse, and recognition that the psychometric qualities of any reduced scale should be treated with caution, until proven otherwise.