The Performance of the Semigeneralized Partial Credit Model for Handling Item-Level Missingness

Abstract

The semi-generalized partial credit model (Semi-GPCM) has been proposed as a unidimensional modeling method for handling not applicable scale responses and neutral scale responses, and it has been suggested that the model may be of use in handling missing data in scale items. The purpose of this study is to evaluate the ability of the unidimensional Semi-GPCM to aid in the recovery of person parameters from item response data in the presence of item-level missingness, and to compare the performance of the model with two other proposed methods for handling such missingness: a multidimensional modeling approach for missingness and full information maximum likelihood estimation. The results indicate that the Semi-GPCM performs acceptably in an absolute sense when less than 30% of the item data is missing but does not outperform the other two methods under any particular conditions. We conclude with a discussion about when practitioners may or may not want to use the Semi-GPCM to recover person parameters from item response data with missingness.

Keywords

missing data semi-generalized partial credit model multidimensional item response theory full information maximum likelihood

Introduction and Proposed Model

Missing data commonly exist in quantitative studies, and such missing data are known to possibly induce bias in parameter estimates when not handled appropriately. A core difficulty in dealing with missing data in social science research is that the underlying mechanism of missing data is often unknown (Schafer & Graham, 2002). Depending on the underlying mechanism, missing data are divided into three types: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR; Little & Rubin, 2014). MCAR is defined as missing data that are independent of any observed or unobserved variables. In this case, parameter estimates are expected to be unbiased. MAR is defined as missing data that are dependent on one or more of the observed variables in the data set. In this case, by fully accounting for the relationship between the missing data and the observed variables, researchers should still be able to obtain unbiased parameter estimates. MNAR, also named as nonignorable missing data, are defined as missing data that are related to one or more unobserved variables.

When applying the three missingness mechanism definitions to scale item data aiming to estimate a unidimensional latent trait, we can be more specific in the missingness mechanism definitions. MCAR would remain consistent with the above definition, that is, any item-level missingness is considered MCAR if the missingness is random. MAR would be defined as missingness within a scale item that is related to an observed variable external to the scale itself (e.g., a demographic variable). MNAR could be defined in two ways. First, item-level missingness that is related to a variable external to the scale (e.g., a demographic variable) that was not measured in the study could be classified as MNAR. Second, item-level missingness that is related to the latent trait that the scale is attempting to measure would also be classified as MNAR, since the latent trait is unobserved. In this study, we consider the second type of MNAR item-level data.

One simplified way of handling missing item-level data is to calculate summated scores (rather than latent trait scores) from the scale data, while handling the missingness within the summation in some particular way, such as listwise deletion. However, these methods are generally not preferred due to issues of parameter estimate bias (Enders, 2010) and loss of available information that is expected when, for example, deleting full cases. For example, in a National Center for Education Research report, Ruby and Doolittle (2010) found that the missing proportion of individual survey item data was less than 5%, yet if they conducted a listwise deletion at the scale level, they would have excluded almost one third of the respondent sample. Broadly speaking, researchers should aim to handle missing data at the item-level rather than at the total scale level.

Huggins-Manley et al. (2018) proposed a semi-generalized partial credit model (Semi-GPCM) to deal with not-applicable (NA) scale responses. They considered NA as a valid response option and analyzed it together with the other common Likert-type scale data using the Semi-GPCM, noting very specific cases of unidimensionality that would have to hold for such a modeling approach. This model was developed based on the relationship between the generalized partial credit model (GPCM; Muraki, 1992) and the nominal response model (NRM; Bock, 1972; e.g., Huggins-Manley & Algina, 2015; Thissen & Steinberg, 1986).

Let y_ij represent a response to an item i by participant j; k represent a particular ordered response categories in a polytomously scored scale item and K represent the total number of ordered response options; $ζ$ and $λ$ represent the intercept and slope parameter, respectively, for each response category; and $θ$ represents an underlying latent trait measured by the scale items. The Semi-GPCM is defined

P_{(y_{ij} = k = 1)} = \frac{1}{1 + \sum_{k = 2}^{K} e^{(ζ_{ik} + [k - 1] λ_{i 2} θ_{j})} + e^{(ζ_{ih} + λ_{ih} θ_{j})}}

(1)

for the first response category in an item with more than two ordered response options,

P_{(y_{ij} = k = g)} = \frac{e^{(ζ_{ig} + [g - 1] λ_{i 2} θ_{j})}}{1 + \sum_{k = 2}^{K} e^{(ζ_{ik} + [k - 1] λ_{i 2} θ_{j})} + e^{(ζ_{ih} + λ_{ih} θ_{j})}}

(2)

for an ordered response category k = g, where g is an ordered response category other than the first category selected by a respondent, and

P_{(y_{ij} = h)} = \frac{e^{(ζ_{ih} + λ_{ih} θ_{j})}}{1 + \sum_{k = 2}^{K} e^{(ζ_{ik} + [k - 1] λ_{i 2} θ_{j})} + e^{(ζ_{ih} + λ_{ih} θ_{j})}}

(3)

for the nominal response category h, which is the category for missing data in this study.

One notable property of this model is its assumption of unidimensionality underlying the scale item responses, in which the Semi-GPCM assumes that the nominal category (here, the missing data on the item) is a function of the underlying latent trait ( $θ$ ). This assumption should be met if item-level missingness is MNAR due to a functional relationship between the missingness and the latent trait variable. However, in other missing mechanism conditions, the model may be robust to the assumption violation because of the freely estimated nominal category slope parameter ( $λ_{ih}$ ). We expect $λ_{ih}$ to be zero if the missing data are not correlated to the underlying latent trait ( $θ$ ). This unique property of the Semi-GPCM may suggest that we can generalize the model to deal with other types of missing data beyond MNAR, such as MAR and MCAR, or at least that the model parameter estimates can inform researchers of the degree to which MNAR is a supported assumption.

Purpose

The purpose of this study is to evaluate the ability of the Semi-GPCM to estimate unbiased person traits from polytomous scale data in which some item responses are missing. We evaluated the model-based parameter estimates with both absolute and relative bias criteria under a variety of data conditions that may influence the criteria in social science research studies that make use of scale data. We achieved this purpose through two simulation studies.

The first simulation study (Study 1) examined if the Semi-GPCM can perform well under three different missingness mechanism conditions: MCAR, MAR, and MNAR. However, we hypothesized that the model is particularly suited to handle MNAR item-level data, which meets a critical need in applied measurement as these types of missing item data tend to be the most challenging type to handle (Finch, 2008; Rose et al., 2010). We evaluated the model performance in light of various test lengths, sample sizes, proportions of missing data, and magnitudes of correlational relationships of the missing data to observed and unobserved variables. The research questions are as follows:

Research Question 1: To what degree can the Semi-GPCM recover person trait estimates under different types of item-level missingness mechanisms?

Research Question 2: To what degree do four factors (test length, sample size, the proportion of missing data, and correlation magnitudes between the missing data and particular variables) affect person trait estimate recovery under the Semi-GPCM?

In Study 2, we compared the model person parameter recovery between the Semi-GPCM and two other methods for estimating person parameters from scale data with missing item responses: fitting a multidimensional item response theory (MIRT) that contains a secondary latent trait intended to represent the propensity to not respond to items, and implementing full-information maximum likelihood (FIML) in the presence of the missing data. Hence, the second simulation addresses the following research question:

Research Question 3: Can the Semi-GPCM provide accurate person trait estimates relative to results from handling missing data with multidimensional IRT or FIML?

Background on FIML and MIRT

Holman and Glas (2005) proposed four general MIRT models for handling nonignorable item-level missing data partially based on previous studies (e.g., Bock et al., 1988; Moustaki & Knott, 2000; O’Muircheartaigh & Moustaki, 1999). Different from the Semi-GPCM, MIRT requires the creation of a missing data indicator matrix,¹ in which elements of the matrix (r_ij) are defined as

r_{ij} = {\begin{matrix} 1 & if y_{ij} is missing \\ 0 & otherwise \end{matrix} .

(4)

Assuming local independence, the four models from Holman and Glas (2005) are

G 1 = \underset{i, j}{Π} p (y_{ij} | r_{ij}, θ_{j}, β) p (r_{ij} | {θ_{missing}}_{j}, δ_{i}) g_{1} ({θ_{missing}}_{j}) g_{2} (θ_{j}),

(5)

G 2 = \underset{i, j}{Π} p (y_{ij} | r_{ij}, θ_{j}, β) p (r_{ij} | {θ_{missing}}_{j}, δ_{i}) g ({θ_{missing}}_{j}, θ_{j} | ϕ),

(6)

G 3 = \underset{i, j}{Π} p (y_{ij} | r_{ij}, θ_{j}, β) p (r_{ij} | {θ_{missing}}_{j}, θ_{j}, δ_{i}) g ({θ_{missing}}_{j}, θ_{j} | ϕ),

(7)

and

G 4 = \underset{i, j}{Π} p (y_{ij} | r_{ij}, {θ_{missing}}_{j}, θ_{j}, β) p (r_{ij} |, δ_{i}) g ({θ_{missing}}_{j}, θ_{j} | ϕ),

(8)

where $θ_{missing}$ is the latent trait representing the propensity to not respond to items; $β$ represents structure parameters; ϕ is the index of the common distribution of $θ$ and $θ_{missing}$ ; and $g_{1} ({θ_{missing}}_{j})$ and $g_{2} (θ_{j})$ represent the density of ${θ_{missing}}_{j}$ and $θ_{j}$ , respectively. The four models are slightly different from each other, depending on the different relationship assumptions between missing data and the underlying latent trait. G1 assumes $θ_{missing}$ is independent of the latent trait of interest ( $θ$ ). G2, G3, and G4, on the other hand, assume $θ_{missing}$ is related to the latent trait of interest ( $θ$ ). As Holman and Glas (2005) demonstrated, all G1, G2, G3, and G4 models can be fit to data in applications to deal with the missingness due to the lack of information on the underlying missing mechanism. Researchers can then choose one of the models based on model fit and also their understanding of the missing data in their data set.

Unlike MIRT, using FIML estimation to handle missing data does not require researchers to compare different models and does not explicitly require researchers to have a good understanding of their missing data. Moreover, FIML is the default estimation approach to analyze measurement data when using the structural equation modeling (SEM) software Mplus (L. K. Muthén & Muthén, 2015), which arguably makes it an easy method to apply. Different from standard maximum likelihood (ML) estimation, FIML computes a casewise likelihood function using all the available information for the particular case first, and then accumulates likelihood functions across the entire sample (Arbuckle, 1996), such that

\log L_{j} = C_{j} - \frac{1}{2} (\log | Σ_{j} | - \frac{1}{2} {(y_{j} - μ_{j})}^{'} Σ_{j}^{- 1} (y_{j} - μ_{j})

(9)

and

\log L (μ, Σ) = \sum_{j = 1}^{N} \log L_{j},

(10)

where C_j is the constant based on the number of observations for participant j; $μ_{j}$ and $Σ_{j}$ are the mean vector and covariance matrix corresponding to observed response vector y_j ; and N is the number of participants. Previous studies compared FIML estimation in the presence of missing variable data with listwise deletion, pairwise deletion, mean imputation, and similar pattern imputation to deal with MCAR and MAR situations in the SEM (Enders & Bandalos, 2001; Hirose et al., 2016; Myrtveit et al., 2001). They found FIML estimation to be superior across all conditions, especially when missing data is not MCAR. However, it should be noted that these particular studies did not focus on categorical item-level missing data in measurement models.

General Method

Overview

We used the same approach to simulate the missing data for both Study 1 and Study 2, but applied different models to analyze them. Below we detail the data generation method that applies to both studies.

Data Generation

All data sets were generated in R (R Core Team, 2013) and were varied on the basis of four factors mentioned in Research Question 2 (test length, sample size, the proportion of missing data, and correlation magnitudes between the missing data and particular variables). The crossing of the four factors resulted in 432 unique simulation conditions (see Table 1). For the missing correlation factor, when the correlation was set to zero, we considered the missing data as MCAR. In MAR conditions, the correlation was not fixed to zero and the missing data were correlated at a specific magnitude to a secondary latent trait ( $θ_{missing}$ ) that was not intended in the measurement (i.e., a nuisance trait). In MNAR conditions, the correlation was not fixed to zero and the missing data were correlated at a specific magnitude to the primary latent trait of measurement ( $θ$ ).

Table 1.

The Count of Different Combinations of the Conditions for the Data Simulation.

	Type of missingness
Simulation factor and levels	Complete	MCAR	MAR	MNAR
Number of items (5, 10, 30)	3	3	3	3
Sample size (200, 500, 1,000, 2,000)	4	4	4	4
Missing proportion (.05, .1, .2, .3, .5)	—	5	5	5
Missing correlation (.3, .5, .7)	—	—	3	3
Number of conditions (product of columns)	12	60	180	180

Note. MCAR = missing completely at random; MAR = missing at random; MNAR = missing not at random.

We used the GPCM (see Equations 1-3) as the data generating model for creating 100 complete data sets under each condition. Each item was generated to have five ordered categories, mimicking a common 5-point Likert-type scale measurement scenario. First, we generated the person parameters ( $θ_{j}$ ) from N ~ (0, 1) and the item parameters (a, b₂, b₃, b₄, b₅) from normal distributions with means fixed to 1, −1.5, −0.5, 0.5, and 1.5, respectively, and standard deviations fixed to 0.1. Note that item parameter a is a discrimination parameter, and the series of b item parameters represent adjacent category difficulty parameters. The data generating distribution means were chosen to be equally spaced and represent progressively more difficult response options, as would be ideal under rating scale measurement (Linacre, 2002).

Next, a binary missing indicator matrix R was generated based on the missing proportion factor and the missing correlation factor. For MCAR conditions, the missing correlation factor was fixed to zero, and hence we randomly sampled the missing indicator for each person based solely on the missing proportion. For conditions with the missing proportion fixed to .1, we generated the missing indicator matrix based on the missing proportion. For example, when the missing proportion equaled to .1, 10% of the values in the missing indicator matrix were 1 and 90% were 0. For MAR and MNAR, the propensity for a score to go missing was calculated based on the normal ogive model (Lord & Novick, 2008; Padilla & Algina, 2004), notated as $Φ (ξ + c)$ , where $Φ$ is a cumulative standard normal distribution function and $ξ$ and c are defined as

ξ = ρ_{missing} * θ_{missing} + \sqrt{(1 - ρ_{missing}^{2}) * e}

(11)

and

c = \sqrt{2} * Φ^{- 1} (ρ_{missing}),

(12)

where $ρ_{missing}$ is the correlation between missing data and $θ_{missing}$ , and e is an error term generated from N ~ (0, 1). The e term is used to introduce noise such that the correlation between missing data and $θ_{missing}$ was a probabilistic outcome rather than deterministic. In MAR conditions, $θ_{missing}$ was generated independently from N ~ (0, 1). In our MNAR conditions, $θ_{missing}$ in Equation 11 was the same as $θ$ because the missing indicator ( $r_{ij}$ ) was defined in relation to $θ$ . We sampled missing indicators (r_ij) based on their corresponding missing propensity scores. Where the missing indicator equaled to 1, we removed the original data point from the item response data matrix.

Study 1

Data Analysis

We fit the GPCM to the complete data sets first to obtain baseline results. As for the incomplete data sets, we fit the Semi-GPCM in Equations 1, 2, and 3, in which we treated the missing data as a nominal category. For all the conditions, we analyzed the data by batching to Mplus (L. K. Muthén & Muthén, 2015) through R with the MplusAutomation package (Hallquist & Wiley, 2018).

Person parameter recovery was evaluated using three criteria: bias, root mean square error (RMSE), and the Pearson correlation between the estimated latent trait ( $\hat{θ}$ ) and the true latent trait ( $θ$ ). Each criterion was calculated for each data set first and then averaged across the 100 iterations, such that

bias = \frac{\sum_{1}^{N} (\hat{θ} - θ)}{N},

(13)

RMSE \sqrt{\sum_{1}^{N} \frac{{(\hat{θ} - θ)}^{2}}{N}}

(14)

and

c o r r e l a t i o n = \frac{c o v (θ, \hat{θ})}{S_{θ} * S_{\hat{θ}}}

(15)

A multiple regression analysis was conducted for each criterion to explore which simulation factor affected the criterion, in which we treated missing type (MCAR, MAR, MNAR) as a categorical factor and all the other simulation factors as continuous variables. All possible interactions between different conditions were included in the analysis. In terms of the missing correlation factor, we disposed of its main effect because the meaning of the factor varied across MAR, MNAR, and MCAR conditions. In MAR conditions, the missing data were correlated to the secondary latent trait $(θ_{missing})$ . In MNAR conditions, the missing data were correlated to the primary latent trait $(θ$ ). In MCAR the missingness was unrelated to any latent trait. Hence, we dummy coded the missing type factor (D_MAR,D_MNAR) and multiplied it with the missing correlation factor. When interpreting the multiple regression outputs, we focused only on the predictors with at least a small effect size ( $η^{2} > . 02$ ; Cohen, 2013).

Results

Complete Data Sets

Table 2 shows the descriptive statistics of person parameter recovery in the GPCM model analysis on complete data sets. The bias was small for all conditions, in which the largest absolute value of bias was less than or equal to |.008|. Neither the number of items nor the sample size meaningfully affected the bias: F(1, 1196) = 0.64, p = .42, $η^{2} < . 001$ ; F(1, 1196) = 0.80, p = .37, $η^{2} < . 001$ . RMSE decreased statistically significantly as the number of items increased, F(1, 1196) = 10170.19, p < .001, $η^{2} = . 89$ , but was not meaningfully affected by sample size, F(1, 1196) = 11.91, p < .001, $η^{2} = . 001$ . As for the correlation between the estimated latent trait and the true latent trait, larger numbers of items were associated with higher correlations, F(1, 1196) = 5252.50, p < .001, $η^{2} = . 81$ , but sample size did not have an impact on this criterion, F(1, 1196) = 0.69, p = .41, $η^{2} < . 001$ .

Table 2.

Descriptive Statistics of the Bias, RMSE, and Correlation Under Baseline (Complete Data) Conditions in Study 1.

Number of items	Sample size	Bias	RMSE	Correlation
5	200	.008	.471	.885
5	500	−.003	.468	.886
5	1,000	.002	.462	.887
5	2,000	−.002	.462	.887
10	200	.002	.357	.936
10	500	.002	.350	.938
10	1,000	.002	.347	.938
10	2,000	.000	.348	.938
30	200	.000	.230	.976
30	500	−.002	.223	.976
30	1,000	.000	.219	.976
30	2,000	−.001	.218	.976

Note. RMSE = root mean square error.

Data Sets With Missingness

Table 3 shows a summary of the multiple regression analyses evaluating the person parameter recovery when using the Semi-GPCM, including only the main and interaction effects associated with statistically significant and practically meaningful results, as defined above. Bias across all the conditions ranged from −.28 to .26. None of the model predictors had a meaningful impact on bias. RMSE was affected by the interaction of the missing proportion and the missing type, F(2, 41960) = 4988.44, p < .001, $η^{2} = . 043$ , and the test length, F(1, 41960) = 12075.01, p < .001, $η^{2} = . 052$ . The correlation between the estimated latent trait and the true latent trait was affected by the interaction of the missing proportion and the missing type simulation factors, F(2, 41960) = 3414.00, p < .001, $η^{2} = . 046$ .

Table 3.

Percentage of Variance Explained (η²* 100) and Statistical Significance Results of the Multiple Regression Predictors Under Missing Data Conditions in Study 1.

Predictor	Bias (%)	RMSE (%)	Correlation (%)
Missing proportion	0.00	54.72^***	46.16^***
Number of items	0.00^***	5.24^***	1.66^***
Missing data type	0.00	4.76^***	3.47^***
Missing proportion * Missing data type	0.00	4.33^***	4.56^***

Note. RMSE = root mean square error.

p < .05. ^**p < .01. ^***p < .001.

Figure 1 shows a visual of the meaningful interaction effects from Table 3, only with regard to RMSE and correlation criteria, as the simulation factors were not meaningfully predictive of bias. The Semi-GPCM performed well under most conditions except when missingness was MAR with a missing proportion equal to .5. Overall, as the missing proportion factor increased, the performance worsened, noting that the person parameter recovery under MCAR was relatively better than the person parameter recovery under MAR and MNAR across all levels of missing proportions, and the person parameter recovery under MAR was worse than the person parameter recovery under MNAR when the missing proportion was .5 (Figure 1A and C). In terms of the number of items, longer test lengths aligned with better person parameter recovery according to the RMSE and correlation criteria (see Figure 1B and D).

Figure 1.

Person parameter recovery using the Semi-GPCM in Study 1.

Discussion

The Semi-GPCM provided acceptable person parameter recovery for all the conditions except for those with 50% of the data missing. Bias was small across all conditions and was not affected by any simulation factor. The maximum absolute value of bias was .28 when using the Semi-GPCM and all bias was in a confined range, which was consistent with the results shown in previous studies (e.g., B. Muthén et al., 1987; Reise & Yu, 1990). In terms of the other two criteria, larger missing proportions of data and fewer numbers of items led to worse person parameter recovery, whereas the sample size did not show a meaningful impact.

The magnitudes of the person parameter recovery results in this study aligned with previous research fitting the GRM to complete data sets. For example, Reise and Yu (1990) conducted a simulation study to explore the parameter recovery in the GRM and they found that the correlations between trait estimates and true traits were over .85 across all conditions when the sample size was greater than 500. We have similar findings in this study, even with sample sizes as low as 200. In this study, the sample size had little influence on the trait parameter recovery.

We found interaction effects between study factors and the missingness type. As the missing proportion increased, MNAR and MAR conditions had increasingly worse person parameter recovery as compared with MCAR conditions. When missing proportion increased to .5, the person parameter recovery under MNAR is better than the person parameter recovery under MAR. This aligned with our hypothesis that, due to the underlying unidimensionality assumption of the Semi-GPCM, the studied approach to missing data is better suited for MNAR data conditions comparing with MAR data conditions. Study 1 supports the notion that missing data that are a function of the trait of interest can be handled well by fitting the Semi-GPCM. As MNAR can be one of the more difficult forms of missing item data to handle, this is an advantage of the proposed modeling approach when MNAR is the theorized and empirically supported missing data mechanism.

Study 2

In Study 2, we compared the person parameter recovery across three different methods: fitting the Semi-GPCM, fitting a multidimensional IRT model, and using FIML while fitting the GPCM.

Method

Data were generated using the same method as Study 1. However, we excluded the sample size of 200 due to convergence issues when fitting multidimensional IRT models under small sample sizes. Multidimensional IRT and FIML were used to analyze the data. Compared with multidimensional IRT, using FIML while fitting the GPCM was a straightforward approach in which we used all the available information in the data set (see Equations 9 and 10). However, applying multidimensional IRT required us to make some decisions before interpreting the results, such as which model to choose and how to identify the model. In our case, we chose model G2 from Holman and Glas (2005; see Equation 6) based on the match it has to our data generation process. Specifically, our missing propensity is conditioned on the latent trait of the missing data ( $θ_{missing}$ , see Equation 11) and $θ$ and $θ_{missing}$ were drawn from a multivariate normal distribution under MAR conditions. For test practitioners, because of the unknown underlying mechanism of the missing data, they may need to apply all the multidimensional models and then choose the model based on the model fit and also their understanding of their operational data (Rose et al., 2010). To make the multidimensional IRT model identified, we can set one of the factor loadings of the missing indicators to zero, which is meaningful in a practical data set if an item has much less missingness than other items, or if that item is not particularly of interest in terms of missingness. However, in our simulated data, the proportion of missingness for each item is approximately the same across items, and there is no content for us to decide which item is more important or not. Hence, we forced the correlation between the two latent traits to zero when the data are MAR and MNAR, which is also the default in the software package “mirt” in R (Chalmers, 2012).

Just as in Study 1, all the data were generated in R and analyzed in Mplus (L. K. Muthén & Muthén, 2015), with 100 data sets per condition. Our evaluation criteria included bias, RMSE, and the correlation between the estimated latent trait ( $\hat{θ}$ ) and the true latent trait ( $θ$ ) as seen in Equations 13, 14, and 15, respectively. We conducted a multiple regression analysis for each criterion to compare the performance of those different missing data handling methods and explored which conditions and interactions of conditions impacted the person parameter recovery when using these different methods. All predictors with the significant impacts are listed in Table 4. All the simulation factors were treated as continuous variables in the regression model except the missing data type and the missing handling approaches. All possible interactions between different conditions were included in the analysis. However, none of the four-way and five-way interaction terms achieved the minimum requirement of small effect size ( $η^{2} > . 02$ ; Cohen, 2013), and hence the results of those interaction terms are not shown or discussed in this article.

Table 4.

Percentage of Variance Explained (η²* 100) by Simulation Factors in the Multiple Regression Analysis for Study 2.

Predictor	Bias (%)	RMSE (%)	Correlation (%)
Missing proportion	1.30^***	28.14^***	23.06^***
Number of items	0.29^***	20.74 ^***	7.43 ^***
Analysis model	2.16^***	12.23^***	11.00^***
Missing proportion * Analysis model	2.16 ^***	11.58 ^***	16.46^***
Missing proportion * Analysis model * Missing data type	0.00	1.53^***	2.26 ^***

Note. RMSE = root mean square error. Boldfaced numbers are the significant impacts reported in the article.

p < .05. ^**p < .01. ^***p < .001.

Results

Similar to the Semi-GPCM, the bias was relatively small across all the conditions using multidimensional IRT and FIML, with a range from −.18 to .16, and −.16 to .30, respectively, and not impacted by the test length. The RMSE and correlation evaluation criteria, on the other hand, were both heavily influenced by the main effect of test length (i.e., number of items), with F(1, 94380) = 158584.9, p < .001, $η^{2} = . 21$ for RMSE outcomes and F(1, 94380) = 31804.45, p < .001, $η^{2} = . 07$ for correlation outcomes. As test length increased, the RMSE results were lower and the correlation evaluation results were higher (see Figure 2A and B).

Figure 2.

Person parameter recovery comparison using Semi-GPCM, FIML, and multidimensional-IRT.

Of core interest to Research Question 3, the analysis models (i.e., Semi-GPCM, multidimensional IRT, and GPCM with FIML) performed differently across different missing proportions of data, with F(2, 94380) = 44279.32, p < .001, $η^{2} = . 12$ for the RMSE outcome and F(2, 94380) = 1257.21, p < .001, $η^{2} = . 02$ for the bias outcome. For RMSE, all three methods followed the same trend such that with an increase in missing proportion, all methods tended to perform slightly worse regarding both outcomes. However, multidimensional IRT and FIML followed almost identical trends and the Semi-GPCM tended to perform worse than the other two as the missing proportion of data increased (see Figure 2C). In terms of bias, among the three analysis models, only the parameter estimate bias of FIML was significantly influenced by the increase of the missing proportion. The estimates from FIML were positively biased when the missing proportion increased (see Figure 2D). We also found a three-way interaction between the missing proportion, analysis model, and missing data type for the correlation evaluation criterion, with F(4, 94380) = 2414.98, p < .001, $η^{2} = . 02$ . When data were MCAR, all three analysis models followed the same trend when the missing proportion increased (see Figure 3A). When MAR and MNAR, FIML and MIRT still performed similarly, but the performance of the Semi-GPCM was worse when the missing proportion was high and MNAR. Also, the performance of the Semi-GPCM was better when data were MNAR, particularly when compared with the performance of the Semi-GPCM when data were MAR and the missing proportion was .5 (see Figure 3B and C).

Figure 3.

The correlation between the true latent trait and the estimated latent trait using Semi-GPCM, FIML and multidimensional-IRT when MCAR (A), MAR (B), and MNAR (C).

Discussion

Similar to Study 1, bias was small across all the conditions and not affected by test length. RMSE and correlation performance, on the other hand, were both affected by the test length (i.e., number of items). Among all factors, the test length plays a relatively prominent role compared with the rest of the factors, but the sample size does not heavily influence the primary latent trait estimates. This finding may be due to the sufficient sample sizes we chose, but regardless it is noteworthy that research on handling missing data with multidimensional IRT or FIML shows similar findings. Specifically, Holman and Glas (2005) used multidimensional IRT to deal with the nonignorable missing data, and their results showed that the mean square error of parameter estimates is more influenced by the number of items than the sample size. Enders and Bandalos (2001) explored how sample size, missing proportion, and factor loadings influenced the parameter estimate bias in their study that explored the use of FIML in SEM, and their results indicated that the sample size heavily influences bias when it is small but not so much when the sample size is over 500.

Returning to Study 2, we explored how the choice of analysis model affects the person parameter recovery and we found a significant interaction between the analysis model and the proportion of missing data for both bias and RMSE, and a three-way interaction between analysis model, missing proportion, and the missing data types. For bias, only FIML showed slightly more positive bias when the missing proportion is .5 (Figure 2D). However, the maximum absolute value of the FIML bias is .03, which is often tolerable when the true person parameter ranges from −3 to 3. Overall, for RMSE and correlation, MIRT and FIML performed either equally well or better than the Semi-GPCM, especially when the proportion of missing data was large. When data were MCAR, all three models performed equally well regarding the correlation evaluation criterion. The average correlation between true theta and estimated theta stayed near .9 for all three models even when the missing proportion was .5 (see Figure 3A). This aligns with overarching theories on MCAR data and parameter recovery in statistical models (Rubin, 1976). FIML and multidimensional IRT often showed similar performance between MCAR conditions and other conditions of MAR and MNAR. However, the performance of Semi-GPCM was noticeably worse than the other approaches for handling missing data when MAR. This aligns with the unidimensional assumption of the Semi-GPCM. In both Study 1 and Study 2, the Semi-GPCM results were largely acceptable in an absolute sense when the missing proportion was equal to or less than .3. However, the relative performance did not support the use of the Semi-GPCM. Next, we discuss instances in which practitioners may want to use the Semi-GPCM, but of course there may be many instances in which other methods for handling missing data are preferred.

Conclusion

Among all three analysis methods that we studied, using FIML while fitting the GPCM to polytomous item response data with missingness seems to be the most user-friendly method that can produce acceptable trait recovery results both in an absolute and relative sense. No missing data preprocessing is required for FIML and it is readily available in many statistical packages, including Mplus. However, there are times when practitioners not only want to recover parameters but also want to have information on the missing data itself. FIML does not provide substantive information about the missing data, which is in contrast to the other two studied methods. Based on the definition of multidimensional IRT, the relationship between the nuisance latent trait (i.e., the latent trait that is correlated with the missing data) and the primary latent trait of interest can inform researchers about the missing data mechanism and, indirectly, the relationship between missing data and the primary trait of measurement.

In terms of the Semi-GPCM, the item parameters for the missing data category (coded as a nominal category in the model) can directly inform researchers of the relationship between the missing data and the latent trait of interest. Figure 4 presents item characteristic curves (ICCs) from three iterations of our simulation study, including one iteration under each of the MCAR, MAR, and MNAR mechanisms. The gray lines represent the nominal category (i.e., missing data category), showing the probability of having missing data as a function of the underlying trait (i.e., theta on the x-axis). As shown in that figure, the missing data category is not strongly related to the trait under the MCAR and MAR conditions, which aligns with the simulated conditions. In contrast, the ICC in Figure 4 shows that when the data were generated as MNAR due to the missingness being related to the trait of measurement, the category response function in the ICC shows a clear relationship between the missing data and the trait of measurement. In this case, students of a higher trait were more likely to provide missing data on the item. These example ICCs demonstrate that applying the Semi-GPCM for missing data in polytomous scale items can directly inform researchers about this critical relationship between the trait they are measuring and the propensity for respondents to provide missing data.

Figure 4.

Item characteristic curves based on item parameters estimated from the Semi-GPCM for three different simulation iterations.

While multidimensional IRT models can also be used to evaluate indirectly the degree to which missing data are related to the trait of interest, we argue that there are more practical challenges to doing so as compared with fitting the Semi-GPCM. Specifically, one must first choose an appropriate multidimensional IRT model out of a family of models, which would likely require model comparisons (Rose et al., 2010). Deciding between the multidimensional IRT models available can be time-consuming and, sometimes, subjective. In our simulations, we were able to match the multidimensional model choice to the data generation choice, but practitioners do not have this luxury. Hence, we view our study results as being the best-case scenario for the multidimensional IRT modeling approach, and had we chosen a more mismatched multidimensional IRT model for Study 2, we hypothesize that the parameter recovery would have been worse for that method. This hypothesis is supported by the fact that even when we used the appropriately specified multidimensional IRT model, we ran into some issues. Specifically, we freely estimated the correlation between two traits when fitting to the MCAR data, but the analysis hit a saddle point, and the estimates for some items were unrealistically large. In that case, we had to fix the correlation to zero for MCAR conditions, but allowed it to be freely estimated for MAR or MNAR. These choices are much easier to make in a simulation where we know when the data are MCAR than in practice. In addition, the sufficient sample size was relatively large for multidimensional IRT. Due to the complexity of the model, the number of parameters estimated using multidimensional IRT is almost twice as much as the number of parameters when fitting the GPCM or Semi-GPCM. Practitioners may not be able to consider MIRT methods for missing item data under particular sample sizes.

Overall, applying the Semi-GPCM to polytomous scale data with missingness can produce unbiased person parameter estimates in some measurement conditions, and the model results can provide useful information for practitioners trying to evaluate the missing data mechanism. However, FIML or multidimensional IRT may be preferred in particular measurement conditions and can provide a similar or better degree of accuracy in person estimates.

All our study results should be interpreted in light of limitations. First, multiple imputation (MI) is a known, viable option for handling missing data both inside (Finch, 2008) and outside (Enders, 2010) of unidimensional latent trait measurement, and it was not considered in this study. Ultimately, MI is a very general term for a process that has many particular decisions to make, and hence there was no single comparison arm that we could justify as representing MI methods in our study. Second, readers should be very attuned to the fact that we generated our MNAR data based on the assumption that the latent trait itself was the unobserved variable that explained missingness patterns. The performance of the Semi-GPCM in recovering trait parameters under MNAR conditions may be quite different if the MNAR mechanism was due to some other unobserved variable (e.g., the motivation of test taker that was not measured by the researcher).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Sherry Zhou

Notes

References

Arbuckle

J. L.

(1996). Full information estimation in the presence of incomplete data. Marcoulides

G. A.

Schumacker

R. E.

(Eds.), In Advanced structural equation modeling: Issues and techniques (pp. 243-277). Lawrence Erlbaum.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29-51. https://doi.org/10.1007/BF02291411

Bock

R. D.

Gibbons

Muraki

(1988). Full-Information Item Factor Analysis. Applied Psychological Measurement, 12(3), 261-280. https://doi.org/10.1177/014662168801200305

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06

Cohen

(2013). Statistical power analysis for the behavioral sciences. Routledge. https://doi.org/10.4324/9780203771587

Enders

C. K.

(2010). Applied missing data analysis. Guilford Press.

Enders

C. K.

Bandalos

D. L.

(2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430-457. https://doi.org/10.1207/S15328007SEM0803_5

Finch

(2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225-245. https://doi.org/10.1111/j.1745-3984.2008.00062.x

Hallquist

M. N.

Wiley

J. F.

(2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling, 25(4), 621-638. https://doi.org/10.1080/10705511.2017.1402334

10.

Hirose

Kim

Kano

Imada

Yoshida

Matsuo

(2016). Full information maximum likelihood estimation in factor analysis with a large number of missing values. Journal of Statistical Computation and Simulation, 86(1), 91-104. https://doi.org/10.1080/00949655.2014.995656

11.

Holman

Glas

C. A.

(2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1-17. https://doi.org/10.1111/j.2044-8317.2005.tb00312.x

12.

Huggins-Manley

A. C.

Algina

(2015). The partial credit model and generalized partial credit model as constrained nominal response models, with applications in Mplus. Structural Equation Modeling, 22(2), 308-318. https://doi.org/10.1080/10705511.2014.937374

13.

Huggins-Manley

A. C.

Algina

Zhou

(2018). Models for semiordered data to address not applicable responses in scale measurement. Structural Equation Modeling, 25(2), 230-243. https://doi.org/10.1080/10705511.2017.1376586

14.

Linacre

J. M.

(2002). Understanding Rasch measurement: Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85-106.

15.

Little

R. J.

Rubin

D. B.

(2014). Statistical analysis with missing data (Vol. 333). Wiley.

16.

Lord

F. M.

Novick

M. R.

(2008). Statistical theories of mental test scores. Information Age.

17.

Moustaki

Knott

(2000). Weighting for item non-response in attitude scales by using latent variable models with covariates. Journal of the Royal Statistical Society: Series A (Statistics in Society), 163(3), 445-459. https://doi.org/10.1111/1467-985X.00177

18.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1), i-30.

19.

Muthén

Kaplan

Hollis

(1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52(3), 431-462. https://doi.org/10.1007/BF02294365

20.

Muthén

L. K.

Muthén

(2015). Mplus. The comprehensive modelling program for applied researchers: User’s guide, 5.

21.

Myrtveit

Stensrud

Olsson

U. H.

(2001). Analyzing data sets with missing data: An empirical evaluation of imputation methods and likelihood-based methods. IEEE Transactions on Software Engineering, 27(11), 999-1013. https://doi.org/10.1109/32.965340

22.

O’Muircheartaigh

Moustaki

(1999). Symmetric pattern models: A latent variable approach to item non-response in attitude scales. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162(2), 177-194. https://doi.org/10.1111/1467-985X.00129

23.

Padilla

M. A.

Algina

(2004). Type I error rates for a one factor within-subjects design with missing values. Journal of Modern Applied Statistical Methods, 3(2), 406-416. https://doi.org/10.22237/jmasm/1099267980

24.

R Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

25.

Reise

S. P.

(1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27(2), 133-144. https://doi.org/10.1111/j.1745-3984.1990.tb00738.x

26.

Rose

Von Davier

(2010). Modeling nonignorable missing data with item response theory (IRT). ETS Research Report Series, 2010(1), i-53. https://doi.org/10.1002/j.2333-8504.2010.tb02218.x

27.

Rubin

D. B.

(1976). Inference and missing data. Biometrika, 63(3), 581-592. https://doi.org/10.1093/biomet/63.3.581

28.

Ruby

Doolittle

(2010). Efficacy of schoolwide programs to promote social and character development and reduce problem behavior in elementary school children (Report from the Social and Character Development Research Program, NCER 2011-2001). National Center for Education Research.

29.

Schafer

J. L.

Graham

J. W.

(2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147-177. https://doi.org/10.1037/1082-989X.7.2.147

30.

Thissen

Steinberg

(1986). A taxonomy of item response models. Psychometrika, 51(4), 567-577. https://doi.org/10.1007/BF02295596