Combined Approach to Multi-Informant Data Using Latent Factors and Latent Classes: Trifactor Mixture Model

Abstract

Although collecting data from multiple informants is highly recommended, methods to model the congruence and incongruence between informants are limited. Bauer and colleagues suggested the trifactor model that decomposes the variances into common factor, informant perspective factors, and item-specific factors. This study extends their work to the trifactor mixture model that combines the trifactor model and the mixture model. This combined approach allows researchers to investigate the common and unique perspectives of multiple informants on targets using latent factors and simultaneously take into account potential heterogeneity of targets using latent classes. We demonstrate this model using student self-rated and teacher-rated academic behaviors (N = 24,094). Model specification and testing procedures are explicated in detail. Methodological and practical issues in conducting the trifactor mixture analysis are discussed.

Keywords

informants mixture congruence raters factor analysis latent class

In educational and psychological studies, it is recommended that data be collected from multiple informants (Achenbach et al., 1987). However, there is often low agreement across raters depending on the construct of interest (Laird & De Los Reyes, 2013) and the correlation between informants is generally low (e.g., .40 or less; Achenbach et al., 2005; Flake & Petway, 2019). The informant disagreement is not necessarily ascribed to informant bias, but rather considered as unique perspectives because multiple informants may observe a target in different settings and hence provide valuable information that cannot be observed if only one of them is surveyed (Roskam, 2018). For example, student self-report can contribute unique and valuable information when combined with teacher ratings in predicting distal academic outcomes (von der Embse et al., under review). Thus, it is recommended to use multi-informant data rather than one source of information (e.g., Standards for Educational and Psychological Testing, American Educational Research Association et al., 2014).

Although collecting data from multiple informants is highly recommended, synthesizing multi-informant data is challenging and methods for multi-informant research are still limited (Kim et al., 2018; Flake & Petway, 2019). Two intertwined challenges include (1) how to appropriately model and assess the discrepancies between informants and (2) how to use such incongruous information (De Los Reyes, 2011). To measure discrepancies between multiple informants, a mean difference, correlation, and interrater reliability between informants are often reported (Achenbach, 2011; Kim et al., 2018). In the use of multi-informant ratings, researchers may choose a sole or optimal informant, simply average scores across informants, conduct separate analyses by informant, or use clinical cutoff scores aggregated across informants (Bauer et al., 2013; Kraemer et al., 2003; van Dulmen & Egeland, 2011). However, previous studies (e.g., Kraemer et al., 2003; van Dulmen & Egeland, 2011) reported the limitations of these methods: for example, choosing only one may lead to incorrect treatment decisions (De Los Reyes, 2011).

On the other hand, Bauer et al. (2013) noted the advantages of a factor analytic approach over the aforementioned methods. The factor analytic approach can afford the disentanglement of effects from various sources (e.g., rater effect as a method factor in the multitrait multimethod or MTMM model) and model the uniqueness as well as commonalities between informants. Specifically, Bauer et al. suggested a trifactor model that decomposes the variances into three primary sources, including (1) a common factor that represents the common view of the targets across informants, (2) informant perspective factors that represent each informant unique view of the targets not shared by the others, and (3) item-specific factors that are unique to items. The trifactor model was used to demonstrate differences in father and mother ratings of child negative affect and show how each item was related to the unique rater factors (father and mother) as well as the common factor, and how these factors were associated with child age, child gender, and parental impairment (Bauer et al., 2013).

The decomposition approach to multi-informant data has inherent assumptions, including that the underlying item and construct structure is dimensional (Clark et al., 2013). However, multi-informant data (e.g., children rated by both parents) could have heterogeneous groups of targets (e.g., children) not only in terms of the construct (e.g., low and high in negative affect) but also due to the disagreement between informants (e.g., children who were rated high by mother but not by father or vice versa). Thus, we employ a latent categorical variable approach to understand the categorization of targets of raters. This approach, usually called mixture modeling or latent class analysis¹, assumes that the underlying structure of items is categorical and is used to extricate heterogeneous groups of individuals based on their responses to the items (Clark et al., 2013).²

The present investigation extends Bauer et al.’s (2013) work on the trifactor model by applying factor mixture modeling to multi-informant data: We introduce the trifactor mixture model that combines the trifactor model and the mixture model. The trifactor mixture model distinguishes the common factor from the informant perspective factors and item-specific factors and simultaneously classifies individuals into different subgroups in terms of those factors. This combined approach allows researchers to investigate not only the dimensionality of the construct the items measure (latent factors) but also the classification of individuals (latent classes).³ Specifically in multiple informant studies, the informant congruence and incongruence are separately modeled as latent factors and further the covariates associated with informant congruence and incongruence can be investigated. In addition, using the trifactor mixture model, researchers can examine how similar or dissimilar the targets are in terms of the common factor and informant perspective factors. If the targets are heterogeneous, researchers can further investigate what predictors are related to such heterogeneity and how the heterogeneous groups affect other outcomes.

The article is structured as follows. We first present the technical overview of trifactor mixture model and discuss its advantages in the use of multi-informant data. Then, we demonstrate the model with student self-rated and teacher-rated academic behaviors. The illustration includes the trifactor mixture model with predictors and outcomes of latent class membership. In this illustration, model specification and testing procedures are explicated in detail. Finally, the article concludes with discussions on several methodological and practical challenges in conducting the trifactor mixture analysis with multi-informant data and suggestions for future research.

Trifactor Mixture Model

Trifactor Model

The trifactor model is a measurement model in which the relations of items with the factors are specified (Jöreskog, 1969). Considering that multi-informant data possibly have more than one sources of variability and those sources are assumed to be independent of each other, Bauer et al. (2013) suggested to build a measurement model with three types of orthogonal factors: common factor that measures the construct of interest, informant perspective factors each of which represents the unique perspective of the corresponding informant, and item specific factors each of which represents the uniqueness of the item conditional on the other substantive factors. Under the assumption of unidimensionality of the common factor and multivariate normality of observed variables for simplicity, the trifactor model is expressed as

y_{irt} = ν_{ir} + λ_{ir}^{(C)} η_{t}^{(C)} + λ_{ir}^{(P)} η_{rt}^{(P)} + λ_{ir}^{(S)} η_{it}^{(S)} + ε_{irt}

(1)

where $y_{irt}$ is the observed response of informant r on item i for target t, $ν$ is an intercept, $λ$ is a factor loading, $η$ is the latent factor score, and $ε$ is the error. The superscripts (C), (P), and (S) represent the common factor, 1, . . ., R informant perspective factors, and 1, . . ., I item-specific factors, respectively. Although we assume the multivariate normality of observed variables for simplicity, this distributional assumption can be relaxed and a different distribution can be modeled depending on the scale of the items (Bauer et al., 2013).

This model is illustrated in Figure 1. Each item loads on three orthogonal factors, that is, the common factor, one of informant perspective factors, and one of item specific factors. In other words, the rating of each target is a combination of unique contributions of three factors. This measurement model can be easily extended to a structural model by including a set of predictors of the latent variables which is expressed in matrix form as

η = α + Γ X + ζ

(2)

where $η$ is the k×1 vector of factors, $α$ is the k×1 vector of intercepts, $Γ$ is the k×p matrix of regression coefficients for the p×1 vector of $X$ , $ζ$ is a k×1 vector of disturbance; k is the total number of factors in the model (i.e., 1 for common factor +R for informant perspective factors +I for item specific factors); and p is the number of predictors. In this conditional trifactor model, the effect of a predictor on each factor can be hypothesized and tested. The trifactor model specification will be explained in detail with an example in the illustration section.

Figure 1.

Conditional trifactor mixture model of academic behaviors with a covariate.

Trifactor Mixture Model

Factor mixture modeling (FMM) is the general approach that combines the confirmatory factor analysis with unobserved continuous variables (i.e., latent factors) and the latent class analysis with unobserved categorical variables (i.e., latent classes) (Lubke & Muthén, 2005). FMM has advantages over latent class analysis because it incorporates a measurement model that not only partials out measurement errors from the observed scores but also allows explicit tests of psychometric properties, including measurement invariance across latent classes (Kim et al., 2018).

The trifactor mixture model is a special case of the general factor mixture model. In mixture modeling, the density function of the outcome variables, $f (Y)$ is a mixture of density functions of J classes (Yung, 1997):

f (Y) = \sum_{j = 1}^{J} π_{j} f_{j} (Y)

(3)

where $π_{j}$ is the probability of being in class j satisfying the assumptions $π_{j} > 0$ and $\sum_{j = 1}^{J} π_{j} = 1$ . For the trifactor mixture model under multivariate normality, $f_{j} (Y)$ within latent class j (j = 1, . . ., J) is written as

(y_{irt} | j) = ν_{irj} + λ_{irj}^{(C)} η_{tj}^{(C)} + λ_{irj}^{(P)} η_{rtj}^{(P)} + λ_{irj}^{(S)} η_{itj}^{(S)} + ε_{irtj}

(4)

where all notations are explained above. Similar to the conditional trifactor model, the probability of each target to belong to a certain latent class can be further explained by regressing the class membership on a set of predictors in the multinomial logistic regression model. On the other hand, the estimated class membership can be used as a predictor of other outcome variables.

Advantages of Trifactor Mixture Model for Multi-informant Research

Bauer et al. (2013) presented three advantages of the trifactor model. In addition to those advantages, the trifactor mixture model will enhance the use of multi-informant data in several aspects. Because the trifactor model is a measurement model, it is useful for item development and validation for a multiple score system. Researchers can assess the relative contribution of each factor on each item. For example, an item with the high factor loading of common factor, relatively low factor loading of informant factor, and no or ignorable impact of item specific factor can be retained for a valid scale. Beyond these, the trifactor mixture model allows researchers to test whether one trifactor model holds for all targets or there are subgroups that have different trifactor structures. For example, the relations of items with a certain informant factor could be higher or lower in one group than the other indicating that the informant perspective is stronger or weaker for a certain group of targets. Relatedly, measurement invariance can be explicitly evaluated across unobserved subgroups for each factor, which will be particularly vital for the use of common factor scores⁴ because measurement invariance is a prerequisite for a meaningful comparison of scores across groups (Kim et al., 2018).

In the conditional model, a researcher can inspect which covariates are associated with class membership as well as each factor. Thus, beyond identifying latent factors and latent classes, the trifactor mixture model allows researchers to describe those unobserved variables and further explain their relations with other variables. More specifically, researchers can investigate which predictors are associated with informant congruence and incongruence and concurrently who is more likely to belong to a certain subgroup. For example, by including covariates on the trifactor, Bauer et al. (2013) showed that parental perspective factors were related with parental impairment such as parental depression. Of note is that the trifactor model is basically confirmatory and thus latent factors are predetermined based on theory and substantive research interest, but mixture modeling is inherently exploratory. That is, the extracted latent classes should be theoretically justified and explained. Hence, the inclusion of covariates is a crucial part in trifactor mixture modeling to understand who belong to each subgroup (latent class) and what their characteristics are. The trifactor mixture model is also flexible in incorporating outcomes of latent factors and latent classes to explain how they affect the outcomes.

Finally, common factor scores are generated which are based on the common view of multiple informants on the target without item specific bias and measurement error (Bauer et al. 2013). Thus, the common factor scores are considered more reliable than observed scores and better aggregation by properly weighting different contributions of items and informants than simply averaging scores across informants (van Dulmen & Egeland, 2011). Also, researchers do not need to choose one of multiple informants.

By allowing heterogeneity, researchers are informed of different groups based on targets’ standings in the factors. This latent categorization of individuals overcomes the limitations of arbitrary grouping that uses a certain cutoff score such as a median split: observed score categorization depends on the researcher’s choice of cutoff and does not take into account measurement errors (MacCallum et al., 2002). For example, the clinical cutoff score method is used with multiple informant data (assigning 1 for an individual whose score rated by an informant is in a clinical range and aggregating clinical scores across informants), but van Dulmen and Egeland (2011) reported that its predictive validity was low. On the other hand, when heterogeneity is detected among targets, the trifactor mixture model yields their probability to belong to a certain class along with their most likely class membership that informs researchers of the placement of targets in different subgroups.

Illustration

In this study, we purported to examine the congruence and incongruence between students and teachers in their ratings of student academic behaviors by building a trifactor mixture model with the common factor and informant perspective factors and in tandem to identify heterogeneous groups of targets based on their scores on such factors. In addition, to explain the extracted latent factors and latent classes, we employed student grade as a predictor in the trifactor mixture model. The mean differences across latent classes in reading and math achievement scores were also examined. In this illustration, we explicate how to specify and identify a trifactor mixture model, to test measurement invariance across latent classes, and to select a model to determine the number of latent classes and the level of measurement invariance. When predictors or outcomes are considered in the trifactor mixture model, when and how to enter those variables are discussed.

Participants and Measures

The data in this study were extracted from FastBridge Learning (FBL; www.fastbridge.org). The sample included 24,094 students across kindergarten through twelfth grade in the 2016-2017 academic year in the United States (age M11.07 years, SD 2.61, 14.7% missing; 49.6% males, 47.0% females, 3.4% missing; 40.3% White, 29.2% non-White, 30.5% missing).

The Social, Academic, and Emotional Behavior Risk Screener (SAEBRS; Kilgus & von der Embse, 2014) assessment suite is a multi-informant universal screening system that includes teacher, parent (Taylor et al., 2019), and student versions (von der Embse et al., 2017) and was well validated in previous studies (e.g., Iaccarino and others 2019). In this study, we used the academic behavior subscale in the SAEBRS–Student Rating Scale (i.e., mySAEBRS) and SAEBRS–Teacher Rating Scale (i.e., SAEBRS-TRS). Academic behaviors are defined as behaviors that influence a student’s ability to benefit from and participate in academic instruction and include both facilitative (production of acceptable work) and inhibitive (distractedness) items (Kilgus et al., 2015). For both student and teacher rating scales, the academic behavior subscale has six items in 4-point Likert-type scale and these items are parallel between student and teacher scales (see Table 1). After reverse-coding of negatively worded items, higher scores indicate better academic behaviors. The Cronbach’s alpha was .63 for students and .92 for teachers.

Table 1.

Item Descriptive Statistics and Correlations of Academic Behaviors Rated by Students (S) and Teachers (T).

Statistic	S1	S2	S3	S4	S5	S6	T1	T2	T3	T4	T5	T6
M	2.04	2.02	2.13	1.76	2.17	2.15	2.23	1.97	2.24	2.23	2.19	2.24
SD	0.94	0.90	0.85	1.10	0.91	0.90	0.93	0.93	0.83	0.82	0.83	0.86
Skewness	−0.78	−0.76	−0.57	−0.26	−0.75	−0.75	−1.03	−0.63	−0.68	−0.66	−0.55	−0.77
Kurtosis	−0.23	−0.10	−0.62	−1.29	−0.50	−0.43	0.09	−0.42	−0.67	−0.65	−0.81	−0.48
Correlation
S2	.32
S3	.13	.18
S4	.03	.12	.28
S5	.09	.10	.36	.27
S6	.10	.19	.46	.41	.35
T1	.19	.21	.26	.09	.18	.19
T2	.16	.22	.25	.10	.16	.20	.66
T3	.18	.21	.33	.14	.21	.22	.62	.60
T4	.16	.20	.30	.18	.21	.23	.55	.56	.74
T5	.18	.22	.30	.16	.22	.23	.59	.63	.77	.80
T6	.17	.21	.31	.14	.21	.24	.60	.63	.81	.74	.76

Note. Correlations are all statistically significant at α = .01. S1/T1 = Difficulty working independently; S2/T2 = Distractedness; S3/T3 = Good grades; S4/T4 = Like school; S5/T5 = Participation; S6/T6 = Preparedness for instruction.

Of note is that although all 24,094 students completed the self-report of academic behaviors without missing responses, only 14,560 students had the corresponding teacher ratings. In other words, almost 40% of students were missing in teacher ratings. However, missingness in teacher ratings was not associated with any items and student demographic variables. Thus, missing data were handled with full information maximum likelihood utilizing all available cases under the assumption of missing at random.

We treated the 4-point Likert-type scale variables as continuous. Although 4-point Likert-type scales can be considered as categorical, estimating the threshold structure of categorical data seems by and large challenging especially as the number of latent classes increases (Lubke & Neale, 2008; Wang et al., 2020), which will be discussed further later. The item statistics did not show any serious departure from normality with reasonable skewness and kurtosis values.

Model Specification and Identification

Three types of factors were constructed in the trifactor model: common factor, informant perspective factor, and item specific factor. All 12 items, six student self-ratings and six teacher ratings, loaded on the common factor that represents academic behaviors. In addition, the six items of student ratings loaded on the student perspective factor; the six items of teacher ratings loaded on the teacher perspective factor. Because each item was rated by both students and teachers, the item specific factor was also modeled to assess item bias. This item specific factor is analogous to the item correlation between two raters. It is assumed that these three types of factors are orthogonal to each other and their factor correlations were fixed at zero. From this orthogonality the variances in the student and teacher ratings are partitioned to multiple sources and the contribution from each of three sources can be evaluated by comparing the magnitude of factor loadings associated to each factor.

For the common factor of academic behaviors, measurement invariance between students and teachers was assumed. Thus, for each item the factor loading of students was constrained equal to that of teachers as illustrated in Figure 1. This invariance constraint is not necessary for informant factors because when informants are distinctive like students and teachers, it is reasonable to assume that student and teacher perspective factors represent different constructs unique to each informant. Based on the previous study of the trifactor model using the current data (von der Embse et al., 2019) the intercepts were constrained equal between the parallel items of students and teachers except one item. The two loadings of each item specific factor should be constrained equal and its variance should be fixed at one for model identification. This equality-constrained factor loading represents the correlation between the parallel items rated by multiple informants conditional on the common and informant factors. With the current data of academic behaviors, we observed that several item specific factor loadings were estimated at zero and thus constrained at zero.

For the identification of the trifactor mixture model with multiple latent classes, the variances of common and student perspective factors were fixed at one for the reference class (the last class in this example) while the teacher perspective factor variance was allowed to be freely estimated across all latent classes. The means of the common factor and the student and teacher perspective factors were fixed at zero for the reference class while those of the other latent classes were freely estimated so that the estimated means of the other classes represent the factor mean differences from the reference class.⁵ All item-specific factor means and variances were fixed at zero and one, respectively, for identification.

Measurement Invariance Testing Across Latent Classes

Because the trifactor mixture model is a measurement model, we can evaluate the level of measurement invariance across latent classes: configural, metric, and scalar invariance. In the configural invariance model, all parameters were allowed to vary across latent classes except the aforementioned constraints made for identification and residual variances. Note that residual variances should be freely estimated across classes in the configural invariance model. However, the free estimation of residual variances often leads to nonconvergence in factor mixture modeling and thus we constrained them equal across classes. For the metric invariance model, the factor loadings were constrained equal across latent classes. For the scalar invariance model, the intercepts were constrained equal across latent classes in addition to the factor loadings. The level of measurement invariance was determined by comparing these models for the best fit as explained in detail in the following section. When scalar invariance was not satisfied in this illustration, we considered partial scalar invariance by relaxing two item intercepts to be different across latent classes.

Testing Procedures and Model Selection

In the trifactor mixture modeling, the number of latent classes is unknown and so is the level of measurement invariance (Kim et al., 2017; Lubke & Muthén, 2005). Thus, both should be determined simultaneously. A common practice in the exploratory approach is to build a series of models with one to a certain number of latent classes in different levels of measurement invariance. Thus, as Kim et al. (2017) demonstrated, we constructed three levels of measurement invariance for each latent class model: one-class, two-class scalar, two-class metric, two-class configural, . . ., four-class scalar, four-class metric, and four-class configural. These models were compared and the number of classes and the level of invariance were concurrently determined by identifying the best fitting model among them.

To select the best fit model in model comparisons, multiple criteria are usually employed. In this example, we used information criteria (IC) such as Akaike information criterion (AIC; Akaike, 1974), Bayesian information criterion (BIC; Schwarz, 1978), and sample-size adjusted BIC (saBIC; Sclove, 1987). The model with the lowest IC values is generally considered as the best fit. When two models are nested by relaxing additional parameters to estimate, for example, two- and three-class models, a likelihood ratio test (LRT) is also a common method of model comparison. However, the LRT statistic (−2*log likelihood) in the mixture modeling does not follow the chi-square distribution given the degrees of freedom difference between two competing models because regularity conditions do not hold (McLachlan & Peel, 2000).⁶ Thus, alternative LRTs were proposed. In this example, we used Lo–Mendall–Rubin likelihood ratio test (LMR LRT) and adjusted LMR LRT. The statistical significance in the LRT indicates that the C class model is supported over the C– 1 class model. The distinctiveness of latent classes and the accuracy of class membership assignment can be evaluated with the entropy but it was reported that the entropy (or the assignment accuracy) tends to decrease as the number of latent classes increases (Collins & Lanza, 2010).

Trifactor Mixture Model With Covariates or Outcomes

The variable we considered for the demonstration of the trifactor model with covariates was student grade. We selected sixth grade as a reference and created 12 dummy-coded variables that represent each grade level from kindergarten to 12th grade in comparison with sixth grade. The 12 student grade variables were entered as predictors in the trifactor mixture model. In the factor mixture modeling with both latent factors and latent classes, the covariates can be modeled in many different ways. The covariates can be introduced to explain the extracted latent classes, which is illustrated as Path (A) in Figure 1. The covariates can explain the variability in latent factor scores (common factor and informant perspective factors), which is illustrated as Path (B) in Figure 1. Finally, by including a path from a covariate to one of the items, we can model and test the measurement noninvariance of the item in terms of the covariate (Path (C)).

In the mixture modeling literature, when to include covariates has been discussed even though no consensus has been reached (Wang et al., 2020). One option is to include covariates when determining the number of latent classes simultaneously. Lubke and Muthén (2007) and Asparouhov and Muthén (2014) found that this simultaneous approach enhanced the correct class enumeration and class assignment by improving the class separation when the covariates were the true predictors of class membership. On the other hand, the inclusion of covariates changes the relationship of the variables and thus, models with and without covariates produce different solutions with respect to the number of classes and the class assignment of individuals. To avoid the shift of latent classes due to the inclusion of covariates, researchers recommended the three-step approach (e.g., Vermunt, 2010). That is, latent classes are identified and estimated first. Then, the estimated class probabilities and measurement errors are saved. In the third step, the effects of predictors on the predetermined class membership are estimated and tested with the uncertainty of class membership taken into account.

In this illustration, we first established the trifactor model and determined the number of latent classes without covariates (namely, unconditional trifactor mixture model). Then, while fixing the number of latent classes to that determined with the unconditional trifactor mixture model, the grade variables were included in three different ways: as predictors of latent classes, as predictors of latent classes and latent factors, as predictors of latent classes, latent factors, and potential DIF (differential item functioning) item intercepts. Note that this approach is different from the three-step approach because the estimated class probabilities from the unconditional model were not saved and used when the covariates were added. Thus, we closely investigated whether or not the class membership estimated in the unconditional model changed notably when the grade variables were introduced as predictors. When the grade variables explained latent classes only, we also employed the three-step approach using the R3step option in Mplus.

Finally, we investigated how the identified latent classes of academic behaviors are different with respect to reading and math achievement scores. The reading and math scores were standardized within grade.⁷ Then, we used the three-step approach to explain reading and math scores with the predetermined class membership as a predictor adopting the BCH option in Mplus.

Throughout this study, we set alpha at .001 given a large sample size. We used Mplus 7.4 and 8.4 (Muthén & Muthén, 2015, 2019) for all data analyses (version 8.4 for the BCH and R3STEP options). See Supplemental Appendix (available online) for the Mplus syntax. For the estimation method, we adopted the Mplus default, robust maximum likelihood. To avoid local maximum which is common in the mixture model estimation, we initially set 200 and 10 for the initial stage random starts and final stage optimization, respectively, but when the best log likelihood was not replicated, we increased the values up to 500 and 50, respectively. Note that all the models up to three classes had the best log likelihood replicated. For four or more classes, we consistently observed one class with a very low or zero proportion and it appeared that the increase of random starts (up to STARTS = 500 50) did not change the solutions. We were cognizant of unstable and untrustworthy solutions especially with local maxima and thus scrutinized the parameter estimates based on our knowledge of the estimates from previous studies and also compared the solutions across different models to check whether adding a class resulted in notable changes in the estimates.

Results

Descriptive Statistics

The descriptive statistics of six items of academic behaviors rated by students and teachers are presented in Table 1. The means of the 4-point scale items ranged from 1.76 to 2.17 for students and from 1.97 to 2.24 for teachers. The absolute values of skewness and kurtosis were all less than 1.5. The item correlations of teacher ratings were moderate to high whereas those of student ratings were lower especially for items 1 and 2. The correlations between student and teacher ratings ranged from .09 to .33.

Unconditional Trifactor Mixture Model: Class Enumeration and Measurement Invariance

The results of trifactor mixture models and simultaneous measurement invariance testing are summarized in Table 2. Note that the models with five latent classes did not yield reasonable solutions: for example, the estimated proportion was zero for one class. Thus, we compared the trifactor models of up to four latent classes.

Table 2.

Class Enumeration and Measurement Invariance of Trifactor Mixture Model for Academic Behaviors.

Parameter	Classes
	1	2	2	2	2	3	3	3	3	4	4	4	4
MI		Scalar	pScalar	Metric	Config	Scalar	pScalar	Metric	Config	Scalar	pScalar	Metric	Config
−LL	263573	261623	261153	260978	259600	260579	255972	259462	258205	259892	250257	254106	256548
Parm	38	42	45	48	62	46	52	58	86	50	59	68	110
AIC	527222	523330	522396	522051	519324	521249	512049	519040	516582	519884	500633	508349	513316
BIC	527530	523670	522760	522439	519825	521621	512470	519509	517277	520289	501110	508899	514206
saBIC	527409	523536	522617	522287	519628	521475	512304	519324	517004	520130	500923	508683	513856
LMR		.00	.00	.00	.00	.00	.00	.00	.00	.02	.04	.69	failed
ALMR		.00	.00	.00	.00	.00	.00	.00	.00	.02	.04	.69	failed
BLRT		.00	.00	.00	.00	.00	.00	.00	.00	.00	.00	failed	failed
Entropy		.85	.86	.86	.63	.75	.73	.77	.72	.63	.77	.66	.60
Prop		.22	.42	.44	.32	.13	.14	.08	.12	.05	.01	.09	.09
Time(hh:mm)	0:01	2:54	5:48	5:45	14:03	6:33	15:45	27:16	140:55	38:08	29:05	0:27	0:12

Note. N = 24,094. MI = measurement invariance; pScalar = partial scalar; Config = configural; LL = log likelihood; Parm = number of free parameters; AIC = Akaike information criterion; BIC = Bayesian information criterion; LMR = Lo–Mendell–Rubin likelihood ratio test p value; ALMR = adjusted LMR p value; BLRT = bootstrap likelihood ratio test p value; Prop = proportion of the smallest class; Time = elapsed time to run the model in hours and minutes.

First, because the scalar invariance models generally fit worse than the metric invariance models, partial scalar invariance was considered. Two of the item intercepts (Items 3 and 4) were allowed to be freely estimated across classes, which improved model fit considerably. In terms of the number of classes, the LRT and adjusted LRT preferred three latent classes when the alpha .001 was applied. On the other hand, all ICs supported the four-class partial scalar invariance model with the lowest values. However, after visually inspecting all fitted models and scrutinizing all parameter estimates, we selected the three-class partial scalar invariance model for several reasons. In the four-class partial scalar invariance model, the proportion of one class was only 1%. Importantly, the results were almost identical between three- and four-class partial scalar invariance models except this 1% class which was a fraction of one class in the three-class solution. Also note that we found several models with lower ICs but did not consider them because they produced unreasonable parameter estimates (e.g., one of factor loadings was unacceptably large and the parameter estimates were wildly different from those in the other models).

The parameter estimates of three-class partial scalar invariance model are shown in Table 3. The patterns and relations of trifactors in the trifactor mixture model were generally consistent with those of the trifactor model that did not analyze the population heterogeneity in the previous study (von der Embse et al., 2019). Item specific factor loadings were either zero or near zero indicating minimum item bias. Common factor loadings were substantial with magnitude between .28 and .52, but lower than teacher perspective factor loadings, which ranged between .43 and .91. Student perspective factor loadings were between −.07 and .59. Interestingly, Items 1 and 2 of student ratings were mostly related with the common factor with minimal contribution from the student unique perspective factor. Note that these two items showed very low correlations with the other student items but higher correlations with teacher items. As observed in item correlations and Cronbach’s alphas, teacher ratings showed much higher R² (.44-.99) than student ratings (.24-.52).

Table 3.

Standardized Item Parameters of Trifactor Mixture Models for Academic Behaviors.

		Factor loading
Item	Intercept	Common	Perspective	Item specific	R ²
S1	2.30 (.01)	.46 (.01)	−.15 (.03)	.00^a	.24
S2	2.25 (.01)	.52 (.01)	−.07 (.02)^b	.00^a	.27
S3	3.05 (.02)	.36 (.02)	.53 (.01)	−.05 (.01)	.42
S4	1.65 (.02)	.28 (.02)	.42 (.01)	.06 (.01)	.25
S5	2.35 (.01)	.37 (.01)	.40 (.01)	.00^a	.30
S6	2.46 (.01)	.42 (.01)	.59 (.02)	.00^a	.52
T1	2.38 (.02)	.48 (.01)	.46 (.01)	.00^a	.44
T2	2.26 (.02)	.52 (.01)	.43 (.01)	.00^a	.45
T3	3.46 (.03)	.41 (.02)	.90 (.01)	−.06 (.02)	.98
T4	2.05 (.02)	.40 (.02)	.91 (.01)	.08 (.01)	.99
T5	2.77 (.02)	.44 (.01)	.71 (.01)	.00^a	.70
T6	2.69 (.02)	.46 (.01)	.71 (.01)	.00^a	.71
Factor means	Class 1	Class 2	Class 3
Common	−0.30 (.04)	−0.43 (.04)	0
Student P	−0.36 (.04)	0.23 (.04)	0
Teacher P	0.08 (.04)^c	−0.31 (.03)	0

Note. N = 24,094. Factor loadings of common factor and item-specific factors and the intercepts of all six items were constrained equal between students and teachers. However, after standardization, those constrained parameter estimates became different between students and teachers. Standard errors are within parentheses. The parameter estimates are all statistically significant with p < .001 with two exceptions.

The factor loading was constrained at zero. ^bp = .005. ^cp = .024.

Under partial scalar invariance, we observed three heterogeneous classes based on the means of three factors (common factor, teacher perspective factor, and student perspective factor), which are reported at the bottom of Table 3 and illustrated in Figure 2. Of note is that one of the classes was a reference for which the means of three factors were all set to zero. The majority of participants (70.3%) turned out to be in the reference class. In other words, the large proportion of participants belonged to the group in which their scores of common factor, student perspective factor, and teacher perspective factor are consistent on average (Class 3, namely, congruent). On the other hand, the other two classes showed obvious disagreement between students and teachers. Interestingly, disagreement happened in the lower tiers of common factor. The class we named as high student perspective (Class 2, 14.4%) showed that student-specific factor scores were high whereas common factor and teacher perspective factor scores were generally low. The class we named as high teacher perspective (Class 1, 15.2%) showed the opposite pattern: Teacher-specific factor scores were high whereas common and student-specific factor scores were low.

Figure 2.

Plot of the estimated trifactor means by latent classes.

Trifactor Mixture Models With Predictors and Outcomes

We constructed the relations of predictors (12 dummy-coded grade variables) with class membership in three different ways as illustrated earlier in Figure 1: Grade explained class membership only; class membership and common and informant factors; and class membership, latent factors, and item intercepts that were deemed noninvariant across grade levels. These three models were compared using model selection criteria as demonstrated in class enumeration. Including grade variables as predictors of class membership did not result in any notable changes from the unconditional trifactor mixture model. However, when we included the effects of predictors on the latent factors in addition to class membership, we observed considerable changes not only in the class assignment but also in the parameter estimates and some of the estimates were not very reasonable. Thus, although the third model in which the grade variables explain latent classes, latent factors, and items showed the lowest ICs, we report the model with predictor effects on only class membership for illustration purposes because this model preserved the three classes we identified in the unconditional model.

The unstandardized coefficients of grade variables on class membership and the corresponding odds ratios (the odds of a certain grade students in comparison to sixth graders to belong to Class 1 or Class 2 instead of the reference Class 3) are shown in Table 4. We observed a general tendency that middle school students in Grades 7 through 9 were more likely to belong to high teacher perspective class than the congruent class. On the other hand, lower grade-level students such as third to fifth graders were likely to belong to high student perspective class, but the effects were not statistically significant at α = .001 with odds ratio less than 1.50. Of note is that we did not control for any other covariates and the effects could be different by modeling other effects such as grade effects on the latent factors. For example, when the effects of grade on latent factors were also modeled, the grade variables showed apparent effects on student perspective factors (but almost no effect on the other two factors), and the grade effects on class membership were not shown. Thus, this conditional model with grade effects on class membership only should be considered as demonstration and the results here should be taken with caution.

Table 4.

Conditional Trifactor Mixture Model With Covariates (Grade).

	Class 1 (High teacher)			Class 2 (High student)
Grade	β	SE	OR	β	SE	OR
GRADE0	−0.15	(0.19)	0.87	0.13	(0.17)	1.14
GRADE1	−0.09	(0.17)	0.92	0.34	(0.15)	1.40
GRADE2	−0.09	(0.12)	0.92	0.28*	(0.11)	1.32
GRADE3	−0.16	(0.10)	0.85	0.31*	(0.09)	1.36
GRADE4	0.07	(0.09)	1.07	0.26*	(0.09)	1.30
GRADE5	0.07	(0.09)	1.07	−0.04	(0.09)	0.96
GRADE7	0.48**	(0.10)	1.61	−0.40*	(0.13)	0.67
GRADE8	0.55**	(0.10)	1.74	−0.37*	(0.13)	0.69
GRADE9	0.53**	(0.14)	1.70	−0.67*	(0.20)	0.51
GRADE10	0.32	(0.15)	1.38	−0.26	(0.18)	0.77
GRADE11	0.12	(0.16)	1.13	−0.28	(0.18)	0.75
GRADE12	−0.02	(0.20)	0.99	−0.70*	(0.25)	0.49

Note. GRADE 6 is the reference and Class 3 (Congruent) is the reference.

p < .01. **p < .001.

Finally, we investigated whether the identified three latent classes differ in their academic achievement, reading and math using the BCH option in Mplus. Chi-square tests showed that all pairwise comparisons as well as the overall mean comparison were statistically significant. The congruent class showed the highest performance in reading and math followed by the high teacher perspective class. Because the reading and math scores were standardized, the mean differences shown in Table 5 can be considered as standardized mean differences or effect sizes. The high student perspective class showed substantially lower reading and math scores on average than the congruent and high teacher perspective classes.

Table 5.

Chi-square Tests of the Differences Between Latent Classes in Reading and Math.

Reading			Math
Class	Mean		Class	Mean
High teacher (1)	−0.08		High teacher (1)	−0.03
High student (2)	−0.32		High student (2)	−0.33
Congruent (3)	0.08		Congruent (3)	0.07
Chi-square test	Mean difference	χ²	Chi-square test	Mean difference	χ²
Overall		158.35	Overall		153.94
1 vs. 2	0.24	41.17	1 vs. 2	0.30	64.89
1 vs. 3	0.16	29.56	1 vs. 3	0.10	13.48
2 vs. 3	0.40	152.71	2 vs. 3	0.40	153.48

Note. N = 24,094. All chi-square tests are statistically significant at p < .001.

Discussion

The trifactor model is useful for separating the common factor from the informant perspective factors taking account of item bias and measurement errors and simultaneously classifying targets into subgroups in terms of their scores in those factors. We also demonstrated that the extracted latent factors and latent classes can be further explained by a set of predictors or can explain other variables of interest. However, generally the factor mixture model including the trifactor mixture model as a special case is very complex because two unobserved components (latent factors and latent classes) should be estimated in tandem. Thus, there are several issues to consider when conducting a factor mixture analysis.

First, nonconvergence and unstable solutions are common issues in factor mixture modeling. Nonconvergence could occur when an additional class is forced to be estimated when there is no further heterogeneity. In this case, the proportion of one class could turn out to be zero even when the model converges. However, the high complexity of the fitted model could also lead to nonconvergence, for example, fully unconstrained model such as configural invariance model. Thus, researchers may have to impose unintended constraints to have the model converged or to obtain an admissible solution. Sometimes the software program, for example, Mplus automatically constrains some parameters to avoid singularity or other nonconvergence problems and presents warning messages.

Future research is called for about the robustness of factor mixture modeling under the unnecessary constraints that are imposed to improve convergence. In growth mixture modeling. McNeish and Harring (2020) proposed a constrained model (i.e., a covariance pattern mixture model in which factor variances are fixed at zero) to increase convergence without sacrificing the ability to address substantive research questions in applied research. Others may argue that any unnecessary constraints could result in model misspecification, which in turn results in biased estimates and other statistical consequences. For example, Morin et al. (2011) noted that the constrained model could result in substantially different solutions from the unconstrained growth mixture model although both models extracted the same number of latent classes. Potentially even more problematic, constraints on different parameters could lead to different solutions (Nylund-Gibson & Masyn, 2016; Vermunt, 2010).

In the illustration, we constrained residual variances equal across latent classes. In confirmatory factor analysis with observed groups, Joo and Kim (2019) found that the misspecification of residual variances and covariances (specifying equal residual variances and covariance between groups when they were heterogeneous) could affect the test of measurement invariance (particularly, metric invariance) especially with misspecified covariances between groups, but such constraints were less consequential in estimating and testing factor means. On the other hand, Enders and Tofighi (2008) reported that misspecifying residual variances equal across latent classes in the growth mixture modeling led to biased and less precise estimates of growth trajectories and variance components. Without established best practice under model nonconvergence, researchers in practice may not know whether the constraints are correct or incorrect specifications (thus help or hurt) necessitating an inspection of the potential impact of the constraints on the solutions to make an informed decision. For example, we observed that some models yielded very different solutions from the original trifactor model (without mixture) and some estimates were not reasonable (e.g., unstandardized factor loading >20; intercept >10). Hence, even when these models showed lower ICs, they were not considered in the model comparisons. Similarly, when the software program shows warning or error messages involving nonconvergence or inadmissible solutions, researchers should inspect what parameter causes such error and how it could be avoided. For example, we often had error messages about the item specific factor loadings and found the estimates were literally zero. The error message disappeared after we constrained the related parameters at zero. Methodologically, future research on which parameter constraint (e.g., equal residual variances or factor variances across latent classes) makes a more or less impact on the performance of FMM is also much needed.

Evaluating and developing proper model selection methods in mixture modeling is another area on which more methodological work is needed. One critical issue is that the model selection criteria that are commonly employed in the mixture modeling such as information criteria and LRTs are known to be sensitive to model complexity and sample size not to mention class separation. Thus, different studies in model selection methodology provide different suggestions without consensus (Clark et al., 2013; Kim et al., 2016; Nylund et al., 2007; Tein et al., 2013). For example, BIC, one of the most widely used criteria, is known to be less sensitive to class separation when sample size is small given model complexity (e.g., Kim et al., 2016; Lukočienė & Vermunt, 2010; Yang, 2006). However, when sample size is extremely large, all ICs and LRTs tend to over-extract latent classes (Kim & Wang, 2019). In other words, simply increasing sample size may not be a solution although FMM requires a large sample for reasonable solutions. In reality, it is difficult to know whether latent classes are underextracted given a relatively small sample size, overextracted due to over sensitivity of ICs to minor heterogeneity, or correctly enumerated. Hence, one consensus among methodologists is that it is a crucial practice to evaluate the extracted classes theoretically. As a side note, bootstrap LRT is sometimes recommended for model selection, but it should be noted that the execution time is tremendously longer, for example, maximum 145 hours for three-class configural invariance model in our example.

The mixture model is inherently exploratory because latent classes are unobserved and unknown in advance. The extracted latent classes do not have specific labels⁸ and should be described on the basis of their characteristics. Thus, thorough investigation of the identified classes is essential in mixture modeling. Often this is done by including a set of predictors that can explain the class membership. In the trifactor model introducing covariates allows the investigation of the effects of predictors (e.g., target and informant characteristics variables) on the latent factors and latent classes and thus enhances the interpretations of the extracted factors and classes (e.g., who is more likely belong to each latent class). However, there is no consensus about when and where to include covariates in the mixture model. For example, it is reported that the inclusion of proper predictors of class membership in the mixture model improves correct class enumeration and classification accuracy (Asparouhov & Muthén, 2014; Lubke & Muthén, 2007). However, it is challenging to identify a good and quality predictor before seeing the extracted latent classes in reality and researchers in practice may naturally take the route of three-step approach (Vermunt, 2010). In addition, it is difficult to know whether the predictors can predict which component of the model such as class membership, latent factors, observed indicators, or any combination of them. Recently, Wang et al. (2020) investigated the role of covariates in factor mixture modeling and found that a model comparison approach using ICs can help search for a correct specification of covariates. That is, researchers can take an exploratory approach to determine which predictor should be included for which component. However, as noted earlier, different ways of incorporating covariates will lead to different solutions of the factor mixture model and the resulting factors and classes could be profoundly different across models. Thus, the decision could depend on research interest, and theoretical considerations will be of importance.

Implications for Multi-Informant Studies

The challenges with multi-informant data include how to model and assess the congruence and incongruence between informants and how to use the scores when they disagree. The trifactor mixture model allows researchers to explicitly separate the agreement from the disagreement using latent factors: common factor and informant perspective factors. Such separation is more meaningful with the trifactor model because item specific factors and measurement errors are also taken into account by employing a measurement model. Furthermore, the explicit modeling of common and informant perspective factors brings on many benefits as discussed earlier. For example, researchers can assess relative contribution of common and informant factors on each item, which is essential to the development and validation of multi-informant scales. In the illustration, we found that two items in the student self-report had very low correlations with the other items. However, the trifactor mixture model showed that these items were fairly related to the common factor with little or no contribution from the student perspective factor. In other words, these items are legitimately valid items of academic behaviors without much impact of student unique perspectives on academic behaviors. This finding cannot be discovered without unraveling the contributions of common and perspective factors on each item.

The trifactor mixture model has another important implication to the multi-informant research. One explanation of the incongruence between informants in the literature is that multiple informants fundamentally interpret and respond to the items of a construct differently (Flake & Petway, 2019). This theoretical explanation can be tested with the trifactor mixture model, that is, measurement invariance testing between informants. Importantly, measurement invariance can be tested for common and informant perspective factors, separately. Measurement invariance between informants in terms of common factor is imperative for the meaningful interpretation and use of common factor scores whereas, as Bauer et al. (2013) noted, measurement invariance is not assumed for informant perspective factors when informants are distinctive (e.g., students and teachers). In addition to the explicit test of measurement invariance between informants on each factor, the trifactor mixture model enables testing measurement invariance across subgroups of targets as demonstrated in this study. Through such investigation researchers are informed of potential groups of targets who are affected more or less by informant perspective factor or informant bias.

Furthermore, employing predictors in the trifactor mixture model enhances the investigation on the factors related to concordance and discordance to address questions such as why informants disagree and what characteristics of informants are related to divergent perspectives. Because common factor and each of informant factors are all separately modeled, researchers have flexibility in including predictors relevant to each factor, for example, student characteristics variables to explain the student perspective factor and both teacher and student characteristics variables for the teacher perspective factor. Also, predictors and outcomes are employed to describe and explain the subgroups of targets that are identified in the mixture modeling. For example, we found that students who scored high in their own perspective factor but low in both common factor and teacher perspective factor of academic behaviors showed lower performance in reading and math achievement compared to the other students.

One interesting finding in this study is that in the two latent classes (i.e., high student and high teacher) students actually scored low for the common academic behavior factor although they scored high for one of informant perspective factors. This could indicate that disagreement between students and teachers occurred more likely when students had poor academic behaviors. This could further imply that selecting one of informants could lead to incorrect decisions about student academic behaviors and demonstrate the importance of using multi-informant data. For example, when student self-report is used for academic behaviors, some students at risk may be excluded from receiving proper intervention at school because their self-ratings are high.

The combined approach to multi-informant data with latent continuous and latent categorical variables can be useful to develop an intervention for a specific group of targets considering the environment where they need such intervention. Suppose multiple informants observe the targets in different settings (e.g., teachers at school and parents at home rate child problem behaviors) and a group of students are identified as high at school in terms of teacher perspective factor but low at home based on parent perspective factor. Then, an intervention could be developed for this specific group considering the environment (e.g., school settings) where problem behaviors more likely occur (Roskam, 2018).

The trifactor mixture model generates factor scores that would be useful for multi-informant research. Researchers can model the predictors and outcomes of the factors directly in the trifactor model or use saved factor scores in subsequent data analysis. However, a note of caution is warranted for the use of factor scores especially for clinical decisions. The quality of factor scores depends on the correct specification of the model from which the scores are derived. It is not known how robust the factor scores are under model misspecification including under- or over-specification in the trifactor mixture model. Similarly, classification accuracy of the trifactor mixture model needs further investigation.

Limitations of the Study

In the demonstration, we treated the 4-point Likert-type scale variables as continuous. Many simulation studies investigated and compared various estimation methods for ordered-categorical data which are commonly used in social sciences. One common recommendation is that treating Likert-type scale data as continuous along with the maximum likelihood estimation is acceptable with five or more categories, but when the number of categories is three or less, categorical data analysis with robust weighted least squares is recommended (e.g., Beauducel & Herzberg, 2006; DiStefano & Morgan, 2014; Lei, 2009; Rhemtulla et al., 2012). However, previous simulation studies on mixture modeling reported estimation challenges with ordered-categorical data and consequently unsatisfactory correct enumeration even with a fairly large sample size (Lubke & Neale, 2008; Wang et al., 2020). The deteriorated performance of categorical factor mixture modeling may not be surprising because estimating one more class adds many more thresholds to estimate instead of one intercept per item, which increases the model complexity tremendously with the complication of threshold invariance across latent classes. However, treating 4-point Likert-type scale data as continuous and applying linear FMM should not be considered as an acceptable practice because the robustness of linear FMM with ordered-categorical data has not been investigated yet. Generally, the performance of FMM with ordered-categorical data warrants further investigation.

Another major limitation of the study is that we could not take into account the data dependency with students nested within teachers when estimating the trifactor mixture model because the teacher identification variable is not available. In addition, methods to handle data dependency specifically for the trifactor model or trifactor mixture model have not been discussed although multilevel factor mixture modeling in general has been demonstrated and investigated. Kim et al. (2018) demonstrated the multilevel confirmatory factor analysis model to test measurement invariance between multiple informants when one group of informants are nested within the other group of informants, for example, students nested within teachers. However, in their demonstration the construct of interest was classroom social environment, specifically, promoting social interaction which was at the teacher level. On the other hand, the trifactor mixture model demonstrated in the current study is for student academic behaviors which is at the student level. Accordingly, a different model specification is necessary, which calls for future research.

Conclusion

In this paper, we demonstrated how the trifactor mixture model that allows the latent continuous and latent categorical variables in tandem could enhance the research with multi-informant data. The trifactor mixture model is useful for multi-informant research to improve understanding of the common and unique perspectives of informants on the targets and enhance the use of scores from multiple informants. However, there are still many methodological challenges with factor mixture modeling in general and the users should be cognizant of such challenges. Also, more methodological work is needed to increase the usability of this model given its profound benefits.

Supplemental Material

sj-pdf-1-epm-10.1177_0013164420973722 – Supplemental material for Combined Approach to Multi-Informant Data Using Latent Factors and Latent Classes: Trifactor Mixture Model

Supplemental material, sj-pdf-1-epm-10.1177_0013164420973722 for Combined Approach to Multi-Informant Data Using Latent Factors and Latent Classes: Trifactor Mixture Model by Eunsook Kim and Nathaniel von der Embse in Educational and Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Eunsook Kim

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Achenbach

T. M.

(2011). Commentary. Definitely more than measurement error: But how should we understand and deal with informant discrepancies? Journal of Clinical Child & Adolescent Psychology, 40(1), 80-86. https://doi.org/10.1080/15374416.2011.533416

Achenbach

T. M.

Krukowski

R. A.

Dumenci

Ivanova

M. Y.

(2005). Assessment of adult psychopathology: Meta-analyses and implications of cross-informant correlations. Psychological Bulletin, 131(3), 361-382. https://doi.org/10.1037/0033-2909.131.3.361

Achenbach

T. M.

McConaughy

S. H.

Howell

C. T.

(1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213-232. https://doi.org/10.1037/0033-2909.101.2.213

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. https://doi.org/10.1109/TAC.1974.1100705

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Joint Committee on Standards for Educational and Psychological Testing.

Asparouhov

Muthén

(2014). Auxiliary variables in mixture modeling: Three-step approaches using Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 329-341. https://doi.org/10.1080/10705511.2014.915181

Bauer

D. J.

Howard

A. L.

Baldasaro

R. E.

Curran

P. J.

Hussong

A. M.

Chassin

Zucker

R. A.

(2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475-493. https://doi.org/10.1037/a0032475

Beauducel

Herzberg

P. Y.

(2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186-203. https://doi.org/10.1207/s15328007sem1302_2

Clark

S. L.

Muthén

B. O.

Kaprio

D’Onofrio

B. M.

Viken

Rose

R. J.

(2013). Models and strategies for factor mixture analysis: An example concerning the structural underlying psychological disorders. Structural Equation Modeling: A Multidisciplinary Journal, 20(4), 681-703. https://doi.org/10.1080/10705511.2013.824786

10.

Collins

L. M.

Lanza

S. T.

(2010). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. Wiley.

11.

De Boeck

Wilson

Acton

G. S

. (2005). A conceptual and psychometric framework for distinguishing categories and dimensions. Psychological Review, 112(1), 129-158. https://doi.org/10.1037/0033-295X.112.1.129

12.

De Los Reyes

. (2011). Introduction to the special section. More than measurement error: Discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. Journal of Clinical Child & Adolescent Psychology, 40(1), 1-9. https://doi.org/10.1080/15374416.2011.533405

13.

DiStefano

Morgan

G. B.

(2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 425-438. https://doi.org/10.1080/10705511.2014.915373

14.

Enders

C. K.

Tofighi

(2008). The impact of misspecifying class-specific residual variances in growth mixture models. Structural Equation Modeling: A Multidisciplinary Journal, 15(1), 75-95. https://doi.org/10.1080/10705510701758281

15.

Flake

J. K.

Petway

K. T.

II ., (2019). Methodologies for investigating and interpreting student–teacher rating incongruence in noncognitive assessment. Educational Measurement: Issues and Practice, 38(1), 63–77. https://doi.org/10.1111/emip.12201

16.

Iaccarino

von der Embse

N. P.

Kilgus

S. P.

(2019). Interpretation and use of the Social, Academic, and Emotional Behavior Risk Screener: A latent transition approach. Journal of Psychoeducational Assessment, 37(4), 486-503. https://doi.org/10.1177/0734282918766650

17.

Joo

S.-H.

Kim

E. S.

(2019). Impact of the strict invariance violations when testing a group mean difference and measurement invariance using MIMIC. Behavior Research Methods, 51(6), 2688-2699. https://doi.org/10.3758/s13428-018-1124-6

18.

Jöreskog

K. G.

(1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34, 183-202. https://doi.org/10.1007/BF02289343

19.

Kilgus

S. P.

Sims

von der Embse

N. P.

Riley-Tillman

T. C.

(2015). Confirmation of models for interpretation and use of the Social and Academic Behavior Risk Screener (SABRS). School Psychology Quarterly, 30(3), 335-352. https://doi.org/10.1037/spq0000087

20.

Kilgus

S. P.

von der Embse

N. P.

(2014). The Social, Academic, and Emotional Behavior Risk Screener—Teacher Rating Scale [Measurement instrument]. FastBridge Learning.

21.

Kim

E. S.

Cao

Wang

Nguyen

(2017). Measurement invariance testing with many groups: A comparison of five approaches. Structural Equation Modeling: A Multidisciplinary Journal, 24(4), 524-544. https://doi.org/10.1080/10705511.2017.1304822

22.

Kim

E. S.

Joo

S.-H.

Lee

Wang

Stark

(2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 870-887. https://doi.org/10.1080/10705511.2016.1196108

23.

Kim

E. S.

Wang

Kiefer

S. M.

(2018). Cross-level group measurement invariance when groups are at different levels of multilevel data. Educational and Psychological Measurement, 78(6), 973-997. https://doi.org/10.1177/0013164417739062

24.

Kim

E. S.

Wang

(2019). Investigating sources of heterogeneity with 3-step multilevel factor mixture modeling: Beyond testing measurement invariance in cross-national studies. Structural Equation Modeling: A Multidisciplinary Joural, 26(2), 165-181. https://doi.org/10.1080/10705511.2018.1521284

25.

Kraemer

H. C.

Measelle

J. R.

Ablow

J. C.

Essex

M. J.

Boyce

W. T.

Kupfer

D. J.

(2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160(9), 1566-1577. doi: 10.1176/appi.ajp.160.9.1566

26.

Laird

R. D.

De Los Reyes

(2013). Testing informant discrepancies as predictors of early adolescent psychopathology: Why difference scores cannot tell you what you want to know and how polynomial regressionmay. Journal of Abnormal Child Psychology, 41(1), 1-14. https://doi.org/10.1007/s10802-012-9659-y

27.

Lei

P.-W.

(2009). Evaluating estimation methods for ordinal data in structural equation modeling. Quality and Quantity, 43, Article 495. https://doi.org/10.1007/s11135-007-9133-z

28.

Lubke

G. H.

Muthén

(2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21-39. https://doi.org/10.1037/1082-989X.10.1.21

29.

Lubke

Muthén

B. O.

(2007). Performance of factor mixture models as a function of model size, covariate effects, and class-specific parameters. Structural Equation Modeling: A Multidisciplinary Journal, 14(1), 26-47. https://doi.org/10.1080/10705510709336735

30.

Lubke

Neale

M. C.

(2006). Distinguishing between latent classes and continuous factors: Resolution by maximum likelihood? Multivariate Behavioral Research, 41(4), 499-532. https://doi.org/10.1207/s15327906mbr4104_4

31.

Lubke

Neale

M. C.

(2008). Distinguishing between latent classes and continuous factors with categorical outcomes: Class invariance of parameters of factor mixture models. Multivariate Behavioral Research, 43(4), 592-620. https://doi.org/10.1080/00273170802490673

32.

Lukočienė

Vermunt

J. K.

(2010). Determining the number of components in mixture models for hierarchical data. In Fink

Berthold

Seidel

Ultsch

(Eds.), Advances in data analysis, data handling and business intelligence (pp. 241-249). Springer. https://doi.org/10.1007/978-3-642-01044-6_22

33.

MacCallum

R. C.

Zhang

Preacher

K. J.

Rucker

D. D.

(2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7(1), 19-40. https://doi.org/10.1037/1082-989X.7.1.19

34.

McDonald

R. P.

(1967). Non-linear factor analysis. Psychometric Monographs, 15, 167.

35.

McLachlan

Peel

(2000). Finite mixture models. Wiley.

36.

McNeish

Harring

(2020). Covariance pattern mixture models: Eliminating random effects to improve convergence and performance. Behavior Research Methods, 52, 947-979. https://doi.org/10.3758/s13428-019-01292-4

37.

Morin

A. J. S.

Maÿano

Nagengast

Marsh

H. W.

Morizot

Janosz

(2011). General growth mixture analysis of adolescents’ developmental trajectories of anxiety: The impact of untested invariance assumptions on substantive interpretations. Structural Equation Modeling, 18(4), 613-648. doi: 10.1080/10705511.2011.607714

38.

Muthén

B. O.

Muthén

L. K.

(2015). Mplus 7.4 [Computer software]. Muthén & Muthén.

39.

Muthén

B. O.

Muthén

L. K.

(2019). Mplus 8.4 [Computer software]. Muthén & Muthén.

40.

Nylund

K. L.

Asparouhov

Muthén

B. O.

(2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 535-569. https://doi.org/10.1080/10705510701575396

41.

Nylund-Gibson

Masyn

K. E.

(2016). Covariates and mixture modeling: Results of a simulation study exploring the impact of misspecified effects on class enumeration. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 782-797. https://doi.org/10.1080/10705511.2016.1221313

42.

Rhemtulla

Brosseau-Liard

P. E.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354-373. https://doi.org/10.1037/a0029315

43.

Roskam

(2018). The clinical significance of informant agreement in externalizing behavior from age 3 to 14. Child Psychiatry & Human Development, 49, 563-571. https://doi.org/10.1007/s10578-017-0775-3

44.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464. https://doi.org/10.1214/aos/1176344136

45.

Sclove

S. L.

(1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333-343. https://doi.org/10.1007/BF02294360

46.

Steinley

McDonald

R. P.

(2007). Examining factor score distributions to determine the nature of latent spaces. Multivariate Behavioral Research, 42(1), 133-156. https://doi.org/10.1080/00273170701341217

47.

Taylor

Allen

Kilgus

S. P.

von der Embse

N. P.

Garbacz

S. A.

(2019). Development and validation of a parent version of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) in an elementary sample. Behavioral Disorders, 44(4), 205-214. https://doi.org/10.1177/0198742918798561

48.

Tein

Coxe

Cham

(2013). Statistical power to detect the correct number of classes in latent profile analysis. Structural Equation Modeling: A Multidisciplinary Journal, 20(4), 640-657. https://doi.org/10.1080/10705511.2013.824781

49.

van Dulmen

M. H. M.

Egeland

. (2011). Analyzing multiple informant data on child and adolescent behavior problems: Predictive validity and comparison of aggregation procedures. International Journal of Behavioral Development, 35(1), 84-92. https://doi.org/10.1177/0165025410392112

50.

Vermunt

J. K.

(2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450-469. https://doi.org/10.1093/pan/mpq025

51.

von der Embse

N.P.

Iaccarino

Mankin

Kilgus

Magen

. (2017). Development and validation of the Social, Academic, and Emotional Behavior Risk Screener—Student Rating Scale. Assessment for Effective Intervention, 42(3), 186-192. https://doi.org/10.1177/1534508416679410

52.

von der Embse

N. P.

Kim

E. S.

Kilgus

Dedrick

Tanaka

Sanchez

. (2019). Multi-informant universal screening: Evaluation of rater, item, and construct variance using a trifactor model. Journal of School Psychology, 77(December), 52-66. https://doi.org/10.1016/j.jsp.2019.09.005

53.

von der Embse

Kim

Jenkins

Sanchez

Kilgus

Eklund

. (under review). Informant discrepancy within universal screening in predicting distal outcomes. Journal of Psychopathology and Behavioral Assessment.

54.

Wang

Kim

Ferron

J. M.

Dedrick

R. F.

Tan

T. X.

Stark

(2020). Testing measurement invariance across unobserved groups: The role of covariates in factor mixture modeling. Educational and Psychological Measurement. Advance online publication. https://doi.org/10.1177/0013164420925122

55.

Wang

Kim

Joo

S.-H.

Chun

Alamri

Lee

Stark

(2020). Reconsidering multilevel latent class models: Can level-2 latent classes affect item response probabilities? Journal of Experimental Education. Advance online publication. https://doi.org/10.1080/00220973.2020.1737913

56.

Yang

C. C.

(2006). Evaluating latent class analysis models in qualitative phenotype identification. Computational Statistics & Data Analysis, 50(4), 1090-1104. https://doi.org/10.1016/j.csda.2004.11.004

57.

Yung

Y.-F.

(1997). Finite mixtures in confirmatory factor-analysis models. Psychometrika, 62(3), 297-330. https://doi.org/10.1007/BF02294554

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB