Visualizing Latent Class Models with Analysis-of-distance BIPLOTS

Abstract

The authors propose using categorical analysis-of-distance biplots to visualize the posterior classifications arising from a latent class (LC) model. Using this multivariate plot, it is possible to visualize in two (or three) dimensions the profile of multiple LCs, specifically both the within- and between-class variation, and the overlap or separation of the classes together with the class weights. Furthermore, visualization of the relative density of each of the data patterns associated with a class is possible. The authors illustrate this approach with real data examples of LC models with three and more classes.

Keywords

biplots latent class (LC) models multidimensional visualization

1. Introduction

In the present article we propose applying biplot methodology to visualize the class-specific response probabilities arising from a latent class (LC) model. The goal is to introduce a multidimensional visualization tool that can help in providing visual insight into better understanding the posterior classifications arising from an LC model. LC models are widely used in the social and medical sciences to group individuals or cases on the basis of their response patterns on a set of (usually categorical) items (Goodman 1974). Examples include the creation of a grouping concerning tolerance toward nonconformity (McCutcheon 1985) or groups of disease severity on the basis of results on a set of medical tests used when a “gold standard” is lacking (Ibironke et al. 2012). Since the use of the LC approach has become more widespread, there is continuous interest in visualizing the LC model (Magidson and Vermunt 2001; McCutcheon 1998; Van der Heijden, Gilula, and Van der Ark 1999). This has occurred because visualization tools can greatly aid in interpreting LC models by giving a graphical insight into the properties of the LC model.

The need for better visualization tools has been motivated by the shortcomings of the currently available tools. There are two types of visualization tools available: univariate and multivariate. The univariate profile plot, the most common tool, represents the class-specific response probabilities on the indicator variables. Although it is frequently used, this plot has many disadvantages. Most important, these plots capture only the class-specific means but not the variance around them. Other problems also may become evident: when the number of categories that need to be presented is large (which can easily happen even with an LC model with five nominal indicators), the plots become too crowded and hard to read. Furthermore, these plots cannot visually represent the distances between the classes or their relative weights (importance), all of these being a consequence of using a univariate plot to visualize multivariate data. An alternative is to use a multidimensional plot known as a ternary plot (Greenacre 2007). In Latent GOLD, a ternary plot is called a tri-plot, and it is shown that it exactly reproduces the posterior profile of a three-class model (Magidson and Vermunt 2001; Vermunt and Magidson 2013). However, a ternary plot cannot visualize more than three LCs. If a ternary plot is used to visualize models with more than three classes, the number of classes first has to be reduced to three classes, although this may be done in various ways to provide different exploratory views. The following procedure is then often followed in practice: in addition to the two largest classes, a third class is formed by taking a (weighted) average over the remainder of the classes. The resulting three classes are then visualized by a ternary plot. In order to be able to visualize all classes separately, or to compare all classes, multiple ternary plots are necessary.

An alternative multivariate approach was proposed by Van der Heijden et al. (1999), who showed the relationship between LC analysis (LCA) and correspondence analysis for the two-variable case and between LCA and multiple correspondence analysis for the multivariate case. On the basis of these relationships, the authors provided bivariate graphs based on a contingency table and multivariate graphs based on the Burt matrix. These plots are evaluated in terms of inner products, which are not easy to visualize or to interpret. Although in this graph, the groups are interpolated, and as such they are supplementary, our goal is different. We want to propose a multivariate plot that separates the class means optimally, instead of simply interpolating them into the plot.

We propose the use of biplots, more specifically of analysis-of-distance (AoD) biplots for grouped categorical data (Gower, Le Roux, and Gardner-Lubbe 2014), to visualize the posterior class assignments arising from an LC model. Using this approach, it becomes possible to use a multivariate visualization of LC models with more than three classes that focuses on the distances between the class centroids, which the visualization tools currently used do not allow.

Furthermore, using this biplot-based approach, it is possible not only to visualize the overall class-specific response probabilities (denoted as class means or centroids in what follows) but also to show the within-class dispersions, thus providing deeper insight into the distribution of the data patterns that belong to the same LC. Closely related to this is also the capacity of the biplots to give a deeper understanding into the quality of classification. AoD biplot methodology allows the construction of classification bags around the class centroids. These classification bags contain the inner 100 $α$ percent of the cases belonging to an LC, where $α$ is a value between 0 and 1. Because the value of $α$ can be specified by the user, the latter is provided with a tool for quantifying the overlap or separation among the different LCs. The same type of measures could be applied to ternary plots as well; however, they are useful only for three-class models.

Using the AoD categorical biplot methodology, we can thus create a classification bag around the LC centroid containing the collection of cases belonging to each LC, respectively. The cases that are on the borders of the bags, or those that are outside the bags or overlapping with the bags of other classes, are the cases that are difficult to be classified, or are misclassified and as such potential causes of overall misfit as well.

Although this methodology for visualization can be used in any situation in which LC modeling is applied, it can be of particular importance in research in which the primary interest is not only in the model parameters but also more specifically in the posterior classifications. For example, in medical research, LCA is used to compare tests for diagnosing the same disease when a gold standard is lacking (Ibironke et al. 2012). In such cases, having a visualization tool that can show which cases are problematic could be advantageous for practitioners in gaining understanding under which circumstances false positives or negatives can occur.

Our article is structured as follows: Section 2 introduces the basic LC model and illustrates it using a real data example, also applying the currently used visualization tools: univariate profile plot, and multivariate ternary plot (which are used in mainstream software, namely, the univariate plot in MPlus, R package poLCA, and Latent GOLD and the ternary plot in Latent GOLD). Section 3 introduces the properties of the AoD biplot methodology pertaining to our application. This is followed in Sections 4 and 5 by a comparison of the AoD biplot for the three-class model with traditional visualizations for such models. Section 6 presents a five-class model in which the proposed AoD bi-plot comes to its own.

2. The LC Model

Let $y_{i k}$ denote the observed response of individual $i$ on one of $K$ categorical items, where $k = 1, 2, . . ., K$ and $i = 1, 2, . . ., n$ . Let $R_{k j}$ denote the $j$ th level (category) of the $k^{t h}$ categorical variable, where $j = 1, 2, . . ., L_{k}$ and $R_{i k j}$ denotes the category chosen by individual $i$ for variable $k$ . The full response vector for individual $i$ can then be written as ${y^{'}}_{i} = (y_{i 1}, y_{i 2}, . . ., y_{i k})$ , where $y_{i k} = R_{i k j}$ . LCA assumes that individuals belong to one of the $T$ categories of an underlying categorical latent variable $W$ that affects the responses (Goodman 1974; Hagenaars 1990; McCutcheon 1987). Denoting a particular LC by $t$ , the model can be formulated as follows:

P (Y_{i}) = \sum_{t = 1}^{T} P (W = t) P (Y_{i} | W = t),

where the $P (W = t)$ is the mixing proportions, or class weights (sizes), and the $P (Y_{i} | W = t)$ represents the class-specific distribution of the responses $Y_{i}$ .

The responses of each case to the $K$ categorical items are usually assumed to be locally independent given the unit’s LC membership. The conditional probability of the $i^{t h}$ case given the LCs can then be written as a product of conditional item responses:

P (Y_{i} | W_{i} = t) = Π_{k = 1}^{K} P (Y_{i k} | W_{i} = t) = Π_{k = 1}^{K} Π_{r = 1}^{R_{k}} π_{k t r}^{I (Y_{i k} = r)},

where the indicator variable $I (Y_{i k} = r) = 1$ if subject $i$ has response $r$ on item $k$ and $0$ otherwise.

Combining equations (1) and (2), the LC model can be written as

P (Y = y) = \sum_{t = 1}^{T} P (W = t) Π_{k = 1}^{K} Π_{r = 1}^{R_{k}} π_{k t r}^{I (Y_{i k} = r)} .

Generally, after estimating an LC model, the interest of researchers is in obtaining the posterior class membership probabilities, $P (W = t | Y = y)$ , calculated in terms of the model parameters defined in equation (3) using Bayes’s theorem:

P (W = t | Y = y) = \frac{P (Y = y | W = t) P (W = t)}{P (Y = y)} .

These posterior class membership probabilities provide information about the distribution over the $T$ classes of the individuals with response pattern $y$ . On the basis of these values, cases are assigned to the various classes (Goodman 2007). Using modal assignment, the most well-known approach, each case is assigned to the class to which it has the highest probability of belonging. Denoting the posterior classification by $S$ , this can be expressed as follows: $P (S = s | Y_{i}) = 1$ if $P (W = s | Y_{i}) > P (W = t | Y_{i})$ for all $s \neq t$ and is equal to 0 for all other classes. An alternative, proportional assignment assigns each case to each of the posterior classes with a weight equal to $P (W = s | Y_{i})$ . The assigned class membership scores can be used in further analysis, for example, in relating assigned class membership to distal outcomes or to predictors of the classification (Vermunt 2010). In the current article we show how these assigned scores can be plotted and how the plots can be used to make inferences about the qualitative description of the classes.

3. A Three-Class Example: Socon Data

As a first illustration of an LC model with accompanying graphics, we consider data coming from the Socio-Cultural Developments (SOCON) study, conducted in the Netherlands, on a sample of 542 respondents. We selected five items that measure the opinion of respondents on women’s liberation. All items have three response categories: “agree” (a), “disagree” (d), and “neutral” (n). The wording of the items is as follows:

A. Women’s liberation sets women against men.

B. It is better for a wife not to have a job because it leads to problems in the household, especially if there are children.

C. It is most natural when the man is the breadwinner and the woman runs the household and takes care of the children.

D. It is not as important for a girl to get a good education than for a boy.

E. A woman is better suited to raise small children than a man.

We report the results of the three-class model, which fitted the data well ( $L^{2}$ = 158.44). Table 1 presents the class weights and estimated class-specific response probabilities on the five indicators. The first row represents the class weights (relative sizes), showing that the largest is class 1 (42 percent of respondents), followed by class 2 (39 percent) and class 3 (19 percent). For a substantive description of the classes, we can look into the class-specific response probabilities on the indicators. The first class, which we label the liberal class, has a high probability of disagreeing on all items, as shown in the second column of Table 1. At the same time, class 3, the conservative class, tends to agree with all items, having more conservative views on women’s liberation. The middle class, called the neutral class, has a high probability of answering the neutral category on most items.

Table 1.

Class-specific Response Probabilities and Class Weights for the Three-class LC Model of the SOCON Study Data

	Class Weight
Category-Level Points	Liberal (.42)	Neutral (.39)	Conservative (.19)
Aa	.05	.14	.54
An	.23	.45	.23
Ad	.72	.41	.23
Ba	.03	.01	.56
Bn	.05	.24	.23
Bd	.92	.74	.21
Ca	.01	.40	.95
Cn	.10	.40	.00
Cd	.89	.20	.05
Da	.02	.00	.17
Dn	.01	.07	.21
Dd	.97	.93	.62
Ea	.19	.65	.84
En	.18	.21	.06
Ed	.64	.13	.11

Note: LC = latent class; SOCON = Socio-Cultural Developments.

Instead of describing the class-specific response probabilities to interpret the LC model on the basis of Table 1, many researchers prefer to first look on the univariate plot as provided in Figure 1. This plot visualizes the class-specific response probabilities and is provided by most software packages for LCA (Mplus, R package poLCA, and Latent GOLD), as discussed by Muthén and Muthén (2012), Linzer and Lewis (2011), and Vermunt and Magidson (2013). In this figure the $x$ -axis contains all the category-level points (CLPs) for each of the indicator variables, and the $y$ -axis contains the class-specific response probabilities. The liberal class shows a consistent pattern on all of the indicators, namely, an ordered tendency to have the highest probability of answering “disagree” on all items, followed by neutral, and having very low probabilities of agreeing on all items at hand. The conservative class also shows a consistent pattern across the indicators, with the exception of the fourth indicator. That is, it mirrors the response probabilities prevalent in the liberal class, it tends to have the highest probability of agreeing on all items, and it has low probabilities of disagreeing and of being neutral. The neutral class shows the highest probability of scoring neutral on almost all items.

Figure 1.

The profile plot of the class-specific response probabilities, $P (Y = y | W = t)$ , for the Socio-Cultural Developments study data.

A weakness of the univariate profile plot is that the variables on the $x$ -axis are not well separated visually. It can easily be seen that with more variables or more categories of the nominal variables, this type of plot can become overpopulated, hence impairing its legibility. Furthermore, this plot can visualize only the overall class-specific response probabilities but not the spread of the data around it.

An alternative is to use a ternary plot, as shown in Figure 2. This is an exact two-dimensional representation of the average posterior class membership probabilities ( $P (W = t | Y = y)$ ; see equation 4 for a definition) of a three-class model. Using this plot, the parameters of the model are essentially re-expressed in terms of row percentages instead of column percentages. In this plot, each variable has a different symbol, and each of its CLPs is plotted. The lines connecting the CLPs to the axes can be interactively drawn for each CLP, and thus the exact class-specific response probability read off. The strong separation of the classes can be spotted as well: all the “disagree” CLPs are in the lower left corner, which corresponds to probabilities close to 1 belonging to the liberal class and probabilities closer to 0 belonging to the other classes. At the same time, the CLPs that “agree” on all items cluster around the upper corner corresponding to the conservative class, and the lower right corner is characterized by the “neutral” categories, belonging to the neutral class. The “agree” and “neutral” CLPs have a larger dispersion than the “disagree” CLPs. It is also possible to interpolate the samples into this plot (not shown here). The biggest problem with this approach is that when applied to models involving more than three classes some form of pooling of the classes must be performed. Inevitably the characteristics of the pooled classes are then lost. Moreover, the pooling might lead to completely wrong conclusions.

Figure 2.

Ternary plot of the posterior response probabilities ( $P (W = t | Y = y$ ) for the Socio-Cultural Developments study data.

4. Employing Aod BiPlots for Grouped Categorical Data to LC Modeling

We propose using biplot methodology to visualize the assigned class memberships arising from an LC model. This methodology allows a multivariate visualization of LC models with more than three classes. These graphical visualizations have the advantage that they often lead to a better understanding of the LC structures when applied to LC models. In its simplest form, a biplot is the simultaneous graphical display of cases and variables contained in a cases × variables data matrix; hence the name biplot. Biplot methodology is primarily not a method of analysis but a convenient way to visualize in one, two (the most common occurrence in practice), or three dimensions multidimensional information (Gower, Lubbe, and Le Roux 2011). Using cases × variables data matrix biplots show patterns in the data, spread of the data, overlap and separation of LCs, overall and within classes variation, and the like. A biplot display is generally an approximation in lower dimensions of data existing in a high-dimensional space. Therefore, the lower dimensional display space that captures or explains most of the variation in the full multidimensional space is usually provided by biplot methodology.

There are many different types of biplots, depending on how dissimilarities between cases (samples) are defined. Overviews of biplots for continuous measurements and for categorical variables are given by Gower, Le Roux, and Gardner-Lubbe (2015, 2016). Well-known methods such as canonical variate analysis and principal components analysis biplots are based on the same principle of intersample distance that forms the basis for the unifying biplot methodology advocated by Gower and Hand (1996). Understanding the similarity between the different approaches provides for a flexibility in the choice of the distance measure most useful for visualizing the data at hand. This points to an AoD underlying the specific choice of biplot to be constructed (Gower et al. 2014; Le Roux, Gardner-Lubbe, and Gower 2014).

The LC models considered in this article are based on categorical variables. Therefore, we propose applying the AoD biplots to visualize LC models. Common choices of dissimilarity measures when dealing with categorical data include chi-square ( $χ^{2}$ ) distance and distances derived from the extended matching coefficient (EMC), but other choices are easy to incorporate. AoD biplots are developed specifically for optimally separating samples belonging to different externally defined groups (or classes). Although our focus is on categorical data, by changing the choice of distance measure, the AoD approach can easily be modified to visualize LC models with continuous indicators or when indicators form a mixture of continuous and categorical variables. It can be shown that the group means (centroids) of $T$ externally defined groups (classes) can be represented exactly in a $T - 1$ dimensional space, and after transformation to a so-called canonical space they can be approximated in a one-, two-, or three-dimensional display space in which distances can be interpreted as ordinary Euclidean distances. The approximation is obtained by performing a principal coordinate analysis, as will be explained below. The $n$ samples, together with the categorical variables in the form of CLPs analogous to calibrated axes for continuous variables, are projected into the display or biplot space, showing the distribution of the samples around the respective group means, together with CLPs that distinguish or characterize a sample or group mean.

Although the ternary plot directly visualizes the posterior classification probabilities, $P (W = t | Y = y)$ , in the AoD biplot, the assigned class membership scores with the corresponding class means are visualized.¹ In this article we consider two approaches for using assigned class membership scores for the biplot: modal assignment and weighted assignment. Modal assignment inherently leads to a crisp partitioning; all cases belonging to the same data pattern are assigned to only one class. To account for the situation in which each data pattern can belong to multiple classes with different probabilities, we also propose weighted assignment, an approach based on proportional assignment. We present here a step-by-step introduction into creating and interpreting an AoD biplot to visualize LC models. We will avoid in-depth technical descriptions, referring instead to articles in which the key derivations can be found. The following five steps can be used to create a categorical AoD biplot:

Derive the coordinates of the group centroids in the $T - 1$ dimensional space and plot them in two (or three) dimensions.

Add the CLPs to the plot.

Add all the cases belonging to the respective groups to the two-dimensional (or three-dimensional) display.

Add enhancement tools such as classification regions and alpha bags.

Calculate measures of fit for the different approximations.

After introducing each step in detail, we show its implementation using a three-class LC model derived for the SOCON data as an example. This is then followed by an application of the AoD biplot approach to a five-class LC model. In the latter case, a ternary plot is not available for a graphical display of the five classes, but a graphical display in the form of an AoD biplot is available.

4.1. Representation of the Group Centroids

Suppose our data are arranged in matrix $X : n \times K$ , where the $k^{t h}$ column gives the category levels recorded for each of the $n$ samples on the $k^{t h}$ nominal categorical variable. We assume that the $k^{t h}$ categorical variable has $L_{k}$ different category levels. Denote by $C_{k}$ the $n \times L_{k}$ indicator matrix with the $i^{t h}$ row consisting of zeros except a single unit in the column pertaining to the actual level recorded for the $i^{t h}$ sample on variable $k$ . We form the matrix

C = [C_{1}, C_{2}, C_{3}, . . ., C_{K}] : n \times L,

where $L = L_{1} + L_{2} + \dots + L_{K}$ . The frequencies of all category levels are then given by the diagonal elements of the $L \times L$ matrix: $L = d i a g (C' C)$ .

AoD requires an $n \times n$ distance or dissimilarity matrix to be computed. Any dissimilarity measure can be used. Chi-square distance or dissimilarities derived from the EMC suggested by Gower and Hand (1996) are commonly used for categorical variables. The $χ^{2}$ -distance between rows $i$ and i′ of $C$ is a weighted Euclidean distance computed as the ordinary Euclidean distance between the corresponding rows of $\frac{1}{\sqrt{K}} C L^{- 1 / 2}$ . The EMC expresses the number of matches for every pair of samples as a ratio of the number of variables $K$ . The corresponding dissimilarities are then given by the matrix

1 1^{'} - CC' / K .

After calculating the dissimilarities, an $n \times n$ matrix $D$ is computed with elements equal to minus a half times the squared dissimilarities.

We assume that the $n$ samples are members of $T$ known groups and define an indicator matrix $G : n \times T$ such that element $g_{i t}$ is equal to 0 except that it is 1 if case $i$ is in group $t$ . The different group sizes are then given by the diagonal elements of the diagonal matrix $N = d i a g (G' G)$ . In particular, we are interested in representing the $T$ group centroids graphically. Geometrically, these $T$ group centroids lie in a $T - 1$ dimensional subspace of the space containing the $n$ samples. After Le Roux et al. (2014), we will call the space containing the group centroids the delta-space. To obtain a graphical display of the group means in the delta-space, we proceed as follows: first, from matrix $D : n \times n$ we form matrix $D^{*} : T \times T$ , which contains the means of the entries in $D$ common to the relevant groups. Gower and Hand (1996) showed that the squared distances between the centroids of groups $t$ and t′ are given by

Δ_{t t'} = D_{t t}^{*} + D_{t' t'}^{*} - 2 D_{t t'}^{*} .

Next, we form matrix $Δ : T \times T$ with elements given by $- 0.50 Δ_{t t'}$ . Last, we obtain the singular value decomposition (SVD)

(I_{T} - 1_{T} {1'}_{T} / T) Δ (I_{T} - 1_{T} {1'}_{T} / T) = V Λ V',

where $1$ represents a vector of ones. We can plot the group centroids in two dimensions using the first two columns of $V Λ^{1 / 2}$ .

4.2. Adding the Reference System of the CLPs

When dealing with categorical items, information about the items is not in terms of linear biplot axes, but in terms of simplices of points, one simplex for each item, one point for each category level (i.e., the categories of the items), called CLPs (Gower and Hand 1996; Gower et al. 2011). The categories of the $k^{t h}$ variable are represented by $L_{k}$ CLPs, and for all $K$ items the total number of CLPs is given by $L = L_{1} + L_{2} + L_{3} + \dots + L_{K}$ . In the full space, the CLPs do not depend on the grouping, but they do need to be projected into the delta-space containing the group centroids.

Le Roux et al. (2014) showed that the approximation in delta-space of the CLPs of the $k^{t h}$ item can be obtained from the matrix $Z_{k (Δ)}$ where

Z_{k (Δ)} : T \times n = Λ^{- 1 / 2} V' (\begin{matrix} {g'}_{1} / n_{1} \\ ⋮ \\ {g'}_{T} / n_{T} \end{matrix}) D_{k} (I_{n} - \frac{1}{n} 1_{n} {1'}_{n}) .

In equation (7), the matrices $Λ$ and $V$ derive from equation (6) while $g_{T}$ denotes the $t^{t h}$ column of the indicator matrix $G$ . The matrix $D_{k}$ is the contribution of item $k$ to the total dissimilarity matrix $D : n \times n$ because it is assumed that the items contribute additively to the total dissimilarity, that is, to $D = \sum_{k = 1}^{K} D_{k}$ . The CLPs for item $k$ are plotted in the two-dimensional biplot by using as coordinates the first two elements of each column of equation (7). Note that this equation has $n$ columns that repeat each of the $L_{k}$ categories in the order of their appearance in the $n$ samples. An important characteristic of the biplot is that the centroid of the CLPs that characterizes each case coincides with the point representing the case. This means that analogous to continuous linear axes the CLPs can be used to infer the category levels chosen by the respective cases.

Figure 3 shows the $T$ group centroids and the CLPs for the SOCON data using the EMC as a measure of distance. Given that in this example we have only three classes, the delta-space is of dimension 2, which means that the group centroids are exactly represented in the two-dimensional biplot. Furthermore, the CLPs are projected into this two-dimensional space. All the “agree” CLPs are in the upper right corner, closest to the mean of the conservative class (symbolized by ▼), while the lower left corner is characterized by the “disagree” CLPs closest to the mean of the liberal class (▲). In the lower right corner the “neutral” CLPs are present, that is, closest to the center of the neutral class (■).

Figure 3.

Biplot showing the category-level points and group centroids (with plotting characters proportional to group sizes) for the three-class Socio-Cultural Developments study data using the extended matching coefficient measure.

For categorical data with more than two categories, the $χ^{2}$ measure is sometimes used. We applied this measure and obtained very similar graphs (not shown here but available upon request).

4.3. Adding the Samples to the Biplot

To interpolate the samples into the biplot display space, the coordinates for the samples in the delta-space are needed. Although these coordinates are exact in the full $n - 1$ dimensional space, they are projected into the delta-space and thus are not exact in the biplot. The following distances are needed for a sample to be interpolated into the biplot: the distance of the case from each group centroid as well as the distances of each group centroid from the overall centroid. When these requirements are met, Le Roux et al. (2014) showed that the coordinates of the projections of all $n$ cases into the delta-space relative to the coordinates of the group centroids are given by the columns of

Z_{(Δ)} : T \times n = Λ^{- 1 / 2} V' [(\begin{matrix} {g'}_{1} / n_{1} \\ ⋮ \\ {g'}_{T} / n_{T} \end{matrix}) D (I_{n} - \frac{1}{n} 1_{n} {1'}_{n})] .

Note that although the $n \times n$ matrix $D$ is required for interpolating the cases, only the SVD of the much smaller $T \times T$ matrix $(I_{T} - \frac{1}{T} 1_{T} {1'}_{T}) Δ (I_{T} - \frac{1}{T} 1_{T} {1'}_{T})$ on the left hand side of equation (6) is needed.

Equation (8) gives the coordinates of the projections of all the cases into the delta-space passing through the centroids of all groups. In this way the representation of the group centroids together with the cases distributed around them is obtained in the delta-space. This can be approximated in the $r$ -dimensional display space ( $r$ = 1, 2, or 3) by using as coordinates only the first $r$ elements in each column in equation (8).

Figure 4 shows the interpolated position in the biplot of a case with “agree” responses to all items. The centroid of the CLPs that characterizes a sample’s response pattern coincides with the sample point. Each case is in the centroid of the area defined by its CLPs, and as such it is closest to these CLPs than any other CLP. Figure 5 shows the AoD biplot with all the interpolated samples. Note that all samples with the same response pattern (a data pattern) are interpolated on top of one another. The spread of the sample points about their respective group centroids gives a visual description of the within-groups variation. Visual evidence of well-separated classes is evident. However, because of the differing numbers of response patterns, the interpretation of the spread of the cases must be done with caution. Section 4.4 introduces tools to aid in the interpretation of Figure 5, especially to quantify overlap and separation as well as to address problems that might occur when dealing with response patterns that have highly divergent frequencies.

Figure 4.

Biplot with an arrow pointing to the centroid of the five category-level points indicating the positions of the “agree” categories of all five items. The tip of the arrow marks the position of a case reporting an “agree” on all five items.

Figure 5.

Biplot of the Socio-Cultural Developments study data with the interpolated samples, shown as empty symbols.

4.4. Enhancing Visualization: Alpha Bags and Classification Regions

A bag plot (Rousseeuw, Ruts, and Tukey 1999) can be considered as a bivariate extension of the ordinary (univariate) boxplot. Although in a boxplot, the box contains the middle 50 percent of the data between the first quartile and the third quartile, the bag of the bag plot contains the deepest 50 percent of a bivariate scatterplot. “Deepest” is defined relative to a bivariate extension of the median, called the Tukey median or depth median (Rousseeuw et al. 1999). When the bag is allowed to contain the inner 100 $α$ percent of bivariate data points (where $α$ is a user-specified value between 0 and 1), it is called an alpha bag (Gower et al. 2011). Alpha bags are useful to show the overlap and separation among different groups as well as the within-group variation in a biplot.

In Figure 6, the biplot of Figure 5 is equipped with 90 percent bags, which separate the three classes of the SOCON data almost completely. There are only a few samples that fall outside the bags or inside the bag of another class. Furthermore, the bags provide evidence that the within-class variation is smaller for the liberal group than for the neutral group that seems to have the largest within-class variation. By varying the value of $α$ , we can express the amount of class separation quantitatively. Note that alpha bags give information regarding overlap or within-group variation that is not obscured by having a large number of cases with the same response pattern or falling on top of one another, as is the case when considering only the plotting characters for the samples. Figure 6 (and also Figure 5) shows that the horizontal dimension (explaining most of the variation) clearly distinguishes the conservative LC from the liberal class, with the neutral class occupying an intermediate position as a transition class having mostly neutral responses, but also some patterns having either “agree” or “disagree” responses. The second dimension shows a large overlap between the three classes as well as a neutral class with a centroid well below the centroids of the other two classes having centroids almost at the same height. Furthermore, it is clear that two dimensions are not adequate to judge the structure of the three LCs satisfactorily.

Figure 6.

Biplot of the Socio-Cultural Developments study data, including classification regions and 90 percent bags; the solid symbols represent the group centroids, and the nonsolid symbols represent the samples.

Figure 6 also shows the biplot superimposed with classification regions as described by Gower et al. (2011). This process involves overlaying the two-dimensional biplot space with a grid and calculating the Euclidean distance from each grid point to each of the group centroids in delta-space. The grid point is then shaded according to its shortest distance to a group centroid in delta-space resulting as a map of convex classification regions, as seen in Figure 6. However, construction of such regions often has limited value when the number of classes is larger than three. This is because the corresponding classification regions in a two-dimensional biplot are then approximated, frequently resulting in some regions that become largely concealed by others. It can be noted here that alpha bags are much more versatile visual instruments for exploring the LC structure.

It is also possible to approximate the posterior profile of a case: For each case $i$ calculate in the $r$ -space or the delta-space the distance to all $T$ category centroids using an appropriate distance measure. Let $d_{i t}$ denote the Euclidean distance (or any other appropriate distance measure) from case $i$ to class $t$ , then estimate the posterior profile of case $i$ by calculating for each $t = 1, 2, . . ., T$ the quantity

(1 / d_{i t}^{2}) / \sum_{t = 1}^{T} (1 / d_{i t}^{2}) .

The quantity in this equation for $t = 1, 2, . . ., T$ represents a normalized squared closeness for case $i$ to each group centroid with values between 0 and 1 summing to unity.

4.5. Measures of Fit Associated with the Biplot Approximations

We have seen that the biplot display of an LC structure involves the approximation of entities in a higher dimensional space in a display space of low dimension. Therefore, in order to reach valid conclusions from inspecting a biplot, it is important to have a measure of how well the biplot display fits the configurations in the full space. What is needed is a quantitative measure of the loss of information incurred by visualizing in $r$ dimensions ( $r$ = 1, 2, or 3) the true higher-dimensional configuration.

The $T$ group centroids originating from the $n$ samples are exactly represented in a $T - 1$ dimensional space, called the delta-space in this article. The $n$ samples in turn need $n - 1$ dimensions to obtain a configuration for the cases in which all the intercase dissimilarities or distances can be represented exactly. Thus the delta-space reproduces the class centroids exactly, but the variation of the individual cases around its mean is not exactly represented. Variation in data can be numerically expressed in terms of the sum of squared deviations from a mean value. Le Roux et al. (2014) show that the total sum of squares (SS) associated with the cases can be expressed in terms of SS distances. Moreover, this total SS can be partitioned analogous to an analysis of variance into a between-groups and a within-group SS. Thus we have the basic identity total = between + within given by

\frac{1' D 1}{n} = \frac{n' Δ n}{n} + \sum_{t = 1}^{T} \frac{{g'}_{t} D g_{t}}{n_{t}},

where the vector $n' = (n_{1}, n_{2}, . . ., n_{T})$ is the vector of group sizes. The AoD table resulting from equation (10) can be broken down further into contributions attributed to the different dimensions, the delta-space, the biplot space, and remaining residual dimensions to provide a quantification of the loss incurred by the approximations associated with the biplot. Thus we obtain measures of how well the group means are approximated as well as the dispersion of the cases about their respective means. Moreover, this can be done individually for each sample in order to provide evidence of the quality of the biplot representation of individual sample points.

Table 2 shows how the total SS for the SOCON data can be broken down. Because we have a three-class model in this example, the first two dimensions explain the full between SS, with 78 percent of the between-groups SS attributed to the horizontal dimension of the biplot and 22 percent to the vertical dimension. Within-group variation has components in the two dimensions of the delta-space and the (540) dimensions orthogonal to the delta-space. Therefore the average per dimension residual within group SS (398/540 = 0.74) is much less than in the delta-space.

Table 2.

The Sums of Squares Partitioning Attributed to the Delta-space per Dimension in Parentheses, and the Residual Part Indicated by $Δ^{⊥}$ , the Space Orthogonal to the Delta-space

	$Δ$ (1) (%)	$Δ$ (2) (%)	$Δ^{⊥}$ (%)	Total
Between SS	163 (78)	45 (22)	0 (0)	208
Within SS	32 (7)	41 (9)	398 (84)	471
Total SS	195 (29)	86 (13)	398 (59)	679

Note: SS = sums of squares.

5. Weighted Assignment

In most of this article, our focus has been on biplots based on modal LC assignment, but here we show how proportional LC assignment may also be accommodated. This type of assignment assumes that each response pattern has a probability equal to $P (W = t | Y = y)$ to be assigned to class $t$ for $t = 1, 2, . . ., T$ . Instead of pursuing the possibility of allowing the indicator matrix $G : n \times K$ to take on values other than 0 or 1, we propose the following procedure: assign to each of the classes a proportion of cases equal to $P (W = t | Y = y)$ . In this way each case is assigned to only one particular class, but the set of cases having the same response pattern is proportionally divided among the $T$ LCs.

Figure 7 shows a biplot that results from applying the proposed weighted assignment method to the SOCON data. Although the relative positions of the CLPs as compared with the class means remain almost unchanged from using modal assignment, the means are closer together, and the 90 percent bags show a higher degree of overlap. Nevertheless, the ordering of the classes (with the neutral class in between the two extremes) stays the same, and the classification regions are still describing the class structure well.

Figure 7.

Biplot of Socio-Cultural Developments study data based on weighted class assignment. The solid symbols represent the group centroids and the nonsolid symbols the samples.

Table 3 shows the SS partitioning for the SOCON data resulting from the weighted assignment method. An important difference between Table 3 and Table 2 (which is based on modal assignment) is that when using weighted assignment, the between-groups SS declines from 208 to 179 (i.e., from 208/679 = 30.6 percent of the total SS for modal classification) to 179/679 = 26.4 percent of the total SS for weighted assignment. This decrease in the between SS is accompanied by an increase in the corresponding within SS showing the fuzziness in the class assignment. Weighted assignment can therefore be used to construct a biplot incorporating information about the uncertainty inherent in the allocation of LCs. In the rest of this article, we focus on LC structures having more than three LCs with accompanying biplots based on modal allocation.

Table 3.

The Sums of Squares Partitioning Attributed to the Delta-space per Dimension in Parentheses and the Residual Part Indicated by $Δ^{⊥}$ , the Space Orthogonal to the Delta-space When Weighted Assignment Has Been Used

	$Δ$ (dim1) (%)	$Δ$ (dim2) (%)	$Δ^{⊥}$ (%)	Total
Between SS	146 (82)	33 (18)	0 (0)	179
Within SS	49 (10)	53 (11)	398 (84)	500
Total SS	195 (29)	86 (13)	398 (59)	679

Note: SS = sums of squares.

6. LCA with Five Classes: Solidarity and Conflict between Adult Children and Parents

The next example is a five-class model based on eight dichotomous indicators measuring multiple dimensions of solidarity and conflict between parents and their adult children, based on data from the first wave of the Netherlands Kinship Panel Study (Van Gaalen and Dykstra 2006). The first two indicators measure the existence of monthly face-to-face contact (A) and other types of monthly contact (B). Financial support was measured by whether the child has received financial support in the past year from the parents (D). Furthermore, emotional support was measured by whether the child has shown interest in the personal life of the parent in the past three months and vice versa (F). Practical support was measured by two variables: (1) whether the child had helped the parent with chores in and around the house, lending things, providing transportation, and moving things in the last three months (C), and (2) vice versa (E). In addition, material conflict (G) and emotional conflict (H) was measured. A more detailed description of the items is available in Van Gaalen and Dykstra (2006).

A five-class model fitted the data well ( $L^{2} = 226.68$ ). Table 4 contains the class weights and class-specific response probabilities of a positive answer on the eight indicators. The first item (i.e., face-to-face contact at least once a month) differentiates strongly between class 1 and class 3 on the one hand and class 4 and class 5 on the other. Class 1 and class 2 have high probabilities of giving a positive answer on all items, thus describing the most engaged parent-child relationships; however, the probability of conflict is also the highest in class 2. On the other hand, class 5 has a low probability for a positive answer on all items considered. Class 3 has a high probability of monthly face-to-face contact but scores lower on the other items, while class 4 has high probabilities for emotional support and has none for face-to-face contact and scores lower on the other items.

Table 4.

The Class-specific Probabilities of a Positive Answer on the Items and Class Weights for the Five-class Latent Class Model of the Kinship Data

	Class 1	Class 2	Class 3	Class 4	Class 5
Class weights	.37	.33	.14	.11	.05
Monthly face-to-face contact (A)	.98	.94	.96	.02	.02
Other types of monthly contact (B)	.88	.89	.50	.83	.08
Practical help given (C)	.64	.86	.52	.19	.07
Financial support received (D)	.14	.30	.08	.18	.05
Practical help received (E)	.48	.56	.18	.07	.01
Emotional support exchanged (F)	.99	.95	.50	.91	.16
Material conflict (G)	.00	.23	.08	.05	.05
Personal conflict (H)	.06	.21	.13	.11	.21

Figure 8 shows the profile plot of the five-class model on the kinship data. Given the small number of answer categories, the class-specific profiles can be seen well. It can also be seen that class 1 and class 2 have a very similar pattern, both having a high probability of responding “yes” on indicators A, B, C, and F and scoring low on the last two indicators. Class 5 clearly stands apart, having a low probability of answering “yes” on all items at hand. Class 3 and class 4 are more in the middle. All classes score low on the last two indicators, signaling that these two variables do not differentiate well between the classes. The negative aspects of this plot, similar to the SOCON data example, are that it does not show the spatial distance between classes or the relative size of the different classes, as compared with each other. Class 5, which has only 5 percent of the respondents, is given the same weights as class 1, which contains 40 percent of respondents. Furthermore, we do not have information about the within-group variation.

Figure 8.

Profile plot of the five-class model of the kinship data.

Figure 9 shows the ternary plot for the kinship data. The shortcoming of this visualization is that it can represent only three classes. It is common practice to visualize separately the two largest classes, and a combined profile of the smallest classes, or alternatively based on the characteristics of the classes to combine them differently into three classes. The choice is arbitrary. Furthermore it is possible to create multiple ternary plots, in which two classes are compared with each other by pooling the other classes into a third class. We chose the standard visualization used in Latent GOLD (Vermunt and Magidson 2013), namely, comparing the two largest classes with the combined profile of the smallest classes. This choice leads to losing most information on the characteristics of the clustering, given that class 1 and class 2 are very similar to each other, and the smaller classes differ more. Most of the CLPs are situated in the middle of the graph, with the “yes” categories toward the bottom, and the “no” categories toward the top. This shows that the first two classes are more prone to answer “yes” on all categories, while the other classes are more negative.

Figure 9.

Ternary plot of the kinship data.

Next we show how the categorical AoD biplot can be used to visualize the five-class model. First we focus on the location of the group means and the CLPs (the only information that the profile plot provides). As can be seen from Figure 10, the first (horizontal) dimension separates class 1 (symbolized by ■) and class 2 (●) on the one hand from class 5 (▼) on the other. The CLPs on the right side characterize class 5: we can see that all CLPs represented there have the “no” (1) answer, while the CLPs on the left side are the “yes” (2) category. The second (vertical) dimension differentiates class 3 (▲) and class 4 (♦) mostly. The CLPs above the middle of the graph describe class 3, while the ones below describe class 4. The respondents who have regular face-to-face contact (A), give practical help (C), and do not receive financial support (D) and emotional support (F) are more likely to be in class 3 than class 4. It is interesting to note that the probability of saying “no” on emotional support for this group is approximately 50 percent, but for the other classes it is much lower. At the same time respondents who do not have face-to-face contact (A) and do not give practical help (C) but have monthly contact other than face-to-face (B) and exchange emotional support (F) are more likely to be in class 4 than class 3. Furthermore, we can see that the CLPs of the last item, conflict on personal issues (H), are plotted next to each other and the overall mean (cross), which shows that this variable does not differentiate well between the classes (a characteristic that can be spotted in Table 4 as well).

Figure 10.

Analysis-of-distance biplot of group centroids and CLPs of the five-class model of the kinship data using the extended matching coefficient and modal assignment.

We look next into the variation around the class means, adding the samples as well as the 90 percent bags for the different classes in the biplot. The results are shown in Figure 11. In the first two dimensions class 1 and class 2 cannot be differentiated easily (only the 20 percent bags, not shown here, did not overlap), while the 90 percent bags of the other classes did not overlap.

Figure 11.

A biplot with 90 percent bags of the five-class model of the kinship data using modal assignment and dimensions 1 and 2 for scaffolding. The solid symbols represent the group centroids, the nonsolid symbols the sample.

As Table 5 shows, most of the between SS is explained in dimension 1, and dimensions 2 and 3 account for a very similar between and total SS. Although the scaffolding is usually done using the first two dimensions, it follows from Table 5 that using dimensions 1 and 3 as scaffolding would not result in a substantial drop in the percentage of the variation captured in the biplot, as shown in Figure 12.

Table 5.

The Sums of Squares Attributed to the Delta-space per Dimension in Parentheses and the $Δ^{⊥}$ Space Orthogonal to It for the Five-class Model (Kinship Data)

	$Δ$ (1) (%)	$Δ$ (2) (%)	$Δ$ (3) (%)	$Δ$ (4) (%)	$Δ^{⊥}$	Total
Between SS	1220 (57)	475 (22)	401 (19)	42 (2)	0 (0)	2138
Within SS	367 (9)	202 (5)	337 (8)	771 (18)	2501 (60)	4178
Total SS	1588 (25)	677 (11)	738 (12)	813 (13)	2501 (39)	6317

Note: SS = sum of squares.

Figure 12.

A biplot with 90 percent bags of the five-class model of the kinship data using modal assignment and dimensions 1 and 3 for scaffolding. The solid symbols represent the group centroids, the nonsolid symbols the sample.

Figure 12 shows that the third dimension substantially separates class 1 from class 2 (using 85 percent bags separates these two classes completely). However, in the third dimension, class 3 and class 4 show a very high degree of overlap.

In summary we conclude that with more than three LCs, biplots can efficiently be used for graphical descriptions of the LC structure. The biplots of the kinship data provide strong evidence against the practice of pooling the smallest three classes. Moreover, Table 5 and Figure 12 show that for a complete description of the LC structure, consideration of different pairs of dimensions to use as scaffolding axes for constructing biplots may not only be useful but needed.

7. Discussion

In this article, we propose to use categorical AoD biplots to visualize LC models. This proposal is motivated by the lack of proper multivariate plots for LC models with more than three classes. The ternary plot, a popular choice of a multivariate visualization to accompany an LCA, can only represent models with three classes. This means that to use a ternary plot for models with more than three classes, an average over some classes is needed, substantively reducing the quality of representation. The proposed categorical AoD biplot visualizes information about the group means and the variation around these means as well as the location of CLPs without being restricted to three classes. In particular, the AoD biplots can be enhanced to give a quantitative description of the overlap and separation of the different LCs. Furthermore, we also propose to use alpha bags and classification regions that give a visual insight about the degree of the separation between the classes. As such, biplots can be used very effectively in describing the properties of LC models.

We compared the proposed methodology with existing plots for LC models: the univariate profile plot and the multivariate ternary plot on two example data sets, namely, a three-class and a five-class LC model. In the example with the three-class model, we show how similar the biplot representation of the group centroids is to the ternary plot, but in addition, the biplot allows a visual appraisal of the variation of cases around each group centroid. Thus the biplot is capable of enhancing understanding of the LC structure. Both multivariate plots give a deeper insight into understanding the model/data than the univariate profile plot that shows only the class-specific response probabilities without showing the class sizes, or the variation around the mean. In the five-class model we show how our approach can be used in situations in which both the ternary plot and the univariate profile plot show only a limited amount of information. It should be noted that although using the AoD biplot enables visualizing LC models with more than three classes, this comes at a price. Namely, because of the projection of higher dimensional data onto a two-dimensional display space, some information is lost, and the group means and the samples are not plotted exactly. However, the best two-dimensional display space can be found in an objective way. To quantify the quality of representation, we also discuss some measures of fit. Although we have not explored the full use of partitioning the total SS into components, it should be noted that this partitioning allows statistical inference on the basis of permutation testing procedures. These procedures are similar to analysis-of-variance procedures but do not depend on the restrictive assumptions of the latter. Using the information derived from the modification indices, it would even be possible to reconsider whether an LC model best describes the data, or whether for instance the discrete factors LC model (Magidson and Vermunt 2001) could be used, with the corresponding graphs introduced in the same article.

An issue that needs further attention in applying AoD biplots to visualize LC models is the selection of the best assignment method to use as a basis for the visualization. To apply our methodology in its current form, each case needs to be assigned to one particular class with a probability (weight) of 1 and with probability (weight) 0 to any of the remaining classes. We proposed using modal assignment and an approach that we call weighted assignment, whereby the set of samples having the same response pattern is allocated proportionally to the different classes according to the posterior class membership probabilities. Another option would be to consider a class indicator matrix with elements modified to take into account the posterior class membership probabilities directly. Such an approach would correspond exactly to proportional assignment. This is a topic for further research as well as reevaluation of the quality of visualization using the different approaches. The goal of this article has been to demonstrate the usefulness of an AoD biplot in providing a visual appraisal of the LC structure.

Finally, further research is needed to investigate the interactive use of biplots to explore dynamically how adding covariates changes the locations of samples and CLPs. Research is also needed to investigate the role of the choice of the distance or dissimilarity measure to use when AoD biplots are considered for visualizing LC models with ordinal or Poisson data.

Footnotes

Acknowledgements

This work is based on research supported by the National Research Foundation of South Africa. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors, and therefore the National Research Foundation does not accept any liability in regard thereof. We thank the editor and two anonymous reviewers whose comments helped improve this article considerably.

Notes

Author Biographies

Zsuzsa Bakk is an assistant professor of methodology and statistics at the Institute of Psychology at Leiden University. Her research interests are in the field of latent class analysis and stepwise modeling approaches for latent variables. Her research has been published in journals such as Sociological Methodology, Political Analysis, and Structural Equation Modeling. Her PhD thesis, “Contributions to Bias Adjusted Stepwise Latent Class Analysis,” was written under the supervision of Dr. J. K. Vermunt and Dr. D. Oberski at Tilburg University. It won the Best Dissertation Award of the Classification Society in 2016.

Niel J. le Roux is a professor emeritus of statistics at Stellenbosch University. He started his academic career in 1966 as lecturer in experimental psychology at Stellenbosch University and joined the statistics department in 1976. Since his retirement, he has been actively involved in teaching postgraduate courses, supervising research students, and conducting research in multivariate statistical analysis. His main research interests include multidimensional scaling and the graphical display of multidimensional data, in particular the development and practice of biplots. He has published widely on the theory and application of these techniques.

References

Goodman

Leo A.

1974. “The Analysis of Systems of Qualitative Variables When Some of the Variables Are Unobservable. Part I: A Modified Latent Structure Approach.” American Journal of Sociology 79(1):79–259.

Goodman

Leo A.

2007. “On the Assignment of Individuals to Latent Classes.” Pp. 1–22 in Sociological Methodology, Vol. 37, edited by Xie

. Hoboken, NJ: Wiley-Blackwell.

Gower

John C.

Hand

David J.

1996. Biplots. London, UK: Chapman & Hall.

Gower

John C.

Le Roux

Niel J.

Gardner-Lubbe

Sugnet

. 2014. “The Canonical Analysis of Distance.” Journal of Classification 31(1):107–28.

Gower

John C.

Le Roux

Niel J.

Gardner-Lubbe

Sugnet

. 2015. “Biplots: Quantitative Data.” Wiley Interdisciplinary Reviews: Computational Statistics 7(1):42–62.

Gower

John C.

Le Roux

Niel J.

Gardner-Lubbe

Sugnet

. 2016. “Biplots: Qualitative Data.” Wiley Interdisciplinary Reviews: Computational Statistics 8(1):82–111.

Gower

John

Lubbe

Sugnet

Le Roux

Niel J.

2011. Understanding Biplots. Chichester, UK: Wiley.

Greenacre

Michael

. 2007. Correspondence Analysis in Practice. 2nd ed.Boca Raton, FL: Chapman & Hall/CRC.

Hagenaars

Jacques A.

1990. Categorical Longitudinal Data: Loglinear Analysis of Panel, Trend and Cohort Data. Newbury Park, CA: Sage.

10.

Ibironke

Olufunmilola

Koukounari

Artemis

Asaolu

Samuel

Moustaki

Irini

Shiff

Clive

. 2012. “Validation of a New Test for Schistosoma haematobium Based on Detection of Dra1 DNA Fragments in Urine: Evaluation through Latent Class Analysis.” PLoS Neglected Tropical Diseases 6:e1494.

11.

Le Roux

Niel J.

Gardner-Lubbe

Sugnet

Gower

John C.

2014. “The Analysis of Distance of Grouped Data with Categorical Variables: Categorical Canonical Variate Analysis.” Journal of Multivariate Analysis 132(1):9–24.

12.

Linzer

Drew A.

Lewis

Jeffrey B.

2011. “poLCA: An R Package for Polytomous Variable Latent Class Analysis.” Journal of Statistical Software 42(10):1–29.

13.

Magidson

Jay

Vermunt

Jeroen K.

2001. “Latent Class Factor and Cluster Models, Biplots, and Related Graphical Displays.” Pp. 223–64 in Sociological Methodology, Vol. 31, edited by Sobel

Michael E.

Becker

Mark P.

Boston: Blackwell.

14.

McCutcheon

Allan L.

1985. “A Latent Class Analysis of Tolerance for Nonconformity in the American Public.” Public Opinion Quarterly 49(4):474–88.

15.

McCutcheon

Allan L.

1987. Latent Class Analysis. Newbury Park, CA: Sage.

16.

McCutcheon

Allan L.

1998. “Correspondence Analysis Used Complementary to Latent Class Analysis in Comparative Social Research.” Pp. 477–88 in Visualization of Categorical Data, edited by Blasius

Greenacre

San Diego, CA: Academic Press.

17.

Muthén

Linda K.

Muthén

Bengt O.

2012. Mplus User’s Guide. 7th ed.Los Angeles, CA: Muthén & Muthén.

18.

Rousseeuw

Peter J.

Ruts

Ida

Tukey

John W.

1999. “The Bagplot: A Bivariate Boxplot.” American Statistician 53(4):382–87.

19.

Van der Heijden

Peter G. M.

Gilula

Zvi

Van der Ark

L. Andries

. 1999. “An Extended Study into the Relationship between Correspondence Analysis and Latent Class Analysis.” Pp. 147–86 in Sociological Methodology, Vol. 29, edited by Sobel

Michael E.

Becker

Mark P.

Boston: Blackwell.

20.

Van Gaalen

Ruben I.

Dykstra

Pearl A.

2006. “Solidarity and Conflict between Adult Children and Parents: A Latent Class Analysis.” Journal of Marriage and Family 68(4):947–60.

21.

Vermunt

Jeroen K.

2010. “Latent Class Modeling with Covariates: Two Improved Three-step Approaches.” Political Analysis 18(4):450–69.

22.

Vermunt

Jeroen K.

Magidson

Jay

. 2013. Technical Guide for Latent GOLD 5.0: Basic and Advanced. Belmont, MA: Statistical Innovations.