Abstract
Journalism and mass communication research is underutilizing structural equation modeling (SEM) for the purposes of specifying, estimating, and evaluating measurement models. The analytical exercises undertaken for this essay reveal SEM-based confirmatory factor analysis (CFA) to be a multifaceted tool that can aid researchers in generating greater understanding of the measurement elements of their research endeavors. Secondary analyses of 2014 World Values Survey (N = 9,901) data emphasize the following: testing competing models, model identification, proper evaluation of covaried error terms, absolute and incremental model fit statistics, the chi-square distributed test statistic, model equivalency, multiple-group models, and assessing equality constraints.
Measurement and scale development have been consistent research interests since the inception of journalism and mass communication as a field of study. 1 Recent issues of leading Association for Education of Journalism and Mass Communication (AEJMC) journals like Journalism & Mass Communication Quarterly and Mass Communication & Society contain works offering measurement advancements, 2 and the creation of a peer-review outlet within the field devoted to this topic (Communication Methods and Measures) has served to further the cause. 3 Efforts have been undertaken to shore up the use of distinct factor analysis procedures (i.e., principal components analysis [PCA], exploratory factor analysis [EFA], and confirmatory factor analysis [CFA]) commonly engaged by communication researchers in the analytical assessment of measurement, with the understanding that a field’s empirical insights will only be as strong as the measures it utilizes. 4
Of particular interest for this essay is a recent surge in the use of CFA to provide empirical support for the introduction and refinement of a wide range of journalism and mass communication measures. 5 A few works of this kind have shown analytical dexterity (e.g., proper assessment of competing models, well-reasoned discussion of model fit, appropriate use of model respecification procedures), 6 but the field remains far removed from a universal application of structural equation modeling (SEM) for the purposes of testing measurement models. 7 Work from the field is replete with examples of SEM-based CFA procedures that are at best inefficient and ineffective, and at worst analytically suspect. Concerning some of the more fundamental aspects of CFA, there are examples in peer-reviewed journalism and mass communication research efforts of authors presenting a series of scale-specific analyses without testing a full measurement model, 8 failing to report formal measurement model fit statistics, 9 embracing the use of poorly performing model fit statistics (e.g., goodness of fit index [GFI]), 10 providing inadequate discussion of potentially problematic covaried error terms 11 and data-driven respecification procedures, 12 and assessing group comparisons in a less than ideal fashion. 13 Given these issues, there is a need for the field to double back and re-focus on the fundamentals of SEM-based CFA analyses. This essay will seek to reinforce well-established practices for the conducting of a proper CFA.
One element of SEM-based CFA emphasized in this work (i.e., the testing of alternative models) is especially problematic within the field. The testing of alternative models is far from universal, 14 and, even when the practice is engaged, there are examples of scholars failing to acknowledge statistically significant differences in competing model fit statistics, 15 using rudimentary methods (e.g., fixing of covariances) for assessing alternative models, 16 and relying on eyeball assessments rather than providing formal analytical comparisons (e.g., chi-square difference test). 17 In addition, the field has yet to put into practice some of the more advanced elements of SEM-based CFA analyses (e.g., the formal testing of measurement-level group differences) that could lead to greater understanding of the measures used in a diverse set of research agendas. 18 Some of the more pertinent advanced CFA techniques that may be of interest to researchers in the field are displayed in this essay.
This essay offers an overview of SEM-based CFA. The key elements of a measurement model (e.g., observable variables, latent variables, factor loadings, error terms) are detailed, and the specific roles of each element and their relations are outlined. Building on these foundational components, the following CFA procedures are given attention: (1) testing competing models, (2) proper model identification, (3) the meaning and assessment of covaried error terms, (4) the appropriate use of recommended model fit statistics and the chi-square distributed test statistic, (5) model equivalency, (6) the comparison of single- and multiple-group models, and (7) the introduction and testing of equality constraints. Secondary analyses of 2014 World Values Survey (WVS) data (N = 9,901) are undertaken to provide a systematic presentation of the procedures focused upon for this essay.
SEM-Based CFA Models
Each structural equation model is like its own ecosystem. There is a hierarchy within the system (e.g., latent variables are specified as more influential than observable variable error terms), each element within a model has a specific role to perform, and every element can affect every other element within the model (with varying levels of [in]direct force, but impactful nonetheless). This simile retains greater expressive value when we look at not just what is being estimated within a specified model, but also what is not being estimated. The former and the latter are components of any assessment of whether a theoretical model fits a given data set and fit is the key determinant of whether a model can be evaluated.
SEM is a multivariate technique that is “a melding of factor analysis and path analysis into one comprehensive statistical methodology.” 19 The focus of this essay is on the former of these two elements, factor analysis. Figure 1 offers an example of a SEM-based CFA model typical of what has been reported in the journalism and mass communication literature. SEM-based factor analysis is defined as confirmatory given that a researcher is making a theoretical argument for a restricted set of relationships between a specified number of latent variables (LVs) and observable variables (OVs), and these relationships are represented as factor loadings (FLs). Associated with each observable variable is an error term (Es), and every latent variable has a variance that is estimated. In addition, all latent variables are covaried within CFA models (CVs).
Example of SEM-based CFA model.
While these elements of a CFA model are most central to what is being hypothesized, a theoretical model is also being judged against the data by what is not being estimated. SEM programs like LISREL or AMOS will estimate model fit if non-specified parameters within a model were to be estimated (presented in output as modification indices). For example, would the Figure 1 model fit improve if LV1 were to be linked to OV4 by means of a factor loading (i.e., OV4 would then serve as a cross-loader of LV1 and LV2), or what would the model fit be if E1 were to be covaried with E2? In short, all associations between various elements of a model are treated as potential points of connection. It is the pervasive nature of potential associations that often proves most troublesome for researchers who engage in the use of SEM-based CFA.
The measurement model offered in Figure 1 will serve as an exemplar from which to address a series of lingering issues for the use of SEM-based CFA in journalism and mass communication research. Some of these concerns are currently being addressed only implicitly in the literature, but need more explicit discussion (e.g., model identification, model equivalency, assessment of covaried error terms). Other issues addressed in this work require further clarification given their importance, but lack consistent implementation (e.g., the appropriate use of recommended model fit statistics and the chi-square distributed test statistic). Still other matters desperately need to become common practice in the literature (e.g., testing competing models). And, finally, there are some advanced SEM-based CFA procedures (e.g., comparison of single- and multiple-group models, the introduction and testing of equality constraints) unused to date by the field, but which could be of potential benefit for researchers. All of these points can serve to clarify and expand the use of SEM-based CFA in journalism and mass communication research.
Clarifying and Expanding the Use of SEM-Based CFA
Model Identification
A SEM-based CFA measurement model needs to achieve a proper level of identification to be estimated and evaluated. The sample variance–covariance matrix for a model is based on the number of observable variables included in the model. The number of unique values in a matrix is defined by the following equation: ρ (ρ + 1) / 2 (ρ = the number of observable variables). So, the model in Figure 1 has nine observable variables, which translates to 9 (9 + 1) / 2 = 45 unique values that could be estimated (i.e., total degrees of freedom that can be used). For a model to be properly identified (i.e., over-identified), the number of parameters being estimated must tally to less than the total number of unique values in the matrix (i.e., leaving at least 1 degree of freedom).
The Figure 1 model is estimating the following: nine observable variable error terms (Es), six factor loadings (FLs), 20 variances for three latent variables, and three covariances (CVs) linking the latent variables. Thus, there are a total of twenty-one parameters being estimated in the measurement model, leaving twenty-four degrees of freedom (i.e., the model is over-identified). Researchers specifying CFA models need to have complete recognition of the number of unique values available in a model’s variance–covariance matrix and the exact breakdown of parameters being estimated. Once more, it would be wise for journalism and mass communication researchers to offer a report in textual form of the model parameters being estimated. Holbert and Stephenson report that the presentation of SEM in communication is often problematic due to an improper display of which parameters are being estimated in a specified model. 21 Audience members consuming an SEM-based CFA research effort should be able to clearly establish (1) the number of unique values in the variance–covariance matrix, (2) the number of parameters being estimated, (3) whether a model is over-identified by subtracting item #2 estimate from the item #1 estimate, and (4) matching up the total from item #3 with the researcher’s reported degrees of freedom for the model (which should be offered when reporting the chi-square distributed test statistic). One of the best means by which to make sure a researcher and an audience member are on the same page in terms of identification is to offer a systematic textual listing of the exact number of parameters being estimated by type.
Alternative Models
The issue of proper identification pertains to the ability to estimate and evaluate a single model. However, it is imperative for journalism and mass communication researchers to engage in the practice of testing multiple alternative models. There needs to be an assessment of whether a proposed measurement model is better than competing, but plausible models which could fit a given data set just as well or better than what is being hypothesized. 22 SEM practitioners need to remember that just because a hypothesized model fits the data well does not mean it is the single best fit with the data. Judging whether a given model is a solid fit is one level of knowledge generation, but the direct analytical assessment of competing theoretical claims (embodied in the creation of alternative measurement models) is a process that retains greater probative force. As Popper argues, the pitting of competing theoretical claims against one another is an advanced form of scientific inquiry that should be practiced in mature fields. 23
The theoretical measurement model offered in Figure 1 can be assessed for model fit, and the testing of a single measurement model of this kind is what we most often see in the journalism and mass communication literature. However, it is often the case that a series of valid alternative models could be formulated if we were to embrace competing theoretical claims. For example, one alternative to Figure 1 may be to create a single latent variable linked to all nine observable variables. Another model to be offered could consist of just two latent variables, with the first LV affecting OV1 through OV6 and the latter LV connected to OV7 through OV9. The number of possible alternative models would be dependent on the range of theoretical arguments that could be put forward by the researcher. It may be the case that competing models are grounded in fundamentally distinct theories and this would be the most robust use of alternative model testing. However, it may be the case that a series of competing models are presented not based on any one theory, but simple claims of face validity. It is rarely the case that a researcher would be unable to put forward at least one mildly plausible alternative model that could be assessed against the theoretically driven measurement model that is the primary focus of the research effort. As a result, there is little excuse not to test alternative models, and the value gained from such an activity warrants journalism and mass communication researchers taking on this task universally.
Model Equivalency
The topic of alternative models raises the tangential issue of model equivalency. Journalism and mass communication researchers should recognize that alternative models can be created that represent distinct theoretical claims, but are identical mathematically. Alternative models that are equivalent mathematically will produce the exact same model fit statistics. For example, an alternative to the Figure 1 measurement model could consist of a single higher-order latent variable that affects all three lower-order latent variables. This new theoretical model is a doppelgänger mathematically of what is outlined in Figure 1. However, the former and the latter models are making distinct theoretical claims. The Figure 1 model is arguing that the three latent variables are distinct entities, while the higher-order latent variable alternative is arguing that the three lower-order latent variables, while on some level distinct, should be formally linked given they are influenced mutually by the same over-arching force (i.e., the higher order latent variable). The choice by the researcher of which model to embrace will need to be made purely on theoretical grounds given that there is no difference between the two mathematically.
Model Fit and Chi-square Distributed Test Statistic
Let us be clear: the chi-square distributed test statistic should not be used in its pure form as an estimate of model fit. As Schumacker and Lomax states, “The chi-square test of model fit can lead to erroneous conclusions regarding analysis outcomes.” 24 In particular, the chi-square distributed test statistic is overly sensitive to mild differences between a proposed theoretical model and the data when sample sizes are even moderate in size (e.g., 200). In addition, this statistic will perform well only under conditions of multivariate normality (which is often difficult to achieve in the social sciences). As a result, researchers are likely to discard perfectly well fitting models that are appropriate for interpretation if there is an over-reliance on a non-significant chi-square distributed test statistic as an indicator of fit. This is especially true for the assessment of measurement models given the number of parameters being estimated. 25 However, the chi-square distributed test statistic has an important role to play in the assessment of measurement models, especially when embracing earlier advice concerning alternative models. Thus, this statistic should be reported whenever an SEM-based CFA is being conducted. The chi-square distributed test statistic gives researchers a common metric from which to assess directly whether one model is better than another, offering a specific statistic (i.e., Δχ2) with critical values for various alpha levels. In summary, the chi-square distributed test statistic in its pure form should not be reported as a measure of fit, but presented for the purpose of model comparison.
The primary reason why a seemingly infinite number of model fit statistics are reported when using any of the major SEM software packages is that there is no universally agreed upon indicators of whether a proposed model fits a data set well. Each model fit statistic retains its own strengths and weaknesses. Some model fit statistics (e.g., GFI) seem to have more weaknesses than strengths, whereas others have been deemed more laudatory (e.g., confirmatory fit index [CFI]). 26 Given that all estimates of model fit retain some weaknesses, it is important for researchers to report a series of statistics to make a proper assessment of fit. If a group of fit statistics indicates a strong fit, then greater confidence can be granted to the evaluation of a model’s parameter estimates.
So, what mix of fit statistics represents best practice? A specific set of fit statistics will be offered in the “Method” section that reflects a well-promoted combination that has fared well under a variety of assessments. Researchers need not use these same fit statistics in their own work, but what is important is that representatives from two classifications of model fit statistics, absolute and incremental, be reported with the testing of any CFA model. 27 Incremental fit statistics (e.g., CFI) compare the chi-square distributed test statistic against a baseline model, whereas absolute fit statistics (e.g., root mean square error of approximation [RMSEA]) do not engage in any type of model comparison. 28 Researchers should become familiar with how each fit statistic is calculated and make their own judgments on which to embrace in their analyses, but, regardless of which statistics are used, there should be a reporting of some combination of absolute and incremental. The two classifications are built from different analytical principles and assumptions, so they retain distinct strengths and weaknesses. The reporting of a string of absolute or incremental fit statistics is not as strong as the presentation of some combination of the two types.
Covaried Error Terms
Although not the universal cause of poor model fit in measurement models, the most usual suspects leading to an inability to evaluate a model of this kind are fixed covaried error terms between observable variables. As shown in Figure 1, each observable variable has an error term. Although the individual error terms are being estimated, the relationships between these terms are not being calculated in the theoretical model. However, if there is substantial covariance between these error terms, and it is not being estimated, then the proposed model is not a solid representation of the data. In turn, the model fit estimates suffer. It is natural for observable variables associated with the same latent variable to covary with one another to some degree, but the question is whether their covariation rises to a level requiring an official acknowledgment in the model (i.e., freeing the association for estimation) to achieve adequate model fit. More troublesome is the covarying of error terms for observable variables associated with distinct latent variables—this is a practice that should be avoided at all costs without the assessment of alternative models. Covaried error terms are often a signal of other measurement issues. For example, two observable variables with significant covariance may be serving as functional equivalents. If they are ultimately measuring the same construct, then their covariance will be strong. Conversely, substantial covariance between error terms may signal a latent association that points to a unique force at work influencing these observable items in a manner distinct from the other observable variables in the model. It is important for journalism and mass communication researchers to recognize that significant covaried error terms are often a manifestation of a more substantive measurement issue and a careful assessment of the root cause of the covariation requires a firm understanding of one’s data.
Multiple-Group Models and Equality Constraints
Journalism and mass communication researchers have the ability to assess moderation within measurement through the testing of multiple-group models and the establishment of equality constraints within these models. There is every reason to believe in the existence of important group differences within measurement models, but the field has failed to address this matter in the literature. For example, let us say that Figure 1 is being used in a study focused on perceptions of participants in a U.S. presidential debate. The three latent variables could be Democratic candidate credibility, Republican candidate credibility, and journalist-as-moderator credibility. The three observable variables for each latent variable could be expertise, trustworthiness, and goodwill. 29 There is little reason to think that the Figure 1 path estimates will be similar for Democrats, Republicans, Independents, and those individuals who are affiliated with some other U.S. political party (e.g., Libertarians). Researchers have the ability to assess whether measurement differences exist between groups and to isolate which groups differ from one another in a statistically significant fashion.
The study of group differences requires the testing of a multiple-group model. SEM handles this endeavor in a parsimonious fashion by creating a single assessment of fit for the multiple-group model. It is inefficient and unsound for researchers to test the same measurement model separately for each group of interest. Returning to the Figure 1 measurement model, an alternative to the single-group, three latent-variable, nine observable-variable measurement model is a four-group model (i.e., Democrats, Republicans, Independents, Libertarians) testing the same set of associations. A researcher could assess the single- and multiple-group models and compare the two to see which fits better. If the latter fits better, then this signals group differences. However, the analyst is unable to isolate through this comparison the specific elements of the model where there are statistically significant group differences. Researchers need to use equality constraints to assess where meaningful (i.e., statistically significant) group differences reside. For example, group difference in the factor loading leading from LV1 to OV1 in Figure 1 could be assessed. The path estimates for all four groups would be treated as equal and this model would be assessed against the four-group model that contained no equality constraints. The Δχ2 estimate would then be used to see if there is a difference in how well the latter fits the data relative to the former. If the chi-square distributed test statistic for the multiple-group measurement model with the equality constraints increases in a statistically significant fashion, then this means the path estimates are not equal between groups. The researcher could then isolate specific group differences (e.g., create a single equality constraint for Democrats vs. Republicans) to find where the most important group differences exist within the model. These group differences may prove to be of tremendous value for journalism and mass communication researchers seeking to gain a more complete understanding of their measurement, but activities of this kind have failed to be engaged by the discipline to date.
Method
Data
The 2014 WVS data were used to display and address the broad range of CFA issues raised in this essay. The WVS is a worldwide effort initiated in 1981 seeking to collect representative survey data from almost hundred countries. 30 Although underutilized within the field of journalism and mass communication, the WVS has been used widely across the social sciences. 31 Of potential interest to the study of journalism and mass communication are WVS efforts to collect self-report measures of various types of political information seeking. The 2014 data include measures concerning seeking political information through newspapers, magazines, television news, radio, mobile phone, E-mail, and Internet (specific details are offered in the “Measures” section). There is increasing interest in the assessment of how various forms of media use relate to one another in the production of democratic outcomes. Holbert uses panel data to assess relationships between different types of outlets over time, 32 and this line of research has culminated in the presentation of a theory of political campaign media connectedness. 33 Similarly, Dutta-Bergman argues for the need to assess complementary associations between various types of media use, rather than approaching this topic from a competitive framework. 34 More recently, scholars are using unique analytical procedures (e.g., network analysis) to assess these types of associations. 35 The assessment of relations between different types of information seeking can also be addressed through the use of CFA, and analyses of this kind can bring added value to this area of study. Rather than looking at the causal influence of one media activity on another or how various outlets relate to one another through shared audience members via network analysis, CFA can assess the degree to which various forces of information seeking that exist beyond direct empirical assessment (i.e., latent variables) serve to shape how various forms of political media use are related to one another (i.e., linked to similar vs. distinct latent variables).
The 2014 WVS data (N = 9,901) include the following countries: Bahrain (N = 1,200), Brazil (N = 1,486), India (N = 1,581), Jordan (N = 1,200), Kuwait (N = 1,303), Libya (N = 2,131), and Yemen (N = 1,000). Representative samples for each country (adult population eighteen years or older) are constructed, and face-to-face interviews are conducted with the goal of collecting a minimum of one thousand respondents from each country. There is a primary investigator in each country who oversees data collection efforts in adherence to guidelines established by the World Values Survey Association (WVSA). 36 A total of thirteen items from the 2014 survey were utilized for our analyses (see Appendix for zero-order correlation matrix).
Missing Data
Missing values were minimal (i.e., less than 5% of responses) for eleven of the thirteen items used in the CFA analyses, 37 but the accumulation of lost cases from a failure to address this issue would have resulted in a sizable drop in the final listwise N. A multiple-imputation hot deck technique was engaged to address the pertinent items’ missing values. 38 The technique involves the identification of a series of items in a data set with a small number of missing values that are of at least tangential interest to what is being hypothesized. These variables form the basis for the multiple-imputation replacement procedure. Cases with missing values for endogenous items are matched with cases without missing values by shared responses to the hot deck variables. If multiple non-missing cases are matched with a single missing values case, then one of the non-missing value cases is randomly selected. The following variables were used to create the hot deck: biological sex (52.6% male, N = 5,203), country, and importance of religion (measured on a scale ranging from 1 = very important to 4 = not at all important; M = 1.35, SD = 0.66). Appropriate matches were able to be identified for all but twenty-two of the cases with endogenous missing values, resulting in a final listwise N of 9,879. The twenty-two cases (0.002% of the sample) retaining at least one missing value for the endogenous observable variables were dropped from the study and the thirteen items detailed below are without missing values.
Measures
Information seeking
Respondents were offered the following introduction: “People learn what is going on in this country and the world from various sources. For each of the following sources, please indicate whether you use it to obtain information 1 = ‘daily,’ 2 = ‘weekly,’ 3 = ‘monthly,’ 4 = ‘less than monthly,’ or 5 = ‘never.’” The following seven forms of communication were offered: daily newspaper (M = 3.15, SD = 1.64), printed magazine (M = 3.77, SD = 1.35), TV news (M = 1.72, SD = 1.19), radio news (M = 2.97, SD = 1.69), mobile phone (M = 2.41, SD = 1.66), E-mail (M = 3.43, SD = 1.68), and Internet (M = 3.06, SD = 1.75).
Confidence in institutions
The following introduction was offered to a battery of items: “I am going to name a number of organizations. For each one, could you tell me how much confidence you have in them: is it 1= ‘a great deal of confidence,’ 2 = ‘quite a lot of confidence,’ 3 = ‘not very much confidence,’ or 4 = ‘none at all?’” Responses in reference to the following three institutions were singled out for this study: police (M = 2.25, SD = 1.02), courts (M = 2.24, SD = 1.02), and government (“in your nation’s capital”; M = 2.57, SD = 1.05). The three items form a reliable scale (Cronbach’s α = .77).
Human rights values
In reference to another battery of items, the following was presented to respondents: “Many things are desirable, but not all of them are essential characteristics of democracy. Please tell me for each of the following things how essential you think it is as a characteristic of democracy. Use this scale where 1 means ‘not at all an essential characteristic of democracy’ and 10 means it definitely is ‘an essential characteristic of democracy.’” The following three items from this battery were isolated for this study: “People choose their leaders in free elections” (M = 7.20, SD = 2.90), “Civil rights protect people from state oppression” (M = 6.89, SD = 2.84), and “Women have the same rights as men” (M = 6.70, SD = 3.05). The three items form a scale of moderate reliability (Cronbach’s α = .65).
Analyses
First, attention will be given to the seven information seeking items. The importance of competing measurement models will be addressed with an assessment of two models (see Figure 2). The first model argues there is a single latent variable of Information Seeking affecting levels of engagement with all forms of communication. The second model contends there should be two latent variables, Traditional Information Seeking (newspapers, magazines, television, radio) and New Information Seeking (mobile phone, E-mail, Internet).
Initial competing CFA models—Information seeking behavior.
The structural equation model addressed in the next step expands the information seeking model by including two additional latent variables and six observable variables. The first latent variable is Confidence in Institutions (police, courts, government) and the second latent variable is Human Rights Values (people choose, civil rights, gender equality). This step allows for the testing of a traditional CFA-based measurement model consisting of a mixture of communication and non-communication latent and observable variables commonly studied in the journalism and mass communication literatures.
The third step focuses on multiple-group modeling. The final measurement model emerging from the first two analysis stages will be subject to a multi-group assessment by nation (i.e., seven groups). A comparison of the single- versus multiple-group models will be undertaken. In addition, a series of equality constraints will be introduced to offer direct analytical assessments of differences in factor loadings by country.
Given the pedagogical nature of this essay, more specific details concerning various analytical elements will be offered within the “Results” section. However, it is important to note that IBM SPSS AMOS, version 22, is the statistical software used for all SEM analyses. In addition, the method of estimation utilized for all analyses is maximum likelihood (ML). A study-wide alpha level of p < .001 is retained given the large sample size (even at the country level).
Three model fit statistics are utilized for all analyses: CFI, RMSEA, and standardized root mean squared residual (SRMR). These statistics are recommended by Hu and Bentler who stress the need to present a mixture of absolute (e.g., RMSEA) and incremental (e.g., SRMR, CFI). 39 The appropriate cutoff values signaling strong model fit are as follows: CFI, .95 or higher; RMSEA, .06 or lower; and SRMR, .09 or lower. Given the relative weaknesses of any one fit statistic, no one cutoff value should be strictly enforced as a necessary, but not sufficient condition for deeming a theoretical model to fit a data set. Instead, the model fit statistics should be judged collectively, and an argument can then be made for why a specific model retains a level of fit that would allow for the evaluation of its parameter estimates. With this being stated, it is recommended that models retaining a RMSEA above .09 or a CFI below .90 need not be evaluated. 40
Results
Information Seeking Models
The competing information seeking measurement models are properly over-identified. There are seven observable variables in each model which translates to a diagonal/sub-diagonal matrix of twenty-eight potential parameters that could be estimated (estimating all twenty-eight would create a just-identified model). The single latent variable model estimates the following: seven observable variable error terms, six factor loadings, and a single variance for the latent variable (number of estimated parameters = 14). This leaves a model with fourteen degrees of freedom (i.e., an over-identified model). The two-latent variable model estimates the following: seven observable variable error terms, five factor loadings, a pair of variances for the latent variables, and a single covariance linking the latent variables (number of estimated parameters = 15). This model results in the retaining of thirteen degrees of freedom (i.e., an over-identified model) and a difference of one in the degrees of freedom between the former and the latter (which will be of importance in the comparison of the two models).
The single latent variable model produces the following fit statistics: CFI = .79, RMSEA = .16 (90% confidence interval [CI] = [.15, .16]), SRMR = .09. This model does not fit the data well. Thus, its estimates should not be interpreted. For purposes of model comparison, the model’s χ2(df = 14) = 3,493.80, p < .001.
The two-latent variable model results in better model fit estimates: CFI = .94, RMSEA = .09 (90% CI = [.08, .09]), SRMR = .05. For purposes of model comparison, the model’s χ2(df = 13) = 960.05, p < .001. The Δχ2(df = 1) = 3,493.80 − 960.05 = 2,533.75, and this is statistically significant at the p < .001 level. Thus, the latter model is shown to be an improvement over the former model. With this being stated, the fit statistics for the latter model are only at the very outer edges of being acceptable to a degree that would allow for interpretation and evaluation (i.e., two of the three fit estimates are close, but not meeting the recommended cutoff values). Thus, we need to peel back the onion further to gain a sense of how best to address the measurement of information seeking. It is essential to attend to this matter prior to introducing the confidence and values portions of the broader measurement model.
The standardized factor loading estimates for the two-latent variable model contain a cautionary tale (see Table 1). All of the factor loadings are statistically significant at p < .001. However, the four loadings emanating from the traditional information seeking latent variable should draw our attention. The loadings associated with the two print-based media, newspapers (.69) and magazines (.78), are strong, but the loadings for the broadcast media, television (.14) and radio (.33), are decidedly weak. In addition, the modification indices indicate that the single greatest change in the model’s chi-square estimate will come with the freeing (i.e., estimation) of a covaried error term that would link TV news and radio. As stated earlier, the covaried error term can signal functional equivalency—the pair of items measuring the same construct. This would be the most likely scenario if the zero-order correlation between the two items was exceedingly high, but the zero-order correlation between these two items is only .235, p < .001 (see the appendix). The other possibility is that the sizable covaried error term is a manifestation of a deeper issue concerning the proper construction of the traditional information seeking latent variable. Perhaps this latent variable should be disaggregated further into broadcast information seeking (television, radio) and print information seeking (newspaper, magazine).
Standardized Factor Loading Path Estimates—Two-Latent Variable Model.
Note. *p < .001.
The newly proposed model (see Figure 3) was tested and the following fit statistics were generated: CFI = .96, RMSEA = .07 (90% CI = [.07, .08]), SRMR = .04. This model can be deemed a solid fit with the data (i.e., CFI and SRMR are solid, and RMSEA is just slightly above .06). In addition, the Δχ2(df = 2) = 344.72, p < .001, comparing the three latent-variable model against the two latent-variable model, indicates a better fit with the data. Thus, we can argue with a higher degree of analytical certainty that the three-latent variable information seeking model is the best fitting. This is the information seeking model that will serve as a base from which to build the larger measurement model that includes both the confidence and values items.
Additional competing CFA model—Information seeking behavior.
There are several important points to stress with the model respecification and testing activities outlined above. All respecification processes must be treated with an appropriate level of caution. Otherwise, data-driven models are constructed that are overfit with a single data set and these models are much less likely to be replicated in future studies. First, it is important that any form of model respecification be justifiable on conceptual/theoretical grounds, not based purely on what the modification indices offer as post hoc insights for where the proposed model does not match the data. Second, it would be tempting in the last step to retain traditional information seeking as a higher order latent variable that affects two lower order latent variables, print and broadcast. However, it is important to recognize that any structural equation model must be properly identified at all levels. If we isolate just the traditional information seeking portion of the proposed higher order factor model (see Figure 4), we have four observable variables that translate to ten elements in the diagonal/sub-diagonal of the covariance matrix. Thus, only nine paths can be estimated in this model for it to be over-identified (i.e., at least 1 degree of freedom). However, the following ten paths would need to be estimated: four observable variable error terms, two-factor path estimates from the lower order factors to the observable variables, a single-factor path estimate from the higher to the lower order factors, a variance for the higher order latent variable, and two disturbance terms for the lower order latent variables. This element of the model is just-identified (i.e., zero degrees of freedom), ruling the entire model improperly identified.
Under-identified portion of the potential competing CFA model.
The introduction of the notion of higher order latent constructs in the context of coming to some conclusion on the proper measurement model representation of the information seeking items brings to light an additional alternative model: a single higher order latent variable (information seeking) affecting three lower order latent variables (traditional–print, traditional–broadcast, new) with the observable variables tethered appropriately to the lower order latent constructs as presented in Figure 3. This model is sound at the conceptual level and represents a viable alternative. However, mathematically, this model is the functional equivalent of the model detailed in Figure 3. The testing of this model produces the exact same model fit as the model containing the three covaried lower order latent factors only (i.e., they are equivalent models mathematically). Ultimately, both models fit the data appropriately and it is up to the researcher to choose which one makes the most sense theoretically (though they are mathematical equivalents, they are unique theoretically)—focusing primarily on a single higher order information construct or treating the three lower order latent constructs as distinct entities. Which model is best should be determined by the study’s hypotheses. If the author is arguing distinct processes of influence for the various types of information seeking, then the use of the three lower order latent variable model is most appropriate. However, if the hypotheses are treating all types of information seeking as functional equivalents, then the use of the single higher order latent variable with three lower order latent variables would best reflect this level of argumentation. For the purposes of this exercise, we will retain the information seeking measurement model represented in Figure 3.
Full Single-Group CFA Model
The full thirteen-observable variable measurement model being tested includes the following latent variables: traditional information seeking–print, traditional information seeking–broadcast, new information seeking, confidence in institutions, and human rights values (see Figure 5). This model fits the data well: CFI = .95, RMSEA = .05 (90% CI = [.05, .06]), SRMR = .04. For purposes of model comparison, the χ2(df = 55) = 1,573.27, p < .001. Several SEM experts advocate a two-step process of hybrid model specification, estimation, and evaluation. The first step is the testing of the measurement portion of the model. The finding of a solid fit with the model detailed in Figure 5 marks the conclusion of this first step. If the intention of the author is to then assess a series of relationships between the latent variables, then the covaried disturbance terms created for the measurement model can be discarded and formal structural paths can be inserted (e.g., an arrow leading from traditional information seeking–print to human rights values) depending on what is being hypothesized. The model containing a mixture of measurement and structural relationships can then be judged against the data using the appropriate fit statistics. If there is a reduction in the quality of fit with the data at this stage, then the researcher knows the issue is isolated to the manner with which the structural relationships between latent variables have been constructed.
Full single-group CFA model.
Full Multiple-Group CFA Model
It is easy to argue that country may serve as a moderator of relationships between various forms of information seeking, confidence in institutions, and the valuing of human rights. There is little reason to believe that the connections between these constructs are the same in Yemen and Brazil. The analyses conducted so far treat the various countries as functional equivalents. However, some researchers may wish to hypothesize why there would be potentially important differences between countries and seek to assess differential associations analytically. In SEM, this process would involve the assessment of a multiple-group model. It is not analytically efficient to isolate cases by country within the data set and run the exact same model for respondents associated with each country. Instead, SEM allows for a single test of a multiple-group model to be performed. An assessment of fit is provided and can be compared with the single-group model to deem which fits better. So, let us engage in this activity by running a multiple-group version of the model outlined in Figure 5.
The multiple-group model will retain a total number of degrees of freedom that is the number of degrees of freedom from the single-group model (the Figure 5 model has fifty-five) multiplied by the number of groups (we have seven country groups in our data). As a result, the chi-square estimate for the multiple-group model tested in this exercise has 385 degrees of freedom. The estimate produced for the model is as follows: χ2(df = 385) = 1,785.016, p < .001. Although the overall chi-square estimate is higher than that for the single-group model, it is important to focus on the chi-square estimate in relation to the degrees of freedom in this case. If we look at the chi-square estimate divided by the degrees of freedom, we see a shifting from 28.61 with the single-group model to 4.64 for the multiple-group model (lower χ2/df estimates signal a better fit). The CFI for the multiple-group model is identical to that for the single-group model (i.e., .95). However, both the RMSEA = .02 (90% CI = [.02, .02]) and the SRMR = .03 reflect an improvement in fit with disaggregation by country. The improved fit with the multiple-group model means there are meaningful differences in measurement by country. As a result, we can begin to assess potential differences in measurement-based path estimates between groups. Country may not only serve as a moderator of the structural relationships between latent variables (i.e., the group differences most commonly assessed in communication research), but can also affect the measurement components of a structural equation model.
To assess whether differences in path estimates are statistically significant between groups, the researcher needs to introduce a series of equality constraints. Let us focus on the factor loading estimate emanating from the information seeking–new latent variable to the Internet observable variable. The standardized path estimates by country are as follows: Bahrain (λ = .86), Brazil (λ = .88), India (λ = .78), Jordan (λ = .74), Kuwait (λ = .66), Libya (λ = .85), and Yemen (λ = .92). The first equality-constraint model to be tested will treat this path estimate as equal across the seven models, meaning the path will be calculated for the first group and then deemed equal to that estimate for the other six groups (creating a difference in degrees of freedom of 6 between the original multiple-group model and this equality-constrained model). The assessment of this type of equality constraint will tell us whether the seven groups differ, but will not indicate which groups differ from one another in a statistically significant fashion. To assess specific country difference, researchers can create equality constraints for specific group pairings. For the purposes of this exercise, we will compare the Kuwait information seeking–new → Internet path estimate with the other six countries in six respective models, each containing a single equality constraint (all of these models will retain a single degree of freedom difference from the original multiple-group model). We can then compare the chi-square estimates of the equality-constrained models against the original multiple-group model. If the chi-square estimate increases in a statistically significant fashion with the introduction of equality constraints, then this means the path estimates are not equal between the groups (i.e., signaling group to be a statistically significant moderator of the LV → OV factor loading).
The model containing the full set of equality constraints for the Internet factor score estimate produced a χ2(df = 391) = 1,939.40, p < .001, an estimate higher than the unconstrained multiple-group model. This results in a Δχ2(df = 6) = 154.38, p < .001, when comparing the initial multi-group model with this constrained model. Thus, this path estimate varies in strength between groups. In terms of the subsequent focus on comparing Kuwait with the other six countries, two statistically significant differences were found to exist at the p < .001 level. The Kuwait estimate was found to differ in strength from the path estimates produced in the India, Δχ2(df = 1) = 57.20, p < .001, and Libya, Δχ2(df = 1) = 29.02, p < .001, models. These analyses show that group differences do exist within the measurement model analyzed for this exercise. Although it is important to recognize that the assessment of group differences of this kind within measurement models is possible, a cautionary tale must be offered that there needs to be a recognition of how quickly a large number of comparisons can be introduced, especially when there is a large number of groups. The assessment of equality constraints should be undertaken carefully and grounded in solid theoretical argumentation.
Discussion
The following aspects of CFA were stressed in this essay: (1) testing of competing models, (2) proper model identification, (3) the assessment of covaried error terms, (4) the appropriate use of recommended model fit statistics and the chi-square distributed test statistic, (5) model equivalency, (6) the comparison of single- and multiple-group models, and (7) the introduction and testing of equality constraints. It is not our recommendation that the analytical depths probed in this essay are required of all journalism and mass communication measurement endeavors. Instead, the analyses presented in this work are meant to serve as an exemplar of the possibilities that exist for journalism and mass communication researchers who wish to improve and refine their measurement-based analytical procedures through the use of CFA.
With this recognition being offered, we need to stress there are some basic CFA elements highlighted in this work that should become more prevalent in journalism and mass communication research. Most importantly, the testing of competing models should be a more common undertaking. It is important for researchers not only to test their primary theoretical measurement model, but also to assess what is being hypothesized against other plausible, nested alternatives. It is just good science. This recommendation is offered with an understanding that traditional journal pages are limited in number and researchers may not wish to devote significant space in the main text of their work to the testing of alternative measurement models, especially if the assessment of measurement is just a means to the end game of testing structural relationships between latent variables. As a result, we would encourage researchers and editors to explore the presentation of analyses of this kind as an addendum to be offered in online versions of the work. Other fields (e.g., political science) are beginning to explore how digital versions of a piece of research can serve to complement traditional print versions. 41
With the testing of competing models comes better focus on the proper role of the chi-square distributed test statistic. This statistic should not be used as an assessment of fit. It is simply too delicate and vulnerable to wild fluctuations as a result of relatively small deviations between model and data in studies containing even moderately sized samples. The chi-square distributed test statistic is best used for the purposes of model comparison. There also needs to be better assessment of model identification and a more explicit discussion of this element of model specification in journalism and mass communication. Although many SEM programs are becoming more foolproof in guarding against the assessment of under-identified models, the onus should remain with the researcher in making sure proper identification exists at all levels of a measurement model. Finally, the testing of multiple-group models remains severely underutilized in the journalism and mass communication literature. The testing of group differences in measurement represents an important potential advancement for the field.
Although this essay has covered important ground for solidifying and advancing the use of CFA in journalism and mass communication research, a wide range of additional topics deserve the field’s attention. For example, the field needs to become cognizant of the construction of reflective versus formative relationships between observable and latent variables. 42 The most common approach to these relations in journalism and mass communication, as well as likeminded fields (e.g., psychology), is for the latent variable to affect a series of observable variables associated with it (i.e., a reflective relationship in that the observable variables are a reflection of the latent variable). Such an approach assumes that “the latent variable is more fundamental than the item responses.” 43 However, it is possible to construct latent variable–observable variable relations that retain a reverse causal ordering, with the observable variables affecting the latent variables (i.e., a formative relationship). In this instance, the latent variable is formed from the observable variable items. The reflective–formative distinction builds directly from how social scientists differentiate a scale (i.e., reflective) from an index (i.e., formative). 44 For example, the single latent variable (i.e., information seeking as reflective) model offered in Figure 2 can be contrasted against another single latent variable (i.e., news media use as formative) model that has all seven types of news media exposure serving as linked observable variables (see Figure 6). News media use does not cause the various types of consumption, but is the sum total of the various types of exposure. The theoretical stance taken in the construction of the Figure 6 latent variable is distinct from the CFA models estimated and evaluated for this study, all of which theorize information seeking as a latent variable that is a larger force at work affecting various types of news media exposure. Journalism and mass communication scholars must construct latent variable–observable variable associations that match their theoretical arguments, and it is possible, and even highly likely, that several latent variables of interest to the field are theorized as formative rather than reflective. 45
News media use as a formative latent variable.
Scholars who wish to incorporate the use of formative measurement in SEM need to be aware that the observable indicators are covaried with one another. As a result, attention should be paid to possible multicollinearity issues, and it is ideal for the formative indicators to covary as little as possible (creating a greater likelihood of each measure having unique added value for the construction of the latent variable). Comprehensive work on the basic procedures undertaken for the specification and estimation of formative measurement has only recently made its way into the literature. 46 However, multiple-indicator multiple-cause (MIMIC) models, which incorporate the use of formative indicators, have been firmly rooted in the SEM literature for some time and represent a well-tested means for the construction of formative measurement. 47 Journalism and mass communication researchers interested in incorporating formative measures should turn to a recent work by Diamantopoulos for a comprehensive guide. 48
It is our hope that this exercise reveals to its audience the broad range of possibilities that exist for the use of CFA to advance the cause of measurement assessment within the field. The importance of measurement cannot be understated: our models are only as good as our measurement. Although much can also be done by journalism and mass communication researchers to improve the conceptual analysis phase of the concept explication process, it is essential to wield as much dexterity and acumen as possible in the empirical analysis phase of measurement construction. As Converse stated over fifty years ago, “no intellectual position is more likely to become obsolete quite so rapidly as one that takes current empirical capability as the limit of the possible.” 49 The analyses undertaken in this essay can serve in some small way to push researchers forward in seeking to maximize what CFA offers them as but one tool in their larger analytical tool kit.
Footnotes
Appendix
Zero-Order Correlation Matrix (n = 9,879).
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | — | ||||||||||||
| 2 | .54 | — | |||||||||||
| 3 | .17 | .08 | — | ||||||||||
| 4 | .23 | .22 | .24 | — | |||||||||
| 5 | .20 | .23 | .14 | .25 | — | ||||||||
| 6 | .29 | .40 | .04 | .17 | .40 | — | |||||||
| 7 | .26 | .35 | .08 | .18 | .38 | .73 | — | ||||||
| 8 | .21 | .26 | −.08 | .04 | .04 | .11 | .06 | — | |||||
| 9 | .17 | .19 | −.07 | .00 | .00 | .08 | .04 | .47 | — | ||||
| 10 | .05 | .11 | −.06 | .04 | .01 | .08 | .06 | .31 | .38 | — | |||
| 11 | .08 | .03 | .06 | .05 | .03 | .05 | .03 | .09 | .03 | .05 | — | ||
| 12 | .15 | .09 | .05 | .09 | .08 | .08 | .08 | .11 | .06 | .07 | .62 | — | |
| 13 | .18 | .14 | −.02 | .02 | −.02 | .05 | .01 | .19 | .13 | .05 | .47 | .50 | — |
Note. 1 = Newspaper, 2 = Magazine, 3 = TV, 4 = Radio, 5 = Mobile Phone, 6 = E-mail, 7 = Internet, 8 = People Choose, 9 = Civil Rights, 10 = Gender Equality, 11 = Police, 12 = Courts, 13 = Government. Values greater than ±.04 are statistically significant, p < .001.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.