Putting the Methodological Cart Before the Theoretical Horse? Examining the Application of SEM to Connect Theory and Method in Public Administration Research

Introduction

Since its origin as a self-identified field of knowledge, scholars have examined public administration from myriad theoretical vantages. As such, it is not surprising that our contemporary understanding of the field coalesces around the belief in, and acceptance of, public administration as an interdisciplinary field of inquiry (Frederickson, Smith, Larimer, & Licari, 2015; Hou, Ni, Poocharoen, Yang, & Zhao, 2011; Raadschelders, 2011a, 2011b; Riccucci, 2010; Wright, 2011). As an interdisciplinary field, we have recognized that many of the problems faced by public administrators are “open-ended, multidimensional, ambiguous, and unstable” and also that our process for researching said problems “cannot be bounded and managed by classical approaches to the underlying phenomena” (Klein, 1996, p. 142; citing Mason & Mitroff, 1981; Rittle & Webber, 1973). Instead, as a field, we have applied diverse theoretical lenses to integrate knowledge and advance our understanding of government (Klein, 1996; Raadschelders, 2011b).

Although interdisciplinarity, with its aim of the accumulation and integration of knowledge, is not without challenges and pitfalls (see, for example, Mainzer, 1994; Pfeffer 1993), the ability to bring multiple perspectives to bear when examining complex phenomena is widely viewed as a necessary advancement in the knowledge production process itself—one that leads to better solutions—and a powerful tool for classifying and organizing intricate concepts and ideas (Blume, 1985; Klein, 1996). Practically, interdisciplinary inquiries in a given field necessitate merging elements from different disciplines to solve problems, which further requires researchers aggregate disparate epistemologies, concepts, theories, and methodologies. Yet despite efforts to integrate disparate perspectives into a coherent whole, an inherent by-product of interdisciplinarity is theoretical and methodological pluralism.

Like many other public administration scholars (e.g., Raadschelders, 2011; Riccucci, 2010), we start from the assumption that interdisciplinarity generates sounder, more robust knowledge in the field. We also assume that the resulting theoretical and methodological pluralism of interdisciplinarity is generally a net gain. Pluralism allows researchers to examine a single problem or classes of problems from multiple perspectives and using multiple tools. In this sense, pluralism is akin to the concept of triangulation in research methods, which assumes applying two or more methodologies in a study allows researchers to compare and validate findings as well as to “map out, or explain more fully, the richness and complexity of human behavior by studying it from more than one standpoint” (Cohen & Manion, 1986, p. 254). Bringing multiple theoretical and methodological perspectives to bear in the field of public administration acknowledges that each perspective “makes a useful, but partial, contribution” to our understanding of core issues (Mainzer, 1994, p. 360).

Although we view theoretical and methodological pluralism as largely positive, we would also argue that the field has, at times, poorly integrated or applied different methodological orientations when grappling with diverse interdisciplinary theories. And, at times, this tendency naturally and unsurprisingly spills over into our evaluation of homegrown theories too. In part, our struggle to integrate different methodological orientations occurs precisely because we are so willing to borrow from other disciplines and fields. At the risk of sound like a broken record, we borrow not only their theories but also their methods and then are required to integrate these methods into something resembling a coherent whole. Unfortunately, the road to integrating diverse methods and methodologies can be fraught with significant challenges—even more so when diverse methodologies are used to evaluate and advance theory. As such, in our greatest strength also resides our principal weakness.

Put another way, with an expanded methodological toolkit, we too frequently assume multiple methodological hammers pounding on one theoretical nail drive it better than a single hammer. It is our central thesis in this article that we should not assume the use of multiple methodological approaches in a single study—no matter how advanced or numerous—results in a better test of theory than the application of a single method.

Yet, in our view, a significant body of current public administration research falls prey to this mistake, too frequently assuming the use of multiple methodological tools leads to inherently superior tests of theory. One area where this problem has grown increasingly pronounced involves the application of psychometric statistical techniques, such as structural equation modeling (SEM).

There are at least two problems associated with the current application of SEM in public administration—neither having anything to do with the relative merits of SEM as a statistical tool itself. First, some studies use SEM as a part of a “kitchen sink” approach to statistical analysis wherein scholars simultaneously use several different data analytic techniques in a single study. Doing so implicitly assumes that using SEM in conjunction with other statistical techniques compensates for the partial contribution offered by any single approach. The application of multiple—often incompatible—statistical techniques does not necessarily enhance research quality nor does it inherently provide a stronger test of theory. Rather, it confuses, muddies, and undermines the potential theoretical and methodological insight offered in a given study.

Second, there are instances where confirmatory factor analysis (CFA) and SEM are used in an exploratory manner. Both CFA and SEM are techniques that are fundamentally confirmatory and not exploratory in nature. At a minimum, the misuse or misapplication of a statistical tool or set of tools produces inaccurate statistical conclusions. Unfortunately, this, in turn, has spillover effects on theory testing and development. When the statistical conclusions rendered in a research study are inaccurate, theory development suffers. As a consequence, public administration research would benefit by acknowledging that SEM, and the type of questions it appropriately addresses, offer a partial, but useful, contribution to public administration knowledge. However, to fully capitalize on the benefits of SEM and related techniques, researchers must place the theoretical horse in front of the methodological cart. To do so necessitates recognizing what SEM is and is not as well as how SEM can and should be used to examine public administration theory.

To address our central argument, we proceed in four steps. First, we describe SEM and its underlying measurement model, CFA. Practitioners of SEM will be well versed in much of this information. However, our discussion is designed to be an elementary introduction for those not well versed in the technique. We will articulate what scholars can expect SEM to reasonably accomplish as well as its advantages relative to other statistical techniques. Second, we will offer an assessment of the rise of SEM in public administration, its early uses, and how scholars commonly use the technique today in public management research. To achieve this aim, we will examine how SEM has been used in two core research areas: public service motivation (PSM) and bureaucratic red tape. Third, we will assess the application of SEM in conjunction with multiple statistical approaches and the necessity of matching method to theory. Finally, we will conclude with some comments on the future of SEM and how it can contribute to theory building and testing in public administration.

CFA and SEM: A Description

Although many applied researchers tend to conceptualize SEM as a single technique, two statistical processes underlie the approach. Prior to estimating a model that examines cause and effect relationships between constructs, researchers must estimate a CFA. The CFA model deals specifically with measurement accuracy and examines the relationships between a set of indicators (e.g., questionnaire items) and the latent, unobservable construct they are intended to measure (Brown, 2015; Kline, 2016). ¹ Indicators are observable and directly measurable, but CFA assumes that any given indicator represents only an approximation for the intended construct. Thus, CFA requires multiple indicators for any given construct. The common variance between items is assumed to measure the construct whereas the unique variance associated with each item is assumed to represent measurement error (Brown, 2015; Kline, 2016).

Ideally, researchers should use a minimum of three indicators to specify a given construct. The application of the three indicator minimum construct ensures that the construct is identified because the known information in the data set equals the number of parameters to be estimated. However, three indicator constructs, or “just identified” constructs, always fit the data perfectly when estimated in isolation. Researchers may decide to select more indicators per construct when the primary research question involves evaluating measurement accuracy. Multiple indicator constructs can empirically test for construct validity when indicators are appropriately selected (Little, Lindenberger, & Nesselroade, 1999). This approach offers more valid and reliable measurement of constructs when compared with traditional regression analysis which assumes that a single indicator is a perfect measure of a construct.

An important distinction between CFA and other factor analytic techniques frequently used in public administration involves specification of the relationships between an indicator and construct (Brown, 2015). In exploratory factor analyses (EFAs), the relationship between construct and indicator, as well as the number of constructs resulting from a given number of indicators, is statistically derived. EFA approaches treat the number of factors to be extracted from a given set of items as an empirical question, and the researcher can rotate the axes to find the best fit to the data. Although EFA results in a model that fits the data, the resulting factors could be the product of chance relationships among indicators. CFA models, on the contrary, require the researcher to specify the relationships between an indicator and construct based on sound theoretical argument. Researchers must also a priori specify the number of constructs in the data, which means that the number of factors for a given set of items is a theoretical—rather than an empirical—question. If those theoretical assertions are not empirically tenable, model fit suffers. One of the primary assumptions behind CFA is that the researcher should possess some prior knowledge concerning the appropriate number of constructs to examine as well as the indicators used to measure each construct.

Following the specification of a CFA, and a determination that the factor model specified adequately fits the data, the researcher can then examine the structural model. SEM analyses amount to regression analysis conducted with latent variables verified through CFA. SEM also offers researchers a few specific advantages over traditional regression models in addition to correcting for measurement error. First, SEM can handle complex patterns of relationships between multiple latent constructs. Given the flexibility of SEM, the researcher is free to establish any pattern of relationships between the constructs examined. As such, SEM is particularly well suited for examining models where mediation is a theoretical possibility. Second, even where mediation is not a possibility, SEM can handle multiple “y” variables in a single analysis. In other words, unlike in traditional ordinary least squares (OLS) regression, the researcher is not limited to examining a single dependent variable. Finally, SEM can offer distinct advantages when constructs change over time. A construct measured at time point 1 can be used to predict the same construct at time point 2, thereby controlling for its prior level (Burkholder & Harlow, 2003). As such, SEM is useful in the context of both cross-sectional and longitudinal data. Likewise, SEM is also useful for examining multilevel/hierarchical data. Nested data structures violate the independent and identically distributed assumption associated with traditional statistical models, which can result in biased parameter estimates when ignored (Snijders & Bosker, 1999). Therefore, the ability to account for nestedness in a multilevel SEM context is a considerable strength. ² Given these advantages, a scholar’s capacity to test theory increases with the flexibility of SEM. Although these advantages offer researchers advantages for testing theoretically relevant questions in public administration, there are also pitfalls associated with SEM. The relative advantages and pitfalls of SEM include the following:

In sum, there are two hurdles researchers must clear when conducting research that applies SEM. First, the researcher must specify a measurement model that theoretically relates indicator to construct. If the measurement model fails to adequately fit the data, the measurement expectations articulated by the researcher are not supported. Second, if the data support the researcher’s theoretical measurement expectations, the researcher may then examine the structural relationships between the constructs. The relationships between constructs can be examined with a latent regression model. By combining an empirical test of measurement accuracy and a statistical examination of the relationships between constructs, CFA and SEM provide valuable tools for theory building and testing in public administration research. The following section examines the early application of these techniques in public administration and how their use has evolved.

CFA and SEM in Public Administration Research: Two Examples

CFA and SEM were little utilized in public administration research 25 years ago, but both approaches are now widely used to address a host of questions. Nevertheless, certain areas within public administration appear to have been more widely colonized by psychometric statistical techniques such as CFA and SEM. Not surprisingly, given the nature of research questions and constructs examined, research streams associated with public management more broadly apply CFA and SEM. Frederickson et al. (2015) identified public management as a subfield of public administration research more closely associated with theories originating in psychology and generic organization studies. Because public management research more frequently adapts theory from psychology and organization studies, the methodology associated with those disciplines also tends to characterize studies conducted within those streams of literature.

Although our list is far from exhaustive, the research traditions associated with two core concepts in public management illustrate nicely the growth of CFA and SEM in public management research: PSM and bureaucratic red tape. Admittedly, CFA and SEM are widely applied to examine other concepts in public administration research. However, a chronicle of all instances of research that uses CFA or SEM is beyond the scope of this article. Given the prominence of these two research topics, they serve as meaningful cases to examine the potential of CFA and SEM to connect theory and method in public administration.

PSM has become one of the most widely researched, if not core, topics in public management (Bozeman & Su, 2015; Ritz, Brewer, & Neumann, 2016; Vandenabeele, Brewer, & Ritz, 2014). The original conceptualization of PSM characterized it as “an individual’s predisposition to respond to motives grounded primarily or uniquely in public institutions and organizations. The term ‘motives’ is used here to mean psychological deficiencies or needs that an individual feels some compulsion to eliminate” (Perry & Wise, 1990, p. 368, emphasis added). Given the initial conceptualization, much of the theoretical base used to understand PSM originates in psychology (Perry & Hondeghem, 2008; but see Le Grand, 2003, for an alternative explanation). Arguably, the widespread application of psychometric statistical techniques, such as CFA and SEM, to examine PSM is due not only to its theoretical foundations in psychology but also to the early application of CFA to examine construct measurement (Perry, 1996). Since that time, CFA procedures have been used to examine the validity of the PSM measurement instrument with different types of populations and in different countries (Coursey, Perry, Brudney, & Littlepage, 2008; Jensen & Vestergaard, 2017; Kim et al., 2013; Vandenabeele, 2008a). Given the initial measurement focus within PSM research, several studies also seek to test research hypotheses with SEM (Davis, 2010; Davis & Stazyk, 2014; Pandey, Wright, & Moynihan, 2008; Park & Rainey, 2008; Stazyk & Davis, 2015; Wright, Moynihan, & Pandey, 2012).

Based on the trajectory of PSM research, many uses of psychometric techniques were designed to assess construct measurement (Coursey et al., 2008; Kim et al., 2013; Perry, 1996). Although PSM researchers continue to focus on construct measurement issues, the research has evolved to use SEM as a hypothesis-testing technique. Admittedly, the application of SEM as a hypothesis-testing mechanism is far from universal in the PSM research community. Nevertheless, the widespread application of psychometric techniques is evident to even the passing observer of the PSM literature. In sum, much like any other research community, PSM researchers face methodological challenges (Wright, 2008), yet this stream of research serves as an example of methodological progress in the sense that it has capitalized on the accessibility of modern statistical techniques such as CFA and SEM. The question of whether the application of psychometric techniques in PSM serves as a mechanism for theoretical advancement, however, remains an open question.

In addition to PSM, bureaucratic red tape serves as a central public management concept examined extensively in volumes of research over several decades (Bozeman & Feeney, 2011; Pandey & Scott, 2002). Unlike PSM, however, the study of bureaucratic red tape was not initially grounded firmly in theories of psychology. The initial conceptualization of red tape included organizational red tape, which was rooted in objective organizational structure, and stakeholder red tape, which included organization members’ subjective interpretation of rules (Bozeman, 1993). Although examining objective organizational structure need not necessarily use psychological theories, exploring individual perceptions is firmly rooted in the realm of psychological inquiry. Given that red tape partially involved individual perceptions of organizational rules, subsequent scholarship sought psychological explanations for how individuals experience rule demands and rule burden (Davis & Pink-Harper, 2016; Pandey & Kingsley, 2000; Pandey & Welch, 2005; Scott & Pandey, 2005). The psychological foundations of red tape eventually contributed to increased application of CFA and SEM to explore its measurement, causes, and consequences (Coursey & Pandey, 2007; Davis, 2013; Davis & Stazyk, 2014; Moynihan, Wright, & Pandey, 2012).

Nonetheless, the application of psychometric techniques to examine red tape has been comparatively slow relative to PSM. Arguably, the adoption of psychometric statistical techniques by the red tape research community was due to limited attention to construct measurement in early studies. Although red tape researchers have consistently examined questionnaire data since the origins of the concept (Bozeman & Feeney, 2011; Buchanan, 1975; Rainey, Pandey, & Bozeman, 1995), scholarship has generally searched for both objective and subjective measures of the phenomenon (see Bozeman & Feeney, 2011; Pandey & Scott, 2002, for an extensive discussion of red tape measurement strategies). Importantly, syntheses of red tape scholarship highlight myriad measurement strategies used over the past 25 years. As such, the concerted emphasis on measurement in the infancy of the concept was not as pronounced as compared with PSM. The lack of early measurement emphasis is not a critique of the red tape literature, rather it highlights why CFA and SEM took longer to become mainstream techniques in this area.

The following section uses these examples to examine how and to what extent have psychometric statistical techniques contributed to sound theory development in public administration research. Specifically, we assess whether the application of SEM in conjunction with other statistical approaches has generally enhanced or hindered theory testing and development in these areas.

A Sound Merging of Method and Theory?

As stated above, CFA and SEM have become widely applied statistical techniques used to examine public management’s homegrown theories. Nevertheless, these techniques could become the methodological cart that precedes the theoretical horse, which could inevitably stall theory development and discourse. In this section, we examine whether the increasing ease with which scholars can use psychometric statistical techniques has given the false impression of reliable and meaningful theory development in the areas considered. The two areas examined are far from exhaustive. However, given the volume of research on PSM and red tape, each serves as a useful venue to evaluate whether methodological advances have resulted in sound theoretical advances.

We take up two issues in our assessment. First, we examine several instances when scholars have used a “kitchen sink” approach to statistical analysis. Using multiple statistical techniques in a given study can give the illusion of providing sounder tests of theory, but there are also good reasons to assume it can lead to inaccurate conclusions. Second, we examine instances where the application of psychometric techniques, particularly in the realm of measurement, deviates substantially from the logic underlying CFA and SEM. Unfortunately, these deviations have, in some cases, been used to inaccurately redefine theory.

Conclusion

The purpose of this article is to broadly examine the extent to which methodological progress—or, more specifically, the use of new or multiple methods in a single study—may be mistakenly assumed to generate stronger tests of theory. We argue that the public administration research community should be open to diversity in both theory and method as it applies to research on public organizations. However, we also argue that the acceptance of theoretical and methodological pluralism requires a better understading of methodological assumptions. There are times when the underlying assumptions of different methodologies are clearly incompatible; using them jointly as a “coherent, integrated” test of theory actually has the effect of slowing and perhaps even harming theory development and progress by producing inaccurate statistical results.

To support these points, we have focused on two streams of literature in public management: PSM and red tape. These areas of research provide a useful avenue to assess how researchers have, at times, applied new and multiple methods in stand-alone studies. There are clearly some excellent examples of scholarship in public administration that effectively advance theory by using sophisticated methodological approaches. Unfortunately, some studies appear to assume the use of more complex and numerous statistical techniques in a single study inherently improves methodological conclusions and, therefore, results in stronger evaluations of theory.

As a research community that hopes to both retain its legitimacy and confront pressing social issues, we should strive to ensure that theory and method mesh in ways that allow us to confront public administration’s deepest challenges. To do this means we must remain open to pluralist approaches. However, if we are to borrow theories and methods from other fields, we must also be exceedingly careful in our efforts to merge and apply these theories and methods, lest we stall theoretical progress and our ability to contribute to the field’s most pressing issues.

Our concern, throughout this article, has been that the application of new and complex methods have stalled theoretical progress by producing inaccurate statistical results—results that are inaccurate because they are generated through incompatible tests. In the face of methodological and theoretical pluralism, we need a field that is both tolerant of different approaches but also conservative and discriminating in its selection and application of them.

Practically, we would suggest the peer review process is the best place to meet these aims, requiring something of both authors and reviewers. First, scholars must themselves (a) be extremely wary of using new and different methods and methodologies until they have a sound grasp of the fundamental assumptions built into them, and (b) craft a compelling case supporting the appropriateness of using new and multiple methodologies in a single study. Second, reviewers must be (a) vocal in challenging and questioning the use of new and multiple methods in a study, and (b) avoid encouraging authors to include additional statistical techniques that may be incompatible with a study’s existing method and theoretical approach.

As regular users of CFA and SEM, we would also note that both are useful tools for advancing theory. So are other techniques and methods. We would, however, argue that researchers who routinely apply CFA and SEM must start by recognizing that measurement is a theoretical endeavor. Measurement models that do not adequately fit the data suggest theoretical rather than empirical problems. Trimming indicators from a construct or dimensions from a second-order construct may be empirically convenient, but those activities must also be theoretically tenable. Successfully merging theory and method means that researcher must be willing to not only trim parameters from a poorly fitting model but also acknowledge the flaws in their theoretical assertions—something CFA and SEM users in public administration too frequently violate.

Based on our examination, some streams of research in public administration invested significant theoretical attention on concept definition prior to turning toward measurement issues. Alternatively, some scholars have attempted to redefine theoretical concepts based largely on matters of empirical convenience. Although theoretical assertions and empirical evidence are intertwined, concept measurement must be firmly grounded in theoretical logic. Stable theoretical foundations serve as the bedrock of high-quality measurement. Given the concepts we examined here, red tape seems to have invested more attention toward theoretical definition in early research, whereas the theoretical definition of PSM has constantly shifted to accommodate empirical inconveniences. We do not intend to suggest that one of these research streams is qualitatively superior to the other. Rather, we offer each as examples of empirical research in public administration that can effectively use psychometric statistical techniques to merge theory and method.

In sum, empirical research applying CFA and SEM should ensure that the foundations of research have a stable theoretical base from which to progress. This is not to suggest that empirical evidence cannot redirect a stream of research based on evidence that fails to confirm theory. Instead, it suggests that the continuous interplay between theoretical and methodological advancement drive the scientific accumulation of knowledge so long as logical theoretical expectations drive empirical modeling strategies. To adequately use CFA and SEM to address the most pressing problems in public administration, scholars must put the theoretical horse in front of the methodological cart. Anything else mistakes methodological choices and progress for theoretical advancement.