The “Levels” Problem in Assessing Organizational Climate: Evidence From the Federal Employee Viewpoint Survey

Abstract

The Federal Employees Viewpoint Survey (FEVS) has gained prominence as the primary vehicle through which agencies assess the work-related attitudes of their employees. Within the discipline, the FEVS results have proven a fertile source of data on the job-related attitudes of public employees. However, concerns have been raised with the instrument and with how the FEVS data has been used by scholars. This article highlights a concern with the use of ambiguous terms which impede interpretation of the FEVS results. An investigation in partnership with officials from the regional office of a federal agency confirmed the divergent meanings respondents assign terms such as “my organization” and brought to the fore the extent to which, in large organizations, employee attitudes are distinctive to the units to which they belong at different hierarchical levels. The literature on organizational climate provides a useful framework within which the phenomenon can be understood and analyzed.

Keywords

organizational climate Federal Employee Viewpoint Survey questionnaire design multi-level analytical techniques

Scholars have found organizational climate to be an important determinant of both individual outcomes such as job satisfaction and organizational commitment and organizational outcomes such as sales performance and customer satisfaction. Climate, defined as the collective perceptions by members of organizational policies and practices (Schneider & Snyder, 1975), is commonly measured through the use of surveys. For example, an organization may administer a survey to its employees asking their perceptions on subjects such as communication, decision-making practices, and managerial support. The results of such surveys can provide managers and executives with important information that can be used to improve work conditions and to positively impact employee attitudes, motivation, and ultimately performance.

A prominent example of a climate survey in public administration is the Federal Employee Viewpoint Survey (FEVS) administered annually to a high proportion of the federal workforce by the U.S. Office of Personnel Management (OPM).¹ The FEVS results have garnered significant interest within the federal community in part as a consequence of the annual, “Best Places to Work” rankings compiled by the Partnership for Public Service (PPS, 2019). Three items from the FEVS are used to compile an index of employee engagement. PPS then ranks agencies in the categories of “large,” “mid-sized,” and “small.” Agencies at the top of each category such as the National Aeronautics and Space Administration in the large agency category become the objects of favorable attention while those at the bottom become the objects of inquiry on the part of, among others, congressional committees (Wagner, 2017). The FEVS has also served as an object of attention within the scholarly community; scholars have successfully mined the results as a means of probing relationships between constructs as diverse as organizational integrity, managerial trustworthiness, and public service motivation (Somers, 2017).

Widespread use of the FEVS data has, in turn, resulted in scrutiny of the instrument and of how the data has been employed for scholarly purposes. In a 2015 article, Fernandez, Resh, Moldogaziev, and Oberfield provide a broad overview of the FEVS and its history. They cite a list of 42 articles that had been published in public administration journals between 2000 and 2013 as evidence of the extent to which scholars have made use of data from the FEVS or its predecessors. While acknowledging the value of the FEVS data to scholars, Fernandez et al. are nevertheless critical of the FEVS on multiple grounds including that the FEVS lacks, “a central focus or guiding research question” (p. 388). They further criticize some aspects of the design of the FEVS instrument including, for example, the inclusion of, “double-barreled” (p. 389) items, that is, items which combine two or more constructs that are appropriately probed separately. In 2017, Somers offered a separate critique directing attention not to the design of the FEVS but to how the FEVS data has been used or misused by scholars. He references multiple constructs that scholars have assembled from FEVS items such as autonomy, intrinsic motivation, leadership, and job involvement for which validation procedures have been inadequate.

This article continues the conversation initiated by Fernandez et al. (2015) and Somers (2017) about the FEVS and its use. Similar to Fernandez et al., we highlight questions with regard to the construction of some FEVS items. Similar to Somers, we highlight the need for scholars to use the FEVS data consistent with its intended purpose which is to serve as a means of assessing organizational climate.

The relevance of that purpose became apparent in the course of an investigation undertaken by the authors in partnership with officials from the regional office of a federal agency. One purpose of the partnership was to investigate issues of work motivation in the office. A second purpose had to do with the difficulties these officials had encountered in interpreting the results of some of the items in the FEVS. A number of the FEVS items, like other climate surveys, use terms such as “your [or my] organization” where the precise referent is unclear. For example, one of the items in the FEVS reads, “Considering everything, how satisfied are you with your organization?” The problem, as cited by these officials, was that, “your organization” could refer to any one of five different organizational levels, section (lowest level), branch, division, regional office, or the agency as a whole. Unable to discern from the FEVS results how employees interpret the term, these officials have been at a loss to know, when the FEVS results signaled a problem that needed remediation, at which level to intervene. Table 1 lists the eleven FEVS items for which the precise referent is unclear.

Our probe revealed that employees within the regional office assign highly divergent meanings to terms such as “your organization.” Although the immediate issue is one of survey design and cognitive processing, the broader problem has to do with the issue of how climate is assessed in large organizations with multiple hierarchical levels. Is climate relevant only to the organization as a whole or can distinctive climates exist within subsidiary units? The answer that emerges from the climate literature is that a distinctive climate exists within any one organizational unit only to the extent that it can be demonstrated (a) that there is some minimal degree of consensus as to the nature of that climate among employees within that unit and (b) there are differences in climate perceptions across units at the same level.

In the section that follows, we present an overview of the climate construct which has received limited attention within the public administration literature. We then provide a review of how the FEVS has evolved and of how OPM is seeking to bring the FEVS into conformity with more conventional climate assessment instruments.

Organizational Climate

Scholars have found organizational climate to be consequential for both individual and organizational outcomes. Kuenzi and Schminke (2009) cite 89 empirical works published between 1980 and 2008 investigating the relationships between climate and other constructs. Individual outcomes to which climate has been found to relate include job satisfaction, organizational commitment, and turnover intention. Another set of studies have found climate to correlate with organization-level outcomes such as sales performance (McKay et al., 2008), injury rates (Hofmann & Mark, 2006), and customer satisfaction (Dietz et al., 2004).

Of central interest to climate scholars has been whether it is meaningful to talk about a single, “global” or, “molar” climate or only about multiple, “facet”-specific climates. Facet-specific climates that have been studied relate to a range of organizational activities. Thus, Hofmann and Mark (2006) looked at the relationship of safety climate to injury rates. McKay et al. (2008) assessed whether diversity climate moderated racial differences in employee sales performance. Chen et al. (2007) investigated how empowerment climate affected the relationship between leader-member exchange and individual performance. On the broader question of whether global climate is a meaningful construct, Ostroff et al. (2003, p. 573) comment,

Schneider (1975) concluded that the molar concept of climate was too amorphous, inclusive, and multifaceted to be useful; that is, attempting to describe organizational situations simultaneously along 10 or so generic facets has no focus, and thus relationships to some specific outcome will be modest at best (Schneider et al., 2000). He proposed that climate be conceptualized and studied as a specific construct that has a particular referent or strategic focus, indicative of the organization’s goals.

Discussions of organizational climate within the public administration literature have conformed with the trend toward a focus on facet-specific climates. In a 2012 article, Choi (2012) investigated the relationship between diversity climate and employee job satisfaction with diversity climate defined on the basis of three questions from the 2008 Federal Human Capital Survey, the predecessor of the FEVS. Oberfield (2016) explored multiple antecedents of diversity climate such as personnel diversity, management diversity, and procedural justice climate, operationalizing diversity climate with the same three questions employed by Choi but using data from the 2012 FEVS. Moon (2016), in turn, looked at transformational leadership climate as a moderator of the relationship between workforce diversity and organizational citizenship behavior.

None of these scholars acknowledge the debate within the broader climate literature as to whether, in the absence of some minimum degree of consensus among respondents, a climate of any type can be said to exist in the first place. The issue was brought to the fore by James and Jones (1974) who make a distinction between, “psychological climate,” with the individual as the unit of analysis and organizational climate with the organization as the unit of analysis. An organizational climate can be said to exist only if there is a, “consensus of perception” (p. 1103) among employees which, “must be verified if accumulated perceptual organizational climate measures are used to describe organizational attributes” (p. 1103). The issue becomes relevant in the context of this study and its focus on how employee perceptions of the organizational environment can vary across hierarchical levels because pursuant to this perspective, the same criterion, consensus of perception, should serve as the basis for determining whether a distinctive climate can be said to exist within an organizational unit at any of multiple organizational levels. Thus, in the regional office that served as the venue for this study, a distinctive climate could be said to exist within an individual division only to the extent that it can be shown that (a) there is some minimal degree of agreement among divisional employees as to the nature of that climate and (b) that there is variation in climate perceptions across divisions within the regional office.

Climate Assessment and the Federal Employee Viewpoint Survey

The first executive branch-wide survey of employees entitled the Federal Employee Attitudes Survey was designed in part to gauge employee sentiment in the aftermath of the Civil Service Reform Act of 1978 (Fernandez et al., 2015). In 2003, Congress passed the National Defense Authorization Act for Fiscal Year 2004 (NDAA 2004) which included a provision mandating an annual survey of executive branch employees. In the law Congress directed that the survey focus on,

“Leadership and management practices that contribute to agency performance,”

“Employee satisfaction with, a) leadership policies and practices, b) work environment, c) rewards and recognition for professional accomplishment and personal contributions to achieving organizational mission, d) opportunity for professional development and growth; and e) opportunity to contribute to achieving organizational mission.” (“Personnel Management in Agencies—Employee Surveys,” 2006, p. 49979).

The law directed OPM to, “issue regulations prescribing survey questions that should appear on all agency surveys” (“Personnel Management in Agencies—Employee Surveys,” 2006, p. 49979). A final regulation published in August 2006 included a list of 45 required questions that OPM deemed consistent with the areas of interest identified by Congress (“Personnel Management in Agencies,” 2016).

In the Technical Report that accompanied the release of the 2018 FEVS results, U.S. OPM (2018) states that, “the FEVS instrument was designed to assess the climate of federal agencies,” defined as, “employees’ perceptions of organizational policies, practices, and procedures and subsequent patterns of interactions and behaviors that support organizational performance” (p. 1). In fact, however, the FEVS departs from conventional climate surveys as a consequence of the fact that the FEVS items derive from multiple sources not all of which were intended to serve climate assessment purposes. For example, some items derive from the Merit Principles Survey administered periodically by the U.S. Merit Systems Protection Board (n.d.), “to assess the health of Federal merit systems.” Other questions originated in the Clinton-era Organizational Assessment Survey. In 2011, a set of questions related to work-life balance were included in the FEVS to accommodate an Obama administration work-life balance initiative. Despite its proclaimed climate assessment purpose, the current version of the FEVS is thus something of an amalgam with items originating from instruments that served diverse purposes.

In 2016, OPM initiated an effort to address flaws with the FEVS instrument including that of coherence. OPM published a rule in the Federal Register (finalized in 2017) replacing the original 2006 list of the 45 items required to be included in the annual employee survey with only 16 thereby allowing flexibility in the redesign of the instrument. The intent, according to OPM officials, is to use that flexibility to ameliorate flawed items (such as the “double-barreled” items) and to bring the instrument into closer conformity with more conventional climate assessment instruments (Personal communication with an OPM official, March 7, 2017).

This study coincides with OPM’s initiative in two respects, one of which is to identify candidate items for rewording and the second of which is to encourage use of the FEVS consistent with its climate assessment purpose. The wording issue on which this study focuses has to do with the use of ambiguous terms such as “your organization,” where multiple, alternative referents are available. The following two sections present an account of the investigation undertaken by the authors in conjunction with officials from the regional office of a federal agency to probe how employees interpret such items as relates to organizational unit/level. The subsequent section includes a discussion of the “levels” issue more generally as has been addressed within the climate literature.

Method

In 2016, the authors entered into a partnership with officials from the regional office of a federal agency in part to probe issues of employee work motivation and in part to assist in the interpretation of their FEVS results. In the fall of 2016, an online survey was sent to all 1,070 employees in the regional office holding positions below that of division head. The subject agency includes five hierarchical levels as depicted in Figure 1.

Figure 1.

Multiple organizational levels in a typical federal bureaucracy.

Questions designed to probe issues relating to ambiguous terminology in the FEVS were part of a larger survey. Table 2 lists the categories of the items included in the survey. The categories relevant to the present discussion include Categories 5—“Best Places to Work Scale,” 10—“Best Places to Work Scale—Organizational Level,” 14—“Job Characteristics,” 15—“Senior Leaders and Managers,” 16—“Prominence of ‘my organization,’” and 17—“Demographics.” Category 5—“Best Places to Work Scale,” replicated three items from the FEVS: “I recommend my organization as a good place to work”; “Considering everything, how satisfied are you with your job”; and “Considering everything, how satisfied are you with your organization?”³ Category 10—“Best Places to Work Scale—Organizational Level,” included four items intended to replicate the first of the items in the Best Places to Work scale except with a more specific referent for organizational level. Thus, the four items in this section read;

I recommend Agency X as a good place to work,

I recommend Region Y as a good place to work,

I recommend my Division as a good place to work,

I recommend my Branch as a good place to work.

In the questions, “Agency X” and “Region Y” were replaced with their appropriate names. They are masked here to preserve the agency’s preference to remain anonymous. Category 15—“Senior Leaders and Managers” included two items that probed employee interpretations of the terms, “senior leaders” and “managers,” respectively. All the items from the six categories of items referred to in this article are included in the Appendix. The order in which the items were listed was the same for all respondents. Thus, the first item in each category referenced the highest organizational level—the agency as a whole, the second item referenced the region, the third, the division, and the fourth, the branch.

The item for which a question order effect (Strack, 1992) would be most likely would be that which asked respondents to rank from 1 to 4, the organizational level, “most prominent in your mind,” when asked about “my organization.” At issue is whether listing the agency first biases the results in favor of agency. The results do not suggest such an effect. For example, branch, which is the last of the items listed, received the second highest number of responses as the, “level of the organization . . . most prominent in your mind,” after agency but before region and division which were both listed above branch.

Several strategies were used to increase response rates. Information about the survey was placed on the agency’s intranet, signs with information on the study were posted in the building, and several reminder emails were sent to the employees. Overall, 567 responses were collected for a response rate of 53%. Respondent demographics were quite similar to the composition of the agency as a whole. Of the respondents, 52% were female (compared to 51% in the agency); 64% were over the age of 45 (compared to 68% in the agency); and 62% had been with the agency for 15 years or more (compared to 60% in the agency).

Findings

My Organization

For the purpose of understanding how employees interpret the term, “my organization” as employed in the FEVS (as well as other scales and surveys), the authors first replicated the following FEVS question: “I recommend my organization as a good place to work.” Those results are shown in Figure 2. Respondents were then asked a series of specific questions using the same terminology but replacing the ambiguous term “my organization” with a specific level of the organization. The results of these questions, shown in Figure 3, reveal a discrepancy in the attitudes of employees pertaining to the different organizational levels. In particular, the chart shows that employees are more negatively disposed toward the most proximate unit, the branch, than toward the higher-level units of which the branch is part. A dependent sample t-test of differences in the proportions between responses that strongly disagree when asked about “Agency X” and responses when asked about their “Branch” was statistically significant (t = 5.78, p <.001).

Figure 2.

Responses to “I recommend my organization as a good place to work.”

Figure 3.

Probing responses to recommendation based on specific organization level.

In the absence of data from the more specific formulation and forced to act on the basis of the results from the generic question asking about one’s organization, the office would act absent the knowledge that, to the extent the results suggest a problem with how the employees feel about the organization, the problem is at the lowest organizational level. It could be that the highest level of dissatisfaction is coming from within a single branch and thus attempts at reform or change at higher levels of the organization would be unnecessary.

A similar approach was taken with regard to the FEVS item, “Considering everything, how satisfied are you with your organization?” First, the question was posed as in the FEVS (see Figure 4) and subsequently the question was posed in relation to each organizational unit/level separately (see Figure 5). Again, the results show greater levels of dissatisfaction with the branch than with the higher levels of which the branch is part. A t-test of the proportion of those who were extremely dissatisfied with the lowest level to those who stated they were extremely dissatisfied with the highest level was statistically significant (t = 6.06, p < .001). With this additional information, not available as a consequence of the FEVS’ current design, region-level management would not know to focus on branch-level management for purposes of eliciting higher levels of satisfaction and engagement on the part of employees.

Figure 4.

Responses to “Considering everything how satisfied are you with your organization?”

Figure 5.

Probing responses to satisfaction based on specific organizational level.

Overall, Figures 3 and 5 show that respondents do not have a general perception of the organization that transcends different levels. Some individuals are more satisfied with their agency while others are more likely to provide positive responses for their branch. The overall variation in satisfaction when asking the same question but when providing a more specific referent underscores the challenges of relying on questions with ambiguous terms for managerial decision-making or reform.

As a means of further probing how questions relating to the term “my organization” are processed and understood, respondents were asked to rank each organizational level (agency, region, division, branch) according to the degree to which the employee interpreted the “my organization” questions as referring to that level. Specifically, the employees were asked,

Several of the preceding questions in the survey asked you about your thoughts and perceptions toward “your organization.” When you see this term or are asked by someone about your organization, which levels of the organization are most prominent in your mind or most stand out to you? Please rank the following from 1 to 4, where 1 is the most prominent and 4 is the least.

The results, shown in Figure 6, reveal significant differences in interpretation among the employees. The two levels that the employees associate most closely with the term “my organization” are the highest level (agency) and the lowest level (branch), respectively. These are the same two levels that the employees associate least with “my organization.” Thus, approximately a third of the employees identify the agency as a whole as the appropriate “my organization” referent, and another 30% identify their branch. Approximately 40% of employees rank the branch as the level least associated with the term “my organization” and other 40% identify the agency as a whole as the least appropriate “my organization” referent. In fact, the largest number of respondents, 128, had the agency ranked as first and the branch ranked as last. The second largest number, 99, had the agency ranked as last and the branch ranked as first. These findings are particularly revealing of the extent of divergence among employees in their interpretations of the term “my organization” and is suggestive of the confusion that can result in trying to interpret the results of ambiguous climate survey terms. Many respondents perceive the term in exactly opposite ways hindering both the measurement of climate and the capacity of managers to effectively act on the basis of that measurement.

Figure 6.

Level of organization most prominent in respondents mind when being asked about “your organization” (1 is most prominent, 4 is least).

As a means of better understanding how employees process these questions, the authors posed an open-ended question as follows: “Can you further elaborate on your ranking above to help us better understand how you think about and conceptualize ‘your organization’”? One set of comments highlighted a relationship between the employee’s response and the nature of the work of the office to which the employee is assigned. An employee wrote, “In my division there are three branches and the work of the three branches are interdependent and inform the work of each other, mostly. This interdependence is in large part why I view the division as the prominent entity.” Another employee wrote, “Agency X as a whole and Region Y are most prominent because I work in the Office of the Regional Administrator and we cross all programs.”

A second set of comments in response to the open-ended question highlighted the nature of the individual employee’s job as a factor in the identification of the level/unit associated with “my organization.” More specifically, an employee whose job duties cause him or her to engage with those from outside his or her immediate branch or division on a regular basis appeared more likely to specify the higher-level unit as “my organization.” One employee wrote, “I find I need to collaborate with members of other divisions in Region Y. Therefore, Region Y, as the locus of my work and the people I work with stands out most prominently as ‘my organization.’”

Senior Leaders

The second term on which our investigation focused was that of “senior leaders”. The two items in the FEVS that include the term “senior leaders” are shown in Table 1, above.

The employees were asked: “ When you hear the term “senior leader” or are asked about your perception of senior leadership, what members of the organization do you consider to be senior leaders? Click all that apply.”

The choices offered were one or more of;

“Those at Agency headquarters in Washington,”

“Those above the division directors in Region Y (Regional Administrator/Deputy Regional Administrator),”

“Division Directors,”

“Branch/Office Managers.”

Table 1.

Ambiguous Terms in the Federal Employee Viewpoint Survey.

Term	Examples
“my organization”	“I recommend my organization as a good place to work.” (p. 40)
	“Considering everything, how satisfied are you with your organization?” (p. 71)
“leaders”	“In my organization, leaders generate high levels of motivation and commitment in the workforce.” (p. 53)
	“My organization’s leaders maintain high standards of honesty and integrity.” (p. 54)
“managers”	“Managers communicate the goals and priorities of the organization.” (p. 56)
	“Managers review and evaluate the organization’s progress toward meeting its goals and objectives.” (p. 57)
	“Managers promote communication among different work units (for example, about projects, goals, needed resources.” (p. 58)
	“Managers support collaboration across work units to accomplish work objectives.” (p. 59)
“senior leaders”	“I have a high level of respect for my organization’s senior leaders.” (p. 61)
	“How satisfied are you with the policies and practices of your senior leaders?” (p. 66)
“management”	“How satisfied are you with the information you receive from management on what’s going on in your organization?” (p. 64)

The results, shown in Figure 7, reveal that employees interpret the term “senior leaders” very differently. The option selected most frequently was, “Those above the division directors in Region Y (Regional Administrator/Deputy Regional Administrator),” followed closely by, “Division Directors,” and, “Those at Agency headquarters in Washington.” The point made above would apply here as well: If the FEVS results show that employees are not satisfied with, “the policies and practices of [their] senior leaders,” and the term evokes such different interpretations, how does management know with which group and at which level to intervene?

Figure 7.

Which members of the organization are considered “Senior Leaders”?

Discussion

The divergent interpretations of the FEVS items by employees led to a review of the organizational climate literature for guidance on how the levels issue is appropriately addressed in climate surveys. Within that literature, two approaches can be identified. One is to specify the intended referent within the item. Patterson et al. (2005, p. 383) comment, “Each climate questionnaire item should clearly focus on the specific collective unit which corresponds to the climate being studied (team, department, or organization),” adding,

Unfortunately, in many studies respondents have not been instructed to focus on a specific organizational unit, but rather to provide descriptions relating to their “work environments” . . . This ambiguity in the frame of reference of climate items can lead to individuals describing perceptions of different parts of the organization, some assuming the questionnaire asks them to describe their department and others assuming the referent is the organization . . . (p. 383)

Similarly, Schneider et al. (2011) comment,

There is consensus . . . that if one is studying the climate of the unit as a whole, one should frame items for respondents such that they describe the unit to which the items will be aggregated, and thus the level of analysis for analysis! (p. 381)

Pursuant to this approach, items in the FEVS would reference specific levels or units such as in the survey employed here.

This approach presents two difficulties as relate to the FEVS however. One is that each agency is structured differently such that the levels and hence the terminology employed would have to vary by agency. The head of OPM’s survey unit commented that her office has identified 60,000 relevant “components” within the federal establishment (Personal communication with OPM official, April 27, 2017). Customizing the survey in those circumstances would present a daunting challenge for the survey administrators. A second concern is that a customized approach would result in a proliferation of items in a survey that already includes over 90 items. Thus, for example, in an agency with five hierarchical levels similar to the subject of this study, each item listed in Table 1 would require four different versions. Such significant increases in survey length would likely have an adverse impact on response rate.

Assessing Level-Specific Climates

A second approach to resolving the levels issue would be to probe the level of agreement among respondents within and across units at each level. The logic of this approach is consistent with that of James and Jones (1974), cited above, who contend that climate as an organization-level construct can be said to exist only to the extent that there is a reasonably high degree of consensus among members as to the nature of that climate.

Pursuant to this logic, a distinctive climate can be said to exist within a subsidiary organizational unit to the extent it can be demonstrated (a) that there exists a high level of agreement among employees within the unit as to the nature of the climate and (b) that there are differences across units at the same organizational level. Thus, within the agency that serves as the object for this study, a distinctive regional office climate could be said to exist only upon demonstration of a consensus among employees within the region as to the nature of that climate and differences in climate results across regional offices. Field and Abelson (1982) comment that, “group and organizational climate will be dependent upon analyses which show small nonstandardized within-group organizational variance and/or significantly larger between-group than within-group variance” (p. 188). Similarly, Awal and Stumpf (1981) comment that,

If a work group climate (WGC) exists, then there should be substantial differences between OC [organization climate] and WGC across organizations and work groups. These differences could take the form of different salient dimensions of climate, mean differences on the same set of dimensions, and; or relationship differences between OC and WGC dimensions and other variables such as job attitudes. (p. 34)

As explained by Chan (1998), what is at issue is a, “compositional model” (p. 234) in which, “ . . . the functional relationships among phenomena or constructs at different levels of analysis (e.g., individual level, team level, organizational level)” (p. 234) are specified. Chan’s interest is in, “elemental composition” (p. 234), “in which data from a lower level are used to establish the higher level construct” (p. 234). The approach traditionally employed by OPM in reporting the FEVS results is what Chan identifies as the, “additive” (p. 236) model whereby the, “higher-level unit is a summation of the lower-level units regardless of the variance among these units” (p. 236). In employing the additive model, OPM omits information on the degree of dispersion in the individual responses which some theorists suggest is a critical determinant of whether a climate can be said to exist.

With the alternative, “direct consensus” (Chan, 1998) model, the, “meaning of the higher-level construct is in the consensus among the lower level units” (p. 236). In other words, the higher level construct (climate) can only be said to exist to the extent that there is minimal level of agreement on its nature. Schneider et al. (2002, p. 221) comment as follows:

The direct consensus model is the one most frequently discussed in research on organizational climate because shared perceptual agreement at the individual level of analysis has been seen as functionally isomorphic to the construct at the organizational level of analysis (Chan, 1998, p. 237). Organizational climate, then, is the average or most typical way that people in the organization describe it, and within-group agreement in this model serves as a prerequisite for the group-level variable (Chan, 1998). The absence of shared perception, or high within-group variability, implies that a group-level construct does not exist; in other words, the group has no shared meaning (Klein et al., 2001).

In the context of the agency within which this study occurred, the direct consensus approach would dictate that one first determine the specific climate to be assessed, identify those items from the FEVS which capture that climate facet and, based on an analysis of variance, identify those organizational levels/units where distinctive climates exist.

Are Divergent Interpretations Consequential?²

It is important to note in the context of the above discussion that, notwithstanding the divergent interpretations that employees assign to ambiguous terms in the FEVS, there is nevertheless a relatively high degree of correlation in employee perceptions across the different organizational levels—see Figure 3. Thus, the percentage of employees who, “strongly disagree” with the “I recommend . . . ” statement is relatively low across all four levels while the percentage of employees who “strongly agree” with the “I recommend . . . ” statement is relatively high across the four levels. That employee perceptions are generally similar across levels suggests a probe into whether variation in employee interpretation of the terms and assignment of different levels of satisfaction to different levels of the organization are consequential with regard to key outcomes.

Although we did not collect data to specifically test the practical implications of our findings, we can explore a few simple models to validate how the interpretation assigned shapes the conclusions that one may draw from the data. For purposes of this example we draw on data from that portion of the larger survey relating to job characteristics (see Table 2). The model tests how demographics and job characteristics shape two outcomes (a) how likely one is to recommend their organization as a good place to work (Recommend) and (b) how satisfied they are with their current organization (Satisfied). For each of the items, we included responses to those items when the agency and branch level were used as the referent level and develop a simple regression model to compare coefficients across a number of predictor variables.

Table 2.

Survey Overview.

Category number	Item categories	Categories relevant to this study
1	Task significance (4 items)
2	Organizational commitment (6 items)
3	Public service motivation (8 items)
4	Job satisfaction (5 items)
5	Best places to work scale (3 items)	x
6	Perceived social impact and perceived social worth (5 items)
7	Citizen contact (3 items)
8	Media bias (1 item)
9	Public perception (1 item)
10	Best places to work scale—organizational level (8 items)	x
11	Psychological safety (7 items)
12	Person-organization fit (3 items)
13	Work engagement (10 items)
14	Job characteristics (6 items)	x
15	Senior leaders and managers (2 items)	x
16	Prominence of “my organization” (2 items)	x
17	Demographics (4 items)	x

For predictor variables, we rely on the basic demographics of tenure (years with agency, broken down into the following categories: 0–1 years (1); 2–5 years (2); 6–10 years (3); 11–15 years (4); 15 or more years (5)), age (broken down into the following categories: 18–24 (1); 25–34 (2); 35–44 (3); 45–54 (4); 55–64 (5); 65–74 (6); 75 or older (7)), gender (Female = 1), and supervisor status (=1 respondent was in a supervisory position). For job characteristics, we used the items developed by Hackman and Oldham’s (1975, 1980) Job Diagnostic Survey. The five job characteristics were as follows (each measured on a 5-point, disagree/agree scale): (a) “My job affords me a good deal of autonomy in how I go about doing my work” (Autonomy), (b) “My job involves doing a ‘whole’ and identifiable piece of work. That is, my job tasks result in a complete piece of work that has an obvious beginning and end” (Task Identity), (c) “There is great variety in my work that allows me to use a variety of my skills and talents” (Use Skills), (d) “My managers let me know how well I am doing my job” (Management Feedback), and (e) “My job provides me with information about my work performance. That is, the actual work itself provide clues about how well I am doing—aside from any ‘feedback’ co-workers or supervisors may provide” (Performance Information). Results of the models are shown in Table 3.

Table 3.

Perceptions of “my organization” and Outcomes.

Variables	Satisfied—agency	Satisfied—branch	Recommend—agency	Recommend—branch
(Intercept)	1.05***	0.55	1.31***	0.64*
	−0.28	−0.34	−0.28	−0.36
Autonomy	0.26***	0.21***	0.26***	0.19***
	−0.05	−0.06	−0.05	−0.06
Task identity	0.02	0.08*	−0.01	0.03
	−0.04	−0.05	−0.04	−0.05
Use skills	0.12***	0.18***	0.11***	0.20***
	−0.04	−0.05	−0.04	−0.05
Management feedback	0.23***	0.45***	0.18***	0.42***
	−0.04	−0.05	−0.04	−0.05
Performance information	0.08*	0	0.11**	0.06
	−0.05	−0.05	−0.05	−0.06
Tenure	−0.14***	−0.21***	−0.14***	−0.23***
	−0.04	−0.05	−0.04	−0.05
Supervisor	0.30***	0.62***	0.27**	0.64***
	−0.11	−0.13	−0.11	−0.14
Age	0.09*	0.08	0.11**	0.11*
	−0.04	−0.05	−0.04	−0.06
Gender	0.15**	0.03	0.20**	0.04
	−0.08	−0.09	−0.08	−0.1
R ²	0.35	0.42	0.31	0.39
Adjusted R²	0.34	0.41	0.3	0.37
Number of observations	491	491	491	491
RMSE	0.83	1.01	0.84	1.06

Note. RMSE = root mean square error.

*p < .1. **p < .05. ***p < .01.

While the results should be similar given the correlation in responses to recommendation and satisfaction at the different levels of the organization, we do find distinct differences in the results of the models. Of note is the size of the coefficients on Use Skills, Management Feedback, and Supervisor. The size of the coefficient on Use Skills increases by approximately 50% for both Satisfied and Recommend when shifting the referent from agency to branch. Similarly, the coefficients on Management Feedback and Supervisor roughly double in size when going from agency to branch. The demographic variable of gender is positive and significant with agency as the referent, but a magnitude smaller and insignificant when branch is the referent. Thus, managers may identify a potential concern for differences in gender perceptions of organizational satisfaction if questions are asked with the agency as the referent but not when the branch is the referent. While these results provide a simple demonstration of possible practical implications of the data, further research is needed to better understand what relationships and findings may be most affected by ambiguous terminology.

Conclusion

Our study follows those of Fernandez et al. (2015) and Somers (2017) in addressing issues with regard to the Federal Employee Viewpoint Survey (FEVS) and use of the FEVS data. Consistent with the concerns of our regional office partners, our specific interest related to the interpretation of the FEVS results in light of the multiple, ambiguous terms relating to organizational level that are included. In the course of our research, we learned that the problem confronted is one that has provoked extensive discussion within the organizational climate literature. The Discussion section above highlights what has been learned from that literature as to alternative approaches to resolving the problem and as to how that knowledge could be applied in the case of the FEVS.

Although one option would be for OPM to specify the unit/level of interest in each survey item, the size and complexity of the executive branch poses significant administrative challenges to such an approach. An alternative approach that emerges from the literature on organizational climate entails an assessment of the extent to which employees share a common perception of the organizational environment via an analysis of the degree of variation in the responses within any one organizational unit and across units at the same organizational level. A complication is that the propensity to focus on facet-specific climates such as diversity, empowerment, or safety highlights the possibility that the extent to which a climate exists within any one unit or at any one level may vary by climate type or facet. As the FEVS instrument is brought into conformity with conventional climate assessment instruments such as, for example, that developed by Patterson et al. (2005) which includes 16 separate facets, agencies will need to be prepared to accommodate this complexity.

Although our study was confined to a regional office within a single federal agency, OPM officials confirm that the issue of ambiguous terminology in the FEVS has been raised by multiple agencies. The problem surfaced in a formal manner when, in 2016, OPM solicited comments in response to its revised rule on the FEVS. The final rule included the following comment:

Two individuals indicated that Federal Employee Viewpoint Survey references to senior leader, manager and supervisory levels in questions are not clear to employees taking the survey, notwithstanding the terms’ definitions in 5 CFR part 250. (“Personnel Management in Agencies,” 2016, p. 89363)

The response to these comments included the following statement:

OPM acknowledges that general terms and definitions for leadership levels (senior leader, manager, and supervisor) may vary greatly from agency to agency and it is imperative to give agencies and respondents a clearer understanding of each level for accurate answers/data.

Although no specific prescription was offered, as the FEVS is brought into greater conformance with conventional climate assessment instruments, analyses of the type highlighted here could serve as one means of resolving the issue.

From a scholarly perspective, our study highlights the necessity, when climate-related constructs are operationalized using items from the FEVS, that the degree of consensus among respondents as to perceptions of organizational practices and policies be analyzed prior to a determination of whether climate as a collective construct actually exists. We hope that our study will alert public administration scholars to the nuances of analyzing and understanding organizational climate as articulated in the climate literature and to the need to interpret the FEVS data consistent with its intended, climate-assessment purpose.

A suitable subject for future research would be to conduct an analysis of the type outlined here for a specific climate type or facet within one or more agencies and to evaluate how differences across units and levels factor into overall employee perceptions of their work environment.³ A second topic for future research would be to probe relationships between demographic factors, job-related factors, and employee interpretations of terms such as “my organization.” As presented in Table 3, an initial test suggests that such interpretations are consequential but further research as to the extent and nature of these relationships is warranted.

Footnotes

Appendix

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

James R. Thompson

Notes

Author Biographies

James R. Thompson is associate professor in the Department of Public Administration at the University of Illinois, Chicago. His research interests center on the civil service, civil service reform, human resource management in public organizations, and organizational change in public organizations.

Michael D. Siciliano is associate professor in the Department of Public Administration at the University of Illinois at Chicago and co-director of the Networks and Governance Lab. Michael studies how humans and organizations collaborate to improve society. His work explores the cognitive, social, and institutional factors influencing the formation and performance of networks in the public sector.

References

Awal

Stumpf

S. A.

(1981). Differentiating between perceived organization and work group climates. Journal of Management, 7(1), 33–42.

Chan

(1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83(2), 234–246.

Chen

Lam

Zhong

J. A.

(2007). Leader-member exchange and member performance: A new look at individual-level negative feedback-seeking behavior and team-level empowerment climate. Journal of Applied Psychology, 92(1), 202–212.

Choi

(2012). Demographic diversity of managers and employee job satisfaction: Empirical analysis of the federal case. Review of Public Personnel Administration, 33(3), 275–298.

Dietz

Pugh

S. D.

Wiley

(2004). Service climate effects on customer attitudes: An examination of boundary conditions. Academy of Management Journal, 47(1), 81–92.

Fernandez

Resh

W. G.

Moldogaziev

Oberfield

(2015). Assessing the past and promise of the Federal Employee Viewpoint Survey for public management research: A research synthesis. Public Administration Review, 75(3), 382–394.

Field

R. H. G.

Abelson

M. A.

(1982). Climate: A reconceptualization and proposed model. Human Relations, 35(3), 181–201.

Hackman

J. R.

Oldham

G. R.

(1975). Development of the job diagnostic survey. Journal of Applied Psychology, 60(2), 259–286.

Hackman

J. R.

Oldham

G. R.

(1980). Motivation through the design of work. In Hackman

J. R.

Oldham

G. R.

(Eds.), Work redesign (pp. 71–98). Addison-Wesley.

10.

Hofmann

Mark

(2006). An investigation of the relationship between safety climate and medication errors as well as other nurse and patient outcomes. Personnel Psychology, 59(4), 847–869.

11.

James

L. R.

Jones

A. P.

(1974). Organizational climate: A review of theory and research. Psychological Bulletin, 81(12), 1096–1112.

12.

Klein

K. J.

Conn

A. B.

Smith

D. B.

Sorra

J. S.

(2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86(1), 3–16.

13.

Kuenzi

Schminke

(2009). Assembling fragments into a lens: A review, critique, and proposed research agenda for the organizational work climate literature. Journal of Management, 35(3), 634–717.

14.

McKay

P. F.

Avery

D. R.

Morris

M. A.

(2008). Mean racial-ethnic differences in employee sales performance: The moderating role of diversity climate. Personnel Psychology, 61(2), 349–374.

15.

Moon

K.-K.

(2016). The effects of diversity and transformational leadership climate on organizational citizenship behavior in the U.S. federal government: An organizational-level longitudinal study. Public Performance and Management Review, 40(2), 361–381.

16.

Oberfield

Z. W.

(2016). Why are some agencies more committed to diversity than others. Public Management Review, 18(5), 763–790.

17.

Ostroff

Kinicki

A. J.

Tamkins

M. M.

(2003). Organizational culture and climate. In Borman

W. C.

Ilgen

D. R.

Klimoski

R. J.

(Eds.), Handbook of Psychology (V. 12 Industrial and organizational psychology (pp. 565–593). John Wiley & Sons.

18.

Partnership for Public Service. (2019). Best places to work in the federal government. https://bestplacestowork.org/

19.

Patterson

M. G.

West

M. A.

Shackleton

V. J.

Dawson

J. F.

Lawthom

Maitlis

. . .Wallace

A. M.

(2005). Validating the organizational climate measure: Links to managerial practices, productivity and innovation. Journal of Organizational Behavior, 26(4), 379–408.

20.

Personnel Management in Agencies. (2016, December 12). 81 Federal Register, Article 89357.

21.

Personnel Management in Agencies—Employee Surveys. (2006, August 24). 71 Federal Register, Article 49979.

22.

Schneider

(1975). Organizational climates: An essay. Personnel Psychology, 28(4), 447–479.

23.

Schneider

Bowen

Ehrhart

M. G.

Holcombe

K. M.

(2000). The climate for service: Evolution of a construct. In Ashkanasy

N. M.

Wilderom

C. P.

Peterson

M. F.

(Eds.), Handbook of organizational culture and climate (pp. 21–36). SAGE.

24.

Schneider

Ehrhart

M. G.

Macey

W. H.

(2011). Perspectives on organizational climate and culture. In Zedeck

(Ed.), APA handbook of industrial and organizational psychology: Building and developing the organization (Vol. 1, pp. 373–414). American Psychological Association.

25.

Schneider

Salvaggio

A. N.

Subirats

(2002). Climate strength: A new direction for climate research. Journal of Applied Psychology, 87(2), 220–229.

26.

Schneider

Snyder

R. A.

(1975). Some relationships between job satisfaction and organization climate. Journal of Applied Psychology, 60(3), 318–328.

27.

Somers

M. J.

(2017). Strategies for improving measurement models for secondary data in public administration research: Illustrations from the Federal Employee Viewpoint Survey. Public Administration Review, 78(2), 228–239.

28.

Strack

(1992). “Order effects” in survey research: Activation and information functions of preceding questions. In Schwartz

Sudman

(Eds.), Context effects in social and psychological research (pp. 23–34). Springer.

29.

U.S. Merit Systems Protection Board. (n.d.). 2016 Merit principles survey data. https://www.mspb.gov/foia/SurveyData.htm

30.

U.S. Office of Personnel Management. (2018). Federal Employee Viewpoint Survey: Empowering employees, inspiring change.

31.

Wagner

(2017, December 6). NASA ranked best place to work in government six years running. Government Executive. https://www.govexec.com/management/2017/12/nasa-ranked-best-place-work-government-six-years-running/144317/

The “Levels” Problem in Assessing Organizational Climate: Evidence From the Federal Employee Viewpoint Survey

Abstract

Keywords

Organizational Climate

Climate Assessment and the Federal Employee Viewpoint Survey

Method

Findings

My Organization

Senior Leaders

Discussion

Assessing Level-Specific Climates

Are Divergent Interpretations Consequential? 2

Conclusion

Footnotes

Appendix

Declaration of Conflicting Interests

Funding

ORCID iD

Notes

Author Biographies

References

Are Divergent Interpretations Consequential?²