Incredibly Good Performance

Introduction

Many citizens think that government agencies do not do a good job in terms of achieving the objectives set for them (Hibbing & Theiss-Morse, 2001). Low performance contributes to lack of trust in government with citizens having perceptions, attitudes, and behaviors that discourage them from putting what they value, including their time and money, in the hands of government (Gambetta, 1988; Rousseau, Sitkin, Burt, & Camerer, 1998; Christensen & Laegreid, 2005). Distrust on the part of citizens lowers support for redistributive policy programs (Hetherington, 2005) and inhibits cooperation with government activities and laws (Tyler, 1990). The reporting of performance information to improve public accountability to citizens is a core part of arguments for enhancing citizens’ trust of government (Yang & Holzer, 2006). This idea, part of a broader attempt to make government more open and transparent, has been promoted by influential non-governmental organizations, including (in the United States) the Government Accounting Standards Board, the International City/County Management Association, and the National Association of State Budget Officers, and (internationally) Transparency International and the National Democratic Institute. Governments around the world have been attempting to increase transparency by providing more information about inputs, processes, outputs, and outcomes (Grimmelikhuijsen & Meijer, 2012; Hood, 2006b; James, 2011). To take the United States as a prominent example, the transparency requirements of the Government Performance and Results Act were strengthened in 2010 with a requirement that agencies make their performance information publicly available on a website (performance.gov), and the U.K. Government published performance against a set of Public Service Agreements for government activity in the 2000s.

To date, there has been little research on whether citizens find the reporting of government agencies’ performance to be credible. In particular, citizens may be skeptical of information about government agencies’ performance of their tasks when it is reported by those agencies themselves. Organizations have an interest in presenting their performance positively; several forms of unreliability and lack of validity, including deliberate misrepresentation, have been documented (Hood, 2006a, 2011; Moynihan, 2008). General distrust of government among the public may compound the lack of credibility (Alford, 2001; Chanley, Rudolph, & Rahn, 2000; Dalton, 2005; Van de Walle, Van Roosbroek, & Bouckaert, 2008; Van Ryzin, 2011). There is some evidence suggesting that simple performance measures provided by government organizations can be as credible to citizens as information from other sources (Van Ryzin & Lavena, 2013), but we do not know if this finding applies to more complex services and whether it applies in different jurisdictional contexts. In addition and especially, information about high levels of government performance may be particularly likely to lack credibility among citizens because of concerns about government agencies’ incentives to show they are performing well.

In this article, we ask the following questions: Do citizens find performance information from government agencies to be credible, or do they trust more in independent sources? Do they believe politically more attractive agencies, serving clients seen as deserving above politically less attractive agencies, serving clients viewed as less deserving? And does credibility depend on the level of performance, particularly when the government itself reports higher performance relative to lower performance? In the next section, we outline theoretical expectations of how the source and level of reported performance may influence credibility. A subsequent section sets out the experiment we designed to evaluate our hypotheses. We then present our findings, which show that credibility is lower for a less popular federal agency and that citizens are especially doubtful about good performance reported by the government agency itself, as opposed to an independent ratings firm. We conclude by drawing out the implications of these observed credibility effects for government performance reporting and democratic accountability.

Credibility and Sources of Performance Information

There is a great deal of published information about the performance of public organizations and public services in many contemporary contexts, reporting inputs, processes, outputs, or outcomes (Hatry, 1999; James, 2011; Moynihan, 2008; Van Ryzin, 2011). These efforts to report on government performance aim at enhancing the reputation of government agencies in the eyes of citizens by making public service efforts and accomplishments more transparent and visible, counteracting the often negative image of government performance in society (Goodsell, 2003). The potential for such attitude change, however, rests on the assumption that citizens will find government performance reports to be credible, which is the basic assumption we test here. We define credibility as a combination of the perceived truthfulness of the level of performance reported on a measure and the perceived believability of the measure itself as a way of gauging performance. Both these elements reflect citizens’ concerns with honesty and competence as a basis for trust and reflect the definition of credibility consisting of these elements that has been used extensively in political psychology ((Hovland & Weiss, 1951).

Research on information and change in audiences’ beliefs and attitudes suggests that the source of information matters for forming beliefs and underpinning consequent attitudes and behavior (Pornpitakpan, 2004). The expertise and competence of the source, having the interests of the audience at heart (and not exploiting them), the attractiveness or likability of the source, and the similarity between the source and the audience have all been identified as factors affecting credibility (Yoon, Kim, & Kim, 2011). In the context of information about public service performance, some of these factors are likely to affect whether citizens trust information from government agencies about their performance. In particular, because in many contexts citizens find government performance information somewhat abstract and difficult to judge, we would expect them to rely on heuristics or summary cues when forming their judgments about the information’s credibility. Thus, government performance information may engage citizens’ quick response, sometimes called “peripheral” route for perception and attitude change, which is typical for many political contexts where citizens do not have a personal, deep involvement in the issue (Lodge & Taber, 2013). In public services, this processing would be typical of perceptions of services that citizens do not regularly use. This peripheral route contrasts with the central route for perception and attitude change, which is more cognitively taxing and time-consuming, involving more systematic, deep reflection, and processing of information (Eagly & Chaiken, 1993; Petty & Cacioppo, 1986).

Information in the form of a summary cue about local government performance from an independent auditor has been found to be credible to citizens in previous experiments, changing both their perceptions of performance and satisfaction (James, 2011; James & Moseley, 2014). But to date, there has been little research contrasting different sources of government performance information. In an experiment about reporting street cleanliness, Van Ryzin and Lavena (2013) found very little difference in the credibility of a graphical report of the percentage of acceptably clean streets provided by government itself, compared with either a non-governmental source or a no-named source. These authors suggest that credibility effects may be more likely when the service is less tangible or the measurement is more abstract, so that sources act as a cue when citizens form their trust judgments. We test source effects precisely in this context, where such effects are potentially more likely to be observed. Thus, for our experiment, we have chosen to look at an index of overall customer satisfaction for government agencies, which is a fairly abstract, global performance measure that depends a great deal on how satisfaction is defined and evaluated. Related research has examined the effects of different forms of transparency, defined as release of information, on whether public organizations are seen as competent, acting in pursuit of the interests of affected people, and whether they tell the truth (Grimmelikhuijsen & Meijer, 2012). However, this work did not vary the source of the information. In our study, we look directly at citizens’ perceptions of the credibility of performance information itself, and also citizens’ assessments of actual performance relative to reported performance, to assess the effect of different sources of information on citizens’ judgments.

There are several potential alternatives to a government agency reporting on its own performance, including performance monitoring by independent public or private sector audit bodies, non-profit watchdog organizations, and private rating firms. For our study, we have chosen to study independent rating firms. These are independent from government agencies in that they are not publicly owned by the government, do not rely mainly on government funding, and are not regulated by government in their measurement and reporting activity (so they are not regulated in the way that some economic activity is subject to rules run by state regulatory bodies). There are many rating agencies, but our focus is on a major private firm that has developed a measure of user satisfaction with public and private services, the American Customer Satisfaction Index (ACSI). ACSI developed its satisfaction index originally in Sweden (Anderson, Fornell, & Lehman, 1994) and later applied the methodology extensively in the United States to measure customer satisfaction with a wide range of private and public services, including services provided by several U.S. federal government agencies. Although these government agencies contract with ACSI to have their services included, the ACSI methodology is highly standardized across both the private and public sectors, and thus individual firms and agencies cannot influence ACSI’s measurement and reporting of their satisfaction ratings.

The resulting satisfaction index information is available publicly and published online both by ACSI and by the various federal government agencies as part of their broader performance reporting efforts. This allows us to frame the information as coming from either ACSI or a federal agency itself. When customer satisfaction is reported by the agency itself, given general distrust in government and concerns specifically about performance measures, we might expect such information to be seen as less credible. This expectation leads to our first, and most basic, hypothesis:

Even when government is reporting on itself, as prior research on information source effects suggests, the political attractiveness of the agency to the audience may influence reception of the message. Attractiveness is often defined as perceived familiarity, likability, and similarity of the source to the receiver in terms of being part of group identity (McGuire, 1985). In the context of government agencies, more attractive agencies that enjoy broader public support because of favorable attitudes toward their programs or clients may be more likely to be seen as trustworthy sources of information, compared with government agencies that are less attractive. In our study, we focus on two U.S. federal agencies actually covered by the ACSI: the Veterans Administration (VA), the main agency serving veterans (a politically popular client group serving predominantly U.S. citizens), and the Citizenship and Immigration Services (CIS), the main agency serving immigrants (a less politically popular client group who are predominantly non-U.S. citizens). Although we recognize that these agencies differ in many complex ways, and we do not have an empirical gauge of attractiveness, we choose the VA and CIS from among the federal agencies included in ACSI as contrasting agencies in terms of their assumed attractiveness in the eyes of the American public because of their very different client groups. Thus, we suggest the following hypotheses about the attractiveness of the source and credibility:

Not all performance reporting by an agency on itself paints the same picture: It can reveal better or worse news about agency efforts and accomplishments, and we would expect that the perceived credibility of the information will vary in response. The literature on performance reporting suggests that citizens are likely to be more skeptical of relatively positive information about performance than information about relatively lower performance. Government sources have an incentive to misrepresent their performance as being high (to gain rewards or avoid criticism), but there is less obvious reason to think that they would misrepresent their performance as being low. Thus, we hypothesize that this suspicion of misrepresentation is likely to be most salient when the agency reports good news about itself. Citizens’ skepticism about an agency self-reporting good performance has some foundation in reality, it should be noted, as there are many examples of cases where performance has been misrepresented by government agencies, including exaggerating or even falsifying the information to look good (Bevan & Hood, 2006).

Thus, we expect that when agencies report positive information about their own performance, it will be seen as less credible by citizens. In turn, when agencies report less positive information about their own performance, it is more likely to be believed. By varying the positivity of the information about performance of the agency (by showing higher and lower performance), and by varying the source (government itself vs. a more independent source), we can thus test the following interaction hypothesis:

Experimental Design and Participants

The experiment we designed to test these hypotheses presents customer satisfaction information about a federal government agency with random allocation of three elements: the source of the information, the agency, and the level of the satisfaction index. Appendix A provides the complete text and images used in an online survey to present the source, agency, and index information.

We modeled the presentation of the information after ACSI, which as discussed above is a private firm that independently tracks customer satisfaction, with reported satisfaction index scores theoretically ranging from 0 to 100 (with 100 representing complete satisfaction). Both VA and CIS contract with ACSI to track their customer satisfaction, and in recent years, they have received similar ratings (in the high 80s to low 90s). As mentioned previously, the agencies and ACSI separately report this information to the public on their websites, enabling us to realistically frame the index as coming from either the government agency itself or from ACSI, which was described to respondents as an independent organization (because ACSI is not widely known and recognized). This manipulation of source (agency vs. ACSI) allowed us to test the central question of whether an agency reporting on itself is seen as less credible (H1).

The random assignment of two different agencies enabled us to evaluate H2, namely, that the VA (which serves veterans and is thus more politically attractive) would be seen as more credible than the CIS (which serves immigrants and is less politically attractive). Each agency’s logo was included on the page, and the full name of the agency was used, rather than the acronym, in the short written presentation of the satisfaction index (again see Appendix A). It should be noted that this study was conducted several months before the most serious and highest profile revelations of mismanagement and misconduct at the VA emerged in May 2014, eventually resulting in the resignation of the VA Secretary Shinseki. Still, the emerging scandal surrounding the VA at the time of our study may have attenuated the difference in its political attractiveness, as compared with CIS. However, more broadly, it is important to note that while using two real agencies is desirable because it makes the study relevant, the two bodies are complex bundles of activities and citizen perceptions of them, so this is a partial test of the issue of perceived agency characteristics.

To evaluate the source by level of performance interaction hypothesis (H3), we randomly varied the presented value of the 0 to 100 satisfaction index within a range of whole numbers from 86 to 93, based on the real range of ACSI scores reported for our two agencies. This allowed us to examine whether credibility declined with higher reported performance, compared with lower performance, especially when the agency was reporting on itself. We contrasted scores in the range 86 to 89, defined as “lower” (left digit 8), with scores in the range 90 to 93, defined as “higher” (left digit 9). Although all these satisfaction scores are based on the real ACSI scores for both agencies and fairly high overall, in absolute terms, the difference still allows us to exploit the left-digit bias effect. Left-digit bias has been found in a wide variety of contexts, including perceptions of government performance information (Olsen, 2013), and it accentuates the lower–higher contrast because recipients of the information concentrate on the left digit of the numbers.

For all three randomly varied factors (source, agency, and level), we tested for covariate balance using measures of age, sex, race, education, income, political ideology, and trust of government (see Appendix B). No significant differences were found across factors at the .10 level using the Bonferroni correction, thus indicating successful randomization on these variables.

There are two dependent variables that come from a series of questions that directly followed the presentation of the randomized source, agency, and satisfaction index. The first is a “credibility scale” composed of two items: a question asking, “Do you think the level of customer satisfaction reported for the agency is . . . ” with responses from 1 = definitely not true to 5 = definitely true; and a question asking, “In your view, how believable is this index as a way of measuring the satisfaction of actual customers with the agency’s services?” with responses from 1 = not believable at all to 10 = completely believable. The combination of these two questions into a scale reflects our conceptualization of credibility as involving both honesty and competence. For ease of interpretation, the credibility scale was transformed to a 0 to 100 range (M = 47.4, SD = 25.3), and the scale has good reliability (α = .86). The second dependent variable is a single question that asked, “What do you think the true level of the satisfaction index really is—your best guess?” Respondents could indicate a response from 1 = much worse to 7 = much better than the reported satisfaction index (again, a random value from 86 to 93), which was inserted at the midpoint of the 1 to 7 response set. This question thus captures respondents’ judgment of the extent to which the satisfaction index under- or overstates the agency’s real customer satisfaction. To facilitate interpretation, we transformed this measure to a −3 to 3 scale (M = −.86, SD = 1.33), with negative values indicating a belief that the real customer satisfaction is lower than the reported level and positive values indicating a belief that it is higher.

Participants were adult residents from across the United States recruited using CivicPanel, a university-affiliated online research panel, and the survey was conducted in March 2014. A total of 8,832 panelists were sent an email invitation, of which 1,680 were confirmed to have opened the email invitation and 689 responded after two contact attempts (representing a 19% contact rate and a 41% cooperation rate). We dropped 49 non-U.S. respondents as well as 41 respondents who completed the survey in under 2 min (because of concerns that they sped through the questions and did not fully read and consider the information presented to them), leaving an analytical sample of 599. Partial completion and item non-response resulted in somewhat smaller samples in the analyses, depending on the variables involved. Participants were predominately non-Hispanic White (81%) and disproportionately female (67%), with an average of 49 years of age, 15 years of education, and US$75,000 of annual income. In terms of political ideology, they are fairly balanced (M = 5.5 on a 1-10 left–right scale), and they have levels of trust of government that are slightly below the midpoint of the 0 to 100 scale and comparable with national statistics from the American National Election Studies (Gershtenson & Plane, 2007). Appendix B provides the precise statistics for the sample overall as well as across experimental factors.

Our experiment involved randomly varying the ACSI scores shown to respondents. The range of these scores was narrow and quite high overall (86-93) and, as mentioned, was based on the real range of ACSI scores for both agencies. The random variation used meant that not all the particular scores shown to individual participants were exactly as reported in the real ACSI. However, the variation from the actual ACSI scores was small and harmless, in terms of potentially misleading participants about the true performance of these agencies, particularly considering that the real ACSI scores likely have some measurement error as a gauge of the true satisfaction of customers of these agencies. The experimental variation of the source was consistent with actual practice, as the ACSI scores are in fact reported by both the agencies themselves and the ACSI organization.

Results

Figure 1 presents descriptive results for our 0 to 100 credibility index in the form of a graph of means by treatment group status. Overall, the perceived credibility is somewhat low, with all but one of the means falling below the midpoint of the 0 to 100 scale. Only the difference between agencies, CIS versus VA, is marginally significant (p < .10, two-tailed test), with CIS being judged to be somewhat less credible than VA. Although the difference is not significant statistically, the reported customer satisfaction information appears slightly less credible when framed as coming from the agency itself as opposed to the independent source ACSI. Figure 2 presents the means by treatment group for the question that asked respondents about their best guess as to the true level of customer satisfaction. As can be seen, in general, respondents thought the true level was lower, implying that the satisfaction index information presented to them overstated the true level of customer satisfaction. Although the mean for the less attractive CIS appears somewhat lower overall, neither the source nor the agency effects are significant statistically (even at the .10 level).

OPEN IN VIEWER

Figure 1.

Credibility scale (means) by treatment group.

OPEN IN VIEWER

Figure 2.

Best guess about true level (means) by treatment group.

To formally test the main effects, as well as to examine interactions, Table 1 presents the results of a regression analysis, including the agency factor (CIS or VA), the source factor (the agency itself or ACSI), and the randomized level of the satisfaction index. To make the ordinary least squares (OLS) regression coefficients equivalent to main effects, the agency factor and source factor were both effect coded as follows: agency (−.5 = VA, .5 = CIS) and source (−.5 = ACSI, .5 = the government agency itself). The satisfaction index level was dichotomized, as mentioned earlier, to capture left-digit bias and also effect coded (−.5 = 86-89, .5 = 90-93). Table 1 presents results for the two dependent variables, showing models both without and with the two-way interaction terms. Although substantively we are interested primarily in the source by level (left digit) interaction, the other two possible interactions (Agency × Source and Agency × Left digit) were also included in the model to be complete and to avoid bias. (The three-way interaction terms were not of substantive interest and not statistically significant in either model, so they were not included.)

OPEN IN VIEWER

Table 1.

Regression Analysis of Main and Interaction Effects.

	Credibility index (0 to 100)				Best guess true level (−3 to 3)
	Model 1		Model 2		Model 3		Model 4
	Coefficient	p value	Coefficient	p value	Coefficient	p value	Coefficient	p value
Source (government)	−2.384	.252	−2.446	.238	−0.057	.605	−0.058	.601
Agency (CIS)	−3.714	.076	−3.558	.088	−0.145	.191	−0.139	.209
Left digit (9)	0.077	.971	0.108	.959	−0.027	.805	−0.027	.809
Source × Agency	—	—	0.569	.891	—	—	0.081	.713
Source × Left digit	—	—	−12.845	.002	—	—	−0.440	.047
Agency × Left digit	—	—	−3.029	.467	—	—	−0.120	.589
Constant	47.338	—	47.361	—	−0.859	—	−0.858	—
R 2	.007		.024		.003		.011
Observation	589		589		587		587

Note. Table shows unstandardized coefficients from ordinary least squares regressions. CIS = Citizenship and Immigration Services.

Model 1 in Table 1 shows just the main effects on credibility. Paralleling the descriptive results from Figure 1, the source effect is negative as hypothesized (H1), but the effect is small and not statistically significant. Only the agency (CIS) effect is marginally significant (p < .10, two-tailed test), with CIS being viewed as somewhat less credible than the VA, as expected from our attractiveness hypothesis (H2). There is no evidence of a left-digit main effect. Model 2 adds the three two-way interactions, one of which is highly significant statistically: When the agency is reporting on itself—and the satisfaction index level is higher (left digit = 9)—respondents judged the information to be much less credible. This is the interaction of relevance to our interaction or positivity hypothesis (3), and the results confirm our expectation that citizens seem especially skeptical of government-provided information about good performance.

Model 3 in Table 1 shows just the main effects for our second dependent variable, regarding the respondents’ best guess about the true satisfaction level. Although the source and agency effects are in the hypothesized direction, neither is statistically significant, as mentioned earlier when presenting Figure 2. The left-digit main effect is again also not significant. Model 4 includes the interactions, and the results indicate that the interaction of source by left digit is negative and statistically significant (p < .05, two-tailed test). Consistent with our interaction or positivity hypothesis (H3), when an agency is reporting a higher satisfaction index for itself, respondents tend to believe that the real level of customer satisfaction is lower, implying that the agency is overstating its performance.

Figure 3 helps illustrate our finding of an interaction of source (the agency reporting on itself) and a higher level of reported satisfaction (left digit = 9), which again is the interaction of interest for our positivity hypothesis (H3). The figure is based on Model 2 in Table 1 and refers to the credibility scale. Interestingly, when the reported satisfaction index is relatively lower (left digit = 8), there is little difference in the credibility of the reported information, and indeed the credibility appears slightly higher even when the agency is reporting on itself. But when the reported satisfaction score is relatively higher (left digit = 9), there is a distinct drop in the credibility of the information when the agency itself is the source. The drop in credibility is about 9 points on the 0 to 100 credibility scale, which is fairly substantial given an overall mean level of about 47. The strength of this interaction effect is especially noteworthy given that the difference between “lower” and “higher” customer satisfaction was only the left digit of a reported index with a relatively narrow range (from 86 to 93). We discuss the implications of these findings in the next section.

OPEN IN VIEWER

Figure 3.

Interaction of source by score level (left digit 8 vs. left digit 9).

Discussion and Implications

Our experimental results provide, at best, only a weak indication of a source main effect, with respondents viewing the reported customer satisfaction as slightly less credible when the agency was reporting on itself, as compared with a more independent source (the ACSI), although the differences were not statistically significant. This lack of much difference between government versus non-government sources is consistent with the one previous experiment along similar lines (Van Ryzin & Lavena, 2013). Still, we designed our experiment to probe this previous finding more deeply by focusing on the federal government and using the ACSI, a reputable and independent firm, and thus expected to see more of a source main effect. One explanation for the lack of a large source main effect may be that, despite being a leader in its field, ACSI is perhaps not well known enough to have much of an independent reputation in the minds of most citizens (such as the well-known consumer rating firm J.D. Power, for example).

Despite finding only a weak main effect, however, our results show a very large interaction effect of source by level of reported performance. This suggests a different, more complex interpretation of the way in which citizens judge the credibility of government-reported performance. When the agency was reporting on itself—and when the level of reported customer satisfaction was higher (left digit = 9)—participants were much less likely to judge the information as credible, and they were more likely to think that the agency was overstating its performance. In contrast, when the reported level of customer satisfaction was relatively lower (left digit = 8), agency self-reporting was found to be more credible. This interaction effect was substantively large and has potentially important implications. For one thing, this interaction effect suggests a problem facing an agency that performs well and has good news to report, namely, that citizens may be especially distrustful of government-provided information precisely when an agency most expects to receive credit and earn public trust by reporting high performance. Paradoxically, elected officials and agency executives often strive to report their performance in highly positive terms, sometimes bending the results to present their efforts and accomplishments in the most positive light possible (Hood, 2006a). Indeed, it may well be the suspicion of such framing or gaming tactics that breeds distrust of glowing government performance information in the first place. Our results suggest that such practices, rather than helping government, may only erode the trust and credibility of the government performance movement more generally in the eyes of citizens.

An important implication of our interaction finding between the agency as a source and the level of performance is to call attention to the value of an outside or independent rating of government performance, especially when performance is high. A development of this line of research would be to consider alternative forms of involvement by bodies independent of the agency whose performance is being reported, for example, to consider a hybrid case where the agency presents the information but the information is independently certified or in other ways validated by an independent organization. In this way, the achievements of government may be made more credible to citizens.

Our experimental results provide partial support for our hypothesis about the attractiveness of a government agency and its influence on credibility, with the VA viewed as somewhat more credible than CIS. This finding is consistent with Hetherington’s (2005) research showing that Americans express less trust of government agencies that serve out-groups and require sacrifice. However, the agencies differ in many respects, not only their clients, so more research on other dimensions is merited, for example, the extent of redistribution involved in the agency’s programs, citizen knowledge about the agency, or the extent of media coverage of the agency. Veterans Affairs experienced a large amount of negative media coverage (relative to historical coverage) in the months around the time our study was conducted. Despite this caveat, the findings suggest that citizens’ trust of government performance information likely varies by the prestige and political popularity of the particular agency or arm of government doing the reporting. Less prestigious or less politically attractive agencies face an uphill struggle in their effort to demonstrate their efforts and accomplishments to the public.