Performance audit as a contributor to change and improvement in public administration

Abstract

The last 30 years of the New Public Management ‘regime’ in many western countries have resulted in increased use of audit and other control mechanisms. These mechanisms are supposed to contribute both to accountability and improvement. In this article, theories of evaluation and organizational learning are used to understand how the dynamics of control affect change processes. We analyse responses to performance audit reports carried out by the country’s Supreme Audit Institution – an important control institution in Norway, as in most other countries. The findings are primarily based on survey data from 353 Norwegian civil servants. The auditees reported that they were mostly positive towards the reports and used them to make improvements in their control systems; however, civil servants also perceived the reports to be unbalanced. The results indicate that the performance audits’ contributions to improvements are less clear than they appear to be at first glance, and that personal and political factors may affect civil servants’ responses.

Keywords

accountability control evaluation for accountability improvement organizational learning Supreme Audit Institutions

Introduction

Over the past 30 to 40 years, western societies have increasingly modernized in line with New Public Management (NPM) doctrines. The NPM ‘regime’ has involved increased delegation, which in turn has resulted in greater use of audit and control mechanisms. In his much-quoted book, The Audit Society, Power (1997) nevertheless claims that controls or audits are just rituals of verification with the purpose of assurance, but without real effect (see also Behn, 2001).

The impact of audits is a contested issue. Some researchers claim that control can work against improvement, whereas others argue that control enhances trust and performance (Furubo, 2011; Möllering, 2006). Trust reduces transaction costs and thereby increases efficiency and performance (Fukuyama, 2014). The actual outcomes are nevertheless likely to depend on the way controls are implemented (Ossege, 2012). In this article, we investigate what types of improvement, if any, performance audit reports produce in the audited organizations, as experienced by the civil servants involved.

The Supreme Audit Institution (SAI) is an external organization to audited entities. The article explores how external pressure – in this case from an SAI – can contribute to learning in an organization. This question has been widely discussed in the evaluation literature (Armytage, 2011). It is also relevant for studies on institutional entrepreneurship and change in public administration. Such studies explore what drives organizational change and how this occurs, with external pressure cited as one possible change trigger (Battilana et al., 2009; Oliver, 1991; Reichborn-Kjennerud, 2014a).

The article is structured as follows. First we present previous research. We explain the mandate of the Norwegian SAI, and the way the latter is organized. We then present our theoretical approach, followed by our data and research methods. Finally, empirical results are presented and discussed.

Research on the impact of performance auditing

There are few empirical studies focused on the actual outcomes of performance audits. In a review of existing research on the subject, only 14 studies pertaining to state audit were found (Van Loocke and Put, 2011). Like the present study, many of these studies are based on the perceptions of those audited. A recent study (Reichborn-Kjennerud, 2013a) found that performance audits were perceived as useful by the auditees in the Norwegian context. The author suggests that the extent to which the auditees considered the reports useful depended on their appraisal of the quality of the report and of the SAI as an institution. It also depended on perception of the extent to which auditees’ comments were taken into account during the process, and on the changes that actually ensued after the performance audit. In another article Reichborn-Kjennerud and Johnsen (2015) demonstrated how holding auditees to account contributed to actual changes in the audited entities. These results are not reflected in research on the Estonian SAI, however, where the auditees did not perceive performance audit reports as useful (Raudla et al., 2015). And Morin’s recent qualitative research similarly indicated that the impact of the French SAI is weak, its audits contributing more to window dressing than to real influence (Morin, 2016). In the Danish context, Justesen and Skærbek (2010) looked into how accountability mechanisms contributed to the impact of audits, suggesting that they represent forceful mechanisms that induce organizations to implement change. All in all, this indicates that the impact of SAIs is likely to vary according to context.

Morin (2004, 2008) has conducted several studies on performance audit, both qualitative and quantitative. From her quantitative research, she concludes that auditors have a certain impact, one aspect of this being that they influence the auditees’ management practices and their perceptions of usefulness. In later qualitative studies, she problematizes the double role played by auditors as both controllers and management consultants. She also discusses the challenges faced by audit institutions in adopting new evaluation methodologies in performance audits (Morin, 2010, 2016). In a qualitative study, Alwardat (2010) suggests that the utility of performance audits – as perceived by the auditees – depends on their expectations, their view of the auditors’ expertise, the materiality of the auditors’ findings, and the truth and fairness of the reports.

Several authors problematize the independent role of auditors as constituting an obstacle to impact. Being involved with the auditees, which would otherwise enhance impact, may interfere with the auditors’ independent role (Reichborn-Kjennerud and Johnsen, 2011; Vanlandingham, 2011).

Other contributions have explored questions of accountability and the reactions of auditees to audits (Reichborn-Kjennerud, 2013b, 2014a, 2014b). These studies suggest that performance audits have greatest impact:

when civil servants consider the reports to be of a high standard,

when they have some influence over the process, and

when they agree to the SAI’s approach.

Furthermore, auditees will tend to make concrete changes when they are held to account politically and when media and interest groups use the reports. However, knowledge about what types of change occur as a result of performance audits is lacking. More than two decades ago, Pollitt et al. (1999) mapped and analysed SAI performance audits in five different countries, but few studies have investigated the actual content of performance audits (Grönlund et al., 2011) and what contributions to change are instigated by the reports. The current article adds to this body of knowledge by examining the relationship between the content of performance audits and the types of changes occurring subsequently in audited entities.

The Supreme Audit Institution in Norway

The SAI is one of the most important control institutions in a country, investigating whether the public administration use their resources economically, efficiently and effectively. According to its own standards, an SAI’s primary task is control. A secondary task, however, is to contribute to improvements in the audited public organizations (Lonsdale and Bechberger, 2011). In their form and purpose, performance audits resemble accountability evaluations in that their purpose is to hold the public administration to account for their use of resources. Both evaluations and performance audits are conducted to assure the efficient and effective use of public funds (Pollitt et al., 1999).

Performance audits are carried out so that the Norwegian Parliament (Stortinget) can control the government administration. The ministries are obliged to report to Parliament how they are carrying out their responsibilities (Bemelmans-Videc et al., 2007: 241). The Parliament, in turn, can impose various sanctions on ministries, such as public hearings or specific demands for implementing measures. Members of Parliament can, for example, question the minister, conduct hearings or make plenary decisions that affect a ministry based on findings in the performance audit reports. The ministries are also answerable to Parliament for the actions of separate or semi-detached public agencies and for expenditures that are not under their direct control (Rose, 1987).

The SAI’s methods of control are: financial auditing, corporate control and performance auditing. Financial auditing involves verification of the accuracy of financial information. Corporate controls are carried out in order to assess whether the state’s interests in companies have been managed in accordance with the decisions and intentions of the Norwegian Parliament. The mandate for performance audit is to establish whether public policies, programmes, projects or organizations have operated with due regard to economy, efficiency, effectiveness and good management practice (Lonsdale, 2000; Pollitt et al., 1999). The guidelines for performance audit in the Norwegian SAI are based on the standards of the International Organisation of Supreme Audit Institutions (INTOSAI).

The Norwegian Office of the Auditor General (Riksrevisjonen) initiates most of its own audits, has independent status and reports the results of its auditing and monitoring activities to Parliament (NOAG, 2011). Ministries are invited to comment on the Office of the Auditor General’s findings and conclusions before a report is sent to Parliament. Subsequently, the final report, including comments by the relevant ministries, is submitted to a permanent supervisory committee of the Norwegian Parliament, the Standing Committee on Scrutiny and Constitutional Affairs (the control committee), which was established in 1993. This committee reviews reports and submits proposals to Parliament to decide whether the responsible ministries should be sanctioned.

The SAI’s logic is broadly based on an instrumental, purposive rationality. SAIs expect ministries to improve following detected deviance from audit criteria. Although this expectation can be questioned (Hollnagel et al., 2007), the urge to improve based on relevant information is an established assumption in classical management theory and in scientific management (Christensen et al., 2004; Morin, 2008; Taylor, 1967; Weber and Andreski, 1983). This assumption is also present in the performance control system in Norway, labelled ‘Management by Objectives and Results’, which was introduced on a general basis in national public administration from 2001 (Lægreid et al., 2006).

In this study, the Norwegian SAI is used as a case example of an SAI organized as a National Audit Office. This case primarily illustrates a Nordic European setting. Still, the actual impact that SAIs have is likely to differ according to institutional arrangements and cultures, and therefore has to be explored empirically in each case. Internationally the SAI’s impacts have been limited due to lack of resources, although Norway has been considerably less struck by the financial crisis than other European countries. This favorable economic situation gives more leeway for civil servants in Norway to follow up on the SAI’s reports.

Theoretical approach

Similarities and differences between audit and evaluation

SAIs are not organized in any direct hierarchical relationship to other public sector organizations, nor do they have formal sanctioning power. They do have indirect power, because Parliament may use the SAI’s assessments to sanction ministries (Bovens, 2007) (see examples above). SAIs that issue publicly available reports can also influence their environment through media attention paid to these reports. Potentially, interest groups and different types of Non-Governmental Organizations (NGOs) may also make use of the reports for advocacy purposes (Noussi, 2012). Nevertheless, in order to have an impact, predominantly SAIs, like other evaluation researchers and consultants, must convince auditees of the validity of their assessments.

One important goal for SAIs is to contribute to improvement, notably through performance audits. When SAIs identify deficiencies in the audited organizations, they expect that the auditees will use the relevant information to make improvements (Riksrevisjonen, 2005). This may also apply to programme evaluations when the aim is to hold the evaluated agency to account. A good example of this is the evaluation of development aid, where the aim is to investigate whether the funding has been used as intended (Armytage, 2011). Performance audit nevertheless differs from programme evaluation in that it is part of an institutionalized arrangement, and carries the latent threat that Parliament is entitled to use the report for accountability purposes.

Performance audit and organizational learning

Different types of control produce different types of organizational learning. In the rational instrumental perspective, two different types of learning can be identified: single- and double-loop learning (Argyris, 1999). In cases when an auditor’s logic focuses on compliance, this will lead to single-loop learning; whereas a logic focusing on explaining and understanding performance has the potential to lead to double-loop learning (Reichborn-Kjennerud and Vrangbæk, 2016). Single-loop learning involves repeated attempts to solve the same problem without any attempt to change the overall problem-solving systems or their goals, whereas double-loop learning helps to reset, change and redefine the systems and goals that define how problems are to be solved.

From a rational instrumental perspective, there is one aspect of organizational learning that has not been adequately addressed: namely, situations where there is a lack of will to implement recommended changes. This can occur if potential changes threaten particular groups’ interests, or if they work contrary to political agendas (Bringselius, 2013; Reichborn-Kjennerud, 2014a). In such cases, the audited entities can make symbolic changes instead of real changes that improve systems and practices. Even in cases where the SAI’s recommendations are highly relevant, there may be political reasons for auditees not to make real changes. We therefore expect disagreement with audit assessments to be an important reason why change does not occur (Brunsson and Olsen, 1997; March and Olsen, 1989); otherwise we would expect improvements. These insights about limitations to organizational learning come from the cultural-institutional perspective (Christensen et al., 2004), where learning is considered to be a socially constructed rather than technical process.

Learning can be both positive and negative. If there is a positive and significant correlation between the auditees’ perception of usefulness and their opinions about the report, the performance audit could lead to positive learning by making them reflect on their current practices. However, if the auditees’ assessments of the performance audit reports point to lack of compliance with these systems, single-loop learning may occur.

One branch of the organizational learning literature looks into what kinds of intervention are likely to be effective in enhancing organizational learning. Many scholars hold that changes in structures and rules in an organization can bring about desired changes in behaviour (Argyris, 1999: 34). Control systems can, on the other hand, also lead to dysfunctional behaviour that hinders improvement. Cyert and March (1963) showed how control systems led to conflicts between rule setters and rule followers. Internal power games, for example between leaders of different subunits, can undermine efforts to measure performance and improve organizational efficiency and effectiveness. Often, there are gaps between the intended use of systems to improve organizational performance – such as new formal structures, information systems and procedures – and their implementation and actual use (Argyris, 1999: 51).

The link between deliberation and action

Fundamental to theories of action are peoples’ reasoning processes. Communication and deliberation are links to action and can explain change dynamics (Schmidt, 2008).

The goal both for performance audit and programme evaluation is to provide objective information on the functioning of political programmes and institutions. Such information can be regarded as a kind of deliberation. Background ideational abilities guide actors’ thinking and change will not take place until these ideas are expressed (Argyris, 1999; Bourdieu, 1977; Searle, 1995). If auditees perceive performance audit reports as useful, they may be persuaded to change their systems and practices (Argyris, 1999: 67). Based on this, we would expect changes to occur if reports are persuasive to the auditees and if the auditees buy into the arguments made by the SAIs. The extent to which auditees will actually make the changes that they promise will nevertheless vary with context.

Audit frameworks and methods

Even though programme evaluation reports, as well as performance audit reports, differ in important ways in their institutional set-ups, they both face challenges in addressing funders’ (in this case Parliament’s) need for information about accountability. One challenge that is well documented in research on performance measurement has to do with problems of measurement (Crawford et al., 2004; Van Thiel and Leeuw, 2002). Another challenge is methodological, and is linked to Logical Frame Analysis (LFA) evaluations. LFA breaks down the logic of a project into a ‘chain of causality’. ‘Inputs’ are invested to facilitate certain predefined tasks in order to produce results. These results are, in turn, expected to have desirable ‘effects’ on society. Discussions of the LFA illustrate how difficult it is to identify causality between political interventions and the changes observed (Crawford et al., 2004). The rise of LFA in ex post evaluation can also be seen as part of ‘the audit explosion’ that prioritizes external checks using easily quantifiable measures (Gasper, 2000; Power, 1997: 288).

In stark contrast to the positivism of the LFA methodology, we find Guba and Lincoln’s (1989) constructivist ‘fourth generation evaluation’, also termed ‘action research’. This type of evaluation is participatory and involves the evaluated party in the evaluation process. The bodies under scrutiny are regarded as collaborators rather than subjects to be investigated. Assuming this role is difficult for SAIs because they must safeguard their independence and role as watchdogs of public resources (Reichborn-Kjennerud and Johnsen, 2011). Still, some SAIs have recognized the need to be more open and responsive and experiment with new methods (Van der Knaap, 2004).

When measuring results there is a risk that only what can easily be measured is measured and that what actually matters in the policy area is overlooked. Another challenge is the tendency of the LFA to develop into a ‘lock frame’, where evaluators are insensitive to change and stick to their initial model. Another problem linked to the LFA model is that it is preoccupied, as is performance audit, with checking whether predefined goals are attained – whereas the most important results were often not preplanned and therefore initially not formulated as goals or reported on during implementation. Because of these weaknesses, Gasper (2000) argues that the LFA model is more suited to planning and communication purposes than ex post evaluation. In his view, LFAs can lead to dangerous simplifications that can easily be misused and are liable to block rather than aid learning.

The official control system in the Norwegian State mentioned above – the Management by Objectives and Results system (MBOR) – is often the object of inquiry in the NOAG’s performance audits (Reichborn-Kjennerud, 2014b). Research has documented that such systems can be mere paper exercises, with plans and reports bearing little relationship to what is really going on. In that case they are largely irrelevant to management (Perrin, 1998; Van Thiel and Leeuw, 2002). A risk for the business of auditing will be that the reports primarily addressing MBOR systems are not actually meaningful for policy. This leads us to expect auditees to be critical of the SAI’s results, because the auditors’ methodological framework primarily concerns the control system (Reichborn-Kjennerud, 2014b). If the SAI undertakes assessments based on predefined goals and systems alone and shows little sensitivity to ongoing adaptations when measuring performance in audited bodies, reports are unlikely to be perceived as balanced and targeted at the important questions. This will reduce the potential for change and improvement.

Methods and data

This article has a theoretical point of departure. It investigates drivers of organizational change, more specifically how external pressure influences learning in organizations. Survey data to explore this was collected in 2011. A total of 520 questionnaires were distributed among civil servants in public institutions that had undergone performance audits by the Norwegian SAI between 2005 and 2010. We did not go further back in time because respondents could have difficulties recalling the performance audit. Many could also have changed their place of work. The institutions that received inquiries were asked to provide the email addresses of employees that had been involved in the performance audit processes. The employees identified in this way received an information letter and a web-based questionnaire.

After sorting out those who were unable to answer because of absence or sickness, or who were wrongly chosen for participation, 471 potential respondents were counted as valid. With a response rate of 74 per cent, the final number of respondents was 353. The majority of the civil servants who answered the questionnaire were middle managers (54%), followed by senior civil servants (35%) and top executives (11%). Some 60 per cent of the respondents worked in a ministry or an agency (approximately 30% in each). The rest worked in regional and local offices, government-owned corporations, counties and municipalities.

The respondents answered questions related to the particular performance audit(s) they had experienced themselves. The questionnaire contained questions both about the civil servants’ perceptions of performance audits, and their professional backgrounds. To avoid bias, the battery of questions on report quality contained both positive and negative statements. To clarify the argument when presenting the data, however, we have separated the positive and negative statements into two tables.

Civil servants’ answers to questions on which changes occurred after the performance audits were used as proxies for actual change. That method can be questioned, as it might be argued that the results merely reflect civil servants’ views on the issue. At the same time, asking experts about change in their organizations is regarded as a relevant and reliable method (Dorussen et al., 2005).

The questions were designed based on previous research (Morin, 2004; Pollitt et al., 1999; Power, 1997) and validated by research colleagues, ministry and agency employees as well as one Director General and two methodology experts from the Norwegian SAI. A pilot study was conducted with nine respondents before the survey was distributed.

In addition to the survey data, seven civil servants holding positions as administrative leaders, five senior advisors in the ministries, as well as 10 civil servants holding positions as administrative leaders in the audited bodies, were interviewed to learn what influence, if any, the reports had had on their organization. The interviews were read several times to grasp their meaning, and structured according to relevant themes (Thagaard, 2003).

Findings: Report assessments and auditees’ responses

First, we present what we learned from the performance audit reports concerning the type and extent of change they triggered. Here, we also include data on whether the respondents perceived these changes as improvements. We also elaborate on how the different conclusions of each report triggered particular types of change and corrections. Second, we link the auditees’ perceptions of the performance audit reports to their perceptions of the latter’s usefulness.

Type and extent of change triggered by the performance audit reports

Table 1 shows the respondents’ answers to the question about the extent to which the audited body made changes as a consequence of NOAG’s report, and the extent to which they used the report in improving operations in the audited entities.

Table 1.

Resulting changes and improvements.

	No change/change to a little or very little extent	Somewhat	To a large or very large extent	Total
The audited entity made changes as a consequence of the report	21%	30%	49%	100% (N = 290)
The report was used in improving operations in the audited entities	19%	27%	54%	100% (N = 267)

Note: the extent to which changes were made as a consequence of performance audit reports, and the extent to which these changes were regarded as improvements.

Table 1 shows that almost half of the respondents reported that the audited body had made changes to a large or very large extent as a consequence of the performance audit. A majority stated that the reports were used to improve operations in the audited entities.

The respondents who replied that no or very little change had occurred (7.6% of the 290 respondents) were asked to give reasons for the lack of change. Their answers are displayed in Figure 1. The respondents were given several alternatives to choose from. That is why the total in Figure 1 does not add up to 100 per cent.

Figure 1.

Reasons for ‘no or very little change’ responses.

As illustrated in Figure 1, the most common reason why no changes were made was that civil servants were aware of the challenges and had already put measures in place. Several of the interviewees explained that they often felt that the reports dealt with questions that were ‘old news’ to them. Around one third did not make changes because they did not feel responsible, and slightly more because they did not have the necessary instruments at their disposal. Between one third and one quarter did not make changes because they felt that the facts in the report were presented inaccurately. One quarter stated that they did not make changes because they disagreed with the SAI’s conclusions. Interviewees explained that if they fundamentally disagreed they stated their disagreement openly in their responses to the SAI. 269 respondents were asked about what type of changes they had made (the 7.6% from Figure 1 were deducted from the total number of respondents). 264 of the 269 respondents chose one or more of the alternatives presented in the questionnaire.

Table 2 displays the changes made by these 264 respondents and shows whether they considered them to be improvements or not. N in the third column in Table 2 represents the number of respondents who checked the given change alternative. The percentage in the second column is calculated as this N divided by 264 respondents. Accordingly, N in the fifth column is the number of respondents who responded to the question about whether improvements had ensued from the changes made. And the fourth column represents the percentage of respondents who perceive the given changes as improvements (‘to a large/very large extent’) divided by the N in the fifth column (the number of respondents who answered the question).

Table 2.

Type of changes and improvements after performance audit.

	What changes have taken place at your place of work since the performance audit?Per cent, total N = 264	N	Which of these changes did you see as improvements?Percentage ‘to a large/very large extent’ of all the respondents who answered this question (n divided by the total N for each question)	Total N for each question
Increased documentation and reporting	45%	121	63%	87
Changes in the strategies, planning or the Management by Objectives and Results system (MBOR)	42%	112	63%	88
Internal control and risk management	40%	107	64%	84
Changes in significant procedures pinpointed by the SAI	37%	99	65%	87
Changes in procedures for coordination	33%	89	64%	75
Changes in the training of the staff	20%	54	45%	39
Increased funding	13%	34	37%	30
Changes in management or organization	12%	32	25%	10

Notes: Percentage and number of civil servants who checked which type of changes took place at their place of work after the performance audit, and the percentage of those who considered these changes to be improvements.

The total sum does not add up to 100%, as several options were allowed.

As can be seen from Table 2, the changes that took place in the audited entities frequently entailed increased documentation and reporting. Sixty-three per cent of respondents regarded these changes as improvements to a large or very large extent. Changes in strategies, planning or changes in the MBOR system took place almost as frequently and were also regarded as improvements by the civil servants. The same tendency applied to changes in procedures. Changes in the training of the staff did not take place as often, but quite a few respondents considered this to constitute an improvement. As Table 2 demonstrates, few auditees had made changes in management and organization or received increased funding, and relatively few regarded such changes as improvements. All in all, Table 2 shows that the NOAG is primarily focused on planning and control systems in its performance audits.

Linking perceptions of usefulness to perceptions of the report

As we have described above, research confirms that audit results are more likely to be used if the auditees consider them useful. Perceiving the performance audit as useful could induce the respondents to think or act differently, possibly leading to positive long-term changes.

Table 3 shows that 60 per cent of the respondents thought the performance audit reports were useful to a large or very large extent. Only 16 per cent thought the reports were useful to a little or very little extent.

Table 3.

Perceptions of usefulness.

	To a little or very little extent	To neither a little nor a large extent	To a large or very large extent	Total
The report was useful	16% (46)	24% (75)	60% (188)	100% (309)

Notes: Prompt: To what extent did you perceive the performance audit as useful?

N in parentheses.

To grasp the reasons why the civil servants responded positively or negatively to the reports, we compared the 188 positive responses (respondents who replied useful to a large or very large extent) and the 46 negative responses (respondents who replied useful to a little or very little extent) on several dimensions. The middle group (neither a little nor a large extent) was excluded from this analysis. We chose to compare groups according to their perceptions of usefulness (and not based on background variables such as policy sector, administrative position or level) because of our quest for factors contributing to possible double-loop learning.

The differences between the two groups are displayed in Figure 2. The bars in the Figure represent the mean scores of the groups in the second and fourth column in Table 3. The vertical axis in Figure 2 displays the scale presented to the respondents in the questionnaire, and the horizontal axis displays questions asked/statements made concerning the performance audit report that they experienced.

Figure 2.

Auditees’ reactions to the reports (1).

Figure 2 shows that there are significant differences between respondents who thought the reports were useful and those who did not. The respondents who thought the reports were useful tended to agree with the audit criteria. They also tended to think that conflicting objectives in the policy sector had been taken into account, that the methods used observed high/scientific standards, and that there was a clear link between audit criteria, facts and assessments. They also tended to have an overall impression of the report as good, to state that adequate explanations were given, and to believe that the report was a key source of information in dealing with important policy areas. The group perceiving the reports as useful was significantly more positive towards all the statements about the reports than the group that had a mean score lower than the average value of 3. Values below 3 indicate a predominance of respondents who were indifferent or negative to the reports.

A quote from an interview we conducted with a director illustrates the significance of a good quality report in the eyes of the auditees:

If the work of the SAI is of good quality it is not a question of agreeing with it or not, because the report demonstrates that the findings are correct. Then it becomes more a question of whether the SAI’s assessments reinforce our own opinion of our challenges. The SAI seldom points to challenges that are completely unknown to us. They can increase or diminish the severity of a problem that we are already aware of.

There could be one source of error in regarding perceived quality as crucial. If the auditee finds the report useful for political reasons she/he could consciously or unconsciously assess the report positively as a result (Nisbett and Wilson, 1977).

Both groups from Table 3 – the group that perceived the reports as useful and the group that did not – were nevertheless somewhat sceptical towards certain aspects of the reports. This is illustrated by Figure 3.

Figure 3.

Auditees’ reactions to the reports (2).

Figure 3 demonstrates that the mean for both groups of respondents who assessed these claims was above the average value of 3. This indicates that many respondents think that the reports have oversimplified conclusions; that the auditors tend to pay too much attention to deviance regardless of its importance; and that they are too critical. The tendency remained significantly higher for respondents who did not consider the report useful, than for those who did consider it useful.

Discussion

Based on the findings presented above, we now turn to the discussion of the relationship between what civil servants are actually held to account for in the performance audit reports; how they responded to the assessments; and whether the reports led to changes. The discussion ends with reflections on the relationship between change, organizational learning and improvement.

Discussing the findings

The findings indicate that the Norwegian SAI primarily holds auditees to account for deficiencies in the control or management systems of the audited agencies (see Røvik, 2007 for a definition of control and management systems). The auditees responded by implementing the required changes. Their inclination to make changes as a consequence of the reports was substantial: a total of almost 80 per cent had made changes. Around half answered that they had made changes to a large or very large extent as a direct consequence of the report, whereas 30 per cent had made some changes. Many of the changes they had made involved increased documentation and reporting and had led to changes in internal control, risk management, strategies, planning or the MBOR system. This indicates that changes did indeed result from the performance audits, and that the auditees regarded these changes as improvements.

Perceptions of the usefulness of the reports were positively linked to perceptions of their quality. This indicates that organizational change is conditioned by the relevance and quality of such reports. The deliberations in the reports may, therefore, spur concrete actions, leading to long-term change, as suggested in the theory section (Schmidt, 2008). The fact that 54 per cent of the auditees considered that the reports led to improvements, and 27 per cent stated that they contributed somewhat to improvements (81% in total) also indicates that the SAI’s reasoning is mostly convincing to the civil servants. Civil servants tended to consider the reports to be useful and of good quality. The changes that the civil servants reported (49% considered that the reports had contributed to change to a large extent, and 30% somewhat) would not have taken place if this initial reasoning had not been convincing enough to win their acceptance. The most important reason civil servants gave for not making changes was that they had already dealt with the problems, or that fixing them was not actually their responsibility or within their power. Many also answered that they did not make changes because they disagreed with the SAI’s facts or assessments. This indicates that disagreements with the reports’ conclusions may explain why changes do not take place in some of the audited entities. It is important to keep in mind that auditees may, for political or practical reasons, have problems with the report’s recommendations even though they agree with the facts presented therein.

Theoretically, we also expected that auditees would be critical of the SAI’s results if they were oriented towards predefined goals and displayed little sensitivity to ongoing adaptations. This expectation was corroborated: considerable resistance to such reports is illustrated in Figure 2. Regardless of the auditees’ perceptions of the report’s usefulness, a majority perceived the conclusions as oversimplified and too critical. Also, when the auditees did not think the reports were good (on several indicators) they did not find them useful. This is in accordance with earlier research findings and demonstrates that there is a certain resistance to the SAI’s assessments. That resistance could potentially inhibit voluntary change and organizational learning.

This resistance to the reports may indicate that some of the SAI’s assessments are in conflict with fundamental internal organizational understandings. Resistance also demonstrates that there is a gap between what the Norwegian SAI considers it relevant to criticize, compared to the auditees. In the public debate, some resistance has been voiced to this effect. As with the critique directed at the LFA, civil servants have argued that the SAI’s assessments are too simplistic. The data used in this article was collected in the spring of 2011, but during summer and autumn 2012, a liberal think tank launched a critical debate of the Norwegian SAI (Kinander, 2012a). Former ministers and researchers subsequently criticized the SAI in national newspapers (Gjerde, 2012; Kinander, 2012a, 2012b; Reichborn-Kjennerud et al., 2012a, 2012b). The audited civil servants also questioned prevailing modes of accountability. Their resistance to the reports can be interpreted as a legitimate form of opposition to the SAI’s practices. Alternatively, the auditees’ reaction may also simply constitute a defensive reaction against the discomfort caused by the criticism.

A note on the relationship between change, organizational learning and improvement

Auditors use both a ‘goal attainment model’ in line with basic evaluation practice, and an ‘internal process model’ in line with the principle of compliance, as a basis for their assessments (Boyne, 2003; Reichborn-Kjennerud, 2014b; Reichborn-Kjennerud and Vrangbæk, 2016). In the goal attainment model, ‘improvement’ may be defined broadly as ‘producing the intended or expected result’. Universal criteria for improvement are, however, hard to define. It is well known that political goals are often ambiguous and unclear. Different groups, such as employees in agencies and ministries, or in different positions, differ in their view of what constitutes a good performance audit (Reichborn-Kjennerud, 2013b). Whether or not changes can be labelled improvements is therefore fundamentally a political question.

In ‘the internal process model’ the presumption is that particular sets of internal processes are strongly related to organizational performance. The challenge, however, is that antecedents to improvement can be confused with improvement itself (Boyne, 2003). The fact that auditees make changes to systems does not necessarily mean that the services in the audited entities actually improve. Findings in this study demonstrate that the SAI’s performance audit reports tend to encourage increased reporting and documentation. Documentation and reporting might increase transaction costs, which could be conceived of as negative in terms of service improvement. They may also be regarded as administrative procedures more or less decoupled from actual practice.

The auditees always have considerable leeway in deciding how to implement changes, even when they face sanctions by Parliament. The nature and extent of changes and improvements that auditees actually make following a performance audit are therefore challenging to gauge. The findings in this study should be read with this in mind.

Conclusion

This article has investigated the content of performance audits, and linked different types of content to the auditees’ reactions, their perceptions of the reports’ usefulness, and accounts of actual changes made in their organizations. Our contribution demonstrates empirically that the auditees actually take the results from performance audits into account (like other forms of evaluation which produce information that feeds into policymaking) and typically regard them as useful.

In the Norwegian SAI’s performance audit reports, civil servants are mainly held to account for deficiencies in steering, management, internal control systems and procedures. Most audit entities were positive and made changes after having experienced performance audits. The most common changes they made were to increase documentation and reporting and make changes to systems and procedures. At first glance, such measures might appear to increase red tape. But the auditees were positive about the auditors’ advice concerning control systems and procedures. Well-functioning systems are a prerequisite for an organization to perform well. The fact that the auditees were less positive about recommendations concerning staff training or management and organization issues suggests that they think the auditors are at their best when assessing control systems. At the same time, the auditees considered the SAI’s assessments to be too critical. Taken together, this suggests that the decisive factor affecting auditees’ perceptions of the SAI’s added value is not necessarily the target of the audit (control systems or organizational results); rather, it is the nature and value of the auditors’ contributions as judged by auditees. This interpretation substantiates prior qualitative research suggesting that it is the importance of the SAI’s findings that matters to the auditees (Reichborn-Kjennerud, 2014a).

The article contributes to the literature on the impact both of evaluation and performance audits as well as the literature on knowledge utilization and policymaking (Wittrock, 1982). We also see this study as contributing to the literature on institutional entrepreneurship and change in public administration, since we illustrate how external pressure can contribute to improving the governance systems of a public organization.

Footnotes

Acknowledgements

This study is part of a four-year comparative project on the influence of performance auditing on politics and public administration in the Nordic countries. We acknowledge comments from participants at the Norwegian National Conference on Political Science, Oslo, 5–7 January 2015, and from three anonymous reviewers.

Funding

The project is financed by a research grant from the Joint Committee for Nordic Research Councils in the Humanities and Social Sciences (NOS-HS), project number 219574.

Kristin Reichborn-Kjennerud is Senior Research Fellow in the Department of Public Management at Oslo and Akershus University College of Applied Sciences. She is also affiliated with the Norwegian Work Research Institute. Her research interests are performance audit in the public sector, organizational control and learning, and innovation and urban studies.

Signy Irene Vabo is Professor in Political Science at the University of Oslo, Department for Political Science. Her research interests are organizational studies, governance and local government.

References

Alwardat

(2010) External auditors and clients: An investigation of perceptions of value for money (VfM) audit practices in the UK public sector. PhD thesis, University of Westminster, Harrow Business School, UK.

Argyris

(1999) On Organizational Learning. Oxford: Blackwell Business.

Armytage

(2011) Evaluating aid: An adolescent domain of practice. Evaluation 17(3): 261–76.

Battilana

Leca

Boxenbaum

(2009) How actors change institutions: Towards a theory of institutional entrepreneurship. The Academy of Management Annals 3(1): 65–107.

Behn

(2001) Rethinking Democratic Accountability. Washington, DC: Brookings Institution Press.

Bemelmans-Videc

M-L

Lonsdale

Perrin

(2007) Making Accountability Work: Dilemmas for Evaluation and for Audit. New Brunswick, NJ: Transaction Publishers.

Bourdieu

(1977) Outline of a Theory of Practice (Vol. 16). Cambridge: Cambridge University Press.

Bovens

(2007) New forms of accountability and EU-governance. Comparative European Politics 5(1): 104–20.

Boyne

(2003) What is public service improvement? Public Administration 81(2): 211–27.

10.

Bringselius

(2013) Organisera oberoende granskning: Riksrevisionens första tio år. Lund: Studentlitteratur.

11.

Brunsson

Olsen

(1997) The Reforming Organization. Bergen: Fagbokforlaget.

12.

Christensen

Lægreid

Roness

(2004) Organisasjonsteori for offentlig sektor: instrument, kultur, myte. Oslo: Universitetsforlaget.

13.

Crawford

Perryman

Petocz

(2004) Synthetic indices: A method for evaluating aid project effectiveness. Evaluation 10(2): 175–92.

14.

Cyert

March

(1963) A Behavioral Theory of the Firm. Englewood Cliffs, NJ: Prentice-Hall.

15.

Dorussen

Lenz

Blavoukos

(2005) Assessing the reliability and validity of expert interviews. European Union Politics 6(3): 315–37.

16.

Fukuyama

(2014) Political Order and Political Decay. London: Profile Books.

17.

Furubo

(2011) Performance auditing: Audit or misnomer? In: Lonsdale

Wilkins

Ling

(eds) Performance Auditing: Contributing to Accountability in Democratic Government. Cheltenham: Edward Elgar Publishing, 22–50.

18.

Gasper

(2000) Evaluating the logical framework approach: Towards learning-oriented development evaluation. Public Administration and Development 20(1): 17–28.

19.

Gjerde

(2012) Tidligere utviklingsminister Erik Solheim (SV): Opptatt av småfeil. Aftenposten, 11 June.

20.

Grönlund

Svärdsten

Öhman

(2011) Value for money and the rule of law: The (new) performance audit in Sweden. International Journal of Public Sector Management 24(2): 107–21.

21.

Guba

Lincoln

(1989) Fourth Generation Evaluation. Newbury Park, CA: SAGE.

22.

Hollnagel

Woods

Leveson

(2007) Resilience Engineering: Concepts and Precepts. London: Ashgate Publishing.

23.

Justesen

Skærbek

(2010) Performance auditing and the narrating of a new auditee identity. Financial Accountability and Management 3(26): 325–43.

24.

Kinander

(2012a) Riksrevisjonen – vaktbikkje uten bånd? Civita-notat nr. 9/2012. Civita.

25.

Kinander

(2012b) Vaktbikkje uten bånd. Aftenposten, 12 June.

26.

Lonsdale

(2000) Developments in value-for-money audit methods: Impacts and implications. International Journal of Administrative sciences 66(1): 74–89.

27.

Lonsdale

Bechberger

(2011) Learning in an accountability setting. In: Lonsdale

Wilkins

Ling

(eds) Performance Auditing: Contributing to Accountability in Democratic Government. Cheltenham: Edward Elgar Publishing, 268–88.

28.

Lægreid

Roness

Rubecksen

(2006) Performance management in practice – the Norwegian way. Financial Accountability & Management 22(3): 251–70.

29.

March

Olsen

(1989) Rediscovering Institutions: The Organizational Basis of Politics. New York: Free Press.

30.

Morin

(2004) Measuring the impact of value-for-money audits: A model for surveying audited managers. Canadian Public Administration 47(2): 141–64.

31.

Morin

(2008) Auditors general’s universe revisited. Managerial Auditing Journal 23(7): 697–720.

32.

Morin

(2010) Welcome to the Court. International Review of Administrative Sciences 76(1): 25–46.

33.

Morin

(2016) Democratic accountability during performance audits under pressure: A recipe for institutional hypocrisy? Financial Accountability & Management 32(1): 104–24.

34.

Möllering

(2006) Trust: Reason, Routine, Reflexivity. Amsterdam: Elsevier.

35.

Nisbett

Wilson

(1977) The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology 35(4): 250–6.

36.

NOAG (2011) Office of the Auditor General – Performance audit methodology. Available at: http://www.riksrevisjonen.no/en/Methodology/Pages/Methodology.aspx (accessed 12 April 2012).

37.

Noussi

(2012) How Public Accountability is Institutionalized. The Case of External Public Auditing in Global Perspective. Applying a Mixed Methods Approach. Vienna: University of Vienna.

38.

Oliver

(1991) Strategic responses to institutional processes. Academy of Management Review 16(1): 145–79.

39.

Ossege

(2012) Accountability: Are we better off without it? Public Management Review 14(5): 585–607.

40.

Perrin

(1998) Effective use and misuse of performance measurement. American Journal of Evaluation 19(3): 367–79.

41.

Pollitt

Girre

Lonsdale

(1999) Performance or Compliance? Performance Audit and Public Management in Five Countries. Oxford: Oxford University Press.

42.

Power

(1997) The Audit Society: Rituals of Verification. Oxford: Oxford University Press.

43.

Raudla

Taro

Agu

(2015) The impact of performance audit on public sector organizations: The case of Estonia. Public Organization Review 16(2): 217–33.

44.

Reichborn-Kjennerud

Lægreid

Johnsen

(2012a) Detaljkontroll bremser nyskapning. Dagens Næringsliv, 6 December 2012.

45.

Reichborn-Kjennerud

Lægreid

Johnsen

(2012b) Er Riksrevisjonen nyttig? Klassekampen, 21 August.

46.

Reichborn-Kjennerud

(2013a) Political accountability and performance audit: The case of the Auditor General in Norway. Public Administration 91(3): 680–95.

47.

Reichborn-Kjennerud

(2013b) Resistance to control—Norwegian ministries’ and agencies’ reactions to performance audit. Public Organization Review 15(1): 17–32.

48.

Reichborn-Kjennerud

(2014a) Auditee strategies: An investigation of auditees’ reactions to the Norwegian State Audit Institution’s performance audits. International Journal of Public Administration 37(10): 685–94.

49.

Reichborn-Kjennerud

(2014b) Performance audit and the importance of the debate. Evaluation 20(3): 368–85.

50.

Reichborn-Kjennerud

Johnsen

(2011) Auditors’ understanding of evidence: A performance audit of an urban development programme. Evaluation 17(3): 217–31.

51.

Reichborn-Kjennerud

Johnsen

(2015) Performance audits and supreme audit institutions’ impact on public administration: The case of the office of the auditor general in Norway. Administration & Society 0095399715623315.

52.

Reichborn-Kjennerud

Vrangbæk

(2016) Hvordan kan forvaltningsrevisjon bidra til organisasjonslæring? – En komparativ studie av Norge og Danmarks riksrevisjoner innenfor arbeids- og helsefeltet. Nordiske Organisasjonsstudier 18(2): 98–121.

53.

Riksrevisjonen (2005) Retningslinjer for forvaltningsrevisjon. Oslo: Riksrevisjonen.

54.

Rose

(1987) Ministers and Ministries: A Functional Analysis. Oxford: Oxford Clarendon Press.

55.

Røvik

(2007) Trender og translasjoner: ideer som former det 21. århundrets organisasjon. Oslo: Universitetsforlaget.

56.

Schmidt

(2008) Discursive institutionalism: The explanatory power of ideas and discourse. Annual Review of Political Science 11(2008): 303–26.

57.

Searle

(1995) Construction of Social Reality. New York: Free Press.

58.

Taylor

(1967) The Principles of Scientific Management. New York: Norton.

59.

Thagaard

(2003) Systematikk og innlevelse: en innføring i kvalitativ metode. Bergen: Fagbokforlaget.

60.

Van der Knaap

(2004) Theory-based evaluation and learning: Possibilities and challenges. Evaluation 10(1): 16–34.

61.

Van Loocke

Put

(2011) The impact of performance audits: A review of the existing evidence. In: Lonsdale

Wilkins

Ling

(eds) Performance Auditing: Contributing to Accountability in Democratic Government. Cheltenham: Edward Elgar Publishing, 175–208.

62.

Van Thiel

Leeuw

(2002) The performance paradox in the public sector. Public Performance & Management Review 25(3): 267–81.

63.

Vanlandingham

(2011) Escaping the dusty shelf: Legislative evaluation offices’ efforts to promote utilization. American Journal of Evaluation 32(1): 85–97.

64.

Weber

Andreski

(1983) Utvalg. London: Allen & Unwin.

65.

Wittrock

(1982) Social knowledge, public policy and social betterment: A review of current research on knowledge utilization in policy making. European Journal of Political Research 10(1): 83–9.