The User Perspective in Performance Auditing—A Case Study of Norway

How Does Performance Auditing Relate to the Evaluation Field?

Performance auditing emerged as a distinct activity within supreme audit institutions (SAIs) in the late 1970s and 1980s (Pollitt & Summa, 1999, p. 4). Today, performance auditing is part of the legal mandate of at least 164 SAIs. However, there is a potential for growth, as 124 of the SAIs have expressed a need for capacity development support in performance auditing (INTOSAI Development Initiative, 2010, Annexes E and H).

A SAI is a national agency responsible for auditing and monitoring government revenue and spending. The SAIs’ overall objective is to promote good governance and effective public services to the users (INTOSAI, 2004, p. 33). For instance, in the United States, the role of the Government Accountability Office (GAO) is to provide the Congress with independent and objective information on government operations and undertakings. Equivalent to the role of SAIs worldwide, GAO and other SAIs serve the needs of legislatures to have government-wide analyses.

Almost all the countries in the world have a SAI; the INTOSAI Development Initiative (2010, p. 14) has put the total number of SAIs in the world at 204, though this includes some institutions that are probably not operational at the moment. The SAI is explicitly mentioned in the constitution in at least 145 countries (ibid., Annex E). Yet, the legal mandate and the functions of the audit institutions vary across countries, and the literature on SAIs generally distinguishes between at least three different models: the Westminster model, the board model, and the judicial model (Blume & Voigt, 2007, pp. 2–3; Dye & Stapenhurst, 1998, pp. 5–6; INTOSAI Development Initiative, 2010, Annex E; World Bank, 2001).

In the Westminster model, the SAI, headed by an Auditor General, submits periodic reports to the legislature on financial statements and the operations of government entities. In this system, there is less emphasis on compliance with laws and regulations and more focus on financial aspects. The Westminster model is by far the most common model, and is found in many European and Commonwealth countries. The model owes its name to the seat of the Parliament of the United Kingdom. In the political science literature, the term Westminster model is commonly used to describe a parliamentary system of government with a strong executive power in single-party majority cabinets, a majoritarian voting system, a unitary state structure, and weak or nonexistent judicial review by the courts (Lijphart, 1999, pp. 9–21). Though the two concepts are closely associated, the Westminster model of SAIs is also found in countries that have not adopted a Westminster model of government. The U.S. GAO, for example, is a Westminster-type audit institution, even though the system of government in the United States does not follow the Westminster model.

The board model is similar to the Westminster model, but the SAI is headed by a board rather than by a single person, and the audit reports are sometimes sent to the cabinet, which in turn submits them to the legislature. The board model is mainly found in Europe, Asia, Latin America, and the Caribbean. In the judicial model, common in the Latin countries of Europe and the francophone African countries, the SAI is part of the judiciary. The SAI is thus independent from both the legislature and the executive, and makes legal judgments on compliance with laws and regulations (Blume & Voigt, 2007, pp. 2–3; Dye & Stapenhurst, 1998, pp. 5–6; World Bank, 2001). However, many SAIs cannot be placed unequivocally in any of these categories (INTOSAI Development Initiative, 2010, Annex E). The SAI of Norway, for example, follows the board model, even though the Parliament is the main recipient of the reports.

A SAI may perform three basic types of audits: (a) financial audits, where the auditor assesses the accuracy and fairness of financial statements; (b) compliance audits, where the auditor checks whether government revenue and spending have been used for approved purposes; and (c) performance audits, where the auditor typically seeks to determine whether the society and the users get value for the money the government spends (Dye & Stapenhurst, 1998, pp. 15–16; World Bank, 2001). A central feature of performance auditing is the aim of leading to improvements in government operations typically by examining the economy, efficiency, and effectiveness of government undertakings (INTOSAI, 2004, p. 11, 2010a, p. 2, 2010b, p. 4). The aspect of economy concerns keeping costs low. It requires that the resources used by the audited entity for its activities shall be made available in due time, in appropriate quantity and quality and at the best price. The aspect of efficiency addresses the direct products and services delivered (outputs) by a government agency. It is concerned with the best relationship between resources employed, conditions given, and the outputs. The aspect of effectiveness is about meeting a set of objectives. It is concerned with achieving the intended results of the products and services (outcomes).

The roots of performance auditing lie in the parliaments’ need for independent, wide-ranging analyses of the economy, efficiency, and effectiveness of government programs and agencies. Performance auditing has a normative basis: any government undertaking should be assessed with reference to the parliament’s decisions and intentions, or the stated objectives and targets for programs and other interventions. For example, the legislature may set standards for certain areas of health care. Such standards will guide both the users’ expectations and how the performance of these services shall be measured and assessed.

Program evaluation analyzes to what extent intended results are achieved and assesses specific causal contributions of activities to results (Morra-Imas & Rist, 2009, p. 108). According to GAO, program evaluations are systematic studies that assess how well a program is working, both in terms of process and outcome (GAO, 2009, p. 3).

There has been some debate on the relationship between performance auditing and evaluation (see, for example, Barzelay, 1996; Brooks, 1996; Chelimsky, 1985, 1996; Leeuw, 1996; Ling, 2007; Pollitt & Summa, 1996). Most agree that the two traditions share many of the same features, and much of the debate centers on determining exactly how similar they are and what, if anything, separates them. While some argue that performance audit and evaluation are distinct but complementary functions, others maintain that the difference between the two lies in the terms that are being used and not in the content. Barzelay (1996, p. 19), for example, states that “[p]erformance auditing is an evaluative activity of a particular sort whose name happens to include the word ‘auditing.’”

The close relationship between program evaluation and performance audit may be illustrated by considering the legislative evaluation offices in American states. Over the past decades, legislative evaluation offices have been established in most states in order to contribute to the legislatures’ oversight of the executive branch. The offices vary greatly in size and organizational setting; some of them are part of the legislative auditor’s office, others operate as independent legislative units, and yet others are established within a legislative oversight or other committee, or as a joint legislative and citizens’ commission (Brooks, 1996, pp. 18–20; Jonas, 1999, pp. 3–4; NCSL, 2004, p. 2). The offices also produce a variety of research products, and almost all of them conduct evaluations or performance audits of state agencies and programs as one of their primary functions (Brooks, 1996, pp. 18–20; Jonas, 1999, pp. 3–4; NCSL, 2004, pp. 2–3; VanLandingham, 2006, 2011). Furthermore, both program evaluations and performance audits operate in a context where the use and influence of performance measurement is increasing. Program evaluation usually adds a value judgment about the worth of a given program (Mohan, Tikoo, Capela, & Bernstein, 2006, p. 89). In a similar way, performance audits also aim to evaluate the long-term effect of the program. One of the main differences seems to be the legal mandate of the institution that performs the study: performance audits are conducted by SAIs with the legislatures as main recipients, while evaluations, tailored to address the clients’ research questions, are carried out by various organizations.

Why is the User Perspective Important?

Over the years, both SAIs and government agencies have become increasingly oriented toward public performance (Pollitt & Summa, 1999, pp. 9–10; Summa, 1999, p. 11). In Norway, the Parliament has stressed that performance audits should be results oriented, and that the audits should examine whether government agencies meet their objectives to the users’ satisfaction. The Parliament maintained that user surveys could be a suitable approach in audits where user satisfaction is a central question, and asked the SAI to increase its focus on the user perspective (Norwegian Parliament, 2003, p. 5). Members of the Norwegian Parliament have commented that the users’ perspectives help in showing the extent of the problems in government services, and how to improve the performance (Norwegian Parliament, 2009b, pp. 790–791).

The increased focus on the user perspective in performance audits can also be seen as a consequence of two trends in public management. First, public management systems are to a greater extent accounting for the outputs and outcomes of government programs and services, rather than focusing only on inputs. The reform elements relevant to the user perspective in performance auditing include the elaboration of performance indicators, service declarations, and user involvement in public services (Pollitt, 1999, pp. 37–39). These changes are vital to the development of performance auditing both in Norway and other countries.

The outcome orientation in the public sector is in part associated with a second trend in public management: the tendency to see the users’ needs and preferences as the guiding principle for developing public services. During the 1990s, the governments in several countries emphasized that government agencies should provide the users with the services they need and request, and that the government should keep track of its performance through user feedback. Being attentive to user needs is to some extent seen as an end in itself—aligning public services with the users’ preferences is a way of giving the users a say in matters that affect them. In Norway, for example, various governments have emphasized the need to move from a top-down perspective, where government agencies are mainly concerned with implementing policy directives from the ministry, to a bottom-up perspective focusing on the users’ preferences (e.g., Ministry of Children & Family Affairs, 1999, Chap. 3.3; Ministry of Government Administration & Reform, 2007, p. 8; Ministry of Planning & Coordination, 1997). As a result of the latest of these initiatives, the Norwegian Government has conducted an extensive survey of the citizens’ and users’ perceptions of the availability and quality of a wide range of public services (Agency for Public Management & eGovernment, 2010). To be sure, some politicians may have overstated the novelty of such programs, but it seems clear that government agencies have become more concerned with their users’ opinions than they used to be.

The users’ satisfaction can also be a useful key performance indicator of the outcome of public services. Knowledge about people’s opinions and preferences can hence be applied to improve public services, as government programs can be redesigned to better meet the users’ needs. In this perspective user satisfaction can be an indicator of effectiveness, and thereby provide a more precise measure of what works and how public programs can be improved, rather than merely measuring efficiency (see also Figure 1).

OPEN IN VIEWER

Figure 1.

The results chain: efficiency and effectiveness. Modified version of figure in Government Agency for Financial Management (2005, p. 17).

How can the User Perspective add Value in Performance Audits?

Focusing on the user perspective may provide unique knowledge about how government programs work and how societal problems can be addressed. We argue that this kind of knowledge can make important contributions to evidence-based policy making, an approach that has become increasingly prominent over the past decades (Head, 2009, pp. 14–16). For some time now, there has been a debate among the advocates of evidence-based policy making over how rigorous the methods need to be: some have argued strongly in favor of sophisticated methods such as randomized control trials, while others have championed the use of more qualitative methods that account for the users’ experiences. However, recently the two sides have started to converge, and many now tend to use mixed methods when appropriate (ibid., p. 17). In the following, we will provide examples from Norwegian performance audits that have incorporated a user perspective to inform decision-making processes, and how the information has been used to improve public services. The examples will illustrate that it is possible to obtain useful performance data about government programs through fairly simple methods.

Disabled People as Users

At the SAI of Norway, it has become increasingly common to convene focus groups of users in order to obtain the users’ views. An example of this method is found in a report on accessibility for the disabled. The SAI of Norway published a report examining the accessibility to public buildings, transport, and open areas for people with various forms of disabilities (SAI of Norway, 2009b). One of the methods used to examine this question was to convene focus groups of participants with disabilities. The participants were asked to describe their experiences concerning accessibility in general, and access to public buildings, transport, and open areas in particular. In this way, we were able to get in-depth information on the many obstructions that people with disabilities meet in their daily lives. Although the limited number of participants meant that it was not possible to draw general conclusions based on the focus groups, they provided in-depth information that would have been difficult to get through other methods, such as surveys.

In the same report, we also used direct observations as a method to incorporate the user perspective. This is a less widely used method at the SAI of Norway, but it turned out to be a very effective way of collecting audit information. In our report on accessibility for the disabled, the observations were carried out by performance auditors along with representatives of various user groups. The auditors and the users tested how accessible public buildings, transport, and open areas are to people with different types of disabilities. Our experience suggests that it is particularly constructive to involve the users in the observations.

The observations were conducted in four different cities, and the barriers to accessibility (physical and other) were recorded by the auditors using field notes, photos, and film recordings. This approach gave the auditors first-hand knowledge about the barriers that people with disabilities face. In addition, photos from the observations were used in the report to illustrate different types of barriers, thus making it easier for the reader to get an understanding of the nature of the problem. Below are two of the photos from the report (see Figures 2 and 3).

OPEN IN VIEWER

Figure 2.

Photo showing a truck (with the loading ramp down) parked above the tactile paving used to assist blind and partially sighted pedestrians (SAI of Norway, 2009b, p. 38). Photo: SAI of Norway.

OPEN IN VIEWER

Figure 3.

Photo showing the difficulties faced by wheel chair users at a bus stop (SAI of Norway, 2009b, p. 32). Photo: SAI of Norway.

The publication of the report coincided notably with another event, as the responsible ministry presented an action plan on how to meet these challenges the same week. Moreover, in the parliamentary debate on the report, an MP stated that it is “totally unacceptable” that no buildings actually fulfilled all the requirements for access for the disabled. Most importantly, the MP stated, the report shows the extent of the problem, how to address it, and that it is important that the various user groups are heard (Norwegian Parliament, 2009b). The Auditor General of Norway noted that the challenges and the situation for the disabled was an area of such importance that it should be followed up on a regular basis by the SAI of Norway. The impact of the report so far has been placing the topic more firmly on the agenda in the societal and political decision-making processes. The challenges the disabled face require fundamental changes in social and physical infrastructure that necessarily will take sustained attention, efforts, and time to implement.

How can the User Perspective in Performance Auditing be Applied to Improve Public Services?

The Norwegian Parliament is increasingly concerned with the user perspective. Our performance audits offer new knowledge about the users’ needs and points of view to both the Parliament and the government. This provides opportunities for change and improvement in public services and management. However, since the SAI of Norwegian does not have any enforcement authority and rarely gives specific recommendations, the SAI’s contribution to improved services is indirect. The SAI’s role in the process of transforming user views into policies can therefore be illustrated by a modified version of Easton’s basic model of political systems (see Figure 4).

OPEN IN VIEWER

Figure 4.

Model showing how feedback from the users is aggregated and communicated to Parliament by the SAI. Modified version of Easton’s basic model of political systems. Easton’s model is reproduced in Birch (2001, p. 237).

The SAI of Norway collects feedback from the users and analyzes the views of several user groups. The Parliament then considers the report and conveys their conclusions to the government. Following publication of the performance audit reports and the Parliament’s considerations, the government is required to respond to the SAI about the implementation of measures needed to improve public services. The SAI thus provides users with a voice and access to the Parliament, and can thereby indirectly contribute to improved public services for the users.

In many other countries, the SAI can influence the parliament and the government even more directly. In a survey recently conducted by the SAI of Norway (see Appendix for details), many SAIs report that they give recommendations to policy makers on the basis of the findings in their reports. Some SAIs even have enforcement authority in some or all policy areas (see Table 1 in the Appendix). Furthermore, several of the SAIs that report that they give recommendations and/or have enforcement authority, also report that they ask the users for their opinions on public services. Accordingly, many SAIs have a unique opportunity to provide the users with a voice in the public sphere. The U.S. GAO, for example, makes explicit recommendations in the performance audit reports and follows up on their implementation. Although GAO does not have any enforcement authority, its reports include recommendations on program improvement and policy options. The SAI of Norway, on the other hand, generally provides assessments rather than recommendations in the audit reports.

However, the SAIs that have adopted a user perspective in their audits have taken on a task that carries important responsibilities. The requirement of independence and objectivity is emphasized by both auditors and evaluators, and both the Government Auditing Standards of the United States (GAO, 2007) and the program evaluation standards recognize the challenge of balancing the needs of different stakeholders (Bernstein, Whitsett, & Mohan, 2002, p. 96; GAO, 2007, pp. 24–27). For instance, the U.S. GAO’s independent role is crucial when providing advice to the Congress on the effectiveness of government operations. In order to safeguard the audit institution’s independence, it is necessary to maintain objectivity and continuously assess the relationship with stakeholders (GAO, 2007, p. 27).

The scholarly literature has emphasized the importance of involving central stakeholders (VanLandingham, 2011, p. 86), but this also means that the SAIs must consider carefully who these stakeholders are. The primary user of a SAI is often committees of parliament, but the media, academia, professional bodies, and individual users should also be considered as central stakeholders (Sloan, 1996, pp. 141–142). These deliberations are even more important when considering the risk that some vulnerable groups go unnoticed also by the SAIs. Resourceful groups may be better placed to influence the public debate (Klausen & Sweeting, 2003, p. 9; Lijphart, 1999, pp. 175–180), and there is a danger that the SAI may overlook marginalized groups or run the risk of being seen as a mouthpiece for various interest groups, thus infringing their reputation as objective and impartial. This is the same challenge evaluators face when operating in a complex network of evaluation sponsors and stakeholders (Bernstein et al., 2002, pp. 95–96). But gathering and analyzing the subjective views of the users does not mean that the auditors must convey these views uncritically. It is therefore important that a broad range of user groups are issued with a voice and that their views are covered in the reports. Thus both the subject areas and the normative basis of the audit or evaluation are important.

Obviously, the auditors must apply information gathered from the users systematically and analytically, as they would with any other kind of data. Furthermore, user-centered approaches should always be used together with other methods according to the principle of triangulation of methods. Still, methods incorporating the user perspective can, when applied in combination with traditional performance audit approaches, provide performance auditors with valuable information on the quality of public services.

Moreover, users may know what’s wrong, but they may not always know what the best solution to the problem is. An example from the Norwegian health sector may illustrate this. Patients are now discharged from hospital far earlier after treatment than they used to be, in part for medical reasons: a longer stay in hospital may increase the risk of hospital-acquired infections (SAI of Norway, 2003, p. 17). Yet some patients express concern that they are being sent home too early. Hence examining the users’ satisfaction with hospital services might suggest that users are unhappy with the current discharge practices, and that a solution to the problem is to allow patients to stay longer in hospital after treatment. But in many cases this would not be in the patients’ best interest, and finding the best solution to the problem might involve using other methods. For example, a report from the SAI of Norway (2003, p. 24) used data from administrative records, and found that patients from counties with a comprehensive provision of rehabilitation services in nursing institutions were, on average, discharged from hospital sooner than patients from counties with poorer rehabilitation services. A possible solution to the problem could therefore be to provide better rehabilitation services, instead of letting patients stay longer in hospital before being discharged.

Another experience is that a high score on user satisfaction does not ensure that the quality of the service is good enough. A good example is the Norwegian Tax Administration’s efforts to simplify the process of determining and filing one’s tax return. Some years ago, this was considered a complicated task and many people spent long nights trying to figure out how to fill in the tax form. Being sensitive to the users’ needs, the government decided to simplify the process, and throughout the 1990s several improvements were made (Ministry of Finance & Customs, 1998, Chap. 21.2). Today the Tax Administration collects all the relevant information from various sources, and sends out a precompleted tax form. Most people file their tax return via the Internet or by text message. The Tax Administration’s user surveys show that people are more satisfied with the service today than they were in the late 1990s (Norwegian Tax Administration, 2007, p. 44). A recent SAI report, however, shows that the quality of the precompleted information in the tax form is not always satisfactory and that it is difficult for the individual taxpayer to find out if the data is correct (SAI of Norway, 2010b). Thus, user satisfaction may be one, but not the only, key performance indicator of the outcome of public services.

The constitutional role and mandate provide the SAIs with access to relevant information and independent means of reporting, and ensures that the reports are given proper consideration. The SAIs have a unique opportunity to improve public services to the benefit of the users, since it brings institutional power to the recommendations. However, there is a risk that focusing on the user perspective may be seen as advocating higher standards and increased public expenses as the only way of improving public services. This applies to both evaluations and performance audits.

Conclusion

According to the principle of participatory democracy, the people’s direct influence on political decisions is viewed as the foundation of democracy. The constituents’ power is associated with the right to participate in or influence political decisions. Democracy is thus realized when everybody is able to use this right (Bergh, 1983, pp. 7–8; Dahl, 1989, p. 109). The users’ preferences have become an important source of legitimacy in modern democracies: Nonresponsiveness to user needs is not acceptable in politics, nor is it compatible with accountable government. The SAI of Norway’s emphasis on the user perspective is accepted and expected by the government, the Parliament, and the users. The performance audits can therefore be seen as an institutional response to a general trend in society.

The role of the SAI is to identify obstacles to good public performance, and to communicate the findings to the parliament on a regular basis. The aim is both to support the parliament’s control and monitoring function, and to provide it with information that can be used for decision making and legislative purposes. The SAI thus seeks to present audit evidence and well-founded conclusions that may contribute to the parliament’s considerations of change and improvement in public services. Eventually, employing the user perspective can become a means to broaden the scope of participatory democracy.