Teaching integrity in the public sector: Evaluating and reporting anti-corruption commissions’ education function

Abstract

Australia’s state-based public sector standing anti-corruption commissions each have an education function. The function is underpinned by a largely unquestioned assumption that integrity education will reduce levels of misconduct and corruption in the public sector. This study explores how these commissions undertake evaluations and publicly report on the effectiveness of their education events to demonstrate a contribution to this purpose. It acknowledges that well-designed and executed education events may deliver public value outcomes, but finds that the evaluative and reporting approaches used by commissions are insufficient to reliably demonstrate whether or not the function actually achieves this purpose. The dominance of exposure-as-success attendance statistics revealed in this study is particularly concerning as they offer no insights that are directly relevant to this purpose. Consistent with the findings of previous studies, there is also a heavy reliance on reaction-based satisfaction surveys that primarily reflect participants’ enjoyment of an education event, but do not provide meaningful evidence of learning, application of learning, organizational impact, or the overall delivery of public value. As such, it is argued that commissions need to transition from largely superficial appraisals to more robust and strategic assessments that measure actual attitudinal and behavioral change in the public sector. This study also raises questions of broader relevance to the evaluation of other types of workplace education, particularly that requiring affective and behavioral learning.

Keywords

Standing anti-corruption commissions integrity education evaluation attitude behavior

Introduction

Integrity in public administration is increasingly in the spotlight. Broadly, behaving with integrity encompasses three aspects—moral awareness, reasoning, and behavior (Van Montfort et al., 2013: 118)—but the application of standards remains contentious, while precisely measuring the dimensions of problems involving contrary behavior is difficult (see, for example, Cochrane, 2018; Sampford et al., 2006). Nevertheless, with evidence of declining public trust in government institutions (see, for example, Appleby et al., 2016; Hetherington and Rudolph, 2008), the demands on the public sector to meet the highest possible integrity standards have never been greater. Australia’s six state-based public sector standing anti-corruption commissions each have an education function, which is underpinned by a largely unquestioned assumption that integrity education will improve attitudes and behavior and reduce levels of misconduct and corruption. Such an assumption is not unique to integrity education. Referring to workplace education in general, Phillips and colleagues (2007: 4) note that “payoffs are assumed to exist and learning programs appear to be necessary,” but “frustration comes from the lack of evidence to show that the process is really working” (see also Brinkerhoff, 2006).

However, more recently academic studies have begun to examine this assumption (see, for example, Steele et al., 2016) with the findings of some studies raising concerns due to their mixed results. For example, a study of two Dutch municipal integrity training programs demonstrated a positive short-term impact resulting from one program but not the other, while neither produced longer-term benefits (Van Montfort et al., 2013). Similarly, a study of a public ethics and integrity training program for civil servants in Hungary produced moderate outcomes overall, but about a quarter of participants declined in their commitment to fighting corruption during the one- or three-day training courses (Pallai and Gregor, 2016). In Australia, understanding whether outcomes match the assumption is important for ongoing policy debates about the functions and objectives of commissions.

In public administration integrity is approached in two ways: a rules-based compliance approach to prevent or punish negative behaviors, and a values-based ethics approach to promote positive behaviors. Sometimes agencies may have a strong bias toward one or the other approach, but elements of both are usually present. The education function accords with both approaches, encompassing all aspects of Piper’s (1993: 10–11) “educational trilogy” for ethics teaching: knowledge, attitudes, and skills. As Piper (1993: 10–11) argues, “each is important; each is insufficient in isolation.” Indeed, teaching for integrity is set apart from most other workplace education by its greater emphasis on impacting attitudes and behavior vis-a-vis knowledge and skills. Although one requires knowledge of one’s duties in order to enact them, this knowledge is only information—not integrity—unless, or until, it is acted upon in the appropriate manner and according to choice (Griffin, 2012: 332). As Griffin (2012: 338) asks rhetorically: “What does it matter if [people] know how to reason well in ethics if they are not motivated to act ethically?”

This dilemma makes integrity education particularly challenging to implement successfully and to evaluate to ensure it achieves its purpose. Integrity requires knowledge of requirements and expectations; an affective disposition, particularly attitude, conducive to meeting requirements and expectations, skills to fulfil requirements and expectations, and, consequently, behavior according to requirements and expectations. Education for these ends is influenced by three main variables: the characteristics of the participants, the organizational or situational context, and the characteristics of the education program (Van Montfort et al., 2013: 119; see also Leberman et al., 2006: 99–108; Smith and Sadler-Smith, 2006). As such, short ad hoc exposures to information may create superficial awareness, but are unlikely to produce measurable changes in affective attitudes and behaviors (Krathwohl et al., 1964: 80; see also Callerstig, 2016: 119–120). The process of achieving highly internalized integrity learning objectives takes time and requires repetition across a range of content areas that together form a “consistent and complex set” of attitudes and behaviors (Krathwohl et al., 1964: 80).¹

Acknowledging that it is possible for well-designed and executed integrity education to contribute to successful outcomes, this study examines the extent to which the six commissions demonstrate that their education function achieves its purpose. It will present the findings of an exploratory survey of how commissions evaluate and publicly report on their education events. It begins with a review of the academic literature on education evaluations to develop a typology to describe their scope. These categories are then used to undertake the exploratory survey. A discussion of the findings follows, with the shortcomings of the approaches used identified and a recommendation made for a greater focus on evaluations that produce meaningful results. It aims to stimulate further debate and discussion of a significant function of commissions that has long been overshadowed by their more high-profile investigative role.

Evaluating education events

Evaluating the performance of a function should be an essential stage of any policy cycle and is crucial to informing decision-making, monitoring progress, justifying budgets, and improving design and implementation (see Althaus et al., 2013; Bardach, 2012; Kirkpatrick and Kirkpatrick, 2006: 16–17). Table 1 draws on a review of the academic literature to present a typology of approaches to evaluating and reporting the effectiveness of education events conducted as part of the education function. It is important to note that each approach has benefits and shortcomings in evaluations. However, in this study these approaches inform the development of a retrospective classification scheme to categorize and understand the scope of evaluations undertaken and publicly reported by commissions; that is, it is not used to evaluate the effectiveness of the education function itself. To this end, the various approaches offer a useful “vocabulary” (Alliger and Janak, 1989: 331) in seeking a deeper understanding of how commissions evaluate their education events.

Table 1.

Categorizing types of evaluation approaches.

What is measured	Source	Description
Exposure	Observed in this study	Compiling statistics, such as the number of events and participants, to indicate exposure to content
Reactions	Kirkpatrick / Phillips	Methods, such as satisfaction surveys, that measure participants’ reactions, opinions, and feelings
Learning	Kirkpatrick / Holton / Phillips	Methods, such as tests, that measure knowledge and/or attitudes before and after an education event to indicate the level of change achieved
Application of learning	Kirkpatrick (behavior) / Holton (individual performance) / Phillips (job application)	Methods, such as interviews with multiple sources, that measure participants’ prior behavior and subsequent behavioral change at appropriate time(s)
Organizational impact	Kirkpatrick / Holton (organizational results) / Phillips (business results)	Measurement of the impact of education on the target organizations’ results against key objectives at appropriate time(s)
Delivery of public value	Phillips (return on investment)	Compare the cost of the education and evaluation to the outcomes achieved in order to determine value

Sources: This table draws on the work of Kirkpatrick (1975: 1–17); Kirkpatrick and Kirkpatrick (2006); Holton (1996); Phillips et al. (2007); Steele et al. (2016); Pallai and Gregor (2016); Alliger and Janak (1989); and Arney (2017).

In the 1950s Donald L. Kirkpatrick developed a four-level approach to education evaluation that focused on measuring reaction, learning, behavior, and results (see Kirkpatrick, 1975: 1–17; Kirkpatrick and Kirkpatrick, 2006). This approach became increasingly well-known during the following decades and is still widely cited today (Phillips et al., 2007: 8). Although “elegant in its simplicity” (Holton, 1996: 6–8), Kirkpatrick’s approach is hierarchical and critics argue it suggests causal relationships between the categories without accounting for other variables (Alliger and Janak, 1989; Holton, 1996: 6–8). Nevertheless, in Holton’s (1996: 9) alternative “outcome measures” approach, three of Kirkpatrick’s levels are recognized as primary considerations, albeit in a framework that is conceptually very different: learning, individual performance, and organizational results. Acknowledging that there is a range of worthwhile education objectives and available resources to achieve them, Table 1 is not intended to adhere to a strict hierarchy, but to broadly describe types of evaluation.

During the 1980s Jack Phillips highlighted an additional objective by separating—and, therefore, emphasizing the importance of—return on investment from business results (Phillips et al., 2007: 8–9). Measuring a return on investment in education is controversial, with some questioning the extent to which it is possible (see Phillips et al., 2007: 3). According to Phillips and colleagues (2007: 23–24), the use of control groups and trend lines are among the strategies that can help ensure reliability. In Table 1 the delivery of public value is used to underscore the purpose of the public sector in delivering value for the public while acknowledging that resources are limited and no function can occur at any cost. In addition to the categories that feature most prominently in the literature it is necessary to include an exposure category in Table 1. In the exploratory survey detailed below, the compilation and reporting of statistics to show the number of people exposed to education events as indicative of effectiveness is striking.

Exploratory survey of cases

Australia’s six state-based public sector standing anti-corruption commissions broadly adhere to the multi-pronged approach popularized by the Hong Kong Independent Commission Against Corruption established in 1974. Although there are variations, they all have functions in investigation, prevention, and education. The precise nature of the education function is somewhat vague, with terms such as education, training, and awareness often used interchangeably. Public sector education events, such as seminars, workshops, and training sessions, are a prominent part of how commissions undertake this function and are commonly the subject of evaluative statements in public reporting. Nevertheless, the education function encompasses a range of other elements including giving advice and guidance to agencies, developing educational resources, and disseminating information via newsletters, web pages, and similar.

The six commissions examined in this survey are the New South Wales Independent Commission Against Corruption (NSW ICAC), the Western Australian Corruption and Crime Commission (WA CCC), the Queensland Crime and Corruption Commission (QLD CCC), the Tasmanian Integrity Commission (TAS IC), the Victorian Independent Broad-based Anti-corruption Commission (VIC IBAC), and the South Australian Independent Commissioner Against Corruption (SA ICAC). Each of these commissions delivers education events to public sector participants in their respective jurisdictions. The participants targeted in each event may be general or role specific, while the audience may be assembled through a variety of approaches. The main documentary sources informing this exploration are annual reports. They are the essential reporting tool for accountability and transparency and play a key role in communicating each commission’s activities, performance, and plans to the executive, legislature, and public. This is supplemented by legislation and commissions’ websites where necessary. Due to variations in jurisdiction a comparison of resources is not included.

NSW ICAC

The first annual report of the NSW ICAC provided to parliament in September 1989 states that the purpose of its work is “to minimize corruption in the public sector” (NSW ICAC, 1989: 12), a statement that is consistent with the underlying assumption supporting the function today. In the early years, education efforts focused on general public awareness of the commission’s establishment (see NSW ICAC, 1989: 50–53, 1990: 73). Specific public sector education events began with a seminar for all NSW government chief executives in June 1990 and this event provided the basis for a roll out of similar seminars for other officers in the years that followed (NSW ICAC, 1990: 71–72, 1991: 83–84). Reporting of the function mainly involved the inclusion of lists of these events to demonstrate occurrence, which infers exposure (see, for example, NSW ICAC, 1996: 63–66). Their primary goal was cognitive, that is, to promote “better knowledge and understanding of probity and accountability” [original emphasis] (NSW ICAC, 1995: 12).

The education function developed over time, but the evaluative reporting remained largely unchanged. Broad-brush evaluative contentions are frequently made without any substantiating details. In 1996 the annual report makes cursory mention of a “performance review” of the regional seminar program, reporting that there were “high levels of acceptance of the corruption prevention message” and that “the increasing demand for more specialised information is an indication of the Commission’s previous success in disseminating the basic corruption prevention message,” but provides no explanation of the process undertaken to reach these conclusions (NSW ICAC, 1996: 13). Similarly, the 1997 report makes the following statement: “The effectiveness of this work is measured through the positive results of strategic sector-wide projects and the adoption by central agencies of ICAC initiatives to promote integrity in policies and procedures” (NSW ICAC, 1997: 13). No evidence is provided to support the claim of positive results, while the adoption of initiatives says nothing of what they actually achieve.

By the late 1990s mentions of participant reaction survey results begin to appear frequently, albeit still with little explanation or context (see examples NSW ICAC, 1998: 44, 1999: 38). The key indicators cited from the reaction surveys are typically usefulness and meeting training needs (see examples NSW ICAC, 2012: 46–47, 2013: 38). Exposure statistics also continue to appear alongside these reaction indicators (see examples NSW ICAC, 2004: 43–45, 2005: 39, 2008: 59, 2012: 46–47), although the inclusion of event lists ceased after 2009. Cognitive learning-based goals remain at the forefront of the function with the aims of increasing reporting of corrupt conduct and improving investigations through knowledge dissemination, both of which are said to elevate the risk of detection and thereby act as deterrents (NSW ICAC, 2008: 56). The number of evaluations undertaken was reduced in 2013 due to the “consistently high ratings…received in recent years” and the need to “reduce the administrative load such evaluations create” (NSW ICAC, 2013: 38).

Most recently, the NSW ICAC’s annual report for 2017–2018 focuses on exposure statistics and reaction surveys:

In 2017-18, the Commission delivered 126 workshops to over 2,300 attendees…Of the 126 workshops delivered in 2017-18, 104 were evaluated to ensure quality. A total of 94% of participants rated these workshops as “useful” or “very useful” and 96% of participants “agreed” or “strongly agreed” that these workshops met their training needs. (NSW ICAC, 2018: 34)

Additionally, NSW ICAC and Australia and New Zealand School of Government (ANZSOG) partner in a four-day course entitled “Strategic responses to corruption.” Reaction survey responses are given using a 5-point scale on the following indicators:

how transferable the learnings from the program are to their workplace—average rating 4.2 the extent that the course’s learning environment and content were stimulating—average rating 4.2 the course overall—average rating 4.3 the presenters—average ratings 4.5 (NSW ICAC, 2018: 36)

WA CCC

In 2002 the Kennedy royal commission² recommended that Western Australia’s Anti-Corruption Commission be replaced by a new Corruption and Crime Commission (WA CCC) with an added function of prevention and education in order to “lift the level of awareness…of the existence of potential problems” (Kennedy, 2002: 104). This occurred on 1 January 2004. Initially, the education function focused on introducing the commission including its roles and responsibilities before shifting attention to specific issues and customized sessions (WA CCC, 2006: 45). Reporting of the function between 2004 and 2015 was heavily skewed to exposure statistics (see examples WA CCC, 2006: 44, 2011: xx, 2013: 8) with some brief mentions of pre-event consultations, reaction-based feedback, and self-reported application of learning responses (see examples WA CCC, 2005: 32, 2007: 16, 2009: 19).

Legislative changes that came into effect on 1 July 2015 shifted the primary responsibility for education to the state’s Public Sector Commission (WA PSC) (see Corruption, Crime and Misconduct Act 2003 , as amended). This was part of wider reforms to focus the WA CCC on serious matters, with the government arguing that the education function would complement the WA PSC’s existing responsibilities (Day, 2014: 1960). Since the changes were introduced the WA CCC has continued to provide education on serious misconduct and corruption while also working collaboratively with the WA PSC to offer joint education events (WA CCC, 2016: 6, 41). Reporting on the revised education function has only cited exposure statistics (see WA CCC, 2016: 41, 2017: 44, 2018: 50). In its 2017–2018 annual report the WA CCC (2018: 50) provides exposure statistics indicating that the commission delivered three joint sessions with the WA PSC to about 140 participants, while the WA CCC commissioner made a further eight presentations to about 460 attendees.

QLD CCC

The Queensland Criminal Justice Commission (QLD CJC) was established in late 1989 with an education function; however during the set-up phase it operated with only “skeleton staff” and was overwhelmed with complaints—indeed, the 1991 annual report indicates that there were 112 police matters per investigating officer (QLD CJC, 1991: 17). As such, the corruption prevention section did not commence work until August 1991 (QLD CJC, 1992: 46). Nevertheless, each annual report provides a list of events—again, inferring exposure—beginning with public addresses (see QLD CJC, 1990: 46) and quickly expanding to include workshops (see QLD CJC, 1991: 97) and lectures and presentations (see QLD CJC, 1992: 121–128). Exposure statistics dominate the reporting (see, for example, QLD CJC, 1997: 42); however, some mentions of reaction survey results do appear in the annual reports. In 1994 the indicator measured is “helpfulness” (QLD CJC, 1994: 50), while in 1995 and 1997 it is “practicality” (QLD CJC, 1995: 79, 1997: 42).

In January 2002 the QLD CJC merged with the Crime Commission to form the Crime and Misconduct Commission (QLD CMC) and the new Crime and Misconduct Act 2001 shifted greater responsibility onto police and public sector managers to prevent and respond to misconduct matters within their workplaces (QLD CMC, 2002: 3). Thus, the QLD CMC’s (2002: 56) new focus was in capacity building through collaboration with agencies, although it continued to conduct education events (see QLD CMC, 2003: 39). The reporting of these activities throughout the existence of the QLD CMC was limited, with exposure statistics the main information provided (see examples QLD CMC, 2004: 44, 2012: 28). An exception is the 2009 annual report that makes the following bold statement about training effectiveness: “Positive feedback from all sessions suggests that the attendees increased their knowledge” (QLD CMC, 2009: 30). There is no evidence provided to support this suggested link between positive sentiments and actual learning.

In 2014 the Liberal National government led by Campbell Newman removed the renamed QLD CCC’s prevention functions as part of wider reforms to the commission (see further Briody, 2015). It argued that the Public Service Commission (QLD PSC) would “provide the requisite advice, support and assistance to public sector agencies” (Bleijie, 2014: 1342), although it was highlighted that some agencies within the QLD CCC’s jurisdiction sit outside the QLD PSC’s area of operations, for example local government and universities (Pucci, 2014: 1372). The change, although passed, was not enthusiastically embraced by parliamentarians, with some Liberal National Party members expressing their support for the retention of a prevention function (see, for example, Davis, 2014: 1377). The basis for the reform was provided by the Callinan–Aroney review that reported to the government in 2013. The effectiveness of the education function does not appear to have been central to its considerations. Instead, the multiplicity of agencies involved in integrity, including the Integrity Commissioner, Ombudsman, QLD PSC, Electoral Commissioner, QLD CMC, and internal divisions within individual agencies underpinned the argument to refine their responsibilities (Callinan and Aroney, 2013: 137). Particular attention was drawn to the Ombudsman’s acknowledgment that its education function overlapped with that of the QLD CMC (Callinan and Aroney, 2013: 141). Callinan and Aroney (2013: 216) argued that the reform would mean “the emphasis upon unnecessary and duplicated integrity education will reduce.” Further:

The existence of overlapping functions and the multiplicity of institutions have been explained on the basis that it is part of the “integrity regime”…to view the arrangements in this manner is to risk assuming that “more is better”, rather than critically to assess their merits and especially their efficiency, utility and necessity. (Callinan and Aroney, 2013: 142–143)

In 2016 the Labor government led by Annastacia Palaszczuk reinstated the QLD CCC’s prevention functions, a move welcomed by Commissioner Alan MacSporran, QC (QLD CCC, 2016: 7) as “enhancing” the capacity of the commission. The minister responsible for the QLD CCC, Yvette D’Ath (2015: 2969), argued that the reinstatement filled a “critical gap in Queensland’s integrity system.” However, within the new corruption prevention strategy, education events are not a significant focus (see QLD CCC, 2018: 8, 32).

TAS IC

The TAS IC has reported exposure statistics from its inception (see examples TAS IC, 2011: 12, 2012: 18). The use of reaction surveys is mentioned from 2013 (TAS IC, 2013: 23–24). The indicators reported are completion of previous training and self-reported opinions of understandings gained in training (TAS IC, 2013: 23–24, 2014: 27, 2015: 20). Since 2013 the commission has focused on producing generic resources with a number of modules developed in an Ethics and Integrity Training Program for delivery within, and by, government agencies themselves, allowing the TAS IC to target its education in priority areas (TAS IC, 2014: 24, 27, 2017: 30). In 2018 education events targeted agencies that were subject to investigations, reaching more than 2500 participants (TAS IC, 2018: 25). Reporting of exposure statistics is comprehensive in the 2017–2018 annual report, but no other form of evaluation is mentioned (TAS IC, 2018: 25–30).

VIC IBAC

Initially the VIC IBAC focused on raising general awareness, before shifting toward a more targeted education approach (VIC IBAC, 2016: 31). From its inception in 2012, VIC IBAC has reported the number of “corruption prevention initiatives,” including training sessions, speaking engagements, and prevention resources, delivered—inferring exposure—and a “satisfaction rating,” although no detail about the methodology for what are presumably reaction survey results is provided. The results show little variation over the years: 2012–2013: 72 initiatives at 98.9% satisfaction; 2013–2014: 74 initiatives at 96.7%³ satisfaction; 2014–2015: 105⁴ initiatives at 95%⁵ satisfaction; 2015–2016: 78 initiatives at 99% satisfaction; 2016–2017: 93 initiatives at 99% satisfaction; 2017–2018: 92 initiatives at 99% satisfaction (VIC IBAC, 2013: 17, 2014: 11, 2015: 1, 27, 2016: 10, 2017: 10, 2018: 10). Additional exposure statistics are also provided in some reports (see, for example, VIC IBAC, 2014: 27–28). The annual reports also refer to surveys regarding self-reported knowledge before and after training on protected disclosures. For example, the 2012–2013 annual report mentions a “self-rated assessment of knowledge pre-and-post seminars”—presumably part of a reaction survey—that indicated a “strong improvement in knowledge” (VIC IBAC, 2013: 17).

SA ICAC

Given its short period of existence, the SA ICAC has only produced four annual reports. Exposure statistics are dominant in each report, with limited reaction survey results and a few selected comments attributed to participants provided (see examples SA ICAC, 2015: 37, 2017: 43). The latest annual report states:

2017-18 saw 66 education sessions delivered to 2,592 public officers, exceeding our yearly goals of 60 education sessions reaching 2,000 public officers. The commissioner personally presented 15 sessions to 1,207 attendees…Education events consistently receive excellent feedback. (SA ICAC, 2018: 46)

Discussion

This exploratory survey suggests that the unquestioned assumption that integrity education will reduce levels of misconduct and corruption is long-held and pervasive among these commissions. It also suggests that there has been minimal—if any—parliamentary or public expectation that the accuracy of such an assumption be tested through meaningful evaluations of the commissions’ education function. Indeed, beyond a broad directive to improve integrity and reduce misconduct and corruption, it is not clear that the legislatures that have given commissions an education function have clear objectives in mind. Is the goal to raise awareness of the role of commissions, to nudge broader cultural change, or to provide a substantial integrity outcome that influences attitudes and changes behavior? Additionally, the question of whether commissions are the agencies that are best placed to deliver this function is unresolved. Currently a range of agencies provide integrity education and this overlap of responsibilities risks churning public sector workers through a maze of superficial and disconnected educational requirements. An emphatic “more is better” view appears to have been broadly adopted, limited only by resources and an occasional political intervention as occurred in Queensland.

The dominant evaluation approaches used are exposure statistics and reaction surveys, with a few cursory mentions that indicate some consideration has been given to learning and the application of learning. Organizational impact and delivery of public value evaluations are absent. The dominance of exposure statistics seems to suggest that demonstrating the productiveness of education staff is a high priority. However, it is concerning that this exposure-as-success approach substitutes, in large part, for more meaningful evaluations of performance. As Alliger and Janak (1989: 333) note, there is no automatic link between attending and learning. Similarly, participant reactions are a weak indicator of effectiveness (Dixon, 1990: 129-130; Pallai and Gregor, 2016: 247), commonly exacerbated by the distribution of survey forms at the end of an event, inviting rushed and ill-considered responses (Arney, 2017: 221). Nevertheless, a positive participant experience can contribute to the effectiveness of an education event, including by increasing learning motivation, interest, and attention; sustaining the future of the event; and signaling the value of participants to the trainer in a way that fosters a mutually supportive relationship (see Holton, 1996: 10; Kirkpatrick and Kirkpatrick, 2006: 22, 27). However, as Holton (1996: 8) argues, although a positive and enjoyable participant experience can act as a “mediator” of successful learning, it is not directly related to, or indicative of, that outcome (see also Alliger and Janak, 1989). Similarly, research indicates that humor does not produce learning, despite being clearly favored by participants (Alliger and Janak, 1989: 334).

Reaction surveys tend to focus on edutainment demands that are increasingly influencing how education is delivered in all settings (see O’Brien, 2015; Wong and Chiu, 2019). As such, it is no surprise that these surveys are often referred to as “smiley” or “happy” sheets (Arney, 2017: 241; Kirkpatrick and Kirkpatrick, 2006: 41; Pallai and Gregor, 2016: 248). O’Brien (2015) argues that the habitual use of technological devices is creating new demands for all facets of life to be entertaining in order to overcome attention deficits and distractions. In reality, learning can be difficult and, as Holton (1996: 10) suggests, confusing and frustrating, in various stages. Indeed, Alliger and Janak (1989: 334) note that some research has indicated a possible negative link between reactions and learning and, as such, they go so far as to suggest “perhaps it is only when trainees are challenged to the point of experiencing the training as somewhat unpleasant that they learn.” Further, edutainment demands in the context of paid employment seem to suggest that such learning opportunities are held in low esteem.

In Pallai and Gregor’s (2016) study of a Hungarian integrity training program, the weakness of reaction survey results in comparison to learning assessments is demonstrated. Participant satisfaction with some trainers is rated at or above 3.85 out of 4 on reaction surveys, even though learning assessments demonstrate they produced “minimal to zero learning impact” (Pallai and Gregor, 2016: 249, 263 at note 4). Indeed, if trainer selection for this program was undertaken on the basis of reaction survey results, two of the nine most effective trainers could be rejected due to having lower satisfaction scores than some of their less effective but more popular colleagues (Pallai and Gregor, 2016: 263–264 at note 9). Similarly, Dixon’s (1990: 136) study of workplace education events related to a new manufacturing process in the United States found no correlation between trainer ratings and learning, further suggesting reactions reflect enjoyment rather than effectiveness.

Additionally, Dixon’s (1990: 135) study found no correlation between participants’ self-reported perceived level of learning in reaction surveys and either the raw test results for learning or the pre- to post-education learning gains shown in independent assessments. As Dixon (1990: 135) acknowledges, “this finding is counterintuitive, in that it is commonly assumed that individuals know whether or not they have learned.” However, Dixon (1990: 135) speculates that the perceived level of learning may be significantly influenced by enjoyment; that is, when participants have enjoyed an education event, they believe they have also learned a lot, or that they seek to reward the trainer regardless. Both studies highlight the weakness of reaction-based evaluations and, as Pallai and Gregor (2016: 262, 263 at note 4) suggest, highlight the need for a “more sound basis” than participant “self-reports” upon which to assess education events.

A lack of serious evaluation may also suggest that the education function provides false confidence and assurance—to agencies, commissions, politicians, and the public—and possibly serves a branding purpose (see, for example, Broad, 2018). Arney (2017: 219) warns against a tendency to use workplace education as a “comfort blanket, offering warmth and protection” without ensuring it is fit for purpose and the results are reliable. Importantly, understanding how learning and the application of learning occurs—if at all—provides valuable insights to improve the function as well as helping agencies to understand broader workplace issues such as relevance of courses, team morale, culture, motivation, and resistance (see Arney, 2017: 226; Lombardo and Mergaert, 2016: 43–61; Parks, 1993: 27–29, 52). Additionally, it can ensure that education develops participants’ agency in integrity matters rather than producing fear or paralysis (see Cochrane, 2018: 512).

The exploratory survey revealed no evidence of more robust approaches to evaluation being implemented even as the education function itself has evolved. Although it is possible that annual reports do not reveal all the evaluative work that is carried out regarding education events, what is—and is not—publicly reported provides a valuable insight into the priorities of commissions. Modern transparency expectations also suggest that the best available evidence of the performance of the education function should be publicly reported. Additionally, all commissions fail to provide sufficient details about the methodological approaches used in the evaluations that are undertaken in order for independent assessments to be made.

Without any track record of alternative approaches being used by any commission in this study, a precise prescription for the way forward is fraught. This study has not tested any evaluation approaches upon which to base recommendations. Instead, there are ample reasons for the commissions to develop and experiment with new strategies for meaningful evaluations adapted to suit the circumstances of their education function. Of course, such a process will not be without significant challenges. As Arney (2017: 227–28) highlights, scarce resources and a lack of appreciation for the value of evaluation among decision-makers hamper efforts. Indeed, commissions have indicated that they face many difficulties in undertaking evaluations (see, for example, NSW ICAC, 2009: 63). However, this should not prevent existing resources being directed to higher-quality evaluations even if this can only occur on a smaller resource-intensive scale compared to current approaches.

While more complex evaluations may still not account for every variable, the aim should be to produce credible “evidence”—rather than “proof”—of the effectiveness of educational events [original emphasis] (Kirkpatrick and Kirkpatrick, 2006: 65–66, 70). Some inevitable ambiguity in evaluating learning, application of learning, organizational impact, and delivery of public value does not diminish the benefits of pursuing deeper insights. To this end, Patton’s (1997: 4) framework for “utilization-focused evaluation” provides a solid foundation for developing broader strategies that produce useful and meaningful results, while Arney (2017, particularly chapter 6) provides an overview of various approaches that may be useful for particular purposes. Additionally, Steele and colleagues (2016) ask “how do we know what works?” and, in doing so, offer pertinent insights and inspiration for pursuing more meaningful evaluation approaches. Modern technology creates many user-friendly options for evaluating learning, including the use of case studies or test scenarios, while interviews and/or systematic observations can focus on the application of learning, organizational impact, and delivery of public value.

Conclusion

This study has conducted an exploratory survey of how commissions seek to demonstrate the effectiveness of their education function through evaluations and reporting. No claim is made as to the effectiveness of the commissions’ education function itself, but rather that the current evaluative and reporting approaches are insufficient to demonstrate that the education events contribute to achieving the function’s purpose. The dominance of reporting exposure-as-success attendance statistics revealed in this study is particularly troubling, while the heavy reliance on reaction-based surveys—although consistent with the findings of previous studies (see Steele et al., 2016: 32)—demonstrates a narrow approach to evaluation. Although there are some cursory mentions of evaluating learning and application of learning outcomes, these need further development as part of a broader strategy that shifts resources from superficial and repetitive reaction surveys to options that produce more meaningful evidence of performance.

It is hoped that this study stimulates further debate about how integrity education occurs in the public sector and how the performance of this function is evaluated and reported. As Pallai and Gregor (2016: 248) argue, agencies delivering publicly funded education should be obliged to demonstrate its effectiveness with meaningful evidence. More complex evaluations will require greater resources and effort, but if undertaken in a strategic and carefully targeted manner they offer much better prospects of contributing to improving the delivery of integrity education and, in turn, reducing misconduct and corruption in the public sector. As Krathwohl and colleagues (1964: 79) caution:

Such objectives are not to be attained simply by someone expressing the desire that they be attained…when [they are] not fully committed to doing the necessary work for the development of appropriate learning experiences and evaluation evidence…Such objectives should not be stated lightly and expected to be achieved by some semi-magical process…Stating an objective and achieving it are two very different things.

The opportunities for further research in this field are immense. Both practitioners and academics should seek to expand the scope of evaluations in all forms of education, including through the development of innovative approaches and experimental designs. Collaborative approaches among agencies and jurisdictions and the public reporting of experiences with, and outcomes of, new methods can greatly enrich knowledge within the field. Further, as Van Montfort and colleagues (2013: 118) highlight, more studies that seek to empirically identify the specific factors that contribute to positive or negative outcomes in educational approaches, among particular participants, or in different learning contexts, are needed to inform both the design of evaluations and the implementation of integrity education itself.

Footnotes

Acknowledgements

This article benefited greatly from the comments and encouragement of the organizers and participants at the APSA Policy Studies Research Group workshop “Teaching Public Policy” and of colleagues and peers. Thank you to Dr Siobhan O’Sullivan and Dr Yvonne Haigh, as well as anonymous reviewers, for their comments on earlier drafts of this article.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by an Australian Government Research Training Program scholarship. Travel to the APSA Policy Studies Research Group workshop “Teaching Public Policy” in Sydney was provided by APSA workshop funding.

ORCID iD

Catherine Cochrane

Notes

References

Alliger

Janak

(1989) Kirkpatrick’s levels of training criteria: Thirty years later. Personal Psychology 42(2): 331–340.

Althaus

Bridgman

Davis

(2013) The Australian Policy Handbook. 5th edition. Crows Nest: Allen & Unwin.

Appleby

Brennan

Chordia

, et al. (2016) Submission 19: The Senate Select Committee into the establishment of a National Integrity Commission. Available at: http://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Establishment_of_a_National_Integrity_Commission/NIC/Submissions (accessed 16 December 2016).

Arney

(2017) Learning for Organizational Development: How to Design, Deliver and Evaluate Effective L&D. London: Kogan Page.

Bardach

(2012) A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving. 4th edition. Thousand Oaks: SAGE.

Bleijie

(2014) Parliamentary Debates, 7 May. Queensland: Legislative Assembly.

Brinkerhoff

(2006) Telling Training’s Story: Evaluation Made Simple, Credible, and Effective. San Francisco: Berrett-Koehler Publishers.

Briody

(2015) Establishing the Crime and Corruption Commission: The reformation of Queensland’s premier crime-fighting agency. Journal of Policing, Intelligence and Counter Terrorism 10(2): 136–152.

Broad

(2018) Made by Humans. Carlton: Melbourne University Press.

10.

Brookfield

(2008) Teaching critical thinking. In: Athanasou

(ed.) Adult Education and Training. Terrigal: David Barlow Publishing, pp.35–50.

11.

Callerstig

(2016) Gender training as a tool for transformative gender mainstreaming: Evidence from Sweden. In: Bustelo

Ferguson

Forest

(eds) The Politics of Feminist Knowledge Transfer: Gender Training and Gender Expertise. Basingstoke: Palgrave Macmillan, pp.118–138.

12.

Callinan

Aroney

(2013) Review of the Crime and Misconduct Act and related matters: Report of the independent advisory panel (redacted version), Government of Queensland, Brisbane, 28 March.

13.

Cochrane

(2018) Boundary making in anti-corruption policy: Behaviour, responses and institutions. Australian Journal of Political Science 53(4): 508–528.

14.

Corruption, Crime and Misconduct Act (2003). Available at: www.legislation.wa.gov.au (accessed 14 May 2019).

15.

Cranton

Tisdell

(2008) Transformative learning. In: Athanasou

(ed.) Adult Education and Training. Terrigal: David Barlow Publishing, pp.35–50.

16.

D’Ath

(2015) Parliamentary Debates, 1 December. Queensland: Legislative Assembly.

17.

Davis

(2014) Parliamentary Debates, 7 May. Queensland: Legislative Assembly.

18.

Day

JHD

(2014) Parliamentary Debates, 2 April. Western Australia: Legislative Assembly.

19.

Dixon

(1990) The relationship between trainee responses on participant reaction forms and post-test scores. Human Resource Development Quarterly 1(2): 129–137.

20.

Griffin

(2012) Integrity as a core value in organizations. In: Wankel

Stachowicz-Stanusch

(eds) Handbook of Research on Teaching Ethics in Business and Management Education. Hershey: Information Science Reference, pp.327–340.

21.

Hetherington

Rudolph

(2008) Priming, performance, and the dynamics of political trust. Journal of Politics 70(2): 498–512.

22.

Holton

(1996) The flawed four-level evaluation model. Human Resource Development Quarterly 7(1): 5–21.

23.

Kennedy

(2002) Royal Commission into whether there has been corrupt or criminal conduct by any Western Australian police officer, interim report, Government of Western Australia, Perth, 20 December.

24.

Killen

(2016) Effective Teaching Strategies: Lessons from Research and Practice. 7th edition. South Melbourne: Cengage Learning Australia.

25.

Kirkpatrick

(ed.) (1975) Evaluating Training Programs: A Collection of Articles from the Journal of the American Society for Training and Development. Madison: American Society for Training and Development.

26.

Kirkpatrick

(2006) Evaluating Training Programs: The Four Levels. 3rd edition. San Francisco: Berrett-Koehler Publishers.

27.

Krathwohl

Bloom

Masia

(1964) Taxonomy of Educational Objectives: The Classification of Educational Goals – Handbook II: Affective Domain. New York: David McKay Company.

28.

Leberman

McDonald

Doyle

(2006) The Transfer of Learning: Participants’ Perspectives of Adult Education and Training. Aldershot: Gower Publishing.

29.

Lombardo

Mergaert

(2016) Resistance in gender training and mainstreaming processes. In: Bustelo

Ferguson

Forest

(eds) The Politics of Feminist Knowledge Transfer: Gender Training and Gender Expertise. Basingstoke: Palgrave Macmillan, pp.43–61.

30.

Matchett

(2009) Cooperative learning, critical thinking, and character. Public Integrity 12(1): 25–38.

31.

Menzel

(2009) Teaching and learning ethical reasoning with cases. Public Integrity 11(3): 239–250.

32.

Mulgan

Wanna

(2011) Developing cultures of integrity in the public and private sectors. In: Graycar

Smith

(eds) Handbook of Global Research and Practice in Corruption. Cheltenham: Edward Elgar, pp.416–428.

33.

New South Wales Independent Commission Against Corruption (1989–2018) Annual report. Available at: www.icac.nsw.gov.au (accessed 2 December 2018).

34.

O’Brien

(2015) Let me edutain you! Why university lecturers need to play to their audience. The Conversation, 15 January. Available at: www.theconversation.com/let-me-edutain-you-why-university-lecturers-need-to-play-to-their-audience-35942 (accessed 22 March 2018).

35.

Pallai

Gregor

(2016) Assessment of effectiveness of public integrity training workshops for civil servants: A case study. Teaching Public Administration 34(3): 247–269.

36.

Parks

(1993) Is it too late? Young adults and the formation of professional ethics. In: Piper

Gentile

Parks

(eds) Can Ethics Be Taught? Perspectives, Challenges, and Approaches at Harvard Business School. Boston: Harvard Business School Press, pp.13–72.

37.

Patton

(1997) Utilization-focused Evaluation: The New Century Text. 3rd edition. Thousand Oaks: SAGE.

38.

Phillips

Stone

, et al. (2007) The ROI Fieldbook: Strategies for Implementing ROI in HR and Training. Burlington: Butterworth-Heinemann.

39.

Piper

(1993) Rediscovery of purpose: The genesis of the leadership, ethics, and corporate responsibility initiative. In: Piper

Gentile

Parks

(eds) Can Ethics Be Taught? Perspectives, Challenges, and Approaches at Harvard Business School. Boston: Harvard Business School Press, pp.1–12.

40.

Pucci

(2014) Parliamentary Debates, 7 May. Queensland: Legislative Assembly.

41.

Queensland Crime and Corruption Commission (2016–2018) Annual Report. Available at: www.ccc.qld.gov.au (accessed 2 December 2018).

42.

Queensland Criminal Justice Commission (1990–1997) Annual Report. Available at: www.ccc.qld.gov.au (accessed 2 December 2018).

43.

Queensland Crime and Misconduct Commission (2002–2012) Annual Report. Available at: www.ccc.qld.gov.au (accessed 2 December 2018).

44.

Rest

Narváez

(eds) (1994) Moral Development in the Professions: Psychology and Applied Ethics. Hillsdale: Lawrence Erlbaum Associates.

45.

Sampford

Shacklock

Connors

, et al. (eds) (2006) Measuring Corruption. Aldershot: Ashgate Publishing Limited.

46.

Smith

Sadler-Smith

(2006) Learning in Organizations: Complexities and Diversities. Milton Park: Routledge.

47.

South Australian Independent Commissioner Against Corruption (2015–2018) Annual Report. Available at: www.icac.sa.gov.au (accessed 2 December 2018).

48.

Steele

Mulhearn

Medeiros

, et al. (2016) How do we know what works? A review and critique of current practices in ethics training evaluation. Accountability in Research 23(6): 319–350.

49.

Tasmanian Integrity Commission (2011–2018) Annual Report. Available at: www.integrity.tas.gov.au (accessed 2 December 2018).

50.

van Montfort

Beck

Twijnstra

(2013) Can integrity be taught in public organizations? Public Integrity 15(2): 117–132.

51.

Victorian Independent Broad-based Anti-corruption Commission (2013–2018) Annual Report. Available at: www.ibac.vic.gov.au (accessed 2 December 2018).

52.

Western Australia Corruption and Crime Commission (2002–2018) Annual Report. Available at: www.ccc.wa.gov.au (accessed 2 December 2018).

53.

Wong

Chiu

YLT

(2019) Let me entertain you: The ambivalent role of university lecturers as educators and performers. Educational Review 71(2): 218–233.