Abstract
Australia’s state-based public sector standing anti-corruption commissions each have an education function. The function is underpinned by a largely unquestioned assumption that integrity education will reduce levels of misconduct and corruption in the public sector. This study explores how these commissions undertake evaluations and publicly report on the effectiveness of their education events to demonstrate a contribution to this purpose. It acknowledges that well-designed and executed education events may deliver public value outcomes, but finds that the evaluative and reporting approaches used by commissions are insufficient to reliably demonstrate whether or not the function actually achieves this purpose. The dominance of exposure-as-success attendance statistics revealed in this study is particularly concerning as they offer no insights that are directly relevant to this purpose. Consistent with the findings of previous studies, there is also a heavy reliance on reaction-based satisfaction surveys that primarily reflect participants’ enjoyment of an education event, but do not provide meaningful evidence of learning, application of learning, organizational impact, or the overall delivery of public value. As such, it is argued that commissions need to transition from largely superficial appraisals to more robust and strategic assessments that measure actual attitudinal and behavioral change in the public sector. This study also raises questions of broader relevance to the evaluation of other types of workplace education, particularly that requiring affective and behavioral learning.
Introduction
Integrity in public administration is increasingly in the spotlight. Broadly, behaving with integrity encompasses three aspects—moral awareness, reasoning, and behavior (Van Montfort et al., 2013: 118)—but the application of standards remains contentious, while precisely measuring the dimensions of problems involving contrary behavior is difficult (see, for example, Cochrane, 2018; Sampford et al., 2006). Nevertheless, with evidence of declining public trust in government institutions (see, for example, Appleby et al., 2016; Hetherington and Rudolph, 2008), the demands on the public sector to meet the highest possible integrity standards have never been greater. Australia’s six state-based public sector standing anti-corruption commissions each have an education function, which is underpinned by a largely unquestioned assumption that integrity education will improve attitudes and behavior and reduce levels of misconduct and corruption. Such an assumption is not unique to integrity education. Referring to workplace education in general, Phillips and colleagues (2007: 4) note that “payoffs are assumed to exist and learning programs appear to be necessary,” but “frustration comes from the lack of evidence to show that the process is really working” (see also Brinkerhoff, 2006).
However, more recently academic studies have begun to examine this assumption (see, for example, Steele et al., 2016) with the findings of some studies raising concerns due to their mixed results. For example, a study of two Dutch municipal integrity training programs demonstrated a positive short-term impact resulting from one program but not the other, while neither produced longer-term benefits (Van Montfort et al., 2013). Similarly, a study of a public ethics and integrity training program for civil servants in Hungary produced moderate outcomes overall, but about a quarter of participants declined in their commitment to fighting corruption during the one- or three-day training courses (Pallai and Gregor, 2016). In Australia, understanding whether outcomes match the assumption is important for ongoing policy debates about the functions and objectives of commissions.
In public administration integrity is approached in two ways: a rules-based compliance approach to prevent or punish negative behaviors, and a values-based ethics approach to promote positive behaviors. Sometimes agencies may have a strong bias toward one or the other approach, but elements of both are usually present. The education function accords with both approaches, encompassing all aspects of Piper’s (1993: 10–11) “educational trilogy” for ethics teaching: knowledge, attitudes, and skills. As Piper (1993: 10–11) argues, “each is important; each is insufficient in isolation.” Indeed, teaching for integrity is set apart from most other workplace education by its greater emphasis on impacting attitudes and behavior vis-a-vis knowledge and skills. Although one requires knowledge of one’s duties in order to enact them, this knowledge is only information—not integrity—unless, or until, it is acted upon in the appropriate manner and according to choice (Griffin, 2012: 332). As Griffin (2012: 338) asks rhetorically: “What does it matter if [people] know how to reason well in ethics if they are not motivated to act ethically?”
This dilemma makes integrity education particularly challenging to implement successfully and to evaluate to ensure it achieves its purpose. Integrity requires knowledge of requirements and expectations; an affective disposition, particularly attitude, conducive to meeting requirements and expectations, skills to fulfil requirements and expectations, and, consequently, behavior according to requirements and expectations. Education for these ends is influenced by three main variables: the characteristics of the participants, the organizational or situational context, and the characteristics of the education program (Van Montfort et al., 2013: 119; see also Leberman et al., 2006: 99–108; Smith and Sadler-Smith, 2006). As such, short ad hoc exposures to information may create superficial awareness, but are unlikely to produce measurable changes in affective attitudes and behaviors (Krathwohl et al., 1964: 80; see also Callerstig, 2016: 119–120). The process of achieving highly internalized integrity learning objectives takes time and requires repetition across a range of content areas that together form a “consistent and complex set” of attitudes and behaviors (Krathwohl et al., 1964: 80). 1
Acknowledging that it is possible for well-designed and executed integrity education to contribute to successful outcomes, this study examines the extent to which the six commissions demonstrate that their education function achieves its purpose. It will present the findings of an exploratory survey of how commissions evaluate and publicly report on their education events. It begins with a review of the academic literature on education evaluations to develop a typology to describe their scope. These categories are then used to undertake the exploratory survey. A discussion of the findings follows, with the shortcomings of the approaches used identified and a recommendation made for a greater focus on evaluations that produce meaningful results. It aims to stimulate further debate and discussion of a significant function of commissions that has long been overshadowed by their more high-profile investigative role.
Evaluating education events
Evaluating the performance of a function should be an essential stage of any policy cycle and is crucial to informing decision-making, monitoring progress, justifying budgets, and improving design and implementation (see Althaus et al., 2013; Bardach, 2012; Kirkpatrick and Kirkpatrick, 2006: 16–17). Table 1 draws on a review of the academic literature to present a typology of approaches to evaluating and reporting the effectiveness of education events conducted as part of the education function. It is important to note that each approach has benefits and shortcomings in evaluations. However, in this study these approaches inform the development of a retrospective classification scheme to categorize and understand the scope of evaluations undertaken and publicly reported by commissions; that is, it is not used to evaluate the effectiveness of the education function itself. To this end, the various approaches offer a useful “vocabulary” (Alliger and Janak, 1989: 331) in seeking a deeper understanding of how commissions evaluate their education events.
Categorizing types of evaluation approaches.
Sources: This table draws on the work of Kirkpatrick (1975: 1–17); Kirkpatrick and Kirkpatrick (2006); Holton (1996); Phillips et al. (2007); Steele et al. (2016); Pallai and Gregor (2016); Alliger and Janak (1989); and Arney (2017).
In the 1950s Donald L. Kirkpatrick developed a four-level approach to education evaluation that focused on measuring reaction, learning, behavior, and results (see Kirkpatrick, 1975: 1–17; Kirkpatrick and Kirkpatrick, 2006). This approach became increasingly well-known during the following decades and is still widely cited today (Phillips et al., 2007: 8). Although “elegant in its simplicity” (Holton, 1996: 6–8), Kirkpatrick’s approach is hierarchical and critics argue it suggests causal relationships between the categories without accounting for other variables (Alliger and Janak, 1989; Holton, 1996: 6–8). Nevertheless, in Holton’s (1996: 9) alternative “outcome measures” approach, three of Kirkpatrick’s levels are recognized as primary considerations, albeit in a framework that is conceptually very different: learning, individual performance, and organizational results. Acknowledging that there is a range of worthwhile education objectives and available resources to achieve them, Table 1 is not intended to adhere to a strict hierarchy, but to broadly describe types of evaluation.
During the 1980s Jack Phillips highlighted an additional objective by separating—and, therefore, emphasizing the importance of—return on investment from business results (Phillips et al., 2007: 8–9). Measuring a return on investment in education is controversial, with some questioning the extent to which it is possible (see Phillips et al., 2007: 3). According to Phillips and colleagues (2007: 23–24), the use of control groups and trend lines are among the strategies that can help ensure reliability. In Table 1 the delivery of public value is used to underscore the purpose of the public sector in delivering value for the public while acknowledging that resources are limited and no function can occur at any cost. In addition to the categories that feature most prominently in the literature it is necessary to include an exposure category in Table 1. In the exploratory survey detailed below, the compilation and reporting of statistics to show the number of people exposed to education events as indicative of effectiveness is striking.
Exploratory survey of cases
Australia’s six state-based public sector standing anti-corruption commissions broadly adhere to the multi-pronged approach popularized by the Hong Kong Independent Commission Against Corruption established in 1974. Although there are variations, they all have functions in investigation, prevention, and education. The precise nature of the education function is somewhat vague, with terms such as education, training, and awareness often used interchangeably. Public sector education events, such as seminars, workshops, and training sessions, are a prominent part of how commissions undertake this function and are commonly the subject of evaluative statements in public reporting. Nevertheless, the education function encompasses a range of other elements including giving advice and guidance to agencies, developing educational resources, and disseminating information via newsletters, web pages, and similar.
The six commissions examined in this survey are the New South Wales Independent Commission Against Corruption (NSW ICAC), the Western Australian Corruption and Crime Commission (WA CCC), the Queensland Crime and Corruption Commission (QLD CCC), the Tasmanian Integrity Commission (TAS IC), the Victorian Independent Broad-based Anti-corruption Commission (VIC IBAC), and the South Australian Independent Commissioner Against Corruption (SA ICAC). Each of these commissions delivers education events to public sector participants in their respective jurisdictions. The participants targeted in each event may be general or role specific, while the audience may be assembled through a variety of approaches. The main documentary sources informing this exploration are annual reports. They are the essential reporting tool for accountability and transparency and play a key role in communicating each commission’s activities, performance, and plans to the executive, legislature, and public. This is supplemented by legislation and commissions’ websites where necessary. Due to variations in jurisdiction a comparison of resources is not included.
NSW ICAC
The first annual report of the NSW ICAC provided to parliament in September 1989 states that the purpose of its work is “to minimize corruption in the public sector” (NSW ICAC, 1989: 12), a statement that is consistent with the underlying assumption supporting the function today. In the early years, education efforts focused on general public awareness of the commission’s establishment (see NSW ICAC, 1989: 50–53, 1990: 73). Specific public sector education events began with a seminar for all NSW government chief executives in June 1990 and this event provided the basis for a roll out of similar seminars for other officers in the years that followed (NSW ICAC, 1990: 71–72, 1991: 83–84). Reporting of the function mainly involved the inclusion of lists of these events to demonstrate occurrence, which infers exposure (see, for example, NSW ICAC, 1996: 63–66). Their primary goal was cognitive, that is, to promote “better knowledge and understanding of probity and accountability” [original emphasis] (NSW ICAC, 1995: 12).
The education function developed over time, but the evaluative reporting remained largely unchanged. Broad-brush evaluative contentions are frequently made without any substantiating details. In 1996 the annual report makes cursory mention of a “performance review” of the regional seminar program, reporting that there were “high levels of acceptance of the corruption prevention message” and that “the increasing demand for more specialised information is an indication of the Commission’s previous success in disseminating the basic corruption prevention message,” but provides no explanation of the process undertaken to reach these conclusions (NSW ICAC, 1996: 13). Similarly, the 1997 report makes the following statement: “The effectiveness of this work is measured through the positive results of strategic sector-wide projects and the adoption by central agencies of ICAC initiatives to promote integrity in policies and procedures” (NSW ICAC, 1997: 13). No evidence is provided to support the claim of positive results, while the adoption of initiatives says nothing of what they actually achieve.
By the late 1990s mentions of participant reaction survey results begin to appear frequently, albeit still with little explanation or context (see examples NSW ICAC, 1998: 44, 1999: 38). The key indicators cited from the reaction surveys are typically usefulness and meeting training needs (see examples NSW ICAC, 2012: 46–47, 2013: 38). Exposure statistics also continue to appear alongside these reaction indicators (see examples NSW ICAC, 2004: 43–45, 2005: 39, 2008: 59, 2012: 46–47), although the inclusion of event lists ceased after 2009. Cognitive learning-based goals remain at the forefront of the function with the aims of increasing reporting of corrupt conduct and improving investigations through knowledge dissemination, both of which are said to elevate the risk of detection and thereby act as deterrents (NSW ICAC, 2008: 56). The number of evaluations undertaken was reduced in 2013 due to the “consistently high ratings…received in recent years” and the need to “reduce the administrative load such evaluations create” (NSW ICAC, 2013: 38).
Most recently, the NSW ICAC’s annual report for 2017–2018 focuses on exposure statistics and reaction surveys: In 2017-18, the Commission delivered 126 workshops to over 2,300 attendees…Of the 126 workshops delivered in 2017-18, 104 were evaluated to ensure quality. A total of 94% of participants rated these workshops as “useful” or “very useful” and 96% of participants “agreed” or “strongly agreed” that these workshops met their training needs. (NSW ICAC, 2018: 34) how transferable the learnings from the program are to their workplace—average rating 4.2 the extent that the course’s learning environment and content were stimulating—average rating 4.2 the course overall—average rating 4.3 the presenters—average ratings 4.5 (NSW ICAC, 2018: 36)
WA CCC
In 2002 the Kennedy royal commission 2 recommended that Western Australia’s Anti-Corruption Commission be replaced by a new Corruption and Crime Commission (WA CCC) with an added function of prevention and education in order to “lift the level of awareness…of the existence of potential problems” (Kennedy, 2002: 104). This occurred on 1 January 2004. Initially, the education function focused on introducing the commission including its roles and responsibilities before shifting attention to specific issues and customized sessions (WA CCC, 2006: 45). Reporting of the function between 2004 and 2015 was heavily skewed to exposure statistics (see examples WA CCC, 2006: 44, 2011: xx, 2013: 8) with some brief mentions of pre-event consultations, reaction-based feedback, and self-reported application of learning responses (see examples WA CCC, 2005: 32, 2007: 16, 2009: 19).
Legislative changes that came into effect on 1 July 2015 shifted the primary responsibility for education to the state’s Public Sector Commission (WA PSC) (see Corruption, Crime and Misconduct Act 2003 , as amended). This was part of wider reforms to focus the WA CCC on serious matters, with the government arguing that the education function would complement the WA PSC’s existing responsibilities (Day, 2014: 1960). Since the changes were introduced the WA CCC has continued to provide education on serious misconduct and corruption while also working collaboratively with the WA PSC to offer joint education events (WA CCC, 2016: 6, 41). Reporting on the revised education function has only cited exposure statistics (see WA CCC, 2016: 41, 2017: 44, 2018: 50). In its 2017–2018 annual report the WA CCC (2018: 50) provides exposure statistics indicating that the commission delivered three joint sessions with the WA PSC to about 140 participants, while the WA CCC commissioner made a further eight presentations to about 460 attendees.
QLD CCC
The Queensland Criminal Justice Commission (QLD CJC) was established in late 1989 with an education function; however during the set-up phase it operated with only “skeleton staff” and was overwhelmed with complaints—indeed, the 1991 annual report indicates that there were 112 police matters per investigating officer (QLD CJC, 1991: 17). As such, the corruption prevention section did not commence work until August 1991 (QLD CJC, 1992: 46). Nevertheless, each annual report provides a list of events—again, inferring exposure—beginning with public addresses (see QLD CJC, 1990: 46) and quickly expanding to include workshops (see QLD CJC, 1991: 97) and lectures and presentations (see QLD CJC, 1992: 121–128). Exposure statistics dominate the reporting (see, for example, QLD CJC, 1997: 42); however, some mentions of reaction survey results do appear in the annual reports. In 1994 the indicator measured is “helpfulness” (QLD CJC, 1994: 50), while in 1995 and 1997 it is “practicality” (QLD CJC, 1995: 79, 1997: 42).
In January 2002 the QLD CJC merged with the Crime Commission to form the Crime and Misconduct Commission (QLD CMC) and the new Crime and Misconduct Act 2001 shifted greater responsibility onto police and public sector managers to prevent and respond to misconduct matters within their workplaces (QLD CMC, 2002: 3). Thus, the QLD CMC’s (2002: 56) new focus was in capacity building through collaboration with agencies, although it continued to conduct education events (see QLD CMC, 2003: 39). The reporting of these activities throughout the existence of the QLD CMC was limited, with exposure statistics the main information provided (see examples QLD CMC, 2004: 44, 2012: 28). An exception is the 2009 annual report that makes the following bold statement about training effectiveness: “Positive feedback from all sessions suggests that the attendees increased their knowledge” (QLD CMC, 2009: 30). There is no evidence provided to support this suggested link between positive sentiments and actual learning.
In 2014 the Liberal National government led by Campbell Newman removed the renamed QLD CCC’s prevention functions as part of wider reforms to the commission (see further Briody, 2015). It argued that the Public Service Commission (QLD PSC) would “provide the requisite advice, support and assistance to public sector agencies” (Bleijie, 2014: 1342), although it was highlighted that some agencies within the QLD CCC’s jurisdiction sit outside the QLD PSC’s area of operations, for example local government and universities (Pucci, 2014: 1372). The change, although passed, was not enthusiastically embraced by parliamentarians, with some Liberal National Party members expressing their support for the retention of a prevention function (see, for example, Davis, 2014: 1377). The basis for the reform was provided by the Callinan–Aroney review that reported to the government in 2013. The effectiveness of the education function does not appear to have been central to its considerations. Instead, the multiplicity of agencies involved in integrity, including the Integrity Commissioner, Ombudsman, QLD PSC, Electoral Commissioner, QLD CMC, and internal divisions within individual agencies underpinned the argument to refine their responsibilities (Callinan and Aroney, 2013: 137). Particular attention was drawn to the Ombudsman’s acknowledgment that its education function overlapped with that of the QLD CMC (Callinan and Aroney, 2013: 141). Callinan and Aroney (2013: 216) argued that the reform would mean “the emphasis upon unnecessary and duplicated integrity education will reduce.” Further: The existence of overlapping functions and the multiplicity of institutions have been explained on the basis that it is part of the “integrity regime”…to view the arrangements in this manner is to risk assuming that “more is better”, rather than critically to assess their merits and especially their efficiency, utility and necessity. (Callinan and Aroney, 2013: 142–143)
TAS IC
The TAS IC has reported exposure statistics from its inception (see examples TAS IC, 2011: 12, 2012: 18). The use of reaction surveys is mentioned from 2013 (TAS IC, 2013: 23–24). The indicators reported are completion of previous training and self-reported opinions of understandings gained in training (TAS IC, 2013: 23–24, 2014: 27, 2015: 20). Since 2013 the commission has focused on producing generic resources with a number of modules developed in an Ethics and Integrity Training Program for delivery within, and by, government agencies themselves, allowing the TAS IC to target its education in priority areas (TAS IC, 2014: 24, 27, 2017: 30). In 2018 education events targeted agencies that were subject to investigations, reaching more than 2500 participants (TAS IC, 2018: 25). Reporting of exposure statistics is comprehensive in the 2017–2018 annual report, but no other form of evaluation is mentioned (TAS IC, 2018: 25–30).
VIC IBAC
Initially the VIC IBAC focused on raising general awareness, before shifting toward a more targeted education approach (VIC IBAC, 2016: 31). From its inception in 2012, VIC IBAC has reported the number of “corruption prevention initiatives,” including training sessions, speaking engagements, and prevention resources, delivered—inferring exposure—and a “satisfaction rating,” although no detail about the methodology for what are presumably reaction survey results is provided. The results show little variation over the years: 2012–2013: 72 initiatives at 98.9% satisfaction; 2013–2014: 74 initiatives at 96.7% 3 satisfaction; 2014–2015: 105 4 initiatives at 95% 5 satisfaction; 2015–2016: 78 initiatives at 99% satisfaction; 2016–2017: 93 initiatives at 99% satisfaction; 2017–2018: 92 initiatives at 99% satisfaction (VIC IBAC, 2013: 17, 2014: 11, 2015: 1, 27, 2016: 10, 2017: 10, 2018: 10). Additional exposure statistics are also provided in some reports (see, for example, VIC IBAC, 2014: 27–28). The annual reports also refer to surveys regarding self-reported knowledge before and after training on protected disclosures. For example, the 2012–2013 annual report mentions a “self-rated assessment of knowledge pre-and-post seminars”—presumably part of a reaction survey—that indicated a “strong improvement in knowledge” (VIC IBAC, 2013: 17).
SA ICAC
Given its short period of existence, the SA ICAC has only produced four annual reports. Exposure statistics are dominant in each report, with limited reaction survey results and a few selected comments attributed to participants provided (see examples SA ICAC, 2015: 37, 2017: 43). The latest annual report states: 2017-18 saw 66 education sessions delivered to 2,592 public officers, exceeding our yearly goals of 60 education sessions reaching 2,000 public officers. The commissioner personally presented 15 sessions to 1,207 attendees…Education events consistently receive excellent feedback. (SA ICAC, 2018: 46)
Discussion
This exploratory survey suggests that the unquestioned assumption that integrity education will reduce levels of misconduct and corruption is long-held and pervasive among these commissions. It also suggests that there has been minimal—if any—parliamentary or public expectation that the accuracy of such an assumption be tested through meaningful evaluations of the commissions’ education function. Indeed, beyond a broad directive to improve integrity and reduce misconduct and corruption, it is not clear that the legislatures that have given commissions an education function have clear objectives in mind. Is the goal to raise awareness of the role of commissions, to nudge broader cultural change, or to provide a substantial integrity outcome that influences attitudes and changes behavior? Additionally, the question of whether commissions are the agencies that are best placed to deliver this function is unresolved. Currently a range of agencies provide integrity education and this overlap of responsibilities risks churning public sector workers through a maze of superficial and disconnected educational requirements. An emphatic “more is better” view appears to have been broadly adopted, limited only by resources and an occasional political intervention as occurred in Queensland.
The dominant evaluation approaches used are exposure statistics and reaction surveys, with a few cursory mentions that indicate some consideration has been given to learning and the application of learning. Organizational impact and delivery of public value evaluations are absent. The dominance of exposure statistics seems to suggest that demonstrating the productiveness of education staff is a high priority. However, it is concerning that this exposure-as-success approach substitutes, in large part, for more meaningful evaluations of performance. As Alliger and Janak (1989: 333) note, there is no automatic link between attending and learning. Similarly, participant reactions are a weak indicator of effectiveness (Dixon, 1990: 129-130; Pallai and Gregor, 2016: 247), commonly exacerbated by the distribution of survey forms at the end of an event, inviting rushed and ill-considered responses (Arney, 2017: 221). Nevertheless, a positive participant experience can contribute to the effectiveness of an education event, including by increasing learning motivation, interest, and attention; sustaining the future of the event; and signaling the value of participants to the trainer in a way that fosters a mutually supportive relationship (see Holton, 1996: 10; Kirkpatrick and Kirkpatrick, 2006: 22, 27). However, as Holton (1996: 8) argues, although a positive and enjoyable participant experience can act as a “mediator” of successful learning, it is not directly related to, or indicative of, that outcome (see also Alliger and Janak, 1989). Similarly, research indicates that humor does not produce learning, despite being clearly favored by participants (Alliger and Janak, 1989: 334).
Reaction surveys tend to focus on edutainment demands that are increasingly influencing how education is delivered in all settings (see O’Brien, 2015; Wong and Chiu, 2019). As such, it is no surprise that these surveys are often referred to as “smiley” or “happy” sheets (Arney, 2017: 241; Kirkpatrick and Kirkpatrick, 2006: 41; Pallai and Gregor, 2016: 248). O’Brien (2015) argues that the habitual use of technological devices is creating new demands for all facets of life to be entertaining in order to overcome attention deficits and distractions. In reality, learning can be difficult and, as Holton (1996: 10) suggests, confusing and frustrating, in various stages. Indeed, Alliger and Janak (1989: 334) note that some research has indicated a possible negative link between reactions and learning and, as such, they go so far as to suggest “perhaps it is only when trainees are challenged to the point of experiencing the training as somewhat unpleasant that they learn.” Further, edutainment demands in the context of paid employment seem to suggest that such learning opportunities are held in low esteem.
In Pallai and Gregor’s (2016) study of a Hungarian integrity training program, the weakness of reaction survey results in comparison to learning assessments is demonstrated. Participant satisfaction with some trainers is rated at or above 3.85 out of 4 on reaction surveys, even though learning assessments demonstrate they produced “minimal to zero learning impact” (Pallai and Gregor, 2016: 249, 263 at note 4). Indeed, if trainer selection for this program was undertaken on the basis of reaction survey results, two of the nine most effective trainers could be rejected due to having lower satisfaction scores than some of their less effective but more popular colleagues (Pallai and Gregor, 2016: 263–264 at note 9). Similarly, Dixon’s (1990: 136) study of workplace education events related to a new manufacturing process in the United States found no correlation between trainer ratings and learning, further suggesting reactions reflect enjoyment rather than effectiveness.
Additionally, Dixon’s (1990: 135) study found no correlation between participants’ self-reported perceived level of learning in reaction surveys and either the raw test results for learning or the pre- to post-education learning gains shown in independent assessments. As Dixon (1990: 135) acknowledges, “this finding is counterintuitive, in that it is commonly assumed that individuals know whether or not they have learned.” However, Dixon (1990: 135) speculates that the perceived level of learning may be significantly influenced by enjoyment; that is, when participants have enjoyed an education event, they believe they have also learned a lot, or that they seek to reward the trainer regardless. Both studies highlight the weakness of reaction-based evaluations and, as Pallai and Gregor (2016: 262, 263 at note 4) suggest, highlight the need for a “more sound basis” than participant “self-reports” upon which to assess education events.
A lack of serious evaluation may also suggest that the education function provides false confidence and assurance—to agencies, commissions, politicians, and the public—and possibly serves a branding purpose (see, for example, Broad, 2018). Arney (2017: 219) warns against a tendency to use workplace education as a “comfort blanket, offering warmth and protection” without ensuring it is fit for purpose and the results are reliable. Importantly, understanding how learning and the application of learning occurs—if at all—provides valuable insights to improve the function as well as helping agencies to understand broader workplace issues such as relevance of courses, team morale, culture, motivation, and resistance (see Arney, 2017: 226; Lombardo and Mergaert, 2016: 43–61; Parks, 1993: 27–29, 52). Additionally, it can ensure that education develops participants’ agency in integrity matters rather than producing fear or paralysis (see Cochrane, 2018: 512).
The exploratory survey revealed no evidence of more robust approaches to evaluation being implemented even as the education function itself has evolved. Although it is possible that annual reports do not reveal all the evaluative work that is carried out regarding education events, what is—and is not—publicly reported provides a valuable insight into the priorities of commissions. Modern transparency expectations also suggest that the best available evidence of the performance of the education function should be publicly reported. Additionally, all commissions fail to provide sufficient details about the methodological approaches used in the evaluations that are undertaken in order for independent assessments to be made.
Without any track record of alternative approaches being used by any commission in this study, a precise prescription for the way forward is fraught. This study has not tested any evaluation approaches upon which to base recommendations. Instead, there are ample reasons for the commissions to develop and experiment with new strategies for meaningful evaluations adapted to suit the circumstances of their education function. Of course, such a process will not be without significant challenges. As Arney (2017: 227–28) highlights, scarce resources and a lack of appreciation for the value of evaluation among decision-makers hamper efforts. Indeed, commissions have indicated that they face many difficulties in undertaking evaluations (see, for example, NSW ICAC, 2009: 63). However, this should not prevent existing resources being directed to higher-quality evaluations even if this can only occur on a smaller resource-intensive scale compared to current approaches.
While more complex evaluations may still not account for every variable, the aim should be to produce credible “evidence”—rather than “proof”—of the effectiveness of educational events [original emphasis] (Kirkpatrick and Kirkpatrick, 2006: 65–66, 70). Some inevitable ambiguity in evaluating learning, application of learning, organizational impact, and delivery of public value does not diminish the benefits of pursuing deeper insights. To this end, Patton’s (1997: 4) framework for “utilization-focused evaluation” provides a solid foundation for developing broader strategies that produce useful and meaningful results, while Arney (2017, particularly chapter 6) provides an overview of various approaches that may be useful for particular purposes. Additionally, Steele and colleagues (2016) ask “how do we know what works?” and, in doing so, offer pertinent insights and inspiration for pursuing more meaningful evaluation approaches. Modern technology creates many user-friendly options for evaluating learning, including the use of case studies or test scenarios, while interviews and/or systematic observations can focus on the application of learning, organizational impact, and delivery of public value.
Conclusion
This study has conducted an exploratory survey of how commissions seek to demonstrate the effectiveness of their education function through evaluations and reporting. No claim is made as to the effectiveness of the commissions’ education function itself, but rather that the current evaluative and reporting approaches are insufficient to demonstrate that the education events contribute to achieving the function’s purpose. The dominance of reporting exposure-as-success attendance statistics revealed in this study is particularly troubling, while the heavy reliance on reaction-based surveys—although consistent with the findings of previous studies (see Steele et al., 2016: 32)—demonstrates a narrow approach to evaluation. Although there are some cursory mentions of evaluating learning and application of learning outcomes, these need further development as part of a broader strategy that shifts resources from superficial and repetitive reaction surveys to options that produce more meaningful evidence of performance.
It is hoped that this study stimulates further debate about how integrity education occurs in the public sector and how the performance of this function is evaluated and reported. As Pallai and Gregor (2016: 248) argue, agencies delivering publicly funded education should be obliged to demonstrate its effectiveness with meaningful evidence. More complex evaluations will require greater resources and effort, but if undertaken in a strategic and carefully targeted manner they offer much better prospects of contributing to improving the delivery of integrity education and, in turn, reducing misconduct and corruption in the public sector. As Krathwohl and colleagues (1964: 79) caution: Such objectives are not to be attained simply by someone expressing the desire that they be attained…when [they are] not fully committed to doing the necessary work for the development of appropriate learning experiences and evaluation evidence…Such objectives should not be stated lightly and expected to be achieved by some semi-magical process…Stating an objective and achieving it are two very different things.
Footnotes
Acknowledgements
This article benefited greatly from the comments and encouragement of the organizers and participants at the APSA Policy Studies Research Group workshop “Teaching Public Policy” and of colleagues and peers. Thank you to Dr Siobhan O’Sullivan and Dr Yvonne Haigh, as well as anonymous reviewers, for their comments on earlier drafts of this article.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by an Australian Government Research Training Program scholarship. Travel to the APSA Policy Studies Research Group workshop “Teaching Public Policy” in Sydney was provided by APSA workshop funding.
