The Continuous Learning Framework: Applying accountability for widespread organisational change

Introduction

This article introduces the Continuous Learning Framework (Alderman, 2014) for quality within higher education and other internal evaluation contexts which acts as a scaffold for this special issue of the Evaluation Journal of Australasia. The Continuous Learning Framework was first formulated in a study which was a program evaluation of a series of Australian government reforms in higher education during the 2000s (Alderman, 2014, 2015a, 2015b, 2016). The framework consists of four elements of quality: accountability; improvement; performance and investment. After a brief overview of the development of the framework, this article discusses examples of the practical application of the quality element of accountability. The three other articles in this special issue explore the framework’s elements of improvement (Forbes et al., 2022, this issue), performance (Harris & Alderman, 2022, this issue) and investment (Quadrelli et al., 2022, this issue).

This article and the special issue more broadly seek to bring the role of the internal evaluator into the spotlight. There is a wealth of literature that documents how to conduct evaluation and the ways evaluation adds value to program appropriateness, effectiveness, efficiency, impact and sustainability (for example, Owen & Rogers, 1999; Parlett & Hamilton, 1972, 1976, Patton, 1975, 2002, Scriven, 1991, 2013; Weiss, 1983). In contrast, much less guidance is available about when to choose an internal or external evaluation (Conley-Tyler, 2005; Scriven, 1996), or about the definition of an internal evaluator’s role (Beere, 2005; Kelly & Rogers, 2022; Rogers et al., 2019a, 2019b; Rogers & Gullickson, 2018). Internal evaluators are employed within a government department, university or non-profit organisation to pursue quality. As identified by Rogers et al. (2019a), non-government organisations with an evaluation literate workforce who adopt evaluative thinking are more likely to achieve the greatest social good. In the case of universities, this social good takes the form of assuring the quality of the student experience, which is of paramount importance to the sector and regulated pursuant to the Tertiary Education Quality Standards Agency Act 2011 (Cth).

For the purposes of clarity, this article adopts the definition of evaluation from the National Health and Medical Research Council (2014) and the Australian Evaluation Society. The National Health and Medical Research Council adopted the definition of evaluation from the Australian Evaluation Society’s (2013) Guidelines for the Ethical Conduct of Evaluations:

Evaluation is a term that generally encompasses the systematic collection and analysis of information to make judgements, usually about the effectiveness, efficiency and/or appropriateness of an activity. The term is used in a broad sense to refer to any set of procedures, activities, resources, policies and/or strategies designed to achieve some common goals or objectives (p. 2).

Thus, internal evaluators and evaluating thinking and practice, informed by ethical principles, are an asset to government, organisations, teams and individuals. Internal evaluation is highly relevant to the higher education sector, as the sector is publicly funded and subject to external Australian Government regulation. As outlined in the next section, the Continuous Learning Framework was originally devised from an evaluation research study by the author.

Initial development of the four elements of quality in higher education

An outcome of the evaluation research study (2014), which focused on the 2003 government reform of higher education, was the analysis of three key national initiatives: the Learning and Teaching Performance Fund, Australian Learning and Teaching Council and the Australian Universities Quality Agency. This evaluation research study was able to classify the activities deployed by these initiatives into four elements: improvement, performance, accountability and investment. The first cycle of the three national initiatives were undertaken between 2002 and 2008 and was intended to be complementary in reshaping a focus on learning in teaching within the sector, thereby reducing the emphasis on research.

The four elements of quality in higher education that emerged through the evaluation research study are shown in Figure 1.OPEN IN VIEWER

To further expand on each quality element, this article will discuss the application of the framework to a government reform package and then to a university setting. The first application details the three national initiatives and the specific activities that were implemented to place a spotlight on learning and teaching in the Australian higher education sector. The second application will discuss how the resulting Continuous Learning Framework from the evaluation research study was operationalised in a specific university setting to address a significant external driver for change, bring about widespread organisational change and improve the quality of the student experience.

Continuous Learning Framework applied to a government policy level

In 2002, the Australian Federal Government established a review of the higher education sector. The government prepared a response to this review authored by then Federal Education Minister Brendan Nelson and titled Our Universities: Backing Australia’s Future (2003). This response includes a suite of reforms, budget outlines and timelines for implementation (the Nelson reforms). The Nelson reforms include a strong focus on learning and teaching enabled through the implementation of three national initiatives, and the first cycle of each initiative formed part of the author’s (Alderman, 2014) evaluation research study:

(i) Learning and Teaching Performance Fund (LTPF) 2003–2006 (Department of Education, Employment and Workplace Relations, 2009; 2010; 2011b, 2011c, 2011d, 2011e, 2011f, 2011g, 2011h, 2011a).

During this time, the Higher Education Support Act 2023 (Cth) allowed for self-regulation within the higher education sector. The focus of the evaluation research study was the government’s ambition to bring learning and teaching to the forefront of higher education and to highlight the importance of quality student learning experiences. The author theorised and documented the impact of the three national initiatives and found observable evidence that they were successful in meeting their objectives. This article is built on the outcomes of this evaluation research, which are well documented in other articles (Alderman, 2014, 2015a, 2015b, 2016).

What emerged from this evaluation research was a clear understanding that the government reform made a commitment to four distinct elements of quality in higher education: accountability, improvement, performance and investment. The following outline maps the three national initiatives against these elements.

Reflecting on Deming’s cycle of quality improvement in higher education

Over the last 30 years, the term ‘continuous improvement’ has become a catch cry in the higher education sector. For internal evaluators, the pursuit of ‘continuous improvement’ appeared to be a pursuit with no end due to the lack of direction for continuous learning and unseen client satisfaction (Alderman, 2014). Without systematic analysis of initiatives and reforms, it was unclear which desired outcomes were being pursued within higher education. The lack of definition for what constitutes ‘quality’ meant that quality was always an elusive pursuit for those working in the higher education sector. This issue piqued the interest of the author to go back and look at how quality is defined and how quality and evaluation emerges in the literature. This is where Deming’s (1986) contribution to theory became an important foundational building block in the development of the Continuous Learning Framework.

In terms of the pursuit for quality improvement, Deming’s theory of an ongoing cycle over time remains a valid construct today. Deming’s (1986) Quality Cycle of Plan, Do, Check and Act (PDCA) has an x-axis of Time and a y-axis of Quality, which come together to focus on Standardisation. In Deming’s cycle, each stage is designed to increase customer satisfaction (Reid et al., 1999). Deming became a recognised leader in quality assurance but avoided providing a clear definition of what constitutes ‘quality’. Rather, he focused on improving quality through minimising uncertainty and reducing variability (Redmond et al., 2008). For example, Deming believed that by studying the variables contributing to a product, it was then possible to gain improvement in the overall quality.

It was interesting for the author to go back and read the original literature and unpack the different interpretations of Deming’s work. For example, Moen (2009) described the foundation and history of the plan, do, check and act cycle for learning and improvement. Moen and Norman (2006) expanded on this work by detailing the evolution of the cycle or Model for Improvement. Despite these efforts, however, somewhere along the way something was lost. In his original work, Deming was interested in developing a model to support continuous improvement leading to continuous learning, with the goal being the pursuit of client satisfaction (Deming, 1986). In the literature, there was a focus on continuous improvement when considering Deming’s work, and this meant there was little understanding that Deming’s ultimate goal was continuous learning.

This notion of continuous learning resonated with the author as something that was achievable and highly amenable to embedding within university settings, given their raison d'être as institutions of higher learning. Further, the cyclical nature of the plan, do, check and act cycle aligns with Kemmis and McTaggart’s (1988) action research spiral – a well-known methodology in education research and evaluation. For example, evaluators examine the outcomes of human service programs, including universities who deliver educational programs in reference to this cycle (Tyler, 2013). Therefore, the work of Deming (1986), who also promoted the idea of moving beyond ‘continuous improvement leading to continuous learning’ to becoming a learning organisation, makes a great deal of sense for universities and internal evaluators.

Development of the Continuous Learning Framework

As shown in Figure 2, the Continuous Learning Framework builds directly on Deming’s (1986) quality improvement cycle (plan, do, check, act). The axes of quality and time remain stable together with standardisation, also known as ‘results maintenance’. The plan, do, check and act stages are deliberate in nature to remind those who are interested in quality in higher education that planning, delivering, checking to see if the delivery is working and acting on improvements will lead to an improvement in quality. These actions resonate with evaluators who build monitoring and reporting frameworks as part of the evaluators’ tools of trade (Markiewicz & Patrick, 2015).OPEN IN VIEWER

This framework therefore emerges from a blending of the Alderman’s four elements of quality in higher education and Deming’s plan, do, check and act cycle. It is deliberate in its structure to reaffirm Deming’s pursuit of ‘quality improvement leading to continuous learning’.

This framework is designed as an internal evaluator’s tool to finding the when, where, why and how to improve quality, guiding strategic decision-making and leading an organisation towards becoming a learning organisation. It offers an opportunity to determine the level and appetite of policy makers and decision-makers for strategic decision-making, leading to widespread organisational change. Although the Continuous Learning Framework is positioned as a hierarchy, it has the flexibility for stakeholders to step into the framework at any point.

The framework’s purpose is to separate and articulate the ways a quality model can be achieved through elements of quality: accountability, improvement, performance and investment (Alderman, 2014). Within the academic literature, these four elements are individually represented, and everyone would agree that they each hold face value. When combined in the Continuous Learning Framework, however, there is an emerging hierarchy from accountability to investment with a common assumption that many organisations understand the notion of accountability and hold a strong concept of what constitutes performance. The strength of this framework is that it sends a clear message that all four quality elements are important to becoming a true learning organisation.

Thus, the Continuous Learning Framework offers internal evaluators, government, organisations and individuals a structured approach to step into quality and progress towards becoming a continuous learning organisation.

The next section will outline how the Continuous Learning Framework was applied to a university setting by an internal evaluation team, with a specific focus on the quality element of accountability.

Continuous Learning Framework applied to a university setting

Following the 7-year review of the University of Queensland in 2017, TEQSA placed a condition on the university’s re-registration regarding governance and accountability reporting obligations. The external regulator was specifically interested in the university demonstrating its academic board received comprehensive diagnostic analysis for all higher education programs under offer (TEQSA, 2018).

The author was initially employed by the university as an external evaluation expert and was then employed as the Director (Academic Quality Unit) and now Dean (Academic Transformation). This pathway of employment is one way to demonstrate the university’s ongoing commitment to improving the quality of the student experience and improving the data that inform strategic curriculum decision-making.

The design, development and implementation of the University of Southern Queensland’s Academic Quality Framework (Alderman, 2019) was underpinned by the Continuous Learning Framework. The emphasis across this work was placed on data and the provision of appropriate information at each level of conversation held across the university, which is directly aligned to the condition on re-registration. Each year had a theme, with systemic use of data for year one, embedded quality conversations in year two and sustainable practices in year three. These three years were then supported through the introduction of standardised reporting at survey, course, major, program and university levels and leading into a five-year curriculum accreditation cycle.

The very early conversations held at the university were focused on where data was stored, who the data custodians were, and where the best place was for providing a single repository where academic staff could go to one place to locate the reports. There were a series of activities placed within each element of quality. These are unpacked below.

The University of Southern Queensland’s Academic Quality Framework (2019–2022) placed deliberate emphasis on the availability, collection and use of data. Although this is the framework presented to stakeholders across the university, it was theoretically underpinned by the Continuous Learning Framework. As the body of work progressed, emphasis moved to implementation of widespread organisational change.

Conclusion

The author has presented the design and development of the Continuous Learning Framework as emerging from an evaluation research study on government reform in the higher education sector, and then demonstrated its application in a university setting. The framework is founded on the Alderman’s (2014) evaluation research study’s four elements of quality in higher education and Deming’s (1986) plan, do, act and check cycle. Although the Continuous Learning Framework was designed through the discipline lens of higher education, the notion of quality, unpacked into four distinct elements, is transferable to all disciplines. Therefore, this framework has utility as an internal evaluator’s tool to step into strategic discussions about quality for government, organisations, teams and individuals. In circling back to the beginning of this article, this strategic decision-making tool offers internal evaluators an opportunity to pursue quality in a structured manner to meet its external obligations.