Abstract
The final issue of the Evaluation Journal of Australasia for 2023 brings together a collection of articles that speak to the transformative role of evaluation and evaluators. My mind goes back to Donna Mertens who comments: ‘Evaluators have a role to play in bringing their data into communities in ways that facilitate its use for social transformation' (Mertens, 2010, p. 9). While it might be true to say ‘evaluation can’t change the world’, many of us have built our work as evaluators on the premise that we can make a difference. Indeed, there are plenty of examples within the Journal where articles show that change was both the intention and outcome of an evaluation (e.g., Harrison, 2023; Rijswijk et al., 2015). Change is front of mind for many of our authors in this issue, whether that be organisational change, system change, change in the way evaluations are commissioned or advocacy for social change more generally. Mertens also speaks of a transformative lens in evaluation (Mertens, 2023), suggesting a role for evaluation in education, mental health, environmental policy, criminal justice systems, refugees, economic development, gender-based violence and agriculture. While evaluation certainly does not necessarily result in change, this issue of the Evaluation Journal of Australasia points to several possibilities.
Alison Rogers and Amy Gullickson offer insights into how individuals can encourage embedded evaluation within their non-profit organisations. Their study shed light on strategies used by evaluation advocates to overcome resistance within organisations. Those of us who have worked with non-profit organisations would have experienced many of the challenges reported in this article. However, this research reports on what people within an organisation do to build an evaluation culture. They comment that for some, ‘their prodding, suggesting, cajoling, promoting, prompting, and encouraging were what kept things rolling' (p. 186). Notably, it seems that people with more influence can use other levers to embed a culture of evaluative inquiry. These include an accountability strategy, and an intentional process of supporting capacity building, encouraging evaluative thinking and cooperative teamwork. The authors conclude with a series of theoretical propositions that could be tested further within the frame of social interdependence theory. However, as I read the findings it seems to me that embedding evaluation depends on the agency and motivation of individuals, either as leaders or as passionate practitioners.
In their practice article, Lucas Moore and Reno French present an argument for the use of Action Research methods as a way of respecting Aboriginal and Torres Strait Islander knowledges and ‘to combat the colonising discourse of mainstream research and uphold the principles of Indigenous Data Sovereignty' (p. 216). The context of their work is child protection, which is one of several important issues that in my mind requires community involvement to find solutions to complex problems. Participatory research methods have long been used for these purposes, particularly in education (Kemmis et al., 2014; White, 1988) and community development (see appendices in Haviland & Pillsbury, 2012; Stringer, 2014). One of the positive benefits of this approach within evaluation, as identified by the authors, is the concept of ‘working in partnership' (p. 215), which allows First Nations people to work together in an equitable and decolonising way. The authors however identify that Western structures often fail to allow sufficient resources and time for research completion. Further, the possibility of a ‘decolonised child protection system' (p. 212) is acknowledged to be fraught with challenges that will require a lot more work.
In the next article, Summer Finlay, Amohia Boulton, Helen Simpson, Bronwyn Fredricks, Yvette Roe, Jenni Judd, James Smith, Janaya Pender and Margaret Cargo present a protocol for a scoping review for commissioning practices used in the evaluation of Indigenous health and wellbeing programs. To my knowledge, this is the first time a protocol of this kind has been published in the Journal. The article follows on from the themes raised in the article by Moore and French, in relation to decolonising practice – in this case the practice of evaluation commissioning. Importantly, the article is led by and has strong input from Aboriginal and Torres Strait Islander academics, which gives the protocol integrity, credibility and strength as does the methodology, which is grounded in Indigenous standpoint. I look forward to seeing the results of the scoping review and its potential for transforming commissioning practices. While the focus here is on health interventions, I suspect that evaluations in other disciplines will also benefit from the learnings that will follow on from this work.
In Anthea Rutter’s Evaluator Perspective, Janet Clinton shares from her experiences over a long professional career in evaluation, and on some of the changes she has seen. I noted with interest the trend she observed towards participatory methodologies. Those of us with less experience would probably be surprised to think that evaluators should remain objective in their practice. I was surprised at Janet’s response to Anthea’s question about social issues evaluators should be thinking about: ‘I don’t believe evaluators or evaluation can resolve social issues' (p. 248). I would lean towards a more transformative view of evaluation, consistent with Moore and French’s article, and consistent with the book Liz Curran reviews.
Liz Curran’s book review introduces us to ‘The Practice of Evaluation: Partnership Approaches for Community Change’ edited by Ryan Kilmer and James Cook. I have often thought of (and used) evaluation as a tool for transformative change and advocacy. This is particularly important in a world where misinformation, conspiracy theories and fake news seem to shape social and political discourses. Evidence matters. As I looked through the table of contents, I can see why the latter sections of the book excited Curran. The first few sections seem like a textbook of evaluation approaches and theory, and I agree with Curran that the foci in the last few chapters on advocacy, policy change and program improvement are critically important if evaluation is to move beyond purposes of value judgement, accountability and decision-making. To me though, there is one aspect of evaluation use that is still missing here – the development of, and contribution to, social theory. It is in this space that we will begin to see changes in discourses and language, changes in evaluative thinking, changes in commissioning and changes in the inclusion of minorities and marginalised groups.
Reflecting on 2023 as Editor of the Journal, I also see positive signs of change. We have had a record number of submissions, strong contributions and support from members of the Australian Evaluation Society, increasing reach and increasing citations for articles published in recent years. The Journal also has a strong team of committed associate editors and an amazing journal manager. One of our biggest challenges this year has been finding reviewers for articles, particularly for specialist topics and for articles that engage with First Nations issues. If you are invited to review for the Journal, and are not sure, can I encourage you to reach out to the associate editor to see what is required. Reviewing is an important task which often goes unnoticed. To those who have reviewed for us this year, can I offer you a huge thank you!