Evaluator Perspective

How did you become involved in evaluation?

I think my experience would be similar to a lot of professionals in the evaluation world. I did not have a light bulb moment and say, ‘I must be an evaluator’, but rather slipped into it because of demands in the job world. I guess to pinpoint it, my decision to move into evaluation came through a contact with Jerry Winston, one of our Australian Evaluation Society (AES) Fellows. It was during my time at Box Hill College of TAFE, around 1984/1985. I was asked to put together a research project which involved evaluation. Jerry was doing some work for Box Hill and we got talking about evaluation. So that was my entry into the field. I then heard about Anona Armstrong who at that stage was running the AES out of her own pocket and good will. I went to one of the first AES conferences and enjoyed it and realised that I needed to know more! This was around the time when the AES was officially formed, and I became a founding member. Again, like a lot of evaluators, my career path has moved in and out of evaluation depending upon the demands of a particular job.

What kind of evaluation are you involved with, and has that changed over time?

Well, I guess after 30 years in a profession, things do change somewhat. I spent 12 years in the TAFE sector, doing little bits of evaluation, but mainly curriculum research. Along the way, I attained my masters in Sociology and Anthropology and also did some post-graduate courses in evaluation. Then I moved into consultancy and into a partnership with another AES Fellow, Zita Unger. Zita and I applied for a grant from the Victorian Government and developed our online training tool called Strategic Training Evaluation Model (STEM).

The STEM was a stakeholder driven process that combined qualitative and quantitative methods to evaluate the training return on investment to the business. Its strategic focus on stakeholders and their information needs, helped ensure that evaluation results were credible and useful for purposes of accountability, planning and continuous improvement. We expanded the concept in 2013 to go beyond training and development.

Time out from evaluation came with a stint in the Mental Health Research Institute, Beyond Blue and The Salvation Army, which was more of a policy analysis type of role.

Back full-circle to the Center for Program Evaluation (CPE) where I tend to be involved with evaluations in the community and, in particular, with disability services.

What have been the major influences which have helped to define your evaluation practice?

So many influences and so many great people in the evaluation world. Although in my early days, I did a lot of quantitative studies; my real interest is in qualitative methods. So, I studied the works of people like Michael Patton, Michael Scriven, as well as Ernie House and Carol Weiss – to name a few. But some of the greatest influences were very much home-grown.

When I did my evaluation post-graduate work at The University of Melbourne, Ros Hurworth, John Owen and Neil Day were my teachers. Truly wonderful teachers. I am indebted to both Ros Hurworth and John Owen, from whom I have learnt an enormous amount. Those who have been taught by Ros, will remember clearly ‘the red pen’. Although after being talked to severely, she would usually end up saying, ‘actually that was quite good really!’ When I joined CPE, most of my work was with Ros and that is where I learnt to hone my qualitative skills. I also worked with John Owen on a couple of projects. Again, I learned a lot from that experience. Australia and New Zealand have produced a large number of evaluators, many of which I have learnt from and all of whom would be the match of other highly ranked evaluators produced by other countries.

Major challenges to your practice?

There are always challenges, and I guess the issue is how you deal with them.

There are people you work with who let you down, or who do not share information, or you just do not get on with! I have not been immune to these types of challenges, but I have been fortunate in being around strong and collegial people who as a group we have overcome this particular challenge. In terms of working with clients, there are challenges when they do not give you all of the information you need. Or there are changes to personnel in client organisations where their evaluation needs are not the same as the previous person. This requires careful negotiation and a re-configuration of the work plan. Another challenge can come from a client who does not really understand what an evaluation can do or cannot do. There are those who ask for everything and baulk at the price you quote! Or the client who keeps saying, ‘can we just do a little more here . . . ?’! At the heart of every evaluation is communication, and it is critical to keep those channels open.

There have been times when reports are submitted and, of course, the client does not like the result and wants you to change the findings. What to do? Yes, the client is paying for a service but as an evaluator you do not want to be compromised by providing information which is false or misleading. Tricky . . . again, communication is key, talk it through and if you can come to a compromise.

What have been the highlights of your career?

I have been associated with the AES for over 30 years and consider this relationship to be a very special part of my evaluation career. I have seen the AES move from a fledgling organisation to now a very sophisticated organisation. Membership is growing as evaluation becomes a bit more ‘flavour of the month’. When I became a Fellow of the AES, it was a very humbling moment for me, and I am grateful for the acknowledgement. Zita Unger and I were awarded the Evaluation Training and Service Award for outstanding contribution to the evaluation profession a number of years ago. It is the greatest moment to be acknowledged by your peers.

Another highlight is the work I achieved with Zita Unger. Zita and I worked together for over 20 years, and particularly in the development phase of our STEM model. It was incredibly challenging, but enormously stimulating and rewarding. To develop our online tool, we worked with the Australian Navy at Cerberus to develop the prototype. Although a challenging time, I consider this a real highlight, mainly because we learnt so much, not just about evaluation but technology. This was around 1999 and technology was not as sophisticated as it is now, so there were plenty of hurdles to overcome. However, it was an exciting time!

Being able to contribute to CPE is another highlight, and I still learn from the experience of working there, which is wonderful. My practice is challenged all of the time, which is great!

Can you share one critical piece of advice from your evaluation experience?

Never think that there is no more to learn. You learn all the time, and sometimes in ways and from people that surprise you. I am reminded of a comment made by one of our Fellows, Chris Milne, which I think is so apt. I am sure he would not mind me repeating it here:

You also should approach evaluation with curiosity and mindfulness and be able to live with ambiguity and uncertainty.

When I reflect on the ups and downs of working with clients, I guess there are a few things that I feel are important to always bear in mind. First, keep a constant check with the client to ensure that you are producing what they want. Never lose sight of the fact that your client is paying for your services and of course that you approach evaluation with integrity. Stakeholders can change, so always do reality checks as you go. Also, be prepared to be flexible and change things as you go along.