Abstract
Each year, the Canadian Evaluation Society (CES) hosts an annual English and French-language conference as a forum for the discussion of major theoretical, philosophical, and practical issues in evaluation. This year, the main conference theme provided a forum for reflecting back on the past 35 years of practice, research, and theory in evaluation in Canada.
Patricia Rogers, currently working on a book on policy and evidence, opened the conference with a reflection on why ‘The evaluation world needs more Canada’. Rogers identified several Canadian contributions and evaluation assets but also reminded us of the many evaluation challenges that face us globally. One of Roger’s observed strengths was the existence of a Canadian federal policy on evaluation but as observed by Robert Lahey – founding head of the Treasury Board Secretariat of Canada’s (TBS) Centre of Excellence for Evaluation (retired) – more work needs to be done to improve evaluation capacity in Canada and to institutionalize the evaluation model at the federal level. John Mayne, well-known originator of Contribution Analysis, cautioned about the poor track record for governments, even those with national policies on evaluation, to use evaluation evidence to support their policies and belief systems; he suggested as a solution shifting our focus from evidence-based policy to policy-based evidence.
Observing that our focus on evidence is a very recent innovation, Dan Gardner – of the Ottawa Citizen, and one of the invited keynote speakers – explored the psychology behind people’s tendency to place a great deal of importance in expert predictions despite the repeated and sometimes catastrophic failure of efforts to forecast the future. As a word of advice to evaluators, more work is also needed here to deconstruct the messages that we provide policy makers so that appropriate decisions can be made.
Perhaps encouraged by a recent endorsement by the TBS of theory-based approaches to evaluation, many of the presentations stressed the importance of opening the program black box to better explain what works, for whom, when, where, how, and why. As Sanjeev Sridharan observed, no amount of statistics will demonstrate the effectiveness of a program without having a full understanding of the program theory, and of the context in which the program is embedded. More attention needs to be given to contextual factors as these can help an intervention achieve, or conversely, impede its objectives.
Context is not only relevant to causality, but also for developing our understanding of how and why stakeholders should embrace evaluation, and integrate it into their organizational cultures. Results of a Canadian research program on evaluation capacity building of Brad Cousins, Isabelle Bourgeois and associates suggests there is value in thinking of evaluation as an organizational learning system: a means to assist organizations and programs to leverage change.
The conference included a new student-dedicated stream hosted by the Consortium of Universities for Evaluation Education (CUEE) with support from CES and the regional organization, the Performance and Planning Exchange (PPX). Seventeen students were given the opportunity to present their research in a series of twenty-minute segments. These sessions, supported by discussant feedback, were well attended reflecting the community’s interest in research about evaluation and interest in supporting the next generation of researchers. Discussants complimented on numerous occasions the high caliber of the student contributions.
In closing, Melvin Mark reminded us that evaluation is a matter of consequences – the consequences of the appropriateness and adequacy of the evaluation being conducted, and of the meaningfulness, relevancy, and utility of evaluation findings and methodologies. Mark counseled unwavering focus on the guiding star of evaluation – downstream social betterment – and the key question ‘What are the pathways for evaluation to improve the problem?’
Lastly, Happy Birthday to CES! In a deadpan voice and to the amusement of all, Mark concluded with a tribute to the Society’s 35-year history, quoting famed Kool and the Gang: “There’s a party goin’ right on here. A celebration to last throughout the years.” Next year, the CES’ 36th annual conference will take place in Montreal, Quebec, May 24 to 27. We hope to see you there!
Current and past conference presentations and papers (CES and other) are available on the CES Website at: http://www.evaluationcanada.ca in the grey literature database.