Broadening the Discussion About Evaluator Advocacy

This forum addresses a set of very important questions about evaluator advocacy, including What do we mean by the word “advocacy?” Should evaluators ever advocate? If so, when and how? What are the payoffs and risks to evaluator advocacy? And perhaps most practically, how can we evaluators become more comfortable with an advocacy role? In the following commentaries, two evaluation leaders, George Grob and Rakesh Mohan, draw upon their wealth of practical experience to address these and other questions. Their insights can be useful to all evaluators. Before they tackle these critical questions, I want to broaden our frame of reference about what exactly an evaluator might advocate for, whether operating in the public, nonprofit, or private sector, domestic, or international. I suggest the following four levels of advocacy (also see figure 1):

1. Advocating for the results of a specific evaluation.

OPEN IN VIEWER

Figure 1.

An evaluator might advocate for ….

I suspect that most evaluators define advocacy as something we do to focus our audiences’ attention onto the results of a specific evaluation. For example, we might advocate that our audiences attend seriously to the findings, conclusions, and recommendations of a recent evaluation. We might advocate that our recommendations be accepted—or, we might advocate that, whatever the decision makers decide about our recommendations, they document those decisions and the reasons for making them. We might even advocate that any agreed-upon follow-up actions are actually implemented and perhaps also that this implementation be monitored closely.

If we feel especially secure in our position, we might even advocate that our findings be represented accurately when repeated elsewhere. I once suffered great angst when a former Secretary of the U.S. Department of Health and Human Services distorted before a Congressional committee, for seemingly political reasons, the findings of an evaluation I had watched him receive just the day before. I still regret not speaking up at the time.

These are examples of study-specific advocacy, and most of us would probably be comfortable advocating at this level. In fact, the commentaries by Grob and Mohan later in this Forum offer tips on exactly when and how to advocate effectively at this level; Grob notes that his comments are focused primarily on this level, and the majority of the advocacy activities that Mohan is comfortable recommending relate to the results of specific evaluations. However, this level of advocacy represents only one of the four possible levels of advocacy in which we evaluators might engage. If we are to discuss advocacy comprehensively, I believe we must also be aware of the other three levels.

2. Advocating for adequate support to evaluations in general.

As Figure 1 shows, advocating for the results of a specific evaluation is important, but, while central, it may be the most narrowly focused form of advocacy. Advocating at this level could enhance the attention given to that particular evaluation, but it will have no wider effect. The second, slightly larger circle represents a broader form of advocacy—advocacy for evaluations in general to be sufficiently equipped to produce useful information. For example, we might advocate that decision makers focus evaluations on important programs and issues, not on relatively trivial topics. After all, evaluations are spotlights, and it matters greatly where a spotlight shines.

Or, we might advocate to both policy makers and to our fellow evaluators that evaluations use the most appropriate methods to answer the evaluation questions. If this means advocating for open-ended qualitative methods to a hard-nosed program manager, or advocating for a randomized field trial to a lifelong skeptic of such approaches, then so be it. This advocacy would be for the most appropriate method in any given situation, not a personally preferred or familiar method.

We might also advocate for evaluations in general to receive the resources required to answer convincingly the questions they are being asked to address. For example, we might advocate for all evaluations to receive the necessary time, personnel, training, funding, data access, and other necessities. This needs to be done thoughtfully, of course, lest our advocacy appear to be simply a self-serving grab for as many resources as possible. If done for the broad set of evaluations in general, advocacy at this second level could improve evaluations far beyond our own specific study.

Some evaluators may feel that advocating at this second level is inappropriate for objective analysts—that is, analysts/evaluators whose role is simply to “provide the facts” and nothing more. However, American Evaluation Association’s (AEA, 2013) official “Roadmap for a More Effective Government” specifically advocates at this level when it recommends how best to support an evaluation’s proper scope, coverage, analytic approaches and methods, resources, professional competence, plans, dissemination of results, and independence.

3. Advocating for the concept of evaluation.

An even broader level of advocacy involves advocating for the very concept of evaluation itself. Advocating at this level is much broader than a specific study and even broader than advocating for adequate support to all evaluations in general. At this level, we evaluators might advocate for government officials, nonprofit leaders, private sector CEOs, legislators, boards of directors, program managers, and the general public to better understand and appreciate the potential value of evaluation. We might demonstrate that evaluation is a powerful, but greatly underutilized, instrument for improving program performance and thereby increasing public benefit.

To do this, we might showcase effective evaluations from a wide variety of settings. We might then advocate that program units create designated evaluation units with independence from political interference. AEA’s Roadmap advocates at this level when it proclaims “Evaluation is an essential function of government. It can enhance oversight and accountability of federal programs, improve the effectiveness and efficiency of services, assess which programs are working and which are not, and provide critical information needed for making difficult decisions about them” (AEA, 2013, p. 2). Needless to say, evaluations have been equally effective in the nonprofit and private sectors as well.

Lest we overadvertise the value of evaluations, we might also advocate to our own colleagues—and, in fact, also to ourselves—that we evaluators recognize and stay within our proper role within our processes of decision making. We evaluators are rarely elected or appointed to make key decisions about programs, and we need to keep that reality in mind. “Advisor to the king (or queen)” is not the same as “king or queen.” We evaluators should aspire to be the former, not the latter.

4. Advocating for effective and open governance.

The outer circle in our advocacy levels framework (Figure 1) represents the broadest level of advocacy—for effective and open governance, whether that be in the public, nonprofit, or private sector. At this most controversial level of evaluator advocacy, we, as evaluators, disagree on whether advocacy is even appropriate, much less necessary. For some, this is politicking pure and simple—possibly the proper arena for political scientists or management consultants, but certainly not for evaluators. Many evaluators might argue that evaluators bring no special expertise to this issue of effective and open governance.

Other evaluators might ask, “What good does it do for evaluation to be a tool for effective governance if our organizations have no interest in being effective—or in telling their audiences whether they are effective or not?” This indifference is typically attributed to less developed countries, but it is also a problem in the United States and elsewhere. To counteract this indifference, some believe that individual evaluators should consistently advocate for organizations, in all sectors and levels, to be transparent and to acknowledge their responsibility to be effective and efficient.

Taking this position a step further, Kate McKegg (2013) wonders in a recent issue of this journal if our professional evaluation organizations might even be obligated to conduct this level of advocacy: “I would argue that it might well be the professional responsibility of our evaluation organizations to play a role in the evaluative deliberation so needed to foster the commitment of countries and societies to democratic participation, equity, and social justice” (p. 581).