Evaluation Practice

Evaluation Schools of Thought

My earliest recollections of our field are the AEA and EERS conferences and listening to our early pioneers who were gradually developing and expounding what we now recognize as the foundations of our profession. Some of the major roots or “schools” of our now mature (but still evolving) practice are listed in Figure 1.

OPEN IN VIEWER

Figure 1.

Early schools of evaluation.

These and other schools of thought not only enlightened the growing ranks of evaluators but also divided us. Outsiders or newcomers reading this list would likely see them as important intellectual foundations upon which one could confidently build a professional career. But participants of our evaluation conferences would have also observed disputes, sometimes even stark emotions, among the titans of our field, who sometimes peppered their remarks with clever, stinging rejoinders.

These disputes caused me to reflect on the significant emotional content in the underlying principles of our practice. This reality of profound differences in opinion about the boundaries of our profession and the legitimacy of our methods has, surprisingly, remained central to our practice. It is not just that it is sometimes difficult for us evaluators to conceal differences among one another, but also that our clients hold similar reservations and display similar reactions to our work. Fortunately, the light emanating from these professional exchanges has been stronger than the heat, which, while sometimes bothersome, has tended to refine our mettle.

I will discuss the reality and relevance of, and some possible responses to, both the analytic and emotional features of evaluations more in the sections below on the meaning and methods of proof. But first I need to make a far more crucial point. I do so by sharing advice I received from Eleanor Chelimsky on these matters. In preparing this presentation, I asked her what she thought about these early schools of evaluation and the controversies surrounding some of them. Her reaction was to emphasize how important it is for new evaluators to study them carefully, remember them as they mature, and especially to put them into practice. She hopes that all evaluators will periodically go back to these sources, refresh their understanding of them, and appropriately apply them.

The Standard Professional Evaluation Model

With the big theories momentarily in our review mirror, it is useful to focus on what most evaluators do but don’t talk about very much—at least not in professional evaluation journals. It is what I call the standard professional evaluation model (Figure 2). This model outlines steps that I believe most evaluators use to scope out and perform their evaluations. It is so natural that even nonevaluators relate to it easily and accept it as a way to reach agreement about what they are willing to pay an evaluator to do.

OPEN IN VIEWER

Figure 2.

Standard professional evaluation model.

In this model, the evaluator and client first agree on a general objective and then identify two or three specific questions the evaluation will address. The evaluator suggests the methods to use and prudently includes both quantitative and qualitative methods, often more than one of each. From the associated fieldwork, the evaluator proposes findings and, if desired by the client, recommendations or options to address them.

Driving these discussions is a fixed budget and time frame within which to conduct the evaluation. Within these broad parameters, the evaluator offers greater detail about schedules for completing certain tasks (such as a detailed design, composition of the evaluation team, fieldwork, report outline, briefings, draft and final reports, etc.).

Key features of this approach are a complete abandonment of the notion that the evaluator will independently derive a “correct” set of questions to be answered, the methods that must be followed, the amount of money that it will take to get the job done, and the amount of time needed to do it right. Instead, the client will want to suggest methods and comment on what the evaluator finds and recommends, perhaps thereby implying the desire to influence the results. While an evaluator may see an evaluation objective as driving the questions and methods, the starting point of an evaluation engagement is too often how much money the client is willing to pay and when the results are needed.

Most of us have learned how to handle these realities on the job. I don’t ever remember being taught how to do these things. Still I seemed to have picked up most of it while simultaneously safeguarding my independence, using professional methods, trying to be helpful to my clients, and earning a living. In fact, most evaluators seem to learn all this on the job—and generally learn it well. I continue to be impressed with the professional skills and conduct of the evaluators I have worked with over the years—young and old, novices and old hands. They receive help and advice from their more seasoned supervisors, and we in turn learn much from them. However, I now believe that this approach and the associated negotiation skills should be taught as part of university and professional evaluation training programs.

Here, taking for granted that this is how we need to work, I wish to reflect on selected ramifications of the standard professional evaluation model. I will focus on:

adequacy of proof,

Adequacy of Proof

Many of us have been formally trained on how to find evidence for proof and to seek the truth. That’s because we are professionals, many with postgraduate training. Until recently, this professional education has largely been in fields other than evaluation. Even today, most professional evaluators did not go to an evaluation college or obtain a graduate degree in evaluation. Lately, this is beginning to change, but most of us have degrees in fields that I call “The Proving Professions.” Figure 3 includes an illustrative, although incomplete, list.

OPEN IN VIEWER

Figure 3.

Examples of the “proving professions”.

Each of these professions has adopted, and train their adherents in, a particular method of proof. Practitioners brought up in such professions typically believe their method is the only correct one. They look for things to be proved “their” way and may be unconvinced by other methods. They may even disdain and ridicule methods used by others. Even within our own field of evaluation, there remain strong differences in opinion regarding the relative values of qualitative and quantitative methods, randomized control trials and other indicators of program success, and the legitimacy of relying on performance indicators to assess program effectiveness, for example.

While the various schools and methods of proof have much in common, they also reflect significant differences. This presents a major challenge to the today’s evaluators. Evaluation results will be viewed through their client’s mind-set, not their own. Evaluation findings that are reported broadly will be subject to numerous interpretations. And an evaluator’s claim of proof may evoke an emotional response (of agreement or disparagement).

What to do? This is a hard question. What not to do is easier to address. Evaluators should not try to prove that their own method is the correct one. They will always lose that argument. Instead, they should present their findings professionally and be prepared to answer questions.

Can we satisfy everybody? I don’t think so. But I would like to share an approach that I personally have found to be convincing in my own worlds of policy analysis and evaluation. Here’s what I call the “Universal Proof” (see Figure 4).

OPEN IN VIEWER

Figure 4.

A universal proof.

A brief explanation may be helpful. I have found the biggest divide among the proving professions is about quantitative and qualitative methods. It’s not just that people like one more than the other, it’s that they seem to passionately hate the other. Some will say: “Numbers always lie. Anyone can use numbers to draw any conclusion or its opposite. I want to know firsthand what is really going on out there and what affected people think.” Others will say: “I don’t see the value of asking people what they think and then telling me what they said. I think I already know what they think. I don’t want their opinions. I want to see some hard proof. Show me the numbers.”

Please understand that I am not trying to describe the real nature and legitimacy of either quantitative or qualitative methods. I am simply describing how people I have worked with react to our reports. When I use both methods, I get a very positive response, albeit from opposing viewpoints. “We love what you do. You don’t just ask people what they think, you give us hard data” Or “We love what you do. You don’t just give us a bunch of numbers, you tell us like it really is out there, what real people have to say.” Hence, I always try to use a combination of both approaches.

The point about anecdotes is that while many people like them, many others don’t trust them. They object to what they think are “one-offs” or biased, unsupported opinions based on unique circumstances. Sometimes news reporters even chase down the facts of the case and interview the affected parties, who then may present a more nuanced, circumstantial view or even contradict the evaluator. Hence, in spite of disagreement among many evaluator colleagues, my advice: don’t use them.

The description of methodological limitations is standard practice among evaluators. I simply wish to note here that the more robust the description of limitations, the more credible the evaluation report seems to be. I can’t fully understand why this is the case, but this counterintuitive effect does seem to be quite potent.

Evaluator Independence

A question that divides evaluators and their clients is whether it is appropriate for evaluators to offer recommendations. A related question is whether evaluators should advocate for acceptance of the recommendations. Assuming they should, subsidiary questions arise about how to do so effectively, that is, how to convince and persuade clients to carefully consider these suggestions. I have concluded that there are no “correct” answers to these questions, as evaluators and their clients simply have strong views on these matters and are rarely persuaded to change their minds. But they do raise the more fundamental question of evaluator independence, which must be addressed.

My own reflections on this broader topic have led me to look inward rather than outward. I have concluded that threats to independence are fundamentally within the evaluator rather than the client. That does not mean that clients will never try to influence evaluation results or that there are no external threats to independence. It simply means that it is up to the evaluator to deal with the threats and, more importantly, to recognize those that arise from within themselves. When I ask evaluators what they think are the biggest threats to independence, they usually cite three factors—money, friendship, and politics. It is not always practical to change the attitudes or behavior of others, but it is not only possible but imperative that evaluators themselves not succumb to these threats.

The “money” threat comes from evaluators’ desire to obtain evaluation contracts and to please their clients. Evaluators themselves may not succumb to this threat, but others believe that they do. When I have been asked who is paying me for an evaluation I am working on, and when I say that it is the program officials or sponsors, the reaction I sometimes get is that questioners will therefore not trust the evaluation results. I tell them that we evaluators are professionals whose independence is one of the important values we provide. I use the analogy of auditors who market their independence and generally take pride in preserving it. This often works, but not always.

The “friendship” threat is a bit subtler, as it is seldom invoked explicitly. Again, I doubt that many evaluators would sell their professional souls for their friends, whom some of their clients may gradually become. However, the appearance of independence is as important as the reality of it. If collusion is suspected, even without warrant, the value of the evaluation is lost. It is for this reason that government evaluation offices are often organizationally located to emphasize their independence. Similarly, commercial evaluation businesses appropriately advertise their independence as a value to their clients.

Most of my evaluation work has been in policy settings. In that environment, the most common talk I hear among evaluators is their concern about interference by politicians. Newcomers to this field, and others not so directly involved in policy making, tend to believe that such interference is common. Based on my own experience as a Federal Government program and policy analyst as well as an evaluator, I believe this is rare. All the policy makers I worked for or with (and that included numerous political appointees) have wanted independent advice from professional evaluators. They really did want to improve programs. Most of them wanted to “make a difference” just as much as evaluators do, and they wanted reliable and independent information to support their deliberations. There may well have been a handful of politicians who disagreed with our findings based on political considerations, but I was never asked to change findings or even shade them one way or another.

That said, looking from the outside now, that political world seems to have changed a lot. The intense partisan rhetoric I see in the news is disturbing, and unlike anything I encountered during a career I was very proud of. I hope this bitterness comes to an end soon. However, evaluators working in government agencies, especially at the highest levels, can still be most effective if they preserve both the reality and appearance of their independence. Even in the calmer times of my career, we worked hard to make our independence well known. I developed a script for doing so, and thought it might be worth sharing with you. It goes something like this:

At the start of a meeting with policy makers or senior program officers to discuss an upcoming evaluation, I would begin with a polite reminder of our independence by saying: “As you know, evaluators are required to remain independent in performing their evaluations.” I would then express openness to other viewpoints. “But it would sure be helpful to get your insights on this project before we start.” Then, at the close of the meeting, I would repeat these thoughts, along the lines of “Again I need to remind you that we will conduct this study independently. But it has been very helpful to get your input and advice. I hope we can stay in touch.” I would repeat this routine at each subsequent meeting, such as when presenting an evaluation design, summarizing a working draft report, and the like. Over the years, I found this worked well for me and I hope it is useful to others.

Still, suspicions arose. I spent my more than 30 years as a federal evaluator and policy analyst working in the Office of the Secretary of what was originally the Department of Health Education and Welfare, later the Department of Health and Human Service. As the various presidential administrations came and went, others would suspect that my colleagues or I were aligned with the current political party. Some would refer to my “buddies” upstairs. I have been called both a Democrat and a Republican and probably many other things. But my professional training and culture has ultimately served me well. I developed the following principle to guide my words and actions while on the job. “Always conduct your professional work and behavior in such a way that those around you cannot figure out where your personal party affiliations lie.”

Clearly, it is useful to obtain input and advice from the evaluation client, along the lines just discussed. But that also raises questions about the specific boundaries of independence within the evaluation process. The relevant question is: What is negotiable? Again, I find differences in opinion among seasoned evaluators. So here I share my own opinion, for your consideration, expressed in Figure 5.

OPEN IN VIEWER

Figure 5.

What is negotiable?

Finally, while hoping I have not worn you out with my personal maxims, I share with you one last thought: “The most effective way to protect your independence is to be independent.”

Evaluator as Coach

Paradoxically, an evaluation challenge that has recently arisen stems from the growing appreciation of the value of evaluation. We are beginning to see requirements for evaluation built into program budgets. However, there is a catch. The amount of money to conduct evaluations may simply not be enough. Lately, this has manifested itself in federal research grants. Some of these grants include a requirement that a fixed percentage of the grant funds must be used to hire an independent professional evaluator. Typically, this might be 10% of the total grant award. This might work for large grants, but for small grants, say of US$100,000–200,000, this would allow only US$10,000–20,000 for the evaluation.

One possible solution to this emerging problem is for the evaluator to help the grantee build in the gathering of relevant data as part of the grant budget and operating plan. Another approach is for the evaluator to serve as a coach to one or more grantee staff who will perform the evaluation internally. These approaches may not always work, but it is both promising and important to watch the emergence of this kind of “evaluation on the cheap” requirement.

Modern Evaluation Reports

Perhaps nothing reflects the “When did this child grow up?” phenomenon introduced at the beginning of this article better than the gradual emergence of innovative ways of reporting evaluation results. Ten or 20 years ago, the standard product of an evaluation was a paper report along with supplementary technical attachments. While the requirement for such paper products remains today, the focus has largely shifted to electronic reports and slide show briefings. The latter is of particular interest. Such slide shows briefings are presented in-person or via conference calls, often employing computer-based visuals, followed by discussions with program and evaluation participants. In fact, in many cases, these brief electronic deliberations have become the de facto product, sometimes with the paper report being treated as the de rigueur, but otherwise neglected, deliverable. At national conferences, paper products may even be forbidden. Participants may request that electronic versions of an evaluation be mailed to them if their interest extends beyond the duration of the conference. Even more generally, evaluation reports are increasingly posted on websites as a primary dissemination strategy. Of course, formal written products are still needed in many cases, especially when considerable prominence is afforded the evaluation. But the lesson for evaluators is clear. A premium is now placed on high-tech presentations.

For these modern reports, evaluators need to understand a lot more than they did previously about how to present evaluation results in appealing and persuasive formats. Evaluators are now being taught such things as how many words can be effectively presented on a single computer screen via an overhead projector at a conference; how to use graphics effectively; how to simplify messages; and how to quickly produce, format, and edit slide shows.

All of this has an important side effect. It forces the evaluator to focus on the substance of the evaluation—to decide what is the core message of an evaluation, what are the concise takeaways, and what will the audience remember about the findings and recommendations days after the conference presentation.

It also facilitates a much broader distribution of evaluation results, with the possibility of reaching many parties who are far more willing to watch a quick slide show than to read a report sent to them in the mail.

Most evaluators have stumbled into this new world. The younger members of our profession are being raised in it and so take to the technology easily. But there is still much to be learned about how to boil messages down in an interesting, compelling, and memorable way. There are not many experts available to teach the latest ways. Those who present themselves as teachers and coaches of this may be adept at the mechanics of the slide shows or knowledgeable about how to find and use attractive graphics. But what the up-and-coming evaluators need primarily is the skill to boil a message down to its essentials, while doing so in an interesting and attractive way. These two very different skills are getting confused, and reliable sources of instruction—especially on the effective boil down of messages—may be difficult to find and even harder to recognize. This is emerging as one of the more important skills to learn and teach in today’s evaluation world.

Evaluator Development and Careers

Membership in AEA now exceeds 7,000—quite a growth spurt since its early days when the count was around 1,000–2,000 members. AEA once sponsored a membership scan to identify such things as age, professional education, and experience. I am not aware of any such recent scan. But I am very much aware of what I see at AEA and EERS conferences and on the evaluation teams of the contractors and clients I work with. I see practitioners of all ages and hear from them about the nature of their work in terms of substance, methods, and working conditions. It reminds me very much of the enthusiastic federal employees I got to know through my work over the years. I know firsthand how important their careers are to them. But I do not hear much discussion about their development. That is what I would like to address here briefly.

People like me stumbled through our evaluation careers. We figured things out on the fly. I am not talking here about technical evaluation skills, for which many of us found training through enlightening new books and conference presentations. But we received little guidance on how to grow up as evaluators and how to plan our careers and nurture our professional development.

By way of contrast, as a federal employee and then a supervisor and manager, I did receive training about how to grow professionally, including both analytic and management skills. In this respect, there is much to learn from the oft-maligned civil and military services. One of the great inspirations of my life was June Gibbs Brown, Inspector General in five large Federal agencies—Health and Human Services, Social Security, NASA, Interior, and the Department of Defense. I remember her explaining how impressed she was with Department of Defense training. She remarked that she met young men in their late teens who were fueling jet planes on aircraft carriers, and how impressed she was with officer training and development programs. She brought much of what she learned to the people in domestic agencies she served, including lucky people like myself who benefited greatly from formal development.

A simplified view of a career development program for evaluators may be seen in Figure 6. Figure 6 is a simplified summary of evaluation, administrative, and management skills that may be needed throughout the professional life of an evaluator. It is noteworthy that many of the essential skills are not specifically focused on evaluation but also on management and personal development. Many of these skills are absorbed naturally in the workplace. However, formal training at the right career stages can be extremely beneficial both to the individual who wants to grow professionally and to the various organizations involved. Readers of this article may wish to ask themselves if they have been trained in how to manage small work groups, motivate employees, assess performance, deal with difficult employees, lead a major division, and, perhaps most important, how to manage the development of their own careers. A detailed treatment of this topic is beyond the scope of this article, except for me to note that based on numerous discussions I have had with up-and-coming evaluators, formal career development programs may be lacking in many cases. That deficit affects individuals, their employers, and our profession.

OPEN IN VIEWER

Figure 6.

Career development program for evaluators.

Of particular interest to many evaluators is how to develop their own abilities and careers. A simple approach that may be independently adopted by any individual, but more effectively in concert with the evaluator’s supervisor, is the creation of annual development plans. These may easily be put together based on the simple consideration of three facets of professional growth:

programmatic knowledge,

Under this approach, the evaluator, preferably in conference with his or her supervisor, would prepare an annual plan to identify one aspect of each of these three categories to focus on during the coming year. The result would be the development of enduring and effective knowledge, habits, and skills. Initially, this may not seem like much, but consider the benefits of doing this for 10 years or even 20 or 30. Little is lost from year to year, as habits are hard to break and much knowledge and skill is transferable to other substantive topics and organizations.

I offer this rather personal insight as a relatively easy approach to a topic that I believe could benefit from more systematic attention within our profession. Of course, approaches like this are often built into evaluation organizations that have formal staff appraisal and development programs. However, perhaps AEA or other organizations could help develop an appropriate framework for such development within our evaluation profession.

Conclusion

Like Tevye and his wife Golde, faced with an uncommon event (the Chelimsky Forum), I tried to look back, around, and forward to notice what might not be so obvious in our day-to-day work. I hope it is useful, even if it seems unnecessarily personal or too prescriptive. I am grateful for all I have learned from those who more successfully bridge the two worlds of research and practice than I do. And I am especially grateful for all those evaluators whom I have learned from over the years. I hope that at least some of these observations are useful for those up-and-coming evaluators that even us old timers will learn from in the future.