Abstract
Eleven years ago, I became the director of the Idaho State Legislature’s Office of Performance Evaluations. I appreciated this opportunity because, as only the second person in that role, I considered the office and its function to be still evolving and because I understood legislative leadership to be interested in seeing evaluation work that could actually inform policy making and budget decisions. However, this opportunity also came with challenges, as I had no prescriptive framework for how to conduct evaluations that policy makers would use. I was given neither a job description nor goals and performance measures. Thus, I had to develop a strategy that would help my office conduct useful evaluations. The strategy was to maximize both our office’s responsiveness to and independence from evaluation sponsors and stakeholders (Mohan & Sullivan, 2006).
Responsiveness is a tricky business for evaluators working in public policy environments. It goes beyond just conducting evaluations and issuing reports for sponsors and stakeholders. Being responsive also involves promoting (and sometimes defending) your approaches, methods, findings, conclusions, and recommendations to sponsors and stakeholders who often have competing or conflicting interests. This act of promoting and defending all aspects of evaluation work by evaluators and evaluation organizations is what I call evaluator advocacy, and the purpose of this advocacy is to increase evaluation use.
The Challenge
The advocacy aspect of responsiveness can be interpreted negatively. The challenge as I see it is to balance my responsiveness with my demonstrated independence—akin to walking a tightrope (Grob, 2006). Being responsive only to the evaluation sponsor or to a select few stakeholders could be perceived by others as a bias or taking sides. Such perceptions could adversely affect the credibility of the evaluation and could slowly over time erode the credibility of the evaluator and the evaluation organization. In the public policy environment, credibility comes, in part, from clearly demonstrating the independence of evaluators and evaluation organizations from sponsors and stakeholders.
Many performance audit offices follow the Generally Accepted Government Auditing Standards promulgated by the U.S. Comptroller General. These standards place heavy emphasis on evaluator independence. Trained in an audit environment, I found maintaining my independence from evaluation sponsors and stakeholders to be relatively easy. The challenge was learning how to be fully responsive to these same sponsors and stakeholders without compromising any part of my independence, not even giving the appearance of a compromise. Five strategies have helped me maximize both responsiveness and independence: Understanding the political context of my evaluation environment by identifying key stakeholders and knowing their relationships with each other—and knowing their competing, conflicting, and complementing interests; establishing trust with stakeholders by involving them in the evaluation process, getting their buy-in on scope and methodology, and keeping the evaluation process transparent; disclosing all conflicts of interest between evaluators and evaluation stakeholders and avoiding the appearance of any conflicts; developing practical recommendations that are sensitive to sociopolitical and economic conditions; and preparing evaluation reports that are balanced in both content and tone.
I did not invent these strategies; I learned them over the years from the works of some of the eminent scholars and practitioners in the evaluation field (Chelimsky, 1987; Grob, 2006; Patton, 2008; Weiss, 1993). Obviously, I could not have implemented these strategies by conducting evaluations in a vacuum. Hence, I began learning about Idaho’s political landscape, getting to know key policy actors, building relationships, and establishing trust with sponsors and stakeholders of our evaluations. These efforts led me to understand the politics of evaluation, the realities of which many evaluators either do not acknowledge or prefer to avoid in order to portray themselves as independent and neutral.
Politics of Evaluation
My mother used to say, “wherever there are two or more people, there will always be politics over resources.” Because evaluations involve making judgment about prioritization, distribution, and use of resources, evaluations will always be inherently political. While my mother was not an evaluator by profession, evaluation experts seem to agree: The evaluator has political influence even when he does not aspire to it. The evaluators’ professional conclusions cannot substitute for the political process. —Cronbach (1980, p. 3) … evaluation is a rational enterprise that takes place in a political context. —Weiss (1993, p. 94) Evaluators are inextricably imbued with value commitments, constrained by the context of the study, the politics of the setting, and the politics of the government. Evaluators are fully ‘situated’ in the deepest sense: value-imbued, value-laden, and value-based. —House (2006, p. 121) The fact that people are involved in evaluation means that the values, perceptions, and politics of everyone involved…impinge on the process from start to finish. —Patton (2008, pp. 530–531)
The Risk of Losing One’s Credibility
George Grob compares advocating for evaluation use while maintaining independence to walking a high wire or tightrope; we can fall off and die if we lean either to the left or to the right. However, “To not walk that wire is to abandon all possibility of making an effective contribution to policymaking through evaluation” (Grob, 2006, p. 104). At the end of the day, credibility is the most precious asset an evaluator has. No amount of advocacy is worth the risk of losing one’s credibility. No level of advocacy will bear fruits if one loses credibility as an independent, neutral evaluator in the process. I believe the extent of evaluator advocacy depends on one’s appetite for risk.
Of course, not all evaluators work in similar environments. Their roles, responsibilities, and authority differ widely—even for those evaluators who work in public policy environments. These differences present evaluators with certain privileges and constraints. For example, independent evaluators hired to conduct evaluation in a public policy environment may not benefit from the same level of trust with and familiarity of stakeholders as do those evaluators who work in a government agency that is regularly responsible for conducting evaluation in that policy environment. Likewise, evaluators working for an agency that is given limited authority for conducting evaluations will be constrained in their advocacy activities as compared to those evaluators who work for an evaluation agency that enjoys wide latitude in deciding what to evaluate, how to evaluate, and how to report evaluation results. Evaluators working independently or working for advocacy groups may feel more comfortable in taking sides on a particular policy issue than those working for nonpartisan government evaluation offices.
These differences in evaluators’ work environments and in their roles, responsibilities, and authority contribute to varying perspectives evaluators have on advocacy. As such, no perspective is right or wrong. However, I believe a lack of understanding of those varying perspectives generates strong emotions and makes advocacy a controversial word that many evaluators don’t want to discuss. Because advocacy requires walking a tightrope, not all evaluators feel comfortable in taking that risk lest they fall and lose their credibility. The loss of credibility could adversely affect the evaluator’s professional reputation among peers and could negatively affect his or her livelihood. It is the fear of losing one’s credibility that keeps many evaluators away from engaging in advocacy activities, especially those who work in public policy environments.
Fortunately, my work environment, the nature of my office, and my position as the director of the office afford me considerable flexibility and discretion in deciding what specific advocacy activities I and, in some cases, my staff can engage in. Much of this flexibility and discretion is derived from state law. Because Idaho Code does not detail each advocacy activity, it is up to me to interpret law, convince others of my interpretation, and defend my interpretation when there is a controversy about it. I am accountable to policy makers and the public—in the end, I must be willing to pay the price of my decisions and actions, even if it means losing my job when I become a liability for my employer.
How I interpreted my role and responsibilities when I started in my current position differed from expectations in the office at that time. Specifically, several senior staff in my office strongly advised me to act as follows: Do not conduct policy analysis; avoid the press and be careful about certain individuals in the press; avoid spending any significant amount of time in the state capitol; keep interactions with legislators and agency officials strictly limited to the evaluation project on hand; and avoid interactions with lobbyists.
These staff were concerned about the risks associated with engaging in such activities. I carefully and respectfully listened to them but rejected their advice as evident in my activities in the next section. I rejected their advice because I did not see how I could produce useful evaluation reports by avoiding key players in my evaluation environment.
Advocacy in Action
My office and the evaluation activities my staff and I engage in to advocate for the use of our work offer illustrative examples of advocacy practices. These activities relate to the five strategies of walking the tightrope of responsiveness and independence, which I mentioned earlier in this article.
The Office of Performance Evaluations (OPE) is an independent agency of the Idaho legislature. 1 OPE receives its evaluation assignments from and reports to an eight-member Joint Legislative Oversight Committee (JLOC)—equally divided between the two major political parties and the two chambers of the legislature. OPE’s mission is to promote confidence and accountability in state government through professional and independent evaluation of state agencies, programs, functions, and activities. Thus, OPE evaluators find themselves advocating for the use of their work.
Any legislator can request an evaluation to be conducted by OPE through JLOC, which considers all requests at a public meeting and decides, of those requests, which will be further studied. Although JLOC approves evaluation requests, it does not approve the evaluation scope or methodology. Instead, JLOC members provide input on the scope drafted by OPE. They are expected to respect the independence of OPE and to stay completely out of the evaluation process. The agency being evaluated is given an opportunity to review the draft evaluation report and provide technical comments relating to the accuracy of facts presented in the report. In addition, both the agency and the governor provide written responses to the evaluation, which are incorporated in the final published report. The entire evaluation process is confidential to the public, press, and policy makers until the report is released to JLOC at a public meeting. In addition to the agency and the governor, we provide the final report to JLOC members, the senate president pro tem, the speaker of the house, the cochairs of the budget committee, the chairs of relevant policy committees, and chiefs of the legislature and the governor’s budget offices in advance of the report’s public release.
All evaluators, consultants, and interns working on the evaluation complete an independence statement disclosing actual and potential conflicts of interest. Everyone at OPE is strictly prohibited from displaying political affiliations and participating in political activities that could damage the office’s credibility for conducting independent, nonpartisan, and objective evaluations. In addition to these formal restrictions and requirements, OPE evaluators are expected not to be involved in any activities that could be seen as going beyond the statutory role and construed as meddling in politics. Idaho’s expectations are similar to those of most, if not all, government evaluation offices performing oversight functions across the United States.
Advocacy activities discussed in the following sections are an integral part of my office operations—these activities are all in a day’s work grounded in professional standards and guided by personal ethics.
Conduct my “daily sojourns”
My office is about two blocks from the state capitol. During each legislative session, I try to visit the capitol every day whether I have a specific purpose or not. I call these visits my daily sojourns. I believe in the saying, “out of sight out of mind”—if I am not representing my office where the policy making action is, my office will be ignored and become irrelevant. These visits to the capitol usually take about a half hour to an hour of my time, but they allow me to run into policy makers, legislative staff, governor’s staff, department heads, members of the press, and lobbyists. Through these short visits in the halls of the capitol, I inform others about our work and at the same time I become informed about the political context in which we work. Furthermore, these face-to-face contacts help me build professional relationships and trust over time.
Keep legislative leadership informed
A key ingredient in building relationships and trust is not to surprise others. Therefore, I update legislative leadership about our work at their semiannual meetings. Informally, I visit with the senate president pro tem and the speaker of the house at least once a year to let them know what my office is doing and to seek their input on how we can be more responsive to them. In rare situations, I also discuss extremely controversial findings with the pro tem and the speaker in advance of releasing our evaluation reports. The purpose of such discussions is to prevent them from being blindsided in a public setting. These formal and informal meetings inform them about results of our evaluations and what to expect from our office in the coming months. Under no circumstances, do I make any promises to these legislative leaders and others that I cannot discuss publicly.
Keep stakeholders informed
We keep the agency and key stakeholders informed during the conduct of an evaluation. This includes sharing with them the study scope, evaluation approach, time lines, and evaluation progress. We also ask agency officials for their input on how we can make our evaluation useful to them, beyond just answering evaluation questions that the legislature has asked of us. We do our best to incorporate their input into the study scope.
Share evaluation results with legislative committees
We present results of our evaluation reports to relevant legislative committees. We also testify to legislative committees, when we are invited by committee chairs. Our testimonies stay strictly within the contents of the evaluation report, and we do not comment in support or opposition of legislation being considered. To avoid being perceived as taking political sides, we never comment on citizen initiatives or referendums, even if they relate to the evaluation.
Assist policy makers with evaluation requests and legislation
We offer assistance to individual legislators with drafting their evaluation requests. This could include answering questions, gathering background information, and suggesting evaluation questions that correspond to their requests. Some legislators might be reluctant to request an evaluation because the process of evaluation might be unfamiliar to them. Through assisting legislators with their requests, we can help them see the value of evaluation, especially if one of their requests ends up being a complete evaluation report. Likewise, if asked, we assist legislators with drafting legislation related to the evaluation we completed. However, under no circumstances, we support or oppose legislation. We always keep the process transparent and strictly limited to the contents of the evaluation we completed, so no one thinks that the office is doing favors to certain legislators or using legislators to serve its own political agenda.
Educate policy makers and others about evaluation
On an ongoing basis, I work toward educating policy makers, stakeholders, and the public about the value of evaluation and the role it plays in government accountability and the public policy process. Examples include my annual budget presentation to the legislative budget committee, semiannual presentations to legislative leadership council, and the orientation of new legislators every 2 years. I also promote the use of performance measures in state government for monitoring purposes by assisting agencies with developing and reporting performance measures and educating policy makers on how to use performance measures to ask questions about programs.
Within the framework allowed by the legislature, I reach out to future policy makers and community leaders by guest lecturing at the local university, professional organizations like the Association of Government Accountant, and community organizations like Rotary Club. During these presentations, I talk about the role my office plays in promoting government accountability. However, I never express my personal preferences in support for or opposition of a political party, candidate, policy, or issue.
Use the press effectively
I believe the press is a good medium to convey the evaluator’s message to the public. It is the public’s role, not the evaluator’s, to put pressure on elected officials for a policy change. Evaluation reports can be a great resource for independent and objective information on issues that interest the public. At the completion of every evaluation report, we widely distribute a press release. We also make ourselves available to the press if they have any questions or if they want to do a special story about a particular topic that we have covered in one of our evaluation reports. Recently, we have also started using the social media to promote our work. However, under no circumstances, either with the traditional or social media, do we take sides on a particular piece of legislation or policy. Also, we do not write opinion pieces for the press, nor do we respond to public comments about our work in the press. Such actions would be seen as political or unprofessional.
Concluding Thoughts
If we do not promote and defend our own work, who will? I do not think we can leave the use of our evaluations to happenstance. Yes, evaluator advocacy is controversial and is difficult to do effectively, but these are not good enough reasons to shy away from advocating for the use of our evaluation findings, conclusions, and recommendations in public policy making. That evaluator advocacy can be difficult or controversial only tells me that I have to be thoughtful and strategic about it. I fully understand that a poor job of advocacy could tarnish—or even ruin—my credibility and that of my office.
I consider advocacy activities that my staff and I engage in an essential part of our jobs. I cannot imagine producing useful evaluations without engaging in these activities. In spite of producing useful evaluations during the past 11 years, I have faced a number of difficult challenges that forced me to continuously assess how we carry out our advocacy role in the highly political public policy environment. These challenges came not only from agency officials, individual legislators, stakeholders, and lobbyists but also from members of my own oversight committee. The presence of challenges emphasizes the risky nature of evaluator advocacy and explains why many evaluators shy away from it. Still, aside from some short-term stresses, these challenges reinforce my respect for the public policy process.
In this article, I have presented ideas based on my experience and what I have learned from others. Every evaluation environment is different; each presents its own unique opportunities as well as complications for evaluators. I do not suggest that my perspective is without flaw, nor do I expect my strategies to work for every evaluator in every situation. Instead, I expect these ideas to serve as a catalyst for evaluators to examine their advocacy role in the context of their own evaluation environment. Walking the tightrope of evaluator advocacy is inherently risky. With the help of the Guiding Principles for Evaluators (American Evaluation Association, 2004), I hope I will make the ethically correct calls on difficult advocacy issues. In terms of producing useful evaluations, the rewards of thoughtful and strategic advocacy are worth the risks.
Footnotes
Author’s Note
The views expressed in this article are of the author and do not necessarily represent those of the Idaho State Legislature.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.