Abstract
Evaluation (like applied social science research and social work research) exists to be practical. This means that its very being and certainly its job in the world is to be used—used to help individuals learn, assess, reflect, and be useful in imagining, proposing, deciding, and recommending whether and how to refine, improve, close, or otherwise respond to the realities of organized efforts to “make a difference” in the material and other worlds of individuals, groups, organizations, programs, or the like. It exists to repair (Spelman, 2002). To do this right and well requires facts, values, beliefs, and ideas which can be placed over against criteria of rightness, goodness, appropriateness, efficiency, and effectiveness so as to conclude that a process, practice intervention, activity was/was not consequential and effective, that is, whether it “worked,” and if so, in the ways expected and desired, without major, troubling, or other anticipated and unanticipated consequences on/for the same or other persons, things, or processes. Conventionally, these ideas have informed the creation of the now widely accepted standards of program evaluation (Yarbrough, Shulha, Hopson, & Caruthers, 2011).
To do this work well requires technical competence, facts and other information, thought, and wisdom. Needed is both Aristotle’s epistome and phronesis—practical wisdom. Doing evaluation well—responsibly, with credibility means to do it accurately, fully, openly, transparently, with awareness of one’s stance, biases in all directions, strengths and limitations, along with those of one’s approach, methods, tools, and hermeneutic practices, at least. It can, indeed should, mean to include as a corrective and for utility, that is, for use, evaluation should reflect multiple perspectives be multivocal, and include the ways of seeing, knowing, and doing, as well as the content of what is seen, known, and done, of multiple others, those with interest in what is being evaluated, how the evaluation is undertaken and completed, and how it can and well be used—for policy, program improvement, decision making, and conceptual clarification and refinement. The involvement of others can be as respondents, informants, observers, evaluation workers, or as advisers. Our focus here is on the latter.
Advice is a primordial, socio-intellectual process—a system of soliciting and providing, of asking others for “input”—perspective, technical “know how,” thoughts, and meanings. And of deciding whether or not to use what is offered. Advice is a way to help correct one’s limitations, or a way to solicit support, a way to increase the likelihood that a study will meet its life purpose by being used in the everyday world in ways practical, useful, meaningful, and consequential.
Most researchers, including evaluators, bring advice to their work. This can be in the form of solicited and unsolicited others or interior self-inquiry—reflective practice, where one interrogates oneself to ensure, self-consciously, that one’s biases are explicit and clear; this is one way to minimize the errors brought by arrogance, incompetence, or normal human failing. Like all advice, it can prove preventive and corrective. Less common and the focus of this essay is the inclusion of advice using formal structures—groups, such as an advisory committee or council, or individuals, who can offer consultations and technical assistance on the evaluation enterprise, from contract through research to report and use.
A review of the literature on the use of advisory structures in social work research texts shows that, just as with the literature on advisory structures in evaluation research texts, more is hortatory than practical, far more promotes this practice than describes, suggests, or evaluates advisory structures. Here are the two literature reviews.
Both literature reviews used a similar method. Texts were found using general searches in Google Scholar, Google, Amazon.com, and the University of Minnesota library system. Publisher-specific searchers were also run on major evaluation and social work publishers. Text book indexes and table of contents were initially searched for the phrases: advice, advisory, advisory board, advisory committee, advisory group, steering committee, committee, and working group. Full text searches were made of texts available on Google Books (http://books.google.com). When such titles were available in table of contents, texts were also searched for phrases like “stakeholder engagement,” “advisory board,” “participatory evaluation,” “evaluation management,” and so forth.
Few textbooks and handbooks available at the time of this survey (2011–2012) included the mention of evaluation or research advisory boards/groups/committees. No specific sections of any books examined contain specific or substantive information about advisory boards for evaluation or research. A few examples discussed the value of advisory structures and processes for evaluation and social work research, although none provided details about creating, managing, supervising, or sustaining these groups. Indexes and table of contents did not specifically mention these topics under “advisory,” “board,” “group,” or “committee.” In evaluation texts, when information was found, it was almost always identified under the topical areas “engagement of stakeholders,” “management of evaluation,” and sometimes “participatory/democratic evaluation.”
Included in the evaluation literature review were 43 U.S. evaluation texts primarily published since 2000. Most (26/61%) mentioned some advisory structure—a board (10 or 23%), committee (8 or 18%), group (7 or 16%), panel (2) task force (1), steering committee (1), or working group (1). Missing from the literature were deeper philosophical, theoretical, methodological, or political discussions about advice structures and their use to support high-quality evaluations. Instead, the texts covered the value of “outside” advice, especially evaluation expertise, insights from outsiders about the program being evaluated, and on the political utility of such groups for gaining legitimacy for the evaluator study. A full review of advisory structures in evaluation texts can be found in our recent volume (VeLure Roholt & Baizerman, 2012).
Most social work texts did not specifically mention advisory structure topics, with some notable exceptions (Engel & Schutt, 2012; Shaw, Briar-Lawson, Orme, & Ruckdeschel, 2010). The literature review of social work included 50 research texts in social work, and related research texts—applied research, action research, and practice research. Fewer than half (22) mentioned advisory structures. Again, as in the literature on evaluation, while advisory structures are often advocated for, there are few details about how to create, manage, sustain, and supervise advisory structures for applied research.
We sought to improve upon these omissions by guest editing a special issue of the journal New Directions for Evaluation (VeLure Roholt & Baizerman, 2012), which contains nine articles on evaluation advisory groups. We and our contributors described, theorized, and proposed guidelines for the creation, use, and evaluation of these structures and processes. Here, in this editorial, we only urge that social work researchers adapt appropriate, usable, and likely helpful advisory structures on the grounds that these can enhance the scientific and political legitimacy of a study, meet moral and political tests of inclusivity of perspective, voice, experience, justice, and importantly for both evaluation research and social work research, enhance the likelihood of appropriate, consequential, and helpful use of studies for program improvement, social betterment, intellectual and scholarly clarity, and understanding. Advertisements aside, our text offers, yes, good advice!