Empowerment Evaluation,Postprofessionalization,and Oligarchy

Empowerment Evaluation

The empowerment approach to evaluation was introduced in 1993 when David M. Fetterman made it his presidential theme in his American Evaluation Association address (Fetterman 1994). He defined empowerment evaluation as, “. . . a form of evaluation practice in which evaluators bring the voice of disempowered citizens to those who have power and facilitate citizens’ control over their own affairs” (p. 297). This approach was later elaborated into the 10 principles guiding empowerment evaluation: improvement, community ownership, inclusion, democratic participation, social justice, community knowledge, evidence-based strategies, capacity building, organizational learning, and accountability (Fetterman and Wandersman 2005:27–39).

Miller and Campbell (2006), in their literature review, note that empowerment evaluation is not without shortcomings, including theoretical underdevelopment (Patton 1997a, 1997b; Scriven 1997; Smith 1999), the lack of unanimity in evaluation practice, that is, fidelity (Cousins 2005; Patton 1997b; Smith 1999), and uncertainty as to its greater effectiveness compared with the expert evaluation model (Patton 2005). Although applied social scientists have always involved their evaluation clients with the formulation of the latter’s central evaluation questions and basic study designs, the more proactive involvement of stakeholders, including those from related agencies and representatives of their service populations, throughout the entire evaluation process places evaluators more in the role of a facilitator and teacher. This requires extensive use of meetings and conference calls to discuss how to best carry out the evaluation with people from different stakeholder groups as to why certain methods are more appropriate than others (Embry and Rodabough 2005).

Deprofessionalization/Proletarianization and Postprofessionalization

To understand the recent concept of postprofessionalization (Randall and Kindiak 2008), it is necessary to first highlight the extensive literature of deprofessionalization and proletarianization from which it emerged. Since the 1970s, much of the focus on the professions has been from the perspective of power and conflict at both the structural and individual levels. However, a recent Canadian article suggests that this perspective is not so relevant, particularly with regard to the notion of deprofessionalization and proletarianization (P. Thomas and Hewitt 2011). One reason for this is that most prior studies have focused on medicine and law within the United States and the United Kingdom. These studies do not necessarily apply to professions in other countries nor other professional occupations within the United States and the United Kingdom.

Deprofessionalization involves the erosion of the monopoly control of specialized knowledge and autonomy as a result of a more knowledgeable public demanding greater accountability of professionals. Modern information technology also plays a role by reducing the information gap between professionals and lay persons. Proletarianization involves the same outcomes; however, it is driven by post-industrial capital seeking to gain control over professional labor. This is partly due to the high cost of production resources and complex organization needed to deliver state-of-the-art services—medicine as the classic example (Randall and Kindiak 2008). For example, the introduction of the electronic medical record (EMR) tends to shift some autonomy away from individual physicians to ongoing peer review (Reich 2012). This is related to the rapid synthesis and reporting of clinical practice made possible by the EMR technology, putting physicians “under the microscope” by their colleagues and administrators. The forces related to each of these theoretical perspectives play a role in changing the nature of professional work. For the sake of simplicity, and because my focus is on the outcome of these two perspectives, I will henceforth refer to both as deprofessionalization.

What defines a profession? There are two core components, according to the trait model (Toren 1975). First, its practitioners must have a body of theoretical and technical knowledge and, second, have a service orientation (e.g., code of ethics). The most important aspect of the trait model is professional autonomy that Toren describes as, “. . . the right accorded by society to members of a profession to determine the nature of the problems with which they will be concerned, the appropriate procedures by which these should be solved, and the evaluation of professional performance” (p. 325).

Freidson (1970:127) discussed the status of the medical profession in terms of what he called professional dominance (PD), comprising two levels—the structural and the individual. The former includes control over the design of medical training, setting the criteria for admission to training, the design of the training of most non-physician providers, state licensure and the control of these boards, and medicine’s stature such that the public accepts it as the ultimate authority on medical matters. The individual level enables a high degree of physician independence and autonomy in their daily clinical activities (p. 136). The reality now, however, is that most physicians practice on teams in more complex provider organizations that monitor their performance and support active patient involvement in decisions related to their care. The result is that physicians are no longer as independent and autonomous as in the past (Feste and Anderson 1995; Salmon and Hall 2003; Segal 1998). In contrast with medicine, the evaluation profession has little PD at the structural level. In fact there is no one training program for evaluators (aside from training in methods and statistics) as they might come from a host of social science, or other disciplines.

Other professional examples of deprofessionalization abound—indicating social-structural change in the professions. In aviation, the dominance of airline and commercial captains has been replaced with crew resource management (Bowers et al. 1999). Although captains still have the final authority for their aircraft, they can no longer independently dictate orders as they must integrate and share flight responsibilities with the flight deck and cabin crews. The same is true within law practice with the development of more collaborative lawyer–client relationships (Boccaccini et al. 2004; Fairman 2005; O’Leary 2005). Collaboration is increasingly true among care coordinators and mental health providers in SOC who work with juveniles. Within this model, family members and significant others are brought into the therapeutic process, with most providers acting more as coordinators. The result in each of these professional areas has been an improved quality of services (Boccaccini et al. 2004; Mouawad and Drew 2013; O’Leary 2005; L. Thomas and Galla 2013).

Not surprisingly, how deprofessionalization is perceived depends on your perspective. From the perspective of professionals, protocols and greater client accountability not only undermine their autonomy, but decisions and action are also more likely to be seen as less rational and evidence based. For example, in evaluation, stakeholders often want simple descriptive statistics rather than multivariate, inferential statistics that control for one or more confounding factors. From the perspective of clients, deprofessionalization reduces the mystery and anxiety about professional services and increases satisfaction with those services. From the perspective of administrators, deprofessionalization results in more uniform quality and decreased costs.

In contrast, the more recent concept of postprofessionalization better explains the changing roles of evaluators conducting empowerment evaluations (Randall and Kindiak 2008). Postprofessionalization involves a change in the scope of practice that involves a much greater emphasis on teaching and diplomatic skills than was required on so-called expert-based evaluations. A similar concept has been termed new professionalism in a discussion of the United Kingdom’s National Health Service (Moffatt, Martin, and Timmons 2014). There, physicians have adopted a more managerial focus in terms of their productivity that includes collective as well as individual responsibility. From this perspective, program evaluations, along with several other professions, are becoming more as, “. . . organized occupations with status whose relations with the state, the public, and other professional groups are structured and regulated” (Adams 2010:66). All this is to say that empowerment evaluation is more democratic than its counterparts. However, once a democratic structure is implemented, there looms the specter of oligarchy.

Oligarchy

The iron law of oligarchy—the rule of the many by the few—has been recognized for a century (Michels [1915] 1999). Michels described this as, “. . . organization which gives birth to the domination of the elected over the electors, of the mandatories over the mandators, of the delegates over the delegators. Who says organization says oligarchy” (p. 15). Oligarchy was a direct criticism of Rousseau’s 1762 [1974] concept of direct popular democracy. Michels believed that organization—be it government or non-government—requires coordination, timelines, and professional expertise—all of which tend to undermine full democracy. Organizations require leaders (experts) who for a whole host of reasons may come to undermine democratic principles by putting their own interests ahead of the democratic organization. Now that I have set the context, let us turn to my experience.

Retrospective

I would like to preface this section by saying that the challenges I faced were not the fault of individuals, but instead the result of social, organizational, and structural issues endemic to creating a more democratic and effective mental health services system with a diverse array of people. ¹ These challenges were common to all SOC sites and were broadly addressed at regional and national SOC meetings. Here is the essence of Monroe County’s SOC, previously reported (Grape et al. 2014):

The Monroe County SOC focused on older, cross-systems youth: Those 14 to 21 years of age having a designation of SED and involvement with probation and/or child welfare. Traditional case management was converted to a more holistic care coordination model using the child and family team process to determine strengths, needs, resources, and referrals. The Mental Health Single Point of Access—which determines eligibility, priority, and assignment to a care coordination provider—was embedded in a newly created, cross-systems program known as the “Family Access and Connection Team” (FACT). Referrals to FACT came from a variety of community-based organizations, as well as schools, hospitals, and families themselves. Part of the care coordination process included a tool for “cultural discovery” and the care coordinators participated in extensive training from the lead cultural broker and cultural competence consultant . . . A multi-stakeholder Community Collaborative was established to review data and provide input regarding service needs and gaps (p. 9).

The demographics from Table 1 in that article show that our sample of the population of focus was 59 percent male, 40 percent non-white, and 16 percent reporting two or more races. The mean age was 13 (±3), 61 percent of the families received Medicaid, and 41 percent were below the official poverty line. The top three reported problems were conduct/delinquency related (71 percent), hyperactive and attention related (57 percent), and anxiety related (52 percent).

There were two broad classes of stakeholders represented in this grant—administrative and community. The former comprised the Office of Mental Health (OMH) Director and staff, staff from two of its sub-contracting organizations, and staff from the several agencies that worked with youth in varying capacities. Community stakeholders consisted of anyone from the population of focus who wished to participate on the grant via four councils. Active solicitation of community members continued throughout the grant period. The underlying philosophy of SOC is that interagency and community involvement are necessary for changing the mental health system in a way that is meaningful and acceptable to both providers and those receiving services. The same holds true for empowerment evaluation. When a wide variety of stakeholders are actively involved in the design, analysis, and reporting of data, the result is more likely to focus on what is important to all stakeholders. Furthermore, active participation is very likely to create buy-in by the stakeholders as they help structure and carry out the evaluation. Thus, even when results might be unfavorable, there is a much greater chance that an effort will be made to address those program deficiencies. In other words, empowerment evaluation reduces the likelihood of creating a final report that ends up “collecting dust on a shelf.”

However, as stakeholders generally have no evaluation expertise, they must be educated on the basics of design, analysis, and reporting by the evaluator. My team and I worked closely with the OMH and its designees, providers, and community members during the first two years to educate them on the basics of program evaluation. However, to do this required the establishment of a governance structure. The initial challenge for the administrative stakeholders was to establish a system of governance that would be similar to the population of focus. The goal was to direct community members and agencies to carry out the mental health transformation, including the evaluation component.

The implementation of Monroe County’s grant, beginning October 1, 2005 (ending on September 30, 2011), was slower than anticipated. It was my opinion during the first year that work should proceed in small groups composed of the then current administrative stakeholders having the relevant expertise—a concept supported by the national representatives of SOC. In mid-March 2006, I suggested we all begin the process of writing the study protocol based on questionnaires common to all the SAMHSA grantees for submission to the University of Rochester’s institutional review board (IRB). We could also begin the development of the time-intensive logic model so that we could better specify our outcomes in the protocol. The logic model is a graphic illustration showing what actions the mental health services community planned to undertake, what specific outcomes were expected, and why; that is, it would spell out their theory of action. The evaluators would then track these outcomes over time using as much information as possible from the data collection instruments provided by SAMHSA to all of its funded sites. As the subsequent designs of the more specific data collection instruments for Monroe County were completed, protocol amendments would be submitted to the IRB.

Although our administrative stakeholders did not adopt my plan, they did agree to begin working on the development of a logic model. However, they and I were working on differing assumptions. I assumed my colleague and I would guide our stakeholders through this process, as helping clients articulate their theory of action is integral to our work. Instead, for purposes of inclusiveness, this task remained with our administrative stakeholders. However, they found the logic model and its development too abstract. We continued to offer whatever support they wanted over the subsequent weeks to assist with its development. It was frustrating for us to not lead this process because my colleague and I saw ourselves as the experts who could have accelerated this process. Our limited autonomy to make evaluation decisions was unsettling because if the evaluation failed it would be seen, perhaps rightly so, as our fault. However, given the premise of empowerment and stakeholder participation, we did not challenge their plan.

By late May, I did convince our administrative stakeholders that we needed to convene the evaluation work group to write the study protocol for the IRB submission of the national questionnaires, as I knew that review and approval would take several months. The evaluation group (including two community members) met in early June 2006 to discuss an initial draft of this protocol, but subsequent difficulties with the logic model and our administrative stakeholders’ continuing concerns about a lack of governance and staff yet to be recruited convinced me that subsequent evaluation meetings should be postponed.

We had a day-long retreat with our administrative stakeholders in mid-June (the first of several over the years) in which each of us described our positions and what deliverables we expected from others. This revealed the team’s dilemma between the idea that each person should take ownership of those tasks for which she or he had expertise, while keeping everything open to discussion and revision with a wide variety of stakeholders—many not yet identified. This contributed to the slow implementation process—resulting in a paralysis of decision making. As of late June, our logic model was still not moving forward. Finally, my colleague and I agreed to press this issue. We asked our stakeholders if my colleague could more actively facilitate the logic model discussions to get things moving. By then, they all agreed—her experience expediting this process.

By late July 2006, it was clear to our administrative stakeholders that if they remained with the initial organizational model of waiting for a strong governance structure to be in place prior to conducting substantive work, the mental health services transformation would not move forward. They redefined the mission of the initial governance structure to be less authoritative and more advisory. They also made some organizational changes to facilitate the specific workgroups in carrying out their charges for later reviews by the councils. The establishment of the new governance structure, along with its four working councils, took about six months longer than anticipated due to the politics of county appointments (the governance committee did not convene until October 30, 2006). This delay created a major obstacle to SOC implementation because some team members felt that doing work before governance was in place risked putting them in the position of “rubber stamping”—that the transformation process would not reflect the needs and desires of the community. Another factor was the amount of time necessary to recruit the proper candidates for all the community stakeholder team positions. These factors further postponed the adoption of the logic model and a strategic plan, and the recruitment and training of the interviewers. Nevertheless, it was critical that we have an initial draft of our evaluation protocol submitted to our IRB by August 1 so that we would be prepared to start the evaluation simultaneously with the transformation start-up (we did, and the protocol was approved by the IRB on January 6, 2007). We would later submit amendments to incorporate any revisions desired by our stakeholders. Our administrative stakeholders accepted this decision.

The administrative stakeholders and I attended the regional technical assistance SOC meetings in mid-September 2006 in Buffalo. When our facilitators asked about our progress on a range of topics, we came up short. This meeting clearly indicated that we had to further review our internal processes, most of which related to how we would get work approved by our system of governance on a timely basis. Our administrative stakeholders agreed that consensus building would be done in small groups and their products would then be sent for review by the four councils, representing a broad community spectrum. As a result, the logic model was formally accepted on October 10, 2006. The delay of the logic model and strategic plan resulted in our administrative stakeholders moving Monroe County’s care coordination start date from October 1, 2006, to January 2007.

My colleague and I spent several meetings discussing the basics of evaluation with community members, but they struggled with the concepts and did not find evaluation to be interesting. Furthermore, professional and cultural differences between ourselves, the administrative stakeholders, and community members of differing cultural backgrounds resulted in a range of disagreements about how the evaluation should be carried out. Even when governance was in place, we could not get a clear sense over the years how the community wanted to focus the analyses. I had discussed this dilemma at conferences with evaluators from a few sites around the country. The consensus was that to move the evaluation forward, I needed to carefully listen to all the stakeholders, and then make my own decisions about which analyses to conduct and how to present them. I would then give the stakeholders their chance to suggest revisions, if any. This was the approach that I adopted for the remainder of the evaluation. However, this strategy placed my team and I in the position of being seen as circumventing the democratic and empowerment-based intent of the grant. My team and I were becoming what Michels referred to as the oligarchs.

Empowerment Evaluation, Postprofessionalization, and Oligarchy

Although past theory would see empowerment evaluation as deprofessionalizing, we had to use other skills such as diplomacy and teaching our stakeholders how to make their own evaluation decisions and carry them out. But this was a formidable task as evaluation skills are learned in graduate school and subsequent “on the job training,” often with a more experienced evaluator. Even though Monroe County’s mental health initiative had been implementing SOC principles for several years prior to receiving its grant, to apply these principles to evaluation was overwhelming. Several brief, descriptive community reports were produced by one of the county’s subcontractors with data provided by us. However, both my Institute and I wanted a comprehensive final evaluation report, given that we had received substantial grant monies as the external evaluating organization. So the question was how to do this within the democratic principles of SOC? Again, I decided on a modified version of what my fellow evaluators from other SOC sites around the country had told me several years prior. I would make my own decisions about how the report should be structured. Consequently, I wrote a detailed outline of the final report and presented it to the grant’s project manager. Although it took a couple of months, a few of our administrative stakeholder reviewed it, making some refinements. My internal team and I then ran the analyses as per the revised outline, wrote the report, and presented a draft to the grant’s project manager. I asked her and the rest of the administrative stakeholder team to review it for accuracy and clarity. They returned the revisions, and we wrote the final report (Fielding and Weber 2012). So, yes, I did make key decisions based on my specialized knowledge and expertise—I was an oligarch. But rather than being deprofessionalized, I was postprofessionalized because the nature of my skills needed had changed. There was, in fact, no substantial change in the degree of my autonomy.

Four Suggestions for a Smoother Empowerment Evaluation

As my discussion involves no more than my retrospective, no generalizations are possible. Rather, I merely offer four suggestions that evaluators might consider before signing any client agreements. Although my experience was in the context of SAMHSA SOC philosophy, it would be applicable to a wide range of social initiatives using empowerment evaluation. First, evaluators need to determine the potential clients’ system readiness for change and empowerment evaluation. Although I believe that empowerment evaluations should be conducted whenever possible, the extent to which this is possible depends on the willingness and ability of stakeholders to carry out evaluation responsibilities (Cousins, Whitmore, and Shulha 2014:151). A recent study confirmed several key dimensions that a SOC should have in place that indicate a readiness for change (Behar and Hydaker 2009). These include having families and youth as partners, a network of local agency partners, interagency collaboration, a plan to expand services, shared goals, leadership, action accountability, and understanding and accepting the importance of evaluation (see Behar and Hydaker, for a complete discussion). How prepared the program and community are on these dimensions would be a good proxy for determining how well an empowerment evaluation is likely to go. If this assessment does not go well, then an empowerment evaluation might not be appropriate and the prospective evaluator should consider refusing the job, or telling the prospective clients that the relevant caveats would have to be included in the reports.

Second, without being too theoretical, make explicit the conundrum posed by the iron law of oligarchy, that is, democracy’s need for political and technical experts to get things done that, paradoxically, undermine democracy. This is not obvious to most people and so needs to be underscored so that stakeholders are prepared for these types of challenges.

Third, hand the prospective clients and key stakeholders a brochure of the American Evaluation Association’s (AEA) Guiding Principles for Evaluators (http://www.eval.org/p/cm/ld/fid=51). Of particular importance for discussion is Item C, Integrity/Honesty, whereby I tell potential clients not only that I have to be honest with them, but also that the evaluation results have to be in strict accord with the data. This is outlined in paragraph 6:

If evaluators determine that certain procedures or activities are likely to produce misleading evaluative information or conclusions, they have the responsibility to communicate their concerns and the reasons for them. If discussions with the client do not resolve these concerns, the evaluator should decline to conduct the evaluation. If declining the assignment is unfeasible or inappropriate, the evaluator should consult colleagues or relevant stakeholders about other proper ways to proceed. (Options might include discussions at a higher level, a dissenting cover letter or appendix, or refusal to sign the final document.)

This sends the message that the evaluator has an ethical and social obligation to “let the chips fall where they may.” In other words, evaluators working with communities on an empowerment evaluation have a duty not to let the evaluation “run off the rails.” The evaluation design that the community comes up with might not go far enough or be rigorous enough in the evaluator’s opinion, but it must not be framed to bias results. Should disagreements arise down the road during the evaluation, evaluators can back up their concerns on the basis of the Guiding Principles that, in effect, support their professional authority. Again, this technique will only work if these principles were explicitly discussed at the evaluation’s outset. Furthermore, I always include the following statement in all my evaluation reports, “This evaluation was conducted in accord with the Guiding Principles set forth by the AEA,” followed by its current web address. Knowing that this statement will go into the report helps keep my “feet to the fire” during the writing process and lends credibility to the report.

Fourth, if the evaluation requires an IRB approval, use it to work to the advantage of both the evaluator(s) and stakeholders. Writing a protocol forces everyone to come up with a sound plan before the evaluation is allowed to proceed (again, after community input, the evaluator will have to write it in accord with his or her IRB guidelines, and then solicit stakeholder comments). However, writing an IRB protocol involves a balance between being too specific (risking the later submission of a greater number of formal protocol amendments) and being too vague (resulting in a protocol revision and/or risking some loss of community focus). Evaluators without IRB experience who expect to write a protocol would benefit from attending one of the latter’s protocol-writing seminars for principal investigators. IRB review is a time-consuming process. The typical review and approval time at the University of Rochester is typically around three to four months, but this will take longer if the Board wants revisions (IRB boards typically meet once a month, with an additional lead time necessary to be slotted into the next month’s queue). Thereafter, even expedited amendments can take a month (expedited always meaning that the amendment can be approved by the IRB Chair only—it does not have to appear before the entire Board. Expedited does not mean fast).

In conclusion, I am not saying that the deprofessionalization of evaluators is not possible, but evaluators are not being deskilled in most cases. Instead, they are using new skills. Rather, as with the rest of the professions, we are in an era of postprofessionalization. Any deprofessionalization will be offset by the enhanced social and teaching skills needed to “pull the evaluation off.” In my opinion, we will continue exercising our technical skills and making decisions on the basis of our specialized knowledge. Internal disagreements among stakeholders about how to proceed, or lack of their time, interest, or ability to focus on evaluation, are likely to hinder their participation. Furthermore, as Cousins (2005) notes, there will always remain the necessity for external, expert evaluations in those cases where program continuation and/or staffing cuts are being considered.