Abstract
For some of us interested in community development, finding a method of evaluation that truly incorporates and addresses community needs, desires, and opinions is the Holy Grail. David Fetterman’s book, Empowerment Evaluation in the Digital Villages, could be a map to that grail, but you have to place a lot of faith in the author. The book describes case studies of how empowerment evaluation was used as a tool for a Hewlett-Packard-funded program to bridge the “digital divide” in three low-income communities in the United States: East Palo Alto, Baltimore, and across tribal reservations in Southern California. Specifically, Hewlett-Packard wanted to help these low-income, disenfranchised communities to access the Internet and productively use digital cameras, computers, scanners, and printers.
Hewlett-Packard purposely selected diverse communities to create these Villages, and the management structures and methods of each Village were also diverse. The Tribal Digital Village comprised 18 Native American reservations. The Baltimore Digital Village involved a collaboration of African American community-based organizations and the public school system. Residents in the East Palo Alto Digital Village were African American, Latino, and Pacific Islanders, and those involved included community resource centers, employment training programs, and others (pp. 19–20). The evaluations involved interested residents (the “group”) and were led by a subset of residents called the empowerment evaluation team.
The digital divide, which I usually think of in the context of the poor in developing countries, is also an issue in the United States, or at least it was in 2001. It refers to the gap between those who have access to technology and the Internet and those who do not.
Empowerment evaluation, an approach developed by Fetterman and used as part of the Hewlett-Packard project, is a three-step method that focuses on building capacity and improving communities. Building capacity is a broadly used term in development that generally refers to strengthening the skills and knowledge of people and community organizations, so that they can overcome their problems and sustain that improvement without further external intervention.
What makes empowerment evaluation “a radically different view of evaluation” (p. 2) is that the community being served is in charge of the evaluation, with support from an outside “critical friend” (an evaluation facilitator). Fetterman believes that people responsible for their own evaluations are more likely to use the findings and follow the recommendations (p. 7). If true, this approach has a strong advantage over independent evaluations. In the international water and sanitation world, programs are usually not ongoing; typically some sort of infrastructure is provided, perhaps along with some training, and then the development organization leaves the area. Sometimes the organization will conduct its own post-program evaluation and share the report with the funder, but not the community that was the beneficiary of the program. Independent evaluations of such programs, when they are done at all, are often conducted after a project is completed. In both cases, the evaluations come too late to use the results to modify the program. Furthermore, the reports often sit on shelves (or hard drives) and are rarely used for decision making moving forward. It seems that empowerment evaluation could be a good tool for adaptive management in international development projects, but I wonder how flexible a funder would be with changing the methods or even outputs of the project. For purposes of accountability, I still believe in the usefulness of independent evaluations.
It is not clear from this book whether empowerment evaluation is intended to last officially after the project is completed. It is supposed to become part of the fabric of an organization (p. 31), but Fetterman does not describe whether the community organizations are still self-evaluating. My particular area of interest is sustainable services, so I was curious to know the current status of the three Digital Villages supported by the Hewlett-Packard program. Have they fully bridged the digital divide? This project began in 2001, and the book was published in 2013. Fetterman claims the projects were successful, and even says that “building capacity is what sustainability is all about” and “capacity building requires repetition”(p. 98), but does not describe what is going on in the communities now. His thesis could be strengthened by sharing data on the levels of access to technology and the Internet before the intervention, and now.
Fetterman says that self-empowerment produces “real world, credible outcomes” (p. 107) and a “long list of impressive outcomes” (p. 110) and that “small businesses around the country flourished as a result of Digital Village training and expertise” (p. 107), but frustratingly provides no data. Out of curiosity, I did quick web searches on the three digital villages. I did not see any current information on the Baltimore Digital Village (other than Fetterman articles), which Fetterman describes as having had the most challenges starting up. The East Palo Alto Digital Village seems to be the most active and has been integrated into a larger effort, One East Palo Alto Neighborhood Improvement Initiative (http://www.oneepa.org/history). They have brought in much additional funding. The website for the Tribal Digital Village (http://sctdv.net/node/129) shows indications of success and current activity, and even some additional grant funding:
We currently have 10% of homes connected in 4 of the 17 inhabited reservations. 270 homes have adopted broadband … . We will continue as long as there are homes that want service connections. We are providing broadband education, awareness, training, access, and support to Schools, libraries, medical and healthcare providers, and other community support organizations
There are 60 community anchor institutions, public safety entities, and critical community organizations passed and/or involved with project. All community anchor institutions are connected for FREE!! They have been for 10 years and will continue to receive service free of charge. New tribal facilities also get this same opportunity to have broadband Internet access at no cost.
It is hard to imagine how they are able to (or will be able to continue to) provide these services for no cost; Fetterman does not provide information on this issue. The sustainability of any community intervention program after the initial funding has ended is often a concern. I would have liked to learn from this book whether an intervention that involves empowerment evaluation is more likely to succeed on its own in the long term.
The author promotes empowerment evaluation as the ideal tool for capacity building. Capacity building is an important area of focus for my field, international development. However, Fetterman does not directly compare the results gained using this tool to those using other methods. Wouldn't it have been interesting if the community members (the evaluators), perhaps with support from Fetterman, had written this book?
Fetterman wrote this book “for citizens who are committed to constructive, progressive social change” (p. 3). He also wrote the book for his academic colleagues, including evaluators, and grabs our attention with a provocative statement: “Many evaluators have long held the belief that only they can conduct evaluations. They are the only ones who can be objective, honest, critical, and accurate. However, as this and many other examples confirm, that view is not only wrong, it is misguided and no longer productive” (pp. 6–7). Perhaps there is room for a spectrum of evaluations to meet different needs.
The issues that empowerment evaluation tries to address relate to a theme that has also plagued the international development sector: broken feedback loops. In the marketplace, businesses provide a good or service, and based on feedback from their customers, refine and improve the good or service, which then benefits the customer. In the charitable world, charitable organizations often treat the funder as the customer. Because resources are scarce, charitable organizations change their behavior more based on how they can get donations versus whether their programs are causing the desired impacts. Opinions and desires of the real customer (the people in the communities who benefit from the program) are rarely meaningfully sought in the program design, and customer satisfaction is almost never solicited or measured after a program has ended. Thus, it is interesting to think of how empowerment evaluation would work in the developing country setting, where rural communities are often remote and spread apart. As I read about the process, I wondered whether communities (rather than organizations) would be able to embrace and implement the principles of empowerment evaluation. How often would the critical friend be able to visit?
If you only have time to read one chapter, read “The Engine,” which describes empowerment evaluation. The guiding concepts are as follows: a culture of evidence (much needed in social development), a critical friend, cycles of reflection and action, and a community of learners. By working together toward a common task, empowerment evaluations create a safe environment to brainstorm and improve practice (p. 33). This echoes the results of my evaluation of a partnership in Ethiopia, where I found a vibrant network of interorganizational knowledge transfer. I found the descriptions of single, double, and triple loop learning (p. 35) useful ways to frame the work I am supporting on a current program.
Fetterman shows some internal conflict about the role of Hewlett-Packard, the sponsor. While “it is important to pull people into every step of the process and to allow them to take control of it” (p. 45), he seems horrified that the Baltimore Digital Village both turned down help from Deloitte and Touche and “uninvited their sponsor” from a site visit, and mentions it throughout the book (pp. 49, 52, 113). He describes the leaders of the Baltimore Digital Village group as dysfunctional, “bordering on organizational arrogance” (p. 49).
While Fetterman says he wrote the book for “the skeptic who no longer believes that social change is possible,” the very first chapter is filled with hyperbole. For example, “It is that insatiable thirst for a better life that sustains the often Herculean effort required to transform the world around us” (p. 1). Or this book can serve to “revitalize the triumvirate of philanthropic, academic, and community forces throughout the United States and the world” (p. 3). Or, “there is something almost magical about people physically placing their dots and ratings on the poster paper” (p. 86).
Speaking of engines, the car-racing analogy that Fetterman uses throughout the book is perhaps most off-track (pun intended) in the title. How can you talk about empowerment and then call it Hewlett-Packard’s “race toward social justice”? If there is anything less analogous to the process of achieving social justice, it is a race. First of all, a race implies one person winning, at the expense of others. That is the opposite of social justice where principles of equality and solidarity for all people or classes are emphasized. Second, the social justice experience is more marathon than sprint. Calling empowerment evaluation “the Engine” of the program seems to imply that the evaluation was more important to achieving the goals than the “bridging the digital divide” initiative itself. Other chapter titles have race-related names like “The Fuel” (what the book is about), “The Sponsor and the Crew” (Hewlett-Packard and Stanford University), “The Starting Line” (the mission of the project), “The Pit Stop” (the process of taking stock), and so on. It comes as no surprise when Fetterman coyly lets us know that he has raced Ferraris (p. 82).
This was the first paperback book I have read in a while (I usually read e-books), so I found the formatting of the endnotes distracting, especially when they just referred to a Wikipedia entry. I read the book on a plane, and did not have ready access to a computer. It is difficult to figure out where to find the right endnote, because you have to flip back to figure out what chapter you were reading (endnotes have the chapter number, but not the title), then back to the endnotes, and then back to the page you were reading.
Overall, I found the premise of the book intriguing, but the evaluator in me wanted some data—or at least some information on how the Digital Villages are doing currently—to support the dramatic claims of success. To extend Fetterman’s analogy, it is a pretty car, but I would like a mechanic to look at the engine before I buy it.