Abstract
This article discusses the work of Judith Tendler, a development economist, consultant, and evaluator from the perspective of her contribution to evaluation research. It argues for her role as a forerunner of many contemporary evaluative concerns and the enduring vitality of her unconventional views. Her reflective practice is built on the doubt and surprise that emerge during fieldwork. It aimed to surface what worked in adverse conditions, and the lessons that could be learned across development interventions and across different domains. The article hinges around the many consequences of searching for “how projects could have worked well”: from challenging accepted theories, to developing specific research methods, and being an ethically oriented evaluator.
Introduction
Judith Tendler (1938–2016), a development economist who also engaged in consultancy and evaluation, is little known among the evaluation community at large, notwithstanding her contributions anticipated many current and enduring evaluation debates. In this article, I review her reflective practice in order to abstract from her diverse outputs (disseminated in gray literature as well as articles and books), 1 themes that are relevant to evaluators today. I will focus, first, on how to understand success and failure in development interventions and, second, on morality in the evaluation profession.
In dealing with these topics, this article has a double aim: vindicating Tendler’s role as forerunner, and acknowledging the enduring vitality of her unconventional views and research methods and their fruitful application to evaluation.
The article is organized as follows. After a brief presentation of Tendler’s intellectual profile, “Part I—Success and failure” deals with the way Tendler put the discovery of what worked (success) at the center of her endeavor. This has both methodological and theoretical implications. Methodologically, Tendler focused on the importance of surprise; a kind of comparative analyses that takes success as a benchmark; evaluation questions framed around problems and solutions and theoretical questions—challenging received theories of social change.
“Part II—An ethically oriented democratic professional” deals with her way of being an ethically-oriented, democratic professional. Given the particular kind of professional she was, this meant a specific way of being independent. Tendler was very open in declaring the values she was pursuing (helping the poor, improving the workings of the public sector in the service of the people), and her efforts favored progress consistent with those values in the programs that she evaluated. She combined interest in the public good (democracy, development, social justice) and a sincere concern for the disadvantaged with appropriate methods of research.
“Part III—Discussion” weighs up Tendler’s contribution, and what we can learn from her. It will highlight the fields for which she has been a forerunner for a large group of evaluators (from fieldwork to theory-based evaluation, to positive thinking) and those for whom she continues to be provocative by proposing something “new and good” for the current evaluation discourse: on independence and ethical issues, on theories of change, and on evaluation designs and methods.
Tendler’s intellectual profile
Tendler was raised as a development economist, and she defined herself as such, notwithstanding her different roles, and the variety of disciplines she utilized, and had a special bent toward linking economic, social, and institutional aspects. Since her PhD dissertation on Electric Power in Brazil (Tendler, 1965, 1968), she collaborated with Albert Hirschman, then her supervisor, in elaborating an original outlook on development that focused on the “personality traits” of programs 2 and their consequences for development: the relationship between politics and economics, between technology and administration (the role of tasks, the opportunities for mobilization). She borrowed from her mentor an attitude to “look where other people would not have looked” (Tendler, 1968: xi): side-effects, unintended consequences. And she shared with Hirschman the desire to praise the efforts of people living in the less developed countries (LDCs), and “to ‘sing’ the epic adventure of development—in challenge, drama and grandeur” (Hirschman, 2015 [1967]: xvi). 3
Her personal way of proceeding along those paths, innovating development theory and practitioners attitude alike—that she combined under the label of “lessons learning evaluation research” (Tendler, 2018: 245)—was forged during the long period (the 1970s and 1980s) in which she lived as a consultant to development agencies. Her way of looking at reality, mainly through field research and direct involvement, gave her the authority to act as a really “independent” professional, who would take international agencies to task on their own declared goals, 4 or support some agencies in keeping to their democratic inspiration. 5 She was a different kind of evaluator: committed, moved by the moral tension of improving the conditions of the disadvantaged, as evidenced in her research methods aimed at discovering what was in their interests. And in her reflective practice, she would continuously turn down opportunities to study development when evaluation commissions that were offered her adhered to contemporaneous evaluation practice as advocated by international aid agencies.
When she took up a teaching position at MIT, she would develop her method of “teaching cum research” 6 based on her special way of transmitting to students who would become her collaborators in research campaigns, mainly in Latin America the ability to get surprised.
Part I—Success and failure
Looking also elsewhere
Tendler’s approach was based on doubt (not taking for granted what in a project would be considered success, or failure) and on evidence (collected in fieldwork, by “observing” 7 projects). She was “looking at” the unexpected—“what surprised you?” she kept asking. This also meant being interested in cases that were “not representative.” 8 They could be positive but also negative cases, although her preference was for the positive ones. 9 In “Bringing Hirschman Back In” (Tendler and Freedheim, 1994a: 177), she (and her co-author Sara Freedheim) “confess to a bias toward the positive, but that does not worry us because it may help to counterbalance the stronger bias toward the negative in the sea of literature that surrounds us.”
Tendler (1993) was keen to discover “how some projects could have worked well,” “how they were able to perform well despite the presence of such adversity,” as she stated in New Lessons from Old Projects. Here she noticed that past evaluations of the World Bank’s (WB) Operations Evaluation Division “have been more illuminating about the causes of failure than about the causes of success,” which has meant throwing “more light on what not to do than on what to do,” and then she announced that her study will seek to do the opposite (Tendler, 2018: 165). 10 However, she knew that projects that had worked well were “episodes of good performance that had come and gone, as distinct from consistently ‘good’ agencies, component or projects.” Good performance had “less to do with the inherent capabilities of an agency itself, than with a set of other factors”: the task, outside pressures, built-in incentives to perform, “the involvement of keenly interested actors and organizations at the local level” (Tendler, 2018: 166, 167). She proposed to inquire into these factors, with no limitation of discipline or topic.
I would characterize her approach as “Looking also elsewhere,” at other aspects of projects than the expected, at other disciplines. She suggested to “look open-handedly at what the organization [or the project, we could add] had accomplished, regardless of its objectives: then compare the reality and the objectives. Does the reality shed light on the objectives?” (Tendler, 2018: 135). A distinctive form of goal-free evaluation.
Received theories of change and surprising findings
Being well aware of what mainstream theory maintained in the cases under scrutiny, Judith was not interested in confirmation or falsification that is, in theory testing. Instead she asked what happened when the theory did not cover what she found: she was aiming at what was missing, where things could go in a different direction. And when she had found something positive or un-expected, she inquired into the consequences that had occurred in people’s well-being and policy improvement, and the set of interrelated aspects that the positive surprise suggested to consider.
In Tendler’s way of looking at things, there are neither pre-requisites for success nor predetermined obstacles, but instances of success or failure to be accounted for. Comments on Evaluations of Bid-Financed Rural Credit Programs in Six Countries (Tendler, 1970) deals with the results and recommendations of six evaluations of country studies conducted for the Inter-American Development Bank. In contrast to “unsurprising” failures (that would confirm the presence of obstacles)—Judith Tendler (2018: 11) suggests—considerable attention should be given to the unexpected cases of failure, where all or most of the pre-requisites were in place. Similarly, successes should not be considered as exceptions (to the rules, and therefore the theories)—why they happened should be explained.
[The problem behind] this kind of evaluation is that one knows, by definition, the answers to why things worked or didn’t work before one starts. The evaluation tends, therefore, toward categorization rather than toward a more open-ended and analytic exploration. One tallies up the problems and the achievements, and then places them into their appropriate box: existence of the classic prerequisites, lack of them, big pushes, exogenous circumstances, and exceptions to the rule. It has long been recognized, however, that prerequisites often turn out to be the result rather than the cause of development, that progress on one front often sparks—rather than being dependent on—progress on another front, that “big push” successes often turn out to be a function of factors unrelated to the push, and that exceptions to the rule often, upon close examination, lead to the discovery of new “rules.” (Tendler, 2018: 9–10)
This sounds like a sweeping criticism of linear theories of change and some impact evaluation methodologies, and an acknowledgment of instances of complexity and contribution that require appropriate theory building and research methods. Such an attitude can be perceived in all of Tendler’s (1973; Tendler and Freedheim, 1994b) writings, sometimes even in their evocative titles: Trust in a rent seeking world that runs against main tenets of new institutionalism, or The trouble of goals with small farmers programs (and how to get out of it), which is a critique of the assumption that efficiency and equity and the like necessarily go together. She favored keeping the tension open between the social and the economic.
In this particular way of challenging contemporary program theories, it is as if she wanted to do what she had reproached the big donors for failing to do. In What Ever Happened to Poverty Alleviation? (Tendler, 1987), she criticized the Ford Foundation for having prematurely abandoned state-sponsored poverty-alleviation programs—of which she found cases of success—for an exclusive focus on markets and macroeconomic policies, as suggested by current economics theories, which “resulted in an abundant chronicling of failures and what caused them, but very little understanding of the more successful efforts and their ingredients” (Tendler, 2018: 162).
Sometimes she approached things the other way round, as when she criticized applied social research in development for not utilizing for a given subject the findings of basic research on developed countries: in this case, the “elsewhere” to look at were theories in other literatures and other contexts. For example, the literature on the public sector in developed countries analyzes the behavior of street-level bureaucrats (Lipsky, 1980), and admits of different rationalities when compared to the stereotype of the ritualistic bureaucrat, while for LDCs there is only the picture of ritualistic and lazy bureaucrats. And in general, “even though the political-science literature has long left behind a view of the state as monolithic and unitary, the applied development literature nevertheless sounds as if it still sees the world that way” (Tendler, 2018: 175). She then proposed to look at the public sector in LDCs with the same eyes as with the public sector in Developed Countries, and she identified cases of responsive workers and flexible agencies, and considered under what conditions this had come about.
Analogously, In Good Government in the Tropics (Tendler, 1997: 5), she noted that the literature on industrial performance and workplace transformation, which refers to developed countries, fosters dedication among workers, while the donor community assumes that civil servants in the LDC are inevitably self-interested. In Social Capital and the Public Sector (Tendler, 1995), she asks, “When the subject of worker participation in restructuring of large firms is in such vogue in the U.S. literature, why hasn’t this interest in workers and their associations spilled over into the development field?” (Tendler, 2018: 177).
Comparative analysis
Tendler rings an alarm-bell for evaluators with her disregard for “evaluation,” and her preference for “studies.” She does it in the name of the research methods that she utilizes most of the time, that is, a personal approach to comparative analysis. In Impressions on Evaluation (Tendler, 1983a: 2), she states that evaluations are limited to the process of making decisions about projects, and addressed to a specific commissioner. Studies, on the contrary, can make comparative analyses across sectors, finding out “what the impact of a project has been, in terms of general lessons that might be learned from this particular project experience.” With an eye to public interest, she stresses that comparative analysis is where it is possible to evidence what worked versus what did not, thus offering the general public clues for improving policies.
Comparative analysis was conducted through an iterative process, aiming at discovering “why a particular problem did not occur, or if it did occur it did not prevent improved outcomes,” and for any such instance looking at where (and why) things worked well or badly (Tendler, 2007, 2018: 249). This is in fact the way she challenged traditional oppositions posited in the literature. Take the dichotomy between private/public: she showed a need for understanding where and when something (private or public) was good and where not, and opportunities for combination; or she would not object that nongovernmental organization (NGOs) are closer to the people than public administrations, but would suggest to look also at cases where they are not (Tendler, 1995, 2018: 172).
But comparative analysis had to be crafted for the task. The correct way of comparing was not dictated by the unit of analysis per se (as in the conventional tenet of comparing apples with apples and oranges with oranges), as it could be if comparing agencies with agencies, projects with similar projects, NGOs with NGOs. On the contrary, if the goal was that of finding out what worked and drawing lessons for improving the life of the poor, Tendler would compare private and public organizations, large and small. The focus could be the issue (urban vs rural poverty), the task (administrative vs productive), technical aspects (labor-intensive vs technology-intensive). Here we can find an anticipation of contemporary cross-evaluation “thematic” assessments, and the reasons for realist synthesis, that is to say the aggregate knowledge that is gained not by cumulating results on the same variable but comparing horizontally across mechanisms and contexts. For example,
In Tendler’s (1968) original research on Electric Power in Brazil, she compared hydro-electric versus thermo-electric systems, and concluded that
the construction and operation of hydro as opposed to thermal in a developing country, although incurring a greater real capital cost than is usually believed, also confers real benefits on the economy, in the form of stimulation of local production, in the creation of skills, and in training for planning. (Tendler, 1965: 237)
In What to Think about Cooperatives (Tendler, 1983b, 2018: 139 ff.), she compared four cooperatives (or groups of) for the tasks that were allocated to different types of participants (technicians, workers, leaders, etc.), the capital needed where some crops required more capital investment, hence discriminating between farmers and poor laborers.
In New Directions in Rural Roads (Tendler, 1979, 2018: 84), she compared labor-based rural roads to equipment-based construction. She showed the advantages for a poverty-alleviation policy of labor-based roads that required traditional skills, greater involvement, and were also cheaper. And then she assessed where they were most suitable.
In What Ever Happened to Poverty Alleviation (Tendler, 1987, 2018: 160), a paper where she argues at length for comparative analysis, she compares public-sector enterprises with NGOs (of different kinds) and assesses their performance as to the number of people reached, the competence acquired in performing their tasks, the influence on policies affecting poor people. She then analyzes the common traits of programs, such as mode of organization and objective that went well.
In Rural Projects through Urban Eyes (Tendler, 1982a, 2018: 122 ff.), Tendler assessed the results of rural projects by keeping in mind the lessons that had been learned in the evaluation of the urban projects of the War on Poverty. For example, urban projects were more successful when focused on areas of concentrated poverty, where rich people were not interested, and this could happen also in dispersed rural areas. Rural projects were more subject to political influence of the rich, unless their success required universal participation, as with vaccination programs.
Judith Tendler was so convinced of the usefulness of comparative analyses for finding out what works in development projects that in What Ever Happened to Poverty Alleviation, she suggested that “if the Foundation’s programs are to strive toward impact, then they will also have to create a record of what has worked and what has not.” And she concluded that funding these comparative evaluation studies would also “restore academic prestige, and therefore power, to this particular subject matter” (Tendler, 1987, 2018: 162)—a way of showing consideration for her craft of evaluation.
Framing the evaluative questions
Tendler did not stop at contemplating discrepancies between theories and discoveries. Whether she was commissioned to undertake research (as in Comments on Partnership for Capacity Building in Africa, Tendler, 1998a) or she originated her own research (as in Good Government in the Tropics, Tendler, 1997), when the cases she was looking at revealed that the relevant mainstream theory underlying the projects did not hold, she would change the questions to be asked, and look at places where things worked differently (a form of possibilism, à la Hirschman), that allowed her to enlarge the scope of possible routes to development.
In The Rule of Law (Tendler, 2007), a paper dealing with the Brazilian intervention of WB and Department for International Development (DFID), Tendler (2018) framed the report around the need to overcome the supposed trade-off between economic growth and reducing inequality: then she identifies promising aspects for inquiry: “I identify four themes together with research questions, implications, and case illustrations” (p. 246). They are linkages and spillovers; the interaction between the rule of law and economic development; institutionalizing the mediation of conflict; and modernizing the state, discretion, commitment, and reform fractions. For each of these themes, she formulated questions based on the result of comparative analysis. With reference to industrial sectors, under what circumstances and in which kinds of sectors is the impact greater and more broadly distributed? With reference to regulatory systems, why some regulatory actions have had these positive-sum outcomes for economic development and for improving the rule of law, and others have not?
In Social Capital and the Public Sector (Tendler, 1995), she challenges the idea of the clear demarcation between the public sector (rife with corruption, laziness, etc.) and the outside world of private associations where social capital thrives. And she deplores the fact that there has been so little research on cases where government performs well and has favored the formation of social capital. Thanks to the pathological view of government bureaucracy, “the question of how positive Social Capital Formation has been able to take place in the public sector becomes quite a mystery—one of the basic ingredients of a good research question” (Tendler, 2018: 175). Then, for example, she proposes to inquire into the different groups that compose a single agency, in order to understand how the ascendance of one group may be more favorable to social capital formation (and she notes “I have seen this happen many times in my own field-work”), or to inquire into how public-sector trade unions could “play a major role in making [and not only on breaking] the reforms needed to be undertaken by Less Developed Countries today” (p. 176).
Part II—An ethically oriented democratic professional
Building on Dzur’s distinction between types of professionalism (social trusteeship, technocratic professionalism, democratic professionalism), Schwandt (2018) has recently inquired into what ethos of public good can characterize specific kinds of democratic professionals, with special reference to evaluators. Tendler’s profile fits perfectly that of the democratic professional. Her ethos of public good was openly spelled out. It was based on two pillars: (1) helping the poor, improving people’s life and ability to solve economic and social problems and (2) improving the working of the public sector in the service of the people. And it was reflected in her personal attitude toward commissioners of evaluations and beneficiaries of programs alike, and in the way she assembled her methodological tools.
Tendler’s attitude to her clients was one of complete autonomy: respectful of their work, but not hesitant to make her criticisms heard. She felt the need to say what she thought worked and what did not in projects with goals of poverty alleviation, fostering economic development, building capacity of the public sector in providing services to citizens. Her judgments would rely on her findings, not on the program’s assumptions, that she often challenged. She knew what she was talking about, based on her reading of specialized literature and on what she had “observed” in her work on the field—interviewing local operatives and professionals, public-sector managers and front-line workers, and meeting and discussing with beneficiaries. The accuracy of her reflections, and the authority that came from her intellectual abilities (Bianchi, 2018), helped keep the relationship with her clients going, although one can question whether the lessons that she drew from her work were indeed learned by her interlocutors. There are of course differences in this regard. While, for instance, Peter Hakim at the Inter-American Foundation (IAF) openly acknowledges the value of her work, 11 as did Picciotto in his WB role heading up Operations Evaluation Department (OED), 12 most big clients appeared to take fewer lessons on board, notwithstanding their continuous engagement with her, and the indirect influence that she has certainly exerted on field operations and territorial team leaders.
This difference between IAF and big donors was well understood by Tendler (1981), as can be seen in Fitting the Foundation Style: the Case of Rural Credit where she appreciated the comparative advantage of the former in its flexible methods that suited its “canons of behavior.” In fact, IAF funded primarily NGOs, supported organizations in which the poor participated in decision-making, and “believes strongly in a donors-grantees relationship with little intervention from the donor” (Tendler, 2018: 108). Hence, she advised IAF against adopting “technically rigorous” evaluation methods that were not fitted to its scope and mission. She suggested rather that IAF concentrates on “how can evaluation be done in a way that maintains the Foundation comparative advantage and respects its way of dealing with grantees—rather than playing havoc with those ways in the attempt to do ‘respectable’ evaluation work” (p. 107).
Helping the poor
Tendler was concerned with improving the way of life of the people targeted by the projects and enhancing their capacities to solve problems. She had a special sense for discovering which programs favored the poor, from the perspective of distributive justice, equity, and democracy. And she took into account the consequences of different resources or project characteristics for development. For example, different tasks (e.g. production vs administration) are better or worse for poor people’s empowerment; some crops are more “democratic” than others (coffee better than rice); some projects allow for more participation than others. New Light on Rural Electrification: the Evidence from Bolivia (Tendler, 1980, 2018: 88 ff.) contrasts productive goals versus social goals in a program of rural electrification that is supposed to favor the poor: social goals are best suited, and they should therefore be pursued with greater vigor.
Her sensibility to these issues translated into equity-oriented research methods. In a section of her “Suggestions to evaluators” (Tendler, 1982b), 13 titled “Participation, benefit distribution, innovation, cost,” she advised evaluators to
Watch for examples of, or opportunities for, targeting the poor by type of activity—for example, low-status activities and goods, absence of elite interest in participation in the activity, class-based organizations, and so on. Are these opportunities being exploited, and if not, how might they be? (Suggestion 13)
With respect to project activities involving women, determine to what extent they augment women’s income-earning capacities and other forms of power, and to what extent they reinforce women’s traditional role as homemaker. If the latter is the case, make suggestions as to what changes in project design would be appropriate. (Suggestion 15)
Where there is a community contribution to projects, ascertain its distributional burden. For example, voluntary labor might fall disproportionately on the poor while contributions in cash or kind might fall, disproportionately, on the rich. (Suggestion 16)
“Looking at groups that have done well in regions or countries undergoing particularly adverse economic conditions”—Tendler (1983a: 58) wrote in Impressions on Evaluation—would provide inputs for improving the lives of poor people. And she added, I think the topic is a neglected one because we are so committed to the idea that the poor sectors of the population suffer more than most, and that they are undergoing particular stress these days. It is contrary to the nature of our political commitment to dwell on success—because it may be looked at as saying that things are not that bad after all. I think it is extremely important to overcome our ignorance about how the successful groups have survived, because it will help us to make better decisions about what kinds of projects to support under adverse economic conditions.
This thrust to offer encouraging insights was not thwarted even when her findings showed that the expected benefit did not take place. In The Fear of Education, Tendler (2002) discusses the current belief that in a globalized economy, there is a necessary link between education of the workers and profitability: on the contrary, she had found that in some developing countries and in some industrial sectors, there is an economic rationality in keeping education low. This led her to the suggestion to find a better reason for educating the workers: “researchers of political economy and policy reform might explore the historical experiences of other countries—including in other times—to find ways out of the trap” (Tendler, 2018: 243). For Judith Tendler expanding literacy is a moral issue that justifies finding out how best to achieve this goal, using a comparative method that allows drawing lessons from different situations and different times.
Improving the working of the public sector in the service of the people
Tendler attributed a crucial role to the public sector in the policies of alleviating poverty. She was looking for the contribution of responsible public servants as well as of dedicated policy-makers. She looked at both of them as people, not as anonymous social categories, and she was happy when she could find examples that supported her perspective.
In this connection, she criticized two types of literature. First, the literature which was premised on the superiority of the market mechanism vis-à-vis the inefficiency of the state (depicted as full of bureaucracy, laziness, corruption). Good Government in the Tropics (Tendler, 1997) is moved by disappointment on how theories of development stemming from that literature have shaped programs (bad theories = bad programs): and she demonstrates that good research can dismiss both the social theory (in general and applied to LDCs) and program theory. 14 This book is based on her research on North Eastern Brazilian regions, where she found cases of good government in different sectors—rural preventive health, employment-creating works, construction programs, agricultural extension, and assistance to small enterprises. In particular, she reflected on the findings of government workers dedicated to their work, state government creating a sense of mission around these programs, workers carrying out voluntarily a larger variety of tasks than usual, trust between public servants and citizens, and good relationships between central government, local government, and civil society (Tendler, 1997: 14–15).
Second, Tendler criticized the literature on social capital, booming after the seminal Putnam (1992) publication, which assumes that it is an offspring of civil society (associationalism) against the government. She showed that government can help the creation of civil society associations that in their turn can mobilize people to have better service delivery. She believed in the necessary alliance between agencies, NGOs, workers associations, and social actors. Examples of cases in which social capital was the result of the collaboration of state agencies and civil society groups had already been found in What Ever Happened to Poverty Alleviation (Tendler, 1987), and are well present in Good Government. All in all, her way of “thinking differently about public-sector reform” should help “formulate advice by drawing on cases of good performance” (Tendler, 1997: 8).
In line with this orientation, Tendler was skeptical of the suggestion—so widespread in international agencies—to keep away from politics 15 and recognized the push that local and regional policy-makers had given to democratic reforms. In Rural Projects through Urban Eyes (Tendler, 2018: 123), she noticed “the positive effect that supportive and powerful political interests may have on the project.” Elsewhere she suggested to find ways for politicians to be interested in good performance, by pushing for results, for instance, aligning the duration of public office with length of programs (4 years) as in New Lessons from Old Projects (Tendler, 2018: 170).
Training evaluators and students
Tendler was very generous in sharing her experience with practitioners and students, in helping them become comfortable with the real world and with messy organizations. This probably harks back to 1968 when she had appreciated Hirschman teaching her to be as a “patient enough” researcher so as to “find a rich complexity of both success and failure, efficiency alongside incompetence, order combining with disorder” (Tendler, 1968: xi). This lesson was evidently carried forward in her career.
The main aspects of her pedagogic mission were spelled out on behalf of evaluators, as expressed in “Suggestions to evaluators,” already referred to.
Appreciating people’s contribution
In recommending fieldwork approaches, Tendler suggested how to elicit a fruitful dialogue:
The way to ask questions: “don’t ask people what the impact was; ask, rather, ‘what happened’ and then ask ‘what happened next?’” (Suggestion 18).
Of whom to ask questions, when and where; “ask the people what difference the new practice made in their lives; don’t ask only the project staff” (Suggestion 27).
Junior and field staff have more contact with and understanding of beneficiaries than persons in managerial position [. . .] try to talk to these persons away from the office. The best opportunity for this is to take jeep trips with these persons to visit faraway beneficiaries or project sites. (Suggestion 54) much of what is to be learned about the project will come from interviews and not documents [. . .] evening hours should be taken advantage of [. . .] “hanging around” in the communities where the project takes place, eating and drinking with local people or local staff. (Suggestion 57)
Venerating success
Success and failure are the result of a combination of elements: the evaluator should treat any successes with a sense of awe. Do not be content to say that something worked well, but venture an explanation as to why it worked. Explain what is happening in the project against a background of what is predictable and what is a surprise. (Suggestion 39) Certain problems experienced by projects are recurrent and therefore not a surprise—e.g. faulty maintenance, lack of coordination between agencies, lack of funds for operating costs, schools without teachers, health clinics without doctors—[. . .] look for cases where the expected problems are not recurring, and then try to explain why they did not appear. (Suggestion 45)
Equity. “do some activities seem more appropriable by elites than by others?” (Suggestion 9). “what aspects of the project, if any, seem to be reaching the poorest stratum of the population? Why?” (Suggestion 11).
Possibilism:
watch for examples of, or opportunities for, targeting on the poor by type of activity—for example, low-status activities and goods, absence of elite interests in participation in the activity, class-based organizations, and so on. Are these opportunities being exploited, and if not, how might they be? (Suggestion 13)
“Be alert to the possibility that certain achievements will have been made because of the disorderliness of the organization, and not despite it” (Suggestion 50).
Tendler had this experience in mind when she took up a teaching position that would have offered her the opportunity to train researchers competent in the type of studies she was promoting. Apart from the memories of people who have benefited from her direct teaching (Bianchi, 2018), she has left hints at this in revealing documents. Read, for instance, this syllabus of a course in “Analyzing projects and organizations” (Fall 1995)
16
: “This course teaches students how to understand real organizational environments [chaos] and to be comfortable and analytical with a live organization”: do not “retreat to the tidier world of numbers and written words.” For this reason, the course would teach students to use spoken words: Being attentive to what people say: developing skills in using spoken words as raw data for analyzing the behavior of organizations, and for understanding the meaning of numbers. Paying more attention to the spoken word can help us decide how to choose the numbers and the writings that will be most important for good and interesting analysis.
At the same time, she would consider the class as a collective, would ask students to interrogate themselves (“what you are really interested in”) while interrogating reality, and to immerse themselves in multidisciplinary literatures. Students were also advised to engage with problems before getting to general principles or rules of proceeding. And a way of doing it was learning from the problems of their colleagues in the class: “during the discussion of the project of any particular one of your classmates certain suggestions that are applicable to your work will emerge.”
. . . the characterization of the African development experience seems too monolithic—painted as a failure, as rife with corruption, administrative and political incompetence, and with civil society nowhere to be seen, either because it is repressed or too weak to make a stand. [. . .] What is problematic about the monolithically negative portrayal for the PCBA is that such an initiative needs to be built on an understanding of why some things have worked (within countries, as well as between countries) and others have not. [. . .] The monolithically negative portrayal is also problematic because it is like typical donor portrayals of Africa that many Africans have found insulting. In that sense, it is somewhat jarring in a report claiming to represent a more African view. [. . .] The “systemic” nature of the portrayal of Africa’s problems—all bad things go together in an analytically neat and closed circle of underdevelopment and incapacity—hinders one’s ability to figure out how to intervene, to identify the point of entry. The similarly “systemic” nature of the report’s recommended approach—“integrated” and “not piecemeal” (p. 3)—also seems unrealistic in light of what has been learned from past experience with “integrated” approaches. If the analysis of Africa’s problems could show contrasts, contradictions, and jagged edges, including some bright spots, this would help indicate paths of entry into the problem. [. . .] Finally, the portrayal of the African continent as basket-case implies a kind of African “exceptionalism” that, regardless of its justifiability, makes it difficult to learn from and apply lessons being learned in other, non-African countries. If Africa is so uniquely bad, and so resistant to development as a continent, then the solution must also be unique. There’s not much to be learned, that is, from the experience of other continents.
Part III—Discussion
At the end of this long journey through Tendler’s ideas and approaches, it is time to draw some lessons from her work and thought. In her times, she was an eccentric figure. She “did development differently” (to borrow the title of an apposite blog) when development evaluation was mainly perceived as a technical expertise about the return of investments. Although, as we have seen, she was highly respected, the full range of her heuristic tools was not shared by her colleagues or commissioners: this is perhaps the reason why she wrote so much about how to do things. Today we live in a different landscape. Now that complexity, emergent traits of programs, or recursive causalities are on all evaluators’ agenda, Tendler’s contribution can at last be appreciated in all the depth and wealth of her insights.
These concluding remarks, sum up, first, some traits that vindicate her as a forerunner of positions held by evaluators today, and, second, those traits that acknowledge the enduring vitality of some of her dissident views, that may help orient us in current debates.
Tendler as a forerunner
There are many instances of evaluation practice and research that have slowly gained credence in the evaluation community for which Tendler offers an antecedent. More than that, going back to Tendler’s ideas can offer strong arguments for justifying their current relevance and use.
Fieldwork
It is well accepted nowadays that evaluators should do fieldwork where the program is implemented, possibly through participatory methods. This notwithstanding, it is striking that an opposition continues to be asserted between qualitative and quantitative research methods, between contingency and regularity, subjectivity and objectivity—dichotomies often reinforced by proponents of Evidence-Based Policy. Tendler went to the field where programs were implemented, and observed what was happening during implementation and how people reacted to programs, in order to draw lessons and possibly generalize about good results. In doing fieldwork and utilizing participatory tools, she was respectful of people’s attitudes and values: she took local knowledge seriously, she appreciated what people said and did, and tried to incorporate it in her judgment of the changes that she was observing. In fact, going to the field was part of her theoretical approach based on an openness to surprise whereby a diverse and complex reality could push her to challenge received theories. Conversely, once Tendler identified a problem through her fieldwork engagement, her evaluation design would also consider utilizing quantitative research methods. In a very real sense, Tendler anticipated mixed-method evaluation designs.
Positive thinking and learning
In a sense, Tendler is a forerunner of positive thinking approaches like Appreciative Inquiry and Most significant change and other ones (Stame, 2014, 2016). Such “positive” approaches are consistent with Tendler’s way of looking at success versus failure and where she may reasonably claim credit for anticipating such approaches in evaluation today. She can definitely be considered in the field of those who think you learn more from success than failure, even though—as we have seen—she is not oblivious of cases of failure, that is, cases when “pre-requisites” for success do not work. To those who would point out that you also learn from failure—from trial and error—it can be pointed out that Tendler teaches us to distinguish between knowing and learning. Knowing has no limitations: a good evaluator has to be able to distinguish what went wrong from what went right in a project (Did it work as you expected? What worked that you didn’t expect? What didn’t work as you expected?), and she or he should be able to become familiar with the domains that are required in a particular case: trespassing disciplinary boundaries, as Hirschman would say. This is knowing. But what Tendler would maintain is that you learn more from positive than negative cases from what worked when it was not expected, or in ways that were unexpected. Negative cases of failure—she would say—are depressing, they tell you what not to do, while positive cases of success are encouraging, because they show that there are ways out of predicaments. Hence she suggested to treat a success with “awe.” This is learning how to improve things, which is the real concern of evaluation.
I find a consonance with this way of thinking in the recommendation by Lauren Kogen not to conflate accountability and learning, that she sees implicit in the Results-based Management (RBM) approach sponsored by the main foreign aid donors. Kogen (2018: 100) questions “that accountability and learning are essentially equivalent.” Accountability (that, in my understanding, can be equated with knowledge) tells us “whether a particular project worked” (p. 102), but “on its own, does not improve aid” (p. 104), while learning is about “how social change works” (p. 102), “is what improves lives.” Measuring the success of a program against the funders’ objectives “should be viewed as a secondary purpose of evaluation—as a means to an end, in which the ‘end’ is learning how to effectively promote development” (p. 105).
Implementation as a long journey of discovery in the most varied domains
A significant group of development evaluators are theorizing the centrality of the implementation process in order to understand the impact of programs (Andrews et al., 2017). Implementation is not seen as a moment when monitoring checks whether things happen as planned (blueprint), but—as Hirschman (2015 [1967]: 32) said in Development Projects Observed, The term “implementation” understates the complexity of the task of carrying out projects that are affected by a high degree of initial ignorance and uncertainty. Here “project implementation” may often mean in fact a long voyage of discovery in the most varied domains, from technology to politics.
17
This quote is particularly relevant for our reconstruction of Tendler’s work. Indeed, the “long journey of discovery” alludes to changes, variations, and inverted sequences that happen during implementation (the moment when the evaluator observes what is happening), and that most evaluators would consider crucial nowadays. But specifying that such discoveries happen “in the most varied domains, from technology to politics” underlines the collaboration and shared understandings that existed at the time of writing between Hirschman and Tendler. Looking at the relationships between the different domains (economics and politics, social reforms and economic reforms, technology and administration), and with multiple disciplinary tools, is what Tendler has done in all the work that she later undertook on her own, 18 that has been recalled above.
Tendler’s as a different voice
As we have seen, Tendler’s positions were often at odds with current social theories and professional guidelines. Think of her findings on social capital or public-sector performance, and of her keeping a distance from traditional evaluation practice. These views were based on strong premises: doubt about received theories, and commitment to a supportive social science, devoted to development. We can trace this orientation by referring to topics that are widely discussed by evaluators at present: theories of change, independence and ethics, and evaluation designs and methods.
On theories of change
Well before Theory-Based Evaluation was thought of, Tendler was concerned with the theory behind programs, something she often contested in the light of what she observed in the course of fieldwork. Elaborating program theory was a natural process with her, and she was not intimidated by the authority of the literature or of high-level science. Instead the actions and feelings of her interlocutors in the field encouraged her to look for alternative theories. As Merton would have said, based as it was on unexpected consequences (the “surprise”), her approach to theory was one of “theory fructifying,” instead of “theory testing.” 19
Nowadays program theories and theories of change have become common currency in evaluation. However, they have often been expressed through rigidly assumed causal relationships that are hardly distinguishable from logical frameworks with their linear logic. A reaction to such a rigid (technical) way of utilizing program theories has been expressed by Peter Dahler-Larsen: “the conventional design of the TBE process helps bury or conceal ambiguities that might otherwise lead to interesting and productive heuristic insights,” where ambiguities mean “the coexistence of multiple interpretations of a phenomenon among reasonable people while there is not necessarily an easy way to choose between the interpretations or eliminate some of them.” By embracing ambiguity—Dahler-Larsen (2018) maintains—“evaluators can help promote collective sense-making about complex interventions as part of a democratic evaluation process” (p. 7).
Tendler’s attitude could be a good antidote to this new rigidity. For Tendler, it often meant—as already noted—finding inverted sequences instead of linear processes, considering things from a different angle, and appreciating the relevance of side-effects of programs for the higher goal of social advancement. Here we can identify another lesson. More and more often, realizing that the world is complex, and linear theories do not hold, evaluators face unexpected results. Traditionally, what is mainly focused on are negative consequences: the destruction of the environment due to big infrastructures, the growing inequality due to globalization, and so on. Much less attention is given to how people, located in specific contexts, and utilizing their own capacities and resources, react creatively to unexpected difficulties. 20 At least, this was the situation up until recently, when a new awareness seems to emerge. As Larson has argued, since a program is operating within a complex adaptive system (CAS), it may not have the changes expected because of the usual grievances (delayed starts, confusion about purpose and roles, lackluster uptake), but “the converse is also true. Programs may succeed precisely because they are sensitive to the complex adaptive nature of the context in which they work” (Larson, 2018: 358). 21 Asking CAS-sensitive evaluation questions “leads the evaluator to focus on how implementers responded to the unexpected challenges and opportunities from CAS properties” and to “draw a convincing narrative of how program outcomes and impacts were (or were not) achieved.”
On independence and ethical issues
A characteristic of an ethical evaluator is his or her independence. As we have seen, Tendler fulfills the role of the democratic evaluator as categorized by Schwandt. Her independence was based on her moral status: working for the public good (helping the disadvantaged, the public sector) coupled with her confidence on the quality of her desk and field research. She was committed to providing strong evidence that changes were possible and people could be empowered.
This kind of independence was not defined by legal norms or professional ethical standards mainly oriented to avoiding conflict of interest as is demonstrated by her attitude toward policy-making. She did not argue for the need to “keep away from politics” and in the same spirit as she was not distrustful of public employees. Instead, she would look at how policy-makers could support positive reforms as a key to the success of programs.
This orientation was reflected in her modus operandi: research tools (questions, observations, statistics) able to detect whose interests the evaluated programs were serving. She was aware of power asymmetries of money, gender, generation, technical capacities, leadership roles, and was interested in discovering whether they were addressed and possible to overcome.
In this sense, Tendler would support the contribution that Rogers (2016) has brought to the current debate on equity in evaluation. In reviewing recent evaluations, Rogers identified ways of expressing a judgment based on equity principles, and offered a concrete example of how is it possible to face the ethical challenges that we are confronted with. This can be articulated during all the phases of an evaluation: from the theory of change that should be formulated involving beneficiaries; to formulating evaluation questions (“to what extent have results contributed to decrease inequities between the best off and the worst-off groups?” p. 206), to the evaluation design including the way social differences are described or measured and the need to select samples that represent the worst-off groups; to evaluation management that follows a collaborative enquiry process that appropriately addresses issues of power imbalance; and how resources are allocated between evaluators, who are paid, and community members, who are not.
As Rogers argued, and Tendler would agree, all these aspects require a will to “contrast the spontaneous tendencies to creaming the benefits of programs to the better-off groups, and to keep the intended beneficiaries of programs distant from the benefits as well as from the evaluation.” If properly equipped with relevant research tools this would ensure “that [the] program is not something done to people but something which supports them to be agents of their own development” (Rogers, 2016: 203).
On evaluation designs and methods
Many tools of Tendler’s reflective practice have been mentioned that matched her ambition to learn the lessons for development from evaluation. Let me just recall two of these.
First: Reframing the evaluation questions. In the debate stirred by the push for impact evaluation, there has been a focus on the different evaluation questions that require appropriate approaches and designs (Stern et al., 2012). However, the definition of the questions, and the choice of the appropriate approach, have always been considered a matter to be decided at the start. This is despite issues that might surface in the course of implementation that can suggest the need to modify established designs. Tendler’s approach is different. Based on what is known about “implementation as a journey of discovery in the most varied domains . . .,” she would start from direct observation and the problems that emerged, and then frame the evaluation questions and designs for an evaluation. Thus, her evaluation design was evolving in line with her observations, in a reverse sequence from the usual evaluation cycle and time-plan. Unexpected difficulties opened up opportunities for deeper inquiry and hopefully new ways to achieve success.
Second: Cross-domain comparisons. In the wake of the Evidence-based Policy and Evidence-based Medicine, there has been an upsurge in systematic reviews, syntheses, meta-analyses, and the like. These aim to sum the results of similar programs, in order to identify what is statistically meaningful. Notwithstanding the importance of cumulating knowledge about “what works” (and as we have seen Tendler herself had suggested the Ford Foundation do this in 1987), she would share the criticism of the limitations of that kind of research. As Pawson (2006: 72) noted, by their “process of simplification, standardization and aggregation [. . .] meta-analysis eliminates most of the evidence that is capable of telling us how interventions work and how we might account for their differential effectiveness.” Judith Tendler also concurred with Pawson’s focus on middle range theory in dialogue with “realist synthesis.” As the latter is based on refining program theories about how interventions work “through highly elaborate implementation processes, passing through many hands and unfolding over time” (p. 83), Tendler’s cross-domains comparative method—based as it is on the merit of finding analogies and differences between mechanisms that work in fields that are separate but interacting—adds a further input to that current debate.
To conclude
This essay has argued that reflecting on the approach, practice, and ethically oriented work of Judith Tendler can help us better understand our own practice as evaluators today, gain the courage to challenge professional rituals and theoretical conformism, and add content and meaning to the improvement mission of evaluation.
Footnotes
Declaration of conflicting interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.