Abstract
This article synthesizes interview data from evaluation directors and top executives of philanthropic foundations on how evaluation might better advance their missions. In key informant interviews, respondents commented on the purposes of evaluation from the foundation’s perspective, challenges to effective evaluation, and the means by which evaluation could be made more valuable to foundations. Informants emphasized promoting accountability, improving programs, advancing organizational learning, and disseminating intervention models as desired uses of evaluation. Application of appropriate research techniques, relevance and timing of evaluation products, and internal thinking and social dynamics within foundations were cited as challenges to evaluation. Emerging concerns include the need to evaluate achievements of foundations as a whole and for evaluation to further develop as a science. In terms of increasing the value of evaluation, the current study indicates the importance of improved relationships and dialogue between evaluators and foundation personnel, stronger leadership consensus within foundations regarding evaluation, and safeguarding of the evaluation process.
Introduction
Evaluation based on systematic assessment of processes and outcomes is valuable in a wide range of organizations and institutions concerned with social change. However, such evaluation is particularly important for foundations. Unencumbered by market forces or electoral politics, foundations are held to a higher standard by the public. They are expected to make unique contributions to the society in the form of innovations beyond the capabilities of either business or government (Anheier, 2005; Anheier & Leat, 2006). Neither profitability nor electoral success is applicable to foundations as an indicator of achievement. Thus, evaluation takes on key significance (Wisely, 2002).
Researchers have historically characterized material support for evaluation by foundations as modest. In a survey of foundations, Alie and Seita (1997) reported that fewer than half of the organizations they contacted conducted or commissioned any evaluation. Funding levels for the evaluation efforts that did take place ranged between 5% and 15% of the project costs, and foundations allocated only 1.6% of their total costs to evaluation. Alie and Seita’s survey also found that interest among foundations in training staff in evaluation or hiring evaluation specialists was low.
A more recent survey of foundation chief executive officers (CEOs) focusing on larger foundations found that 90% of the foundations contacted carried out or contracted for evaluation. Still, evaluation was done on half or fewer of the grants made. The median spending on evaluation was reported to be 2% of the grant-making budgets (Buteau & Buchanan, 2011).
Moreover, foundations that conduct or contract for evaluation do not generally appear to use it for program management and strategic planning. Slater, Constantine, and Braverman (2004) comment that most foundations do not “systematically or regularly make use of evaluation.” According to Kramer and Bickel (2004), “few evaluations appear to produce significant changes in the behavior of foundations … or grantees.” Patrizi and McMullan (1999) note that foundations do not use evaluation to assess success or plan strategy . Other observers reinforce the impression that evaluation has not achieved its full potential for helping guide the work of foundations (Lenkowsky, 2007; Wallace, 2008). More bluntly, Bolman and Deal (2008) have stated in a different but related context:
Evaluation consumes substantial time, effort, and money. It typically yields a lengthy report presented with considerable ceremony. Yet rarely are insights or recommendations implemented. Results typically disappear into the recesses of people’s minds or the far reaches of administrators’ file cabinets. (p. 304)
Researchers and commentators have suggested several reasons for the infrequency with which foundations utilize evaluation findings for management and planning. These include lack of technical acquaintance with and time for evaluation-related concerns among staff (Kramer & Bickel, 2004), perceived costs associated with evaluation (McNelis & Bickel, 1996), and tensions between program staff and evaluators (Patrizi & McMullan, 1999). It has also been observed in the literature that foundation boards have not been satisfied with the evaluation products they receive due to an absence of conclusive findings regarding outcomes and clear evidence that outcomes can be attributed to their programs (Patrizi & McMullan, 1999).
The current study helps identify ways in which evaluation can be made more useful to foundations. It utilizes in-depth interviews of key informants who are well positioned in foundations to reflect on current thinking and practice regarding evaluation. These informants are also potential advocates for supporting evaluation more generously and making fuller use of evaluation products. Their views add to previously published explanations of why support for and utilization of evaluation by foundations have typically been modest. Most important, this research provides explicit recommendations offered by a particularly relevant group of informants and builds upon them.
Methods
The data were obtained by interviewing evaluation directors or chief executive officers (CEOs) of relatively large foundations in the United States. This purposive sample was obtained in a multistage process. First, 10 individuals were identified as evaluation directors at the 100 largest U.S. foundations by asset size. Second, in the course of the ensuing interviews, these individuals were asked to identify other evaluation directors who could contribute to developing an understanding of the use of evaluation by foundations. An additional 10 individuals were identified through this snowball sampling procedure. Four foundation CEOs or individuals who had recently left such positions, and who were regarded as knowledgeable about and concerned with evaluation, were also interviewed. Two of these CEOs functioned as evaluation directors of their foundations. A total of 24 individuals were interviewed from 22 foundations. None of those asked to be interviewed refused.
Key features of the foundations whose personnel were interviewed appear in Table 1. Foundations established under bequests of individuals or families but which later became independent predominate. These are denoted as heritage foundations. Second most frequent were conversion foundations established and capitalized through conversion of nonprofits (typically health care entities) into profit-seeking corporations. The “other” category includes one “re-granting” foundation that receives funds from other foundations to conduct specific programs, and one “corporate” foundation funded largely by or through a parent corporation. The sample included six of the oldest foundations in the United States (founded before 1950) as well as some of the newest (founded after 1990). Assets of the foundations ranged from US$50 million to US$9.2 billion. Of the foundations contacted, 10 were primarily concerned with health, 2 with education, and 1 with public affairs. The remainder had multiple concerns, including areas such as climate, environment, economic security, and human services reform. All of the foundations had programs that awarded funds to external grantees.
Characteristics of Foundations Contacted.
Semistructured interviews containing principally open-ended items were conducted via telephone with each individual. Interviewees were considered to be key informants on thinking and activity regarding evaluation in their own as well as other philanthropic foundations. Key informant interviews are distinguished from those comprising opinion surveys in that individuals interviewed are asked not just about their personal views and behavior but “about patterns of behavior … either observed or expected” in their social surroundings (Seidler, 1974). Key informants are not selected to be representative of a population in a statistical sense; they “are chosen because they are … knowledgeable about the issues being researched and able and willing to communicate about them” (Kumer, Stern, & Anderson, 1993).
Each interview lasted from 20 min to 1 hr. Informants were asked to focus on evaluation defined as the “systematic assessment of the implementation or outcomes of a grantee’s work, through qualitative or quantitative means.” The interview schedule contained items on resources committed to the evaluation; methodological preferences among foundation officials; uses made of and impact of evaluation; challenges to evaluation; and ways in which evaluation might be made more useful. The core questions used in the interviews appear in the appendix.
Responses to open-ended questions were analyzed through open coding (Strauss & Corbin, 1990). Words and phrases associated with each question reflecting distinct concepts were identified and given descriptive codes. As a second step, the concepts initially identified were aggregated into broader categories. Assignment of new codes was stopped when theoretical saturation (Strauss & Corbin, 1990) was reached, that is, when no new concepts emerged in responses to the interview questions. The processing of responses to the question “How could program evaluation be made more useful to foundations such as yours?” illustrates the manner in which coding and aggregation took place. Answers to this question included “delivery in real time” and “contemporaneous feedback.” These were aggregated into a category designated as “timeliness.” Through the same process, additional categories were derived from informant comments regarding features of evaluation not specifically solicited by interview items, philanthropic foundations other than their own, or foundations in general. These categories are characterized as themes in the text, grouped under descriptive subheads, and selectively accompanied by illustrative quotations.
As a follow-up to the interviews, all the informants were asked to review an initial draft of the study findings in order to identify any errors in information or quotations from their interviews. Informants were encouraged to provide additional comments they considered relevant, and the comments received were incorporated into the results reported here.
Findings
The interviews yielded information on current practice and outlooks regarding evaluation in foundations, including degree of support for evaluation, perceived uses of evaluation, and challenges associated with evaluation. Such challenges reduce the ability of evaluation to contribute to the uses identified for it by the key informants. Recommendations offered by key informants to address these challenges concentrate on the capabilities and practices of evaluators.
Support for Evaluation
Of the 22 foundations contacted, all conducted some evaluation, either by in-house personnel or by outside contractors. Most informants reported that support for evaluation in their foundations ranged from less than 1% to 15% of extramural funding in the preceding year, although one indicated that the level of support in her foundation was 30%. Some informant comments suggest that these figures may represent underestimates. Four informants, for example, said they did not know the percentage of extramural funds their foundations allocated for evaluation. Several others indicated that funds for evaluation sometimes appeared as budget items under other labels or were allocated to intervention grantees and thus not reflected in a global evaluation budget.
Uses of Evaluation
Questions regarding perceived value and uses of evaluation yielded both reports of actual use and comments on potential though unrealized benefits. Evaluation was seen as potentially contributing to the accountability of foundations by demonstrating the efficacy of their actions. The assessment of interventions or grantees tended to emphasize modification of programs rather than simply judging their efficacy. Also, among the purposes of evaluation that informants mentioned was helping improve the performance of the foundation as a whole in achieving its goals over time. Finally, evaluation was seen as an asset in promoting wider adoption of the methods of an intervention.
Accountability
Accountability was a recurring theme in the interviews. On an elementary level, evaluation was seen as a means of determining whether a grantee was acting in a manner consistent with its commitments. More fundamentally, accountability was expressed as a belief that a foundation needed to demonstrate that it was doing useful things with its resources. As one informant commented, “the purpose of evaluation is to inform a foundation’s work, so [foundations] can become better grantmakers and have better accountability to their missions and the communities they serve.”
Informants cited a number of stakeholder groups to which they considered themselves accountable. Stakeholders internal to foundations included members of governing boards and staff. Informants associated the concept of accountability with functions such as quality control and continuous quality improvement in grant making. Evaluation was seen as an essential resource in the process of improvement.
External stakeholders to which interview subjects considered their foundations accountable included “the public” and “the community.” The evaluation director of a prominent foundation considered it important to demonstrate his organization’s efficacy to groups such as “grantees, thought leaders, business leaders.” He observed:
[As evaluation director], I am responsible for holding [the foundation] accountable. We’re responsible for saying how we’re doing as an organization. We are also responsible for seeing whether our strategies are having impact.
Modification of interventions
As an objective of evaluation, informants often cited its use as a means of identifying the features of an intervention which were working and those which were not. Several mentioned that they were more interested in helping an apparently foundering grantee than “pulling the plug.” Aid to such a grantee might take the form of additional funding or technical assistance. Few individual grantees appear to be defunded as a result of evaluation. Only one informant said that evaluation findings led to defunding of a grantee, two said that evaluation had led reduction in a grantee’s level of funding, and two indicated that evaluation had resulted in a grantee’s failure to receive funding in a subsequent program phase.
Learning about implementation of and fidelity to objectives, as well as acceptance by the affected community, seem particularly important in multiyear programs or initiatives in which awards are made to successive cohorts of applicants. According to one informant,
Evaluation has not affected the funding of individual grantees. But lessons from evaluation are immediately incorporated in technical assistance and reviews of new grant applicants. A strong feedback loop is established between interim results and further work by later grantees.
Organizational learning
Several interview subjects emphasized that evaluation helps them build a model of how the world works and how to change it. Accordingly, evaluation contributes to a continuous process that builds understanding of factors that promote, facilitate, or challenge achievement of a foundation’s goals. Greater understanding of how their initiatives fit with the educational establishment, politics of a profession, or history and structure of a community raises the likelihood that an intervention will be effective.
Development of such understanding constitutes a form of organizational learning (Levitt & March, 1988). Organizational learning is a process by which an organization develops core beliefs and principles of operation over time. Both successful and unsuccessful experiments and ventures constitute valuable lessons in the learning process. Such lessons are incorporated into the organization’s culture and transmitted to new generations of members.
The comments of several informants provide illustration. One characterized evaluation in her foundation as tightly linked with an iterative learning process. This process helps the foundation clarify its thinking about what it really wants to accomplish and to develop strategy for future grant making. In describing a long-term program to which her foundation had committed significant resources, another informant noted that evaluation of the program had helped the foundation “learn to execute a fundamental strategy” regarding social change.
Even disappointing evaluation findings were seen as having value in promoting organizational learning. A former foundation president described an intervention intended to create a “huge national network of home helpers.” However, the program proved infeasible because there were not enough home helpers available nationally to complete the hoped-for network. In this instance, evaluation contributed to the foundation’s understanding of the social reality into which its intervention was inserted. Generally, the informant concluded, evaluation helps to “assess the foundation’s understanding of the field [in which it contemplates intervening] and whether [the intervention] is planned and framed appropriately, not naïve, underpowered, [or otherwise unlikely to succeed].”
Dissemination
Many foundation officials consider the interventions they fund as primarily pilot efforts whose approaches and methods, if they appear successful, will become widely known and supported by other sponsors. Government, it is often pointed out, has resources far beyond those of even the best endowed foundation and can carry an innovation much farther forward. Evaluation is seen as a key vehicle in this process.
Informants expressed satisfaction with instances in which an approach they had funded was taken up by government or otherwise influenced policy. A foundation program called Pathways to Teaching Careers was cited as an example:
The initiative involved partnerships between organizations that train teachers and school districts in need. [In the process of evaluation] a non-traditional source of personnel [was found], teacher aides. Many couldn’t complete college due to financial constraints but had qualifications [relevant to becoming teachers]. Evaluation findings in this area were presented to staff members of a Congressional committee and resulting legislation provided public support for the Pathways model.
Challenges to Evaluation
Informants cited a number of reasons why evaluation often fails to achieve the objectives they identified. Challenges may arise from technical limitations of current evaluation methodology. The products delivered by evaluation professionals may be poorly matched to the decision-making needs of foundations and thus fail to achieve impact. Challenges may arise from within the foundations themselves, involving attitudes toward evaluation, interpersonal relations, and organizational dynamics.
Methodological challenges
Informants seemed open to the use of a wide range of methodologies. Most commented that the substance of an intervention should determine the methodology used. Openness to a variety of methods, including mixed methods applied to the evaluation of a single intervention, had special value to some. In this connection, one informant noted that his foundation used “a combination of quantitative and qualitative methods and triangulated to identify converging lines of evidence.” Another commented that a single methodology could not “get the best overall picture.”
Both traditional and recently developed evaluation methods appeared to be in widespread use. Randomized controlled trial methodology was reportedly used to assess the relative efficacy of two alternative approaches intended to promote voting among young people. Logic model case studies (Lin, 1994) were mentioned as having the advantage of being useful for both evaluation and planning. Participatory evaluation and photo-journalism-based data collection was valued by some not only as a means of evaluation but also for getting young people involved in an intervention.
Informants also cited issues regarding data used by evaluators. Some drew attention to limitations in applicability of data from sources such as the U.S. Census. Such data, for example, were seen as difficult to apply to specific communities targeted by interventions. Special concern was expressed about the difficulty of obtaining data representative of the most relevant populations. Informants observed that concern with confidentiality, particularly regarding vulnerable populations, discouraged collection of data on individuals targeted by interventions. Limited language ability of some individuals, such as children, was cited as a challenge.
These difficulties, it was said, encouraged evaluators to focus data collection on the most accessible individuals. Remarking on the work of an evaluation contractor in an education intervention, one key informant identified this practice as particularly troublesome:
They never looked at data [from or about] students. Same baloney as in many foundations. Lots of crap and never get to the central point about what we should be measuring no matter how difficult it is. I don’t give a crap about whether [evaluators report that a school] administrator feels good.
Challenges associated with these broadened evaluation objectives include issues of measurement. Indicators reflecting the performance of a foundation as a whole are necessarily more abstract than measures regarding an individual grantee, program, or initiative. As one informant indicated, foundations need:
… better knowledge capture around lessons learned, sharing findings among initiatives, [and] turning data into knowledge so we can learn to stop making the same mistakes [over and over again].
Evaluation products and foundation needs
Methodology notwithstanding, two features of the products that evaluators deliver to foundations was identified as having particular impact on their usefulness: timing and accessibility.
Timing
Informants commented that feedback from evaluators would be of greater value if it were delivered sooner—for example, early or at the midpoint of a program rather than after its conclusion. During an implementation period, evaluation may pinpoint problems in launching an initiative. Timely evaluation may identify grantees who have encountered problems but might achieve their objectives with the aid of additional resources such as technical assistance.
Evaluation can be particularly valuable at the midpoint of an initiative. Often, foundations seek to assemble a second cohort of grantees at this time. Input from evaluators may contribute to revision of an initial request for proposals (RFPs) regarding qualifications, resources, and expectations of new grantees.
Informants, however, felt that evaluators tend to concentrate on detailed, rigorous, outcome-oriented final reports. The fact that the initiative has often been completed by the time the final report appears greatly reduces the usefulness of the evaluation. As one informant noted, “evaluation comes too late for decision making.” According to another who reflected on significant foundation experience, “Some [reports] came too late in the day to advise us for anything concrete. We had to keep moving.”
Accessibility
Also prominent were comments urging evaluators to produce findings that are clearer and less guarded. Informants remarked that evaluators often produce reports that are excessively technical. Several commented that the evaluation reports they receive are too jargon laden and “academic” to be meaningful to board members and other stakeholders. Comments illustrating the importance of clear, readily comprehensible findings included the following:
[Evaluation findings should be] more digestible and delivered faster. After benchmark events, a bullet-point memo on findings should be delivered, raising issues and perhaps recommendations. [Evaluators need to write] reports that are not jargonistic. There must be scientific precision … but tell the story in a way that people can understand. [Evaluators should] be able to boil things down to elevator speech findings…. If you want attention from anyone in the hierarchy you need to do this. Consultants are often technical but you need to distill things even to [the degree that they] sacrifice precision in service of communication.
Social and interpersonal relationships
Relationships among individuals involved in the evaluation process and differences in outlooks may reduce both the resources for evaluation and the value that may be obtained from it. Foundation officials may devalue an evaluation effort because the ensuing reports do not meet their expectations or jibe with their impressions. Members of a foundation’s governing board may expect more definitive and quantitative findings than are available and discount results that contradict their hopes. Illustrative comments from an informant who occupied roles of both evaluation director and board member include:
Foundations are bastions of self-congratulatory work. People don’t like bad news … I’m lucky. I have [a] board that wants evaluation and a staff that listens. I’ve seen lots of other foundations that don’t have that same set of conditions. I’m amazed that some foundations don’t change things as a result of the evaluation reports they receive. [In my foundation] a separate evaluation unit reported to the senior director for programs. There was built-in tension, with the evaluation unit responsible … [for] having to negotiate with program directors, who held the budget for evaluation. There was back and forth [competition] for responsibility. Tensions around these struggles exacerbated the problem of using evaluation findings. [It was a challenge] to do the job and at the same time be accepted as colleagues. Personalities of key individuals added to [the] difficulty. … never got clarity about what they wanted to get from evaluation, and the CEO was reluctant to engage them on this. [The CEO] was very conflict averse. Individual board members had different conceptions of evaluation. One was experimental oriented. Another said he just wanted stories. Program staff see [evaluation] as a distraction. They are action people and so are grantees—sure that what they’re working on is going to be a good thing… . From this perspective, the whole idea of evaluation in a formal sense is disruptive. From another angle, program directors are not interested in having results that show their investment in staff and [grant] money was not effective. [Program directors] are passionate and didn’t think what they did could or needed to be measured.
Historical factors affect the importance placed on evaluation and the level of resources available for this purpose. Describing a large foundation whose benefactor had died many decades earlier, an informant reported a relatively weak concern for accountability and hence for evaluation:
When you [as a foundation executive] have exclusive control over several billion dollars [do you] ask yourself how well you are doing? If you can pick all board members, ask yourself whether they will really be able to ask you anything. Will you ever evaluate your own performance? Everything follows from that basic issue.
Recommendations for Improving Evaluation
Informants were asked how evaluation “could be made more valuable to a foundation such as yours.” Explicit recommendations tended to focus on steps that might be taken by the evaluators and addressed challenges such as relationships between evaluators and foundation personnel, timeliness and relevance of evaluation products, and evaluation methodology.
Develop team relationship
A general recommendation emerging from the interviews is consistent with, and in fact subsumes, these comments: evaluators should work with foundations in a collaborative, real-time process. The best evaluators were identified as those able to serve as full team members, working with both foundation personnel and grantees to promote the success of an intervention. The ability of an evaluator to carry out effective group process was considered a valuable asset in this connection.
Informants recommended that evaluators develop and apply a better understanding of the interests and needs of foundation personnel whose specialties are outside evaluation. One commented that “the quality of [evaluation] depends on the ability of the evaluator to facilitate reflection and feed back results.” Evaluators were thought to generate the greatest value when they helped foundations go through a self-reflective learning process aided by formative feedback. To promote this process, an informant suggested that the evaluator needed to “actually talk about evaluation and not just rely on written reports.”
Support of the commitments of foundation personnel to the interventions they developed or managed was also considered conducive to high-value evaluation. As one informant remarked,
The wonderful thing about working in a foundation is that it employs smart people who really care about their work. Sometimes an evaluation consultant can get into [a] nerdy place. Instead, they need to appreciate the passion that program managers have about their work. Evaluators need to support this passion—evaluation should be in service to the goals of the initiative, to help it achieve impact—this approach has a better chance of success than being a dispassionate third party. The evaluator has to be a part of the change effort by being part of the feedback loop.
Advance evaluation as a science
The second explicit recommendation posed by the informants was to increase the scientific basis of evaluation. Science conceived in this fashion involves recognition of techniques with which all practitioners should be conversant and development of standards to which all practitioners adhere. Several informants expressed concern regarding the strength and cohesiveness of the evaluation field itself. The diversity of methods and approaches used by evaluators seems to have reduced the confidence of some foundation personnel in evaluation. As one informant noted,
Tremendous variation exists in use of the term “evaluation.” The term doesn’t mean much now [because it includes so many] widely different approaches. There needs to be … more clarity and consistency in use of the term and a more standardized set of skills and competencies.
Discussion
Study Limitations
This study was not intended to obtain information that could be generalized across all foundations in the United States or even among those of any given size or orientation. Rather, it aimed at illustrating a range of activities, approaches, and issues potentially present in any foundation’s efforts to obtain and make use of evaluation. Hence, conflict between program and evaluation personnel reported by some informants must not be thought of as necessarily prevalent in all foundations. Rather, it is the potential for such conflict in many foundations that must be recognized.
As Table 1 suggests, the sample of foundations includes larger foundations (by asset size) than most foundations in the United States, limiting the direct generalizability of comments made by the key informants. It should be acknowledged, however, that the greatest expertise regarding evaluation is likely to be concentrated in larger foundations. The sampling strategy focused on evaluation directors. Foundations with significant resources are the most likely ones to have evaluation directors and other evaluation specialists on their staffs.
It is important to acknowledge that the comments reported here may differ from those that would have been obtained from program officers, foundation staff not specialized in or familiar with evaluation, or individuals outside the philanthropic community such as policy makers and member of the public. Among those interviewed, however, a high degree of agreement prevailed, an indicator of the reliability of key informant reports (Kumer, Stern, & Anderson, 1993). Although interviews suggested important differences among individual foundations, no informants contradicted each other regarding challenges and potential solutions to making evaluation more useful to foundations.
Variation Among Foundations
In areas that some foundations identified as problems, others reported outstanding practices and achievements. Although resource allocation for evaluation by certain foundations appeared quite small, for example, others have made evaluation an object of generous funding and significant attention. One informant suggested the existence of an “evaluation elite,” whose members use practices such as:
… commissioning numerous, large-scale evaluations; making evaluation reports public, resulting in hundreds of thousands of downloads of these reports; using timely evaluation findings to strengthen foundation grantees’ projects; and using evaluation results to contribute to a more informed public policy discourse. … the [evaluator] presents the proposal and the review panel comments on it. It is usually a lively dialogue that helps everyone avoid the pitfalls that you usually recognize after the study is completed.
Recommendations for Foundations
In addition to the recommendations offered by informants in response to the relevant interview question, several can be inferred from answers to other questions. These suggestions tend to be most applicable inside the foundations themselves.
Institute effective dialogue
Dialogue between program staff and evaluation specialists can bring the perspectives of these groups closer together and enable the work of each to better contribute to the foundation’s goals. Such dialogue should involve program officers, foundation executives, and governing board members. The fundamental objective of dialogue should be the development and maintenance of partnership between evaluation specialists and others in a foundation.
Dialogue should enable evaluation specialists to educate program officers and board members about the potential contributions and limitations of evaluation. It should also provide evaluators with clear direction about the foundation’s expectations and help maintain their commitment to fulfilling these expectations. It is instructive to recall an informant’s complaint about an evaluation contractor’s failure to collect student-level data in an urban education intervention. When confronted, the contractor responded “we [only] did what we were told.”
Develop leadership consensus
Consensus within a foundation regarding purposes and methods of evaluation cannot be assumed to exist. Evaluation personnel require a consensus among decision makers about the objectives of an evaluation effort and the methods and data required by decision makers. Differences in expectations, whether explicit or hidden, mean that at least some decision makers will receive findings that they consider of limited value. Lack of consensus may cause confusion and result in poor focus on the evaluation.
Of course, development of consensus on evaluation is likely to be difficult in many foundations. Differences in approaches and opinions among governing boards, selected as they often are to ensure diversity of outlook, constitute a challenge. However, forging a consensus in such situations is a core function of leadership.
Safeguard the evaluation process
The comments of informants, as well as theory in the social sciences (Greenwald, 2008), suggest that the structural settings in which evaluation specialists are placed may reduce the value of results they provide. The dependence of evaluators on program personnel for resources may motivate evaluators to produce more positive results than they otherwise might. This principle holds for both in-house evaluation staff and outside contractors.
Foundations will obtain more useful results from evaluators if they ensure that evaluators are not punished for reporting negative findings, either by losing favor with program officers or, in the case of outside contractors, by losing future business. Foundations might consider instituting practices for obtaining continuing, independent input on approaches to evaluation, the quality of the evaluation products, alternative interpretation of evaluation findings, and overall success in attaining broad foundation goals.
Appropriately specify evaluation tasks and schedule
Foundations can take instrumental actions in the areas of organization and contracting to help them obtain more useful contributions from evaluators. Early involvement of evaluators can help configure an intervention in ways that make it more readily evaluable, for example, by identifying outcomes that lend themselves to concrete measurement. Calls for proposals by potential grantees can then specify requirements compatible with the foundation’s decision-making needs and schedule. RFPs for outside evaluation contractors can also be formulated to reflect the value foundations place on timely results and formative feedback.
Directions for Implementation
Some recommendations for foundations and evaluators in this article are relatively straightforward, such as those regarding timeliness and accessibility of evaluation products. Implementation of others, however, is problematical, requiring changes in outlook and behavior by multiple actors. Observation of practices in fields other than evaluation suggests potential directions for addressing the more challenging recommendations.
The Council of Economic Advisors (CEA) may serve as a model for safeguarding the evaluation process. The CEA, which has been cited as a model for “tapping knowledge in the social sciences” for high-level policy making (Wilensky, 1967), illustrates the concept of independent advisement. Comprising a small group of highly qualified individuals directly responsible to the President, the CEA is given resources to independently review data, conduct studies on its own initiative, and consult with other outside experts. Each member receives a nonrenewable term appointment. Although few foundations may opt for such an approach in its entirety, a standing panel of evaluation experts (with fixed, non-renewable terms) may contribute the essential element of independent review.
Potential models for development of measures capable of assessing achievement of foundation goals rather than individual interventions exist in biomedical research. Scientists in the field of evidence-based medicine use meta-analytic techniques to synthesize findings from studies utilizing different approaches and instruments. In this fashion, scientists represent findings from individual studies as higher or lower degrees on widely recognized, standard, high face validity instruments (Assendelft, Morton, Yu, Suttorp, & Shekelle, 2003). Another approach to measurement might rely on “cross-cutting indicators,” metrics that represent degrees of success in reaching recognizable milestones (e.g., implementing an action plan) applicable across multiple initiatives and grantees. Compatible research designs and data collection methods would have to accompany either of these approaches to measurement.
Although development of more standard, traditionally scientific methods and measures in evaluation would be desirable, foundations may legitimately hesitate to use them. Methods that consciously depart from classical science are now in widespread use. Participatory evaluation, for example, reduces the traditional distance between researcher and subject (Fetterman & Wandersman, 2005). A foundation may favor a participatory evaluation approach to promote community engagement and capture the experiential knowledge and insights of community members. Participation by community members in evaluation planning might militate against the use of standard measures and designs. Thus, a foundation’s objectives in evaluation could be incompatible with traditional science. Evaluators and nonevaluators need to collaborate in determining the most appropriate evaluation methods and the best mix of approaches on a case-by-case basis.
In addition, the objectives of an intervention may not always lend themselves to evaluation design along the lines of traditional science. Interventions with broad objectives or intangible outcomes may have value but defy standardized measurement. In approving programs, foundation decision makers may have to weigh the relative merits of substantive appeal and objective evaluability.
Conclusions
Key informants interviewed for this study generally reported strong commitments to evaluation in their foundations. Most cited instances in which internal use was made of evaluation findings, only two informants stated that their foundations made no use of evaluation or declined to provide illustrative instances of such use. The comments of many informants, however, included disclaimers regarding their foundations’ utilization of evaluation findings. These, plus suggestions by informants about how evaluation might be improved, support the conclusion that much evaluation currently performed or commissioned does not serve the needs of foundations as well as it might.
Findings on support for and use of evaluation presented here confirm those of several earlier investigators (Kramer & Bickel, 2004; Slater, Constantine, & Braverman, 2004). Financial support for evaluation continues in the approximate range reported by Alie and Seita (1997) in the 1990s. This remains true even within a sample of foundations having resources significantly greater than the average foundation in the United States.
Similar factors identified by earlier investigators as barriers to better supported and more useful evaluation appear to remain influential. For example, the factors identified by Patrizi and McMullan (1999), including dissatisfaction with the methodology used by evaluators and tension between program staff and evaluators, are still important. Additional barriers identified in the current study include the timing of evaluation results and the accessibility of evaluation products to nonspecialists.
Still, significant impetus for more useful (and greater use of) evaluation can be discerned in the foundation community. Accountability is a strong driver of interest in evaluation and in the desire to see evaluators produce more scientifically valid and widely applicable findings. The readiness of informants to make concrete suggestions for further development of the evaluation field reflects their attention to and concern with the issue.
It is important to emphasize that implementing the recommendations emerging from this research represents challenges for both foundations and evaluators. The technical problems associated with traditional scientific designs and the development of more standardized measures may be the most tractable. Establishing a culture of evaluation within the field of philanthropy and reducing the potential for conflict between evaluators and program specialists within foundations will be more difficult. Changing the practice of evaluation to better serve foundation needs will require time and effort. It is hoped that this study will contribute to a dialogue between evaluation professionals and foundation personnel regarding these and other steps to increase the value of evaluation.
Footnotes
Appendix
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The foregoing research was sponsored by the University of Southern California Center for Philanthropy and Public Policy.