The Potential of Outcome Measurement for Strengthening Nonprofits’ Accountability to Beneficiaries

Abstract

This article considers one mechanism that could create a clearer accountability path between nonprofits and their beneficiaries: Outcome measurement. Outcome measurement focuses attention on a nonprofit’s beneficiaries and whether they are better off as a result of the nonprofit’s work. The article analyzed 10 outcome measurement guides targeted to nonprofits, totaling more than 1,000 pages of text. The analysis shows that the guides were neither uniform in the conceptualization of nonprofit beneficiaries nor in how they directed nonprofits to use outcome measurement with their beneficiaries. Despite scholars’ suggestion that a nonprofit’s relationship to their beneficiaries is a key accountability relationship, the guides suggest that beneficiaries have an ambiguous standing, relative to other stakeholders, in the nonprofit accountability environment.

Keywords

outcome measurement accountability nonprofits performance measurement nonprofit beneficiaries

Nonprofits operate in a complex accountability environment (Kearns, 1996). Scholars have described this environment in terms of two primary relationships: Accountability up to trustees, donors and government and accountability down to beneficiaries, communities, and partners (Edwards & Hulme, 1996, p. 8). Regulations, policy documents and funding requirements ensure nonprofits respond to the concerns of governments, donors, and trustees. In contrast, nonprofits are not legally accountable to beneficiaries (Chisolm, 1995). Although studies suggest that nonprofits see their relationship to beneficiaries as important and even their primary accountability relationship (Chaskin, 2003, p. 182; Ospina, Diaz, & O’Sullivan, 2002), nonprofits have no clear accountability path to beneficiaries (Kilby, 2006, p. 951). This is true even though nonprofits often advocate and make claims on behalf of beneficiaries in the policy process and their effectiveness is premised on their responsiveness to those beneficiaries (Kissane & Gingerich, 2004).

This article considers one mechanism that could create a “clearer accountability path” between nonprofits and their beneficiaries: Outcome measurement. Outcome measurement focuses attention on a nonprofit’s beneficiaries and whether they are better off as a result of the nonprofit’s work. As one early article on nonprofits and outcome measurement explained, “Outputs are about the program while outcomes are about the participants” (Plantz, Greenway, & Hendricks, 1997). Indeed, rationales offered to nonprofits about why they should measure outcomes often include better serving their beneficiaries. For example, the United Way Outcome Measurement guide states that one reason nonprofits should measure outcomes is to, “seek the most and best for their customers” (United Way of America, 1996, p. 3). Researchers have also pointed to the, “significant potential of performance assessment and evaluation to strengthen downward accountability” if it is used in a way to increase participation by these groups (Ebrahim, 2003b, p. 825; Edwards & Hulme, 1996).

Despite outcome measurement’s focus on beneficiaries and its potential to strengthen downward accountability, most nonprofit staff see measuring outcomes as something they do for funders rather than something they do to ensure they are responsive to the needs of beneficiaries (e.g., Carman, 2007; Christensen & Ebrahim, 2006; Hwang & Powell, 2009, 294; http://www.innonet.org). In some respects this is not surprising when we consider that funders are the ones requiring data on measurable outcomes and most nonprofits started to collect outcome data as a result of these requirements (see Fine, Thayer, & Coghlan, 2000). Yet, nonprofits have long reported activity and output measures to funders so reporting to funders is nothing new. In contrast outcome measurement focuses on results for beneficiaries not the activity of the organization. So even if nonprofits are required to report this data to funders presumably the focus on outcomes should spur nonprofits to lean more toward beneficiaries.

So why do nonprofit staff feel outcome measurement is something they do for funders? The analysis presented here suggests that part of the answer can be found in the guidance given to nonprofits about how to “do” outcome measurement. The article examines 10 outcome measurement guides targeted to nonprofits. The guides, which total more than 1,000 pages of text, were identified after a systematic scan of more than 500 websites of organizations that encourage nonprofits to measure performance (e.g., funders, management support organizations, member-based organizations). The guides are intended to augment nonprofit capacity to develop and use outcome measurement and thus are a source of normative messages about how nonprofits should “do” outcome measurement (Meyer & Rowan, 1983; Powell & DiMaggio, 1991; Scott, 1995). The analysis maps the content of the guides against what we know about how outcome measurement can support greater downward accountability. The analysis shows that the guides were neither uniform in the conceptualization of nonprofit beneficiaries nor in how they directed nonprofits to use outcome measurement with their beneficiaries. Despite scholars’ suggestion that a nonprofit’s relationship to their beneficiaries is a key accountability relationship, the guides suggest that beneficiaries have an ambiguous standing, relative to other stakeholders, in the nonprofit accountability environment.

This article makes two contributions to a growing body of work on outcome measurement and accountability in nonprofit studies. First, this article drills down into outcome measurement material to examine the guidance given to nonprofits about how to develop an outcome measurement system for their program or organization. Most existing studies treat outcome measurement as a generic technical framework. Yet outcome measurement is a practice, just like other management practices, and the guides offer insight into how this practice is conceptualized in the sector. Sociologists remind us that documents are not simply sources of facts or technical resources, but rather they are carriers of normative ideas about what to do (Powell & DiMaggio, 1991; Prior, 2003; Scott, 1995). In this respect, these guides both reflect dominant conceptions of how nonprofits should develop and use outcome measurement and direct nonprofits in this process.

Second this article looks at how and in what ways outcome measurement could strengthen downward accountability and then analyzes the outcome measurement guides targeted to nonprofits along these dimensions. Existing studies of outcome measurement in the nonprofit sector either focus on the use of outcome measurement by nonprofits (see Carman & Fredericks 2008; http://www.innonet.org) or look at outcome measurement in the context of funding relationships (Benjamin, 2008; Cutt & Murray, 2000). Although funders may be the ones asking for measurable outcomes, nonprofits usually have discretion in how they develop their outcome measurement systems. Who should be involved, how, and in what ways nonprofits use the data are decisions that have implications for nonprofit accountability and the outcome measurement guides are critical resources in these efforts.

The article proceeds in three steps. First the article reviews the relevant literature on outcome measurement as a mechanism to strengthen downward accountability. After briefly summarizing what we know from the nonprofit literature I turn to the literature in public management; this literature has given much more attention to how performance measurement can strengthen downward accountability. In the second section, I present the analysis of the outcome measurement guides. The findings presented here focus on the development and use of outcome measurement and the implications for accountability relationships (other findings, which focus on what nonprofits are encouraged to measure, are reported elsewhere). The final section discusses the implications of these findings with recommendations for further research.

A few clarifications and caveats are necessary. First, this analysis is not intended as a critique of these guides. The guides are important resources for nonprofits. Rather this analysis recognizes that how these guides direct nonprofits to develop and use their outcome measurement system reflects normative ideas about the role of beneficiaries in the process and has implications for whether outcome measurement strengthens accountability to those beneficiaries. Second, by examining whether these guides could strengthen accountability to beneficiaries, I am not suggesting that outcome measurement should always be used for this purpose. My purpose here is to explore whether outcome measurement provides a mechanism to strengthen this relationship in the absence of any other clear accountability path (Kilby, 2006). Third, throughout the article, I use the term beneficiary for clarity even though nonprofits refer to those they serve using a variety of terms, including client, participants, constituents, and users. Finally, by examining the potential of outcome measurement to strengthen accountability to nonprofit beneficiaries, I am not suggesting that nonprofits prioritize accountability to beneficiaries above accountability to other stakeholders.

The Potential of Outcome Measurement

The potential of outcome measurement to strengthen downward accountability has not received direct empirical attention from nonprofit scholars, although many scholars have identified accountability to beneficiaries as one important accountability relationship and have noted the absence of strong mechanisms to support greater accountability to those served (see Ebrahim, 2003b; Najam, 1996; Ospina et al., 2002). As noted above, the bulk of the discussion on outcome measurement in the nonprofit sector, particularly as an accountability mechanism, has been normative in nature either pointing to the necessity of outcome measurement for nonprofits or outlining potential pitfalls. Empirical work in this area focuses on the negotiation between funders and grantees over outcome measurement requirements (Benjamin, 2008; Cutt & Murray, 2000; Ebrahim, 2003a); the limits of using measurable outcomes for accountability purposes (Campbell, 2002; Speckbacher, 2003); and the tension between outcome measurement and organizational learning (Ebrahim, 2003a, 2005). Other work examines how nonprofits are using outcome measurement to improve programs, make management decisions, or report to specific stakeholders like funders and the board (Carman, 2007; Houchin & Nicholson, 2002; United Way of America, 2000). One study did include questions about users (e.g., “to promote the program to potential clients” or “to increase participants’ investment in achieving possible outcomes”; United Way of America, 2000, p. 11).

What is surprising in all of this work is the limited attention given to beneficiaries in studies of outcome measurement. Much of the empirical work on outcome measurement and its potential to strengthen downward accountability can be found in the public management literature. Here the conversation about using outcome measurement to strengthen downward accountability rests against a backdrop of a larger set of global reforms intended to ensure government agencies are more attentive to the concerns of citizens when implementing policy, not simply professional norms and bureaucratic rules (see Behn, 1998; Ho, 2007, p. 1158; Peters & Pierre, 2000). This literature suggests that outcome measurement can strengthen downward accountability by improving transparency, increasing responsiveness and ensuring that organizational priorities are aligned with citizen concerns.

Drawing on outside theory and literature, while limited, can shed light on accountability issues in the nonprofit sector. Scholars have drawn on principal agent theory, a theory originally developed to understand the firm (e.g., Miller, 2002), whereas others have drawn on the idea of negotiated accountability, a concept developed by public management scholars, to shed light on accountability issues in the nonprofit sector (see Kearns, 1994, p. 186; Ospina et al., 2002, p. 9). The public management literature is particularly helpful here because it discusses how outcome measurement is used to strengthen downward accountability. After reviewing this literature, I discuss possible limitations of drawing on the public management literature to understand how outcome measurement can be used to strengthen downward accountability in the nonprofit sector.

Theme 1: Identifying performance measures

Governments around the world are engaging citizens in the performance measurement process, e.g., identifying priorities, collecting and analyzing data, contributing to decision making (see Holzer & Rhee, 2005). Involving citizens in the identification and prioritization of outcomes is one way that government ensures that it measures what matters to citizens. Involving citizens in performance measurement can increase participation in public problem solving more broadly (see Behn, 1998; Callahan, 2007; Irvin & Stansbury, 2004; Roberts, 2004; Thomas, 1990; Vigoda, 2002). Early research shows that if citizens are involved in this process, it is more likely that performance measures will be considered in policy conversations and budget decisions, while citizen trust in government will increase (Ho, 2007).

Theme 2: Reporting performance data

Policy and scholarly discussions also suggest that regularly reporting performance data to the public can strengthen downward accountability in two ways. First, information on performance—particularly when this data is comparative against a baseline, promised targets, or comparable jurisdictions or units—allows citizens to raise questions or voice concerns to policymakers or directly to public managers. Second, by regularly reporting performance data to citizens, citizens are in a better position to make educated choices about what they are willing to pay for or where they would like to receive services. Whether performance reporting is used to encourage citizens to hold government to account by exercising choice or encouraging greater citizen deliberation and engagement has been a central tension in these discussions (Hirschman, 1970). Regardless, without easily accessible information about government performance, citizens are in a weak position to hold government accountable, with implications for the public’s trust in and willingness to support government. In the United States, a number of initiatives have been undertaken to improve performance reporting to the public. For example, the Government Accounting Standards Board offers 16 standards for reporting this data so that it is easily accessible to citizens and publications targeted to public managers offer suggestions for how to report to citizens (see Fountain et al, 2003 Poister, 2003).

Theme 3: Measuring the quality of the encounter

In addition to involving citizens in identification and prioritization of outcomes and reporting results in a way that improves transparency, outcome measurement can also strengthen the accountability relationship between government agencies and citizens by measuring the quality of citizens’ experience with government personnel. Service quality measures have been advocated in the public sector for quite some time as a way to improve responsiveness to citizens receiving the services, whether it’s the taxpayer seeking clarification from the Internal Revenue Service or the driver renewing a license at the Department of Motor Vehicles (see Hatry, 1977). Service quality measures include timeliness, accessibility, and accuracy. Recent discussions suggest that using performance measures to improve responsiveness to citizens have to be coupled with efforts to allow staff the flexibility they need to respond effectively to individual circumstances. Yet the emphasis on customer satisfaction has raised concerns about whether elevating satisfaction and responsiveness to users of government services might undermine broader commitments to democratic accountability, specifically the responsibility of government employees to treat all citizens equally (Behn, 1998). Moreover, other scholars have suggested that individual satisfaction may conflict with broader public values (Moore, 2002; Radin, 2006).

Summary

Reviewing the public administration literature, we can make three observations about how outcome measurement can strengthen downward accountability. First, the adoption of outcome measurement could strengthen downward accountability if it encourages the involvement of beneficiaries in the process of identifying and prioritizing outcomes. Second, the adoption of outcome measurement could also strengthen downward accountability if nonprofits report their performance to beneficiaries in ways that allows them to raise questions about services. Third, the adoption of outcome measurement could strengthen downward accountability if nonprofits measure the quality of beneficiaries’ experience with the nonprofit in ways that spur greater responsiveness to their concerns or enables them to make informed choices.

However, before examining whether outcome measurement will strengthen nonprofit accountability to beneficiaries, we might ask whether outcome measurement should be used to strengthen nonprofit accountability to beneficiaries generally or if it should be used in the same way that it has been used in the public sector. After all, nonprofits are not legally accountable to beneficiaries but rather for their use of charitable assets for purposes stated in their organizing documents. And unlike taxpaying citizens, beneficiaries do not own a nonprofit the way citizens own government nor do they usually contribute financially to the nonprofit (King & Stivers, 1998).

Bracketing for a moment the larger question of whether nonprofits should be accountable to beneficiaries, the mechanisms for strengthening downward accountability identified in the public administration literature are similar to the mechanisms for strengthening downward accountability identified in the nonprofit accountability literature. For example, Kilby (2006) suggests that the greater the depth of the feedback arrangements the nonprofit has in place for its beneficiaries and the more formal the processes to secure “constituency voice” in the organization, the stronger the accountability relationship to beneficiaries. Others echo this point. Ospina et al. (2002), in their study of accountability among identity based nonprofits, found that nonprofits took active steps to get feedback from their beneficiaries around the mission and priorities for the organization, through surveys and meetings, informal visits and conversations, and encouraging staff to report needs to management. At the same time, nonprofits themselves have undertaken efforts to strengthen this accountability relationship in ways similar to those suggested above. For example Action Aid developed the Accountability Learning and Planning System (ALPS), which includes using a participatory review and reflection process with their beneficiaries and making information like appraisals and reviews open to all (see http://www.actionaid.org).

Coming back to the larger question about whether nonprofits should be accountable to the beneficiaries of their programs and services, the nonprofit accountability literature has not discussed the rationale for downward accountability directly. However two rationales are implicit in these conversations. First, because nonprofits make claims about and on behalf of beneficiaries when appealing to donors, talking with funders and advocating in public policy forums, nonprofits should be answerable to these beneficiaries (see Anheier, Kaldor, & Glasius, 2005; Guo & Musso, 2007; Kissane & Gingerich, 2004; LeRoux, 2009). Second, because users receive services of the nonprofit, users can exercise “choice” and “voice” to ensure nonprofits are accountable to their concerns (Ebrahim, 2003b; Williams, Webb, & Phillips, 1991). With that said, the rationale for downward accountability in the nonprofit sector deserves more attention. For the purposes of this article, the analysis rests on two previous observations made by scholars: (a) Beneficiaries are one important accountability stakeholder for nonprofit organizations and (b) evaluation is an important mechanism that could be used to strengthen this relationship (see Ebrahim, 2002; Edwards & Hulme, 1996, p. 8; Najam, 1996). The next section examines outcome measurement’s potential to strengthen downward accountability by examining how nonprofits are guided to develop and use their outcome measurement system.

Outcome Measurement in the Nonprofit Sector

Outcome measurement involves the ongoing process of defining, monitoring, and using performance indicators to improve organizational effectiveness (Poister, 2003). Outcome measurement started taking off in the nonprofit sector during the 1990s and over the course of 20 years many tools and guidebooks have been developed to support nonprofits in identifying and measuring outcomes. This section examines the content of 10 outcome measurement guides targeted to nonprofit organizations in the United States. The findings reported here focus on the how nonprofits are told to develop and use outcome measurement and the role of beneficiaries in this process. This analysis is informed by new institutional theory in sociology, which reminds us that documents are not simply sources of facts or technical resources; they are carriers of normative ideas about practice (Powell & DiMaggio, 1991; Prior, 2003; Scott, 1995). In this regard, these guides both reflect dominant conceptions of how nonprofits should develop and use outcome measurement as well as direct nonprofits in this process. To understand the potential of outcome measurement to strengthen nonprofit accountability to beneficiaries, the analysis below presents examples of text from the guides organized into the three thematic areas identified in the literature reviewed above: (a). Identifying performance measurement; (b). reporting performance data; and (c). measuring the quality of the encounter between nonprofit staff and those they serve. Before moving to this analysis, I briefly report how the guides were selected, analyzed, and give a general description of the guides.

Selection and analysis

The 10 guides selected for this analysis were identified through a scan of organizational websites conducted between August 2006 and March 2007. The organizational websites included funders from the Neighborhood Funders Group, local, and national professional associations of nonprofits, management support organizations including those that focused on evaluation (e.g., Harvard Family Research Project, Innonet, TCC Group) as well as nonprofit professional publications (e.g., Chronicle of Philanthropy, Nonprofit Quarterly, and Nonprofit Times). The assumption was that if nonprofits were interested in or required to develop an outcome measurement system, they would be guided by organizations that would have some expertise or authority on outcome measurement in their environment. In total close to 500 websites were scanned using key words: accountability, performance, evaluation, and outcome.

Four publications—a Harvard Family Research Project newsletter, a publication by The Rensselaerville Institute, a report by NeighborWorks and a guide for adopting Results Oriented Management and Accountability—reviewed common outcome measurement models (see Table B1 in the Appendix B). Of the 19 resources listed in these publications, 10 of them were guides that appeared on more than one list and were targeted to nonprofits addressing a range of social problems (in contrast to resources targeted to government or nonprofits addressing a specific issue like substance abuse). These guides were chosen for analysis (see Appendix A). These 10 guides are not inclusive of all available outcome measurement material out there nor are these guides representative. This would assume that there is a population of outcome measurement material that can be readily identified and selected. Having attempted such an effort, the 10 guides collectively provide a reasonable proxy of how nonprofits are directed to measure their performance. These guides include some of the most well known outcome measurement resources targeted to nonprofits. By analyzing common outcome measurement models, we can have more confidence that nonprofits have been exposed to these models directly or indirectly and consequently the analysis can address the puzzle explored here.

Once the guides were identified, the texts were then converted into Microsoft Word and uploaded into ATLAS.ti. The coding involved three analytical steps. First, the content was sorted into three categories using the following questions: (a) What are nonprofits told to measure? (b) how are they told to measure it? and (c) how are they guided to use this information? This article examines the second and third question (the results of the first question are examined in another article). Second, this coding scheme was refined. For example, “outcomes” were further coded into types of outcomes: knowledge, conditions, behavior, and so on. Finally, in addition to combing through the texts multiple times to further refine the codes, a search and automatic code function in ATLAS.ti was used to identify all the instances of customer, client, beneficiary, participant, constituent, resident, and community to detect any unanticipated themes or messages about the nonprofit beneficiaries. In addition, all instances of terms such as outcome (to examine how the term was defined and the direction given to nonprofits about identifying their outcomes) and accountability (to examine how the term was used and in relationship to what set of stakeholders) were examined.

General description of the guides

The 10 guides all shared a common purpose: To encourage and support nonprofits in measuring outcomes to improve their effectiveness in addressing social problems. The 10 guides explained the rationale for measuring outcomes and then detailed how nonprofits should develop their outcome measurement system. This included positing questions and using models or templates to identify outcomes. Despite the similarities, each of the guides had their unique emphasis. For example, Results Based Accountability emphasized the distinction between population and program accountability. The Balanced Scorecard, originally developed by Kaplan and Norton for firms, looked at the internal drivers of external performance (internal process, employee learning and growth). The Success Measures guide offered a roadmap for a participatory approach to evaluation and performance measurement. At the same time, while seven of the guides targeted nonprofits broadly, two of the guides targeted specific subfield of nonprofits. Success Measures (SM) is targeted to community development organizations and Scales and ladders/ROMA (ROMA) targeted community action agencies. One guide targeted organizations that used training and education as a strategy applicable to any number of issues (Targeting outcomes of programs [TOPS]). Another guide (Outcomes funding framework [TRI]) targeted funders, with nonprofit organizations as the secondary audience (TRI, p. 104).

Theme 1: Identifying performance measures

All the guides offered technical tools and frameworks to identify programmatic and organizational outcomes and suggested that nonprofits consider outside stakeholders in this process. Who to involve and how they should be involved in the overall process of outcome measurement and specifically in identifying outcomes varied and was not always explicit. Two guides placed emphasis on funders only, with no mention of involving beneficiaries in identifying outcomes (ROMA; TRI). For example, one identified policy makers, funders (investors) or implementers (nonprofits) as the source of performance targets (TRI, p. 118). Others suggested that nonprofits consider stakeholders’ perspectives, such as beneficiaries’ experiences, when identifying outcomes without explicitly directing nonprofits on how to consult them (Balanced scorecard [BSC]; Outcomes engineering/result mapping [OE]; Results-based accountability [RBA]). Two of the guides listed beneficiaries as one stakeholder that could be a source of ideas for outcomes (Urban institute key steps in outcome management [UI]; United Way of America [UW]). Two guides emphasized using outcome measurement in a participatory way, in part to strengthen the voice of the community or ensure accountability to beneficiaries (Logic model [KF]; SM). For example, one guide stated “Developing and using logic models is an important step in building community capacity and strengthening community voice” (KF, p. 3). Here beneficiaries were considered one stakeholder group to involve in developing the logic model: “engaging as many stakeholders as possible in the process of developing a logic model” (KF, p. 7). The other guide, which was developed for community development organizations, suggested that the residents were most important in identifying indicators: “The process of involving community members in indicator selection and success measurement will serve to hold organizations accountable to meeting the needs of the community” (SM, p. 8). This guide suggested that funders did not need to be involved but that the nonprofit should share the evaluation plan, indicators and targets with major funders (SM, p. 22).

This analysis suggests that outcome measurement may lead nonprofits to get feedback from beneficiaries regarding programmatic outcomes, and this may be important if the nonprofit does not do this currently, but this message was not coherent or forceful in most guides. The extent to which nonprofit beneficiaries should be considered in the outcome measurement process and how their input should be considered relative to other stakeholders was unclear—including whether they should be involved in the more substantive work of prioritizing outcomes or in simply sharing their experiences about program services. Moreover, no direction was given about what to do when beneficiaries’ priorities or perspectives differed from other stakeholders’ views or from what may be possible for the organization to address. One guide explicitly noted the confusion over who the primary customer is for a nonprofit—funders or beneficiaries (BSC).

Theme 2: Reporting performance data

The outcome measurement guides suggested reporting outcome measurement data to external stakeholders, sometimes as a general directive, sometimes to demonstrate accountability and sometimes for marketing. Directives about using outcome measurement data to report to stakeholders were in the beginning of the guides as a part of the rationale for why nonprofits should spend the time developing an outcome measurement system as well as in sections of the guides dedicated to analyzing, reporting and using data. Most guides associated reporting outcome measurement data, including for accountability purposes, exclusively to the public and/or funders, not to beneficiaries (BSC; KF; OE; RBA; ROMA; TOPS; TRI). Looking at Table B2 in the Appendix B, four of the guides specifically mentioned beneficiaries when discussing reporting results, but in three of the guides the focus was on using the data to attract or retain beneficiaries (UW; UI; KF). Sharing results with beneficiaries to ensure that the nonprofit is accountable to these beneficiaries was suggested in only one guide: “Sharing this information will help keep your organization accountable to the people it serves” (SM, p. 37). Aside from reporting outcome data, two other guides suggested involving beneficiaries in the data analysis process by asking beneficiaries about the reasons for specific outcomes (UI; UW). Although involving beneficiaries in analysis is not the same as reporting performance, this process could be easily serve this purpose. Three guides also suggested sharing desired outcomes, versus outcome data, with beneficiaries to motivate them to achieve desired outcomes or as one guide stated “to provide participants with a map for the road ahead, ensuring they are less likely to stray from the course” (KF, p. 3; TRI; UW).

This analysis suggests that outcome measurement seems less likely to strengthen downward accountability because the emphasis is more on using this data to attract, retain or motivate beneficiaries rather than reporting data to help them raise questions about performance or make informed choices about services. The guides gave nonprofits quite a bit of advice on how to use outcome data internally to inform practice—an important contribution given that nonprofits typically treat outcome measurement as a reporting requirement for funders—but the guides gave little direction for how nonprofits could augment this internal learning through external reporting. One guide suggested that organizations develop the “inside story”—an honest account of how the program or organization is performing, before reporting externally “Without developing the inside story, nonprofits just move to using the data for external promotion with little internal gain” (RBA).

Theme 3: Measuring the quality of the encounter

Capturing the quality of the beneficiaries’ encounter with the nonprofit was one suite of measures, among many, discussed in the guides. Three themes were evident here. First, the guides suggested that beneficiary satisfaction was an important precursor of desired outcomes. Second, although some guides suggested that nonprofits measure beneficiary satisfaction specifically (BSC; KF; ROMA; UW; UI) others suggested capturing their immediate response to the encounter (OI; TOPS; TRI). For example, one guide asked: Did they react to the activities as intended? Did they rate the activities as informative, interesting, and applicable? Did they perceive any immediate benefits? Do they anticipate potential benefits? Third, the guides pointed nonprofits to different dimensions of this encounter, including staff compassion, respect, and treating beneficiaries with dignity, access, timeliness of service, and accuracy. When discussing why nonprofits should measure beneficiaries’ experiences with the quality of the encounter, one guide noted that by measuring satisfaction and “asking participants what they think creates a power balance between the nonprofit and those served” (ROMA). Another guide gave satisfaction surveys as an example of a mechanism to institutionalize accountability to beneficiaries, helping ensure programs reflected the interests and priorities of these beneficiaries (SM).

Of all the ways that outcome measurement may serve as a mechanism to strengthen downward accountability, the directive that nonprofits should measure the quality of the interaction between staff and beneficiaries was the least ambiguous in the guides. Yet, the analysis also calls attention to the diverse way in which this encounter is conceptualized and measured, raising a question about the relationship between what is measured and the extent to which this strengthens nonprofit’s accountability to beneficiaries. For example, if nonprofits measure beneficiaries’ perceptions of staff compassion will this better ensure responsiveness than measures of timeliness? Research in the public sector shows that beneficiaries can give high satisfaction marks to the staff that serve them, believing these staff are doing the best job they can, and still have an overall experience with the organization that is negative (Soss, 2000, p. 121). At the same time, nonprofits have been using satisfaction and feedback surveys for quite sometime but we do not know the content of these surveys or how the data has informed nonprofit practice.

The analysis presented above offers a glimpse of the rich content of these guides and also shows that outcome measurement is far from a homogenous practice. The diversity among the guides served to point to gaps in the other guides and raised questions about whether this guidance would strengthen downward accountability. The next section discusses the implications of this analysis for the accountability relationship between nonprofits and their beneficiaries.

Potential of Outcome Measurement for Strengthening Accountability to Beneficiaries

This analysis shows that although the guides do encourage nonprofits to engage in certain practices, like conducting focus groups with beneficiaries and using satisfaction surveys that could strengthen downward accountability, the guides leave certain issues unaddressed. Specifically, the guides offered limited direction regarding (a) How much to engage beneficiaries in the process of identifying and prioritizing outcomes and specifically how to consider their perspective relative to the perspectives of other stakeholders (b) how to integrate internal and external uses of outcome data, thereby limiting the potential for outcome measurement to strengthen accountability to beneficiaries through greater openness and (c) how to best conceptualize the quality of the encounter between nonprofits and beneficiaries in ways that lead to greater responsiveness. As a reflection of normative ideas about how to do outcome measurement, the guides suggest that beneficiaries have an ambiguous standing in the nonprofit accountability environment, sometimes portraying beneficiaries as one potential customers, sometimes as strong accountability partners but more often as one of several stakeholder groups that nonprofits should attend to in the process of developing and using outcome measurement. Given the absence of other mechanisms to strengthen beneficiary voice and downward accountability this lack of clear direction is noteworthy.

However, this analysis is a starting point. Clearly more research is necessary. This work could proceed along several lines. First, research could examine how nonprofits are using outcome measurement and if they are involving beneficiaries, if this strengthens their strategy and improves performance. Second, research could examine the different types of outcome measurement models adopted by nonprofit organizations and any resulting changes in the nonprofit-beneficiary relationship: Do the guides that emphasize partnership with beneficiaries lead nonprofits to strengthen this relationship more than those that do not? Does the measurement of the beneficiary encounter and its conceptualization have any bearing on how well aligned staff perceptions are with beneficiaries’ perspectives? A third line of research could explore the relationship between existing accountability practices in nonprofits and the adoption of outcome measurement. Nonprofits are far from monolithic in how they engage beneficiaries. What are the implications of adopting outcome measurement in these different settings? Perhaps organizations with more participatory cultures will use outcome measurement accordingly or perhaps nonprofits will struggle with the lack of alignment between the guides and how they work with their beneficiaries (see Benjamin, 2008). The analysis presented here lays the groundwork for more informed and nuanced research on how the adoption of outcome measurement may be reshaping the nonprofit-beneficiary relationship.

In considering whether outcome measurement could create a path to strengthen accountability between nonprofits and beneficiaries, dilemmas remain. First, using outcome measurement to strengthen downward accountability in the nonprofit sector, unlike the public sector, may be impossible or undesirable at times. Nonprofits have to remain attentive to other important stakeholders. For example, when nonprofits are contracted by the state to provide certain services, the outcomes are set and may be non-negotiable. At the same time, beneficiaries may have demands that are out of line with the organization’s mission and the direction set by the board. Second, related to the point above, the rationale for downward accountability deserves more direct attention by nonprofit scholars. Third, if nonprofits were to use outcome measurement to strengthen their accountability to beneficiaries this would require additional time and resources. Many of the guides analyzed here represent efforts to keep the outcome measurement process simple and manageable. Fourth, engaging beneficiaries can raise questions about organizational priorities or programmatic assumptions. Because nonprofits’ ability to actually respond to beneficiary perspectives can often be constrained by funding requirements, board priorities and limited resources, this can just lead to unmet expectations and meaningless participation (see Hickey & Mohan, 2004). In the end, without strong and clear guidance—about the role of beneficiaries in the identification of outcomes, about how to report outcomes to beneficiaries in ways that move beyond attraction and retention or without a clearer understanding of the relationship between satisfaction measures and the responsiveness of the organization to beneficiaries—outcome measurement seems unlikely to strengthen downward accountability. This is not an insignificant point when we consider that outcome measurement seems to offer the most potential for strengthening accountability to beneficiaries.

Footnotes

Appendix A

Appendix B

Table B2.

How Should Outcome Measurement Data Be Used?

	Example guidance	Report outcome results to beneficiaries? For what purpose?
Balanced scorecard	pp. 38-39 Benefits of using a balanced scorecard (7 reasons)	Beneficiaries not specifically mentioned
	*Demonstrate accountability and generate results.
	*Attract scarce resources.
	Enhanced Trust. I’ve already discussed how the balanced scorecard can help you demonstrate accountability and attract scarce resources to your organization. The driver of both those outcomes is enhanced trust: Trust from the community, trust from your funders, and trust from your employees*. (the 4 other reasons: self preservation; create a focus on strategy; produce information and not data; drive change)
Logic model	pp. 5-6 Why use a logic model? (5 points made; 1st point):	Yes. So beneficiaries can decide whether to continue to participate
	1. Logic models better position programs for success. . . .We all know the importance of reporting results to funders and community stakeholders alike.
	p. 40 What information will your program’s audiences want? (5 stakeholder groups listed in table with information interests):
	Participant’s Typical Questions: Use this to make decisions about continuing participation
	Funders Typical Questions: Use this to make decisions about accountability and improvement of future grantmaking efforts
Outcomes engineering/results mapping	Tool 3. Establishing the outcome challenges (out of 5 tools)	Beneficiaries not specifically mentioned
Outcomes engineering/results mapping	pp. 30-31 . . . At some point, there will be enough journeys in the mix to produce a very clear picture in the mind of the funding source and others of the impact domain of the initiative and of the specific ways it has worked within this domain to contribute to life changes in others.	Beneficiaries not specifically mentioned
Scales and ladders/ROMA	p. 52 report your results in way that helps the investors “tell the story” to other key players	Beneficiaries not specifically mentioned
Results-based accountability	p. 99 Reporting progress on population and performance efforts:	Beneficiaries not specifically mentioned
Results-based accountability	Whether you are working on population or performance accountability, it will be necessary to periodically report progress to your stakeholders. . . . . Combining numbers and stories is the most powerful way to report progress. . . . All three types of progress should be used in report cards, strategic plans, grant applications and any other form of reporting to stakeholders.	Beneficiaries not specifically mentioned
Success measures	p. 37	Yes. To ensure your program is accountable to beneficiaries.
	The main reasons for collecting outcome information are to inform planning, to improve program management and to document the results of your work for a wide range of audiences* and ultimately to hold your organization accountable to its mission, constituents and funders. . .
	Once you are routinely collecting this information to guide program management and decision making, you will find it useful to share much of it with funders*….
	It is equally important to share this information with users* of your program and the broader community. They are after all, the primary reason for being business as a community development organization. Sharing this information will help keep your organization accountable to the people it serves.
Targeting outcomes of programs	p. 2 TOPS 9 steps.9.	Beneficiaries not specifically mentioned
Targeting outcomes of programs	Use outcome tracking and program performance evaluation to improve subsequent programs, document accountability, and market programs.	Beneficiaries not specifically mentioned
Outcomes funding framework	(Note: TRI emphasizes reporting on milestones achieved as a way to manage for outcomes. Milestones are critical markers that must be achieved to hit performance targets).	Beneficiaries not specifically mentioned
	p. 85. Brief quarterly reports should stress literal and specific performance in achieving milestones. . . .A final step in the project close out is a written or oral report in which implementers contribute learnings to their colleagues and the investor.
United Way of America	External uses of outcome findings (TP 35 in Training Kit; pp.142-146 in a Practical Approach. 5 uses)	Yes. To market program to beneficiaries.
	There are many ways to use outcome findings to enhance a program’s interactions with its various publics. . . .
	2. Promote program to potential participants and referral sources. . . .Once participants decide to use a particular service, the outcome information helps define what they can expect from the program so that they and program staff are working toward the same objectives.
	5. Retain and increase funding. Many human service programs are implementing outcome measurement to meet the requirements of funders. . . .Programs can respond to this increased concern with accountability by including information on outcomes, measurement approaches, and findings in their funding requests and progress reports.
Urban institute key steps in outcome management	In addition to the internal uses. . .p. 28	Yes. To Market programs to users
Urban institute key steps in outcome management	Improving fundraising and communicating with the public. Outcome information can also be used in proposals and other materials seeking funding. It is assumed that organizations that can document the beneficial outcomes of programs to clients and the community are more likely to receive support. Such information can also encourage potential clients to use the services of the nonprofit.	Yes. To Market programs to users

Acknowledgements

The author wishes to thank Laurie Goldman, Lisa Ranghelli, Jennifer Shea and the anonymous reviewers for their comments on earlier drafts of this article. The author bears sole responsibility for the contents

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: I received funding from the “George Mason University Provost’s Summer Research Fund”.

Author Biography

Lehn M. Benjamin is an associate professor with the Center for Nonprofit Management, Philanthropy and Policy at George Mason University. Her research examines the contributions of nonprofit organizations to democracy and how efforts to ensure greater accountability in the nonprofit sector may support or constrain this contribution. Her current project focuses on front line staff and how they experience accountability and performance expectations.

References

Anheier

H. K.

Kaldor

Glasius

(Eds.) (2005). Global civil society 2004/2005. London, UK: Sage.

Behn

R. D.

(1998). The new public management paradigm and the search for democratic accountability. International Public Management Journal, 1(2), 131-164.

Benjamin

L. M.

(2008). Bearing more risk for results: Performance measurement and nonprofit relational work. Administration & Society, 39, 959-983.

Callahan

(2007). Citizen participation: Models and methods. International Journal of Public Administration, 30, 1179-1196.

Campbell

(2002). Outcomes assessment and the paradox of nonprofit accountability. Nonprofit Management & Leadership, 12, 243-259.

Carman

J. G.

(2007). Evaluation practice among community based organizations: Research into the reality. American Journal of Evaluation, 28(1), 60-75.

Carman

J. G.

Fredericks

K. A.

(2008). Nonprofits and evaluation: Empirical evidence from the field. In Carman

J. G.

Fredericks

K. A.

(Eds.), Nonprofits and evaluation: New directions for evaluation (pp. 51-72). San Francisco, CA: Jossey Bass.

Chaskin

R. J.

(2003). Fostering neighborhood democracy: Legitimacy and accountability within loosely coupled systems. Nonprofit and Voluntary Sector Quarterly, 32, 161-189.

Chisolm

L. B.

(1995). Accountability of nonprofit organizations and those who control them: The legal framework. Nonprofit Management & Leadership, 6(2), 141-156.

10.

Christensen

Ebrahim

(2006). How does accountability affect mission? The case of a nonprofit serving immigrants and refugees. Nonprofit Management and Leadership, 17, 195-209.

11.

Cutt

Murray

(2000). Accountability and effectiveness evaluation in nonprofit organizations. London: Routledge.

12.

Ebrahim

(2002). Information struggles: The role of information in the reproduction of NGO- funder relationships. Nonprofit and Voluntary Sector Quarterly, 31(1), 85-113.

13.

Ebrahim

(2003a). NGOs and organizational change: Discourse, reporting and learning. Cambridge: Cambridge University Press.

14.

Ebrahim

(2003b). Accountability in practice: Mechanisms for NGOs. World Development, 31(5), 813-829.

15.

Ebrhaim

(2003c). Making sense of accountability: Conceptual perspectives for northern and southern nonprofits. Nonprofit management and Leadership, 14, 191-212.

16.

Ebrahim

(2005). Accountability myopia: Losing sight of organizational learning. Nonprofit and Voluntary Sector Quarterly, 34(1), 56-87.

17.

Edwards

Hulme

(1996). Beyond the magic bullet: NGO performance and accountability in the post-cold war world. West Hartford, CT: Kumarian Press.

18.

Fine

A. H.

Thayer

C. E.

Coghlan

(2000). Program evaluation practice in the nonprofit sector. Nonprofit Management & Leadership, 10(3), 331-339.

19.

Fountain

Campbell

Patton

Epstein

Cohn

(2003). Reporting performance information: Suggested criteria for effective communication. Washington, DC: Governmental Accounting Standards Board.

20.

Guo

Musso

J. A.

(2007). Representation in nonprofit and voluntary organizations: A conceptual framework. Nonprofit and Voluntary Sector Quarterly, 36, 308-326.

21.

Hatry

H. P.

(1977). How effective are your community services? Procedures for monitoring the effectiveness of municipal services. Washington, DC: Urban Institute Press.

22.

Hickey

Mohan

(Eds.). (2004). Participation: From tyranny to transformation? Exploring new approaches to participation in development. London, UK: Zed Books.

23.

Hirschman

A. O.

(1970). Exit, voice and loyalty. Cambridge, MA: Harvard University Press.

24.

(2007). Exploring the roles of citizens in performance measurement. International Journal of Public Administration, 30, 1157-1177.

25.

Holzer

Rhee

S. K.

(2005). Citizen driven government performance. National Center for Public Productivity: Rutgers University.

26.

Houchin

Nicholson

H. J.

(2002). Holding ourselves accountable: Managing by outcomes in girls incorporated. Nonprofit and Voluntary Sector Quarterly, 31, 271-277.

27.

Hwang

Powell

W. W.

(2009). The rationalization of charity: The influences of professionalism in the nonprofit sector. Administrative Science Quarterly, 54, 268-298.

28.

Irvine

R. A.

Stansbury

(2004). Citizen participation in decision making: Is it worth the effort? Public Administration Review, 64(1), 55-65.

29.

Kearns

K. P.

(1994). The strategic management of accountability in nonprofit organizations: An analytical framework. Public Administration Review, 54, 185-192.

30.

Kearns

K. P.

(1996). Managing for accountability: Preserving the public trust in public and nonprofit organizations. San Francisco, CA: Jossey-Bass.

31.

Kilby

(2006). Accountability for empowerment: Dilemmas facing nongovernmental organizations. World Development, 34, 951-963.

32.

King

Stivers

(1998). Government is us: Public administration in an anti-government era. Thousand Oaks, CA: Sage.

33.

Kissane

R. J.

Gingerich

(2004). Do you see what I see? Nonprofit and resident perceptions of urban neighborhood problems. Nonprofit and Voluntary Sector Quarterly, 33, 311-333.

34.

LeRoux

(2009). Managing stakholder demands: Balancing nonprofit responsiveness to clients and funding agents in nonprofit social service organizations. Administration & Society, 41, 158-184.

35.

Meyer

J. W.

Rowan

(1983). Institutionalized organizations: Formal structure as myth and ceremony. In Meyer

J. W.

Scott

W. R.

(Eds.), Organizational Environments: Ritual and Rationality (pp. 21-44). Beverly Hills, CA: Sage.

36.

Miller

(2002). The board as a monitor of organizational activity: The applicability of agency theory to nonprofit boards. Nonprofit Management & Leadership, 12, 429-450.

37.

Moore

M. H.

(2002). Privatizing public management. In Donahue

John D.

(Ed.) Market based governance: Supply side, demand side, upside, downside (pp. 296-322). Washington, DC: Brookings Institution Press.

38.

Najam

(1996). NGO accountability: A conceptual framework. Development Policy Review, 14, 339-353.

39.

Ospina

Diaz

O’Sullivan

J. F.

(2002). Negotiating accountability: Managerial lessons from identity-based nonprofit organizations. Nonprofit and Voluntary Sector Quarterly, 31(1), 5-31.

40.

Peters

B. G.

Pierre

(2000). Citizens versus the new public manager. The problem of mutual empowerment. Administration & Society, 32(1), 9-28.

41.

Plantz

M. C.

Greenway

M. T.

Hendricks

(1997). Outcome measurement: Showing results in the nonprofit sector. In Newcomer

K. E.

(Ed.), Using performance measurement to improve public and nonprofit programs (pp. 15-30). San Francisco, CA: Jossey-Bass.

42.

Poister

(2003). Measuring performance in public and nonprofit organizations. San Francisco, CA: Jossey Bass.

43.

Powell

W. W.

DiMaggio

P. J.

(Eds.). (1991). The new institutionalism in organizational analysis. Chicago: University of Chicago Press.

44.

Prior

(2003). Using documents in social research. Thousand Oaks, CA: Sage.

45.

Radin

(2006). Challenging the performance movement: Accountability, complexity and democratic Values. Washington, DC: Georgetown University Press.

46.

Robers

(2004). Public deliberation in an age of direct citizen participation. The American Review of Public Administration, 34, 315-353.

47.

Scott

W. R.

(1995). Institutions and organizations: Foundations for organizational science. Thousand Oaks, CA: Sage.

48.

Soss

(2000). Unwanted claims: The politics of participation in the U.S. welfare system. Ann Arbor, MI: University of Michigan.

49.

Speckbacher

(2003). The economics of performance management in nonprofit organizations. Nonprofit Management & Leadership, 13, 267-279.

50.

Thomas

J. C.

(1990). Public involvement in public management: Adapting and testing a borrowed theory. Public Administration Review, 50, 435-445.

51.

United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: Author.

52.

United Way of America. (2000). Agency experiences with outcome measurement: Survey findings. Alexandria, VA: Author.

53.

Vigoda

(2002). Administrative agents of democracy? A structural equation modeling of the relationship between public sector performance and citizen involvement. Journal of Public Administration Research and Theory, 12, 241-272.

54.

Williams

H. S.

Webb

A. Y.

Phillips

W. J.

(1990). Outcome funding: A new approach to targeted grantmaking. Rensselaerville, NY: The Rensselaerville Institute’s Center for Outcomes.