Abstract
Research on performance measurement by nonprofit organizations increasingly focuses on the use of outcome measurement (OM) to assess organizational effectiveness. This article applies a strategic choice framework to analyze how nonprofit managers’ evaluation of the importance of organizational stakeholders is associated with patterns of OM. The article introduces a multidimensional measure of nonprofits’ implementation of OM that incorporates its extent of program use, as well as whether resources are specifically allocated for this evaluative practice. This multidimensional measure is examined using data from a new survey of service providing nonprofits in the city of Boston. Our multivariate analysis investigates how three sets of influences—resource providers, networks, and internal stakeholders—impact patterns of OM. The findings indicate that the factors driving program use (internal stakeholders) are distinct from those that impact resource allocation (resource providers).
Introduction
A growing body of scholarship examining the turn to accountability in the nonprofit sector focuses on the use of performance measurement by nonprofit organizations.
Performance measures are defined in various ways in different studies but generally include attention to measures of effectiveness, efficiency, workload, and productivity (Ammons, 2002; LeRoux & Wright, 2010; Zimmerman & Stevens, 2006). Increasingly, research on performance measurement by nonprofit organizations examines the use of outcome measurement (hereafter “OM”) to assess organizational effectiveness (Alexander, Brudney, & Yang, 2010; Mitchell, 2012). The use of outcomes as the optimal sign of organizational performance replaced prior efforts to measure inputs (e.g., organizational revenue), throughputs (e.g., number of staff), and outputs (e.g., number of programs and/or clients) as other indicators of organizational success (Packard, 2010; Plantz, Greenway, & Hendricks, 1997).
This article draws on insights from a strategic choice framework in organizational and nonprofit studies (Bradshaw, 2009; Brown & Iverson, 2004; Child, 1972, 1997; Maranville, 1999) to examine how nonprofit managers’ identification of three sets of influences—resource providers, networks, and internal stakeholders—is associated with implementation patterns of OM. It advances our understanding of how nonprofit organizations are using OM by introducing a multidimensional measure that focuses on two important ways in which the implementation of this practice varies across nonprofit organizations. This typology of OM implementation incorporates its extent of program use as well as whether resources are specifically allocated for this evaluative practice. We examine this multidimensional measure using data from a new survey of service-providing nonprofit organizations in the Boston area. Focusing only on those nonprofits that report the use of OM, we find that these two dimensions, program use and resources, are driven by different sets of organizational stakeholders. Managers’ identification of the importance of external actors explains the extent of use of OM at the program level, whereas the influence of internal actors is associated with the allocation of resources for OM.
We begin by discussing the literature on OM in the nonprofit sector. We then consider the applicability of a strategic choice perspective to explain nonprofits’ employment of OM and introduce a multidimensional measure of OM that builds on previous research. After introducing our data and methods, we summarize the results of our statistical model in which patterns of OM implementation are explained by a range of influences identified by nonprofit managers, in addition to organizational controls. We conclude with a discussion of our findings and their implications for future research.
Outcome Measurement in the Nonprofit Sector
Outcome measurement refers to an organizational practice that seeks to quantify the impact of a nonprofit’s programs/services on clients (Benjamin, 2008; Hatry, 2002; Speckbacker, 2003). Outcomes are “benefits or changes for individuals or populations during or after participating in a nonprofit’s activities” (Morley, Vinson, & Hatry, 2001, p. 5). Nonprofit organizations measure their impact by quantitatively comparing the level of a particular attribute before and after the client receives the nonprofit’s good or service, in order to demonstrate the effectiveness of a nonprofit intervention, program, or service (Benjamin, 2008; Speckbacker, 2003).
The use of OM by nonprofits to evaluate organizational effectiveness is a widespread practice (Carmen, 2007; Mitchell, 2012; Morley et al., 2001) that is present across a variety of nonprofit subsectors (Thomson, 2010; Zimmerman & Stevens, 2006). The increasing prevalence of OM is the subject of much concern and interest among practitioners, researchers, and nonprofit stakeholders. Some praise the practice for increasing organizational accountability (Hatry, 2006; Plantz et al., 1997), whereas others raise concerns that nonprofits’ use of OM inhibits innovation and contributes to mission drift (Alexander, Nank, & Stivers, 1999; Poole, Davis, Reisman, & Nelson, 2001). 1
Although much is known about the consequences of OM, we know less about the determinants of this move toward quantifying effectiveness in the nonprofit sector. A burgeoning body of work examines how organizational or environmental characteristics are associated with the use of OM. This research finds organizational mission/type, size (e.g., budget and staff), and funding sources, such as philanthropic foundations (e.g., Benjamin, 2010; Carson, 2000; Sheehan, 1996), the United Way (e.g., Carman, 2009; Julian & Kombarakaran, 2006), and government agencies (e.g., Carlson, Kelley, & Smith, 2010; Thomson, 2010) to be associated with levels of OM. However, we know less about how nonprofit managers account for their organizations’ decisions to implement this form of performance measurement and how managers’ perceptions shape patterns of use of OM.
A Strategic Choice Perspective
When faced with a complex environment including multiple funding sources, increasing demand for accountability, and competition from the private sector, nonprofit managers must make a range of choices about how to structure and operate their organizations. Within the scholarly and practitioner-oriented literature, many authors have sought to explain why organizations implement particular policies or programs; examples include collaboration (Guo & Acar, 2005), strategic positioning (Chew & Osborne, 2009), interorganizational restructuring (Campbell, 2008), and organizational dissolution (Fernandez, 2008).
Much of this research accounts for the adoption of new organizational practices via attention to environmental determinants. In this literature, organizational structure, processes, and effectiveness are deterministically driven by objective environmental factors (Barman & MacIndoe, 2012; Chew & Osborne, 2009; Fernandez, 2008; Guo & Acar, 2005; LeRoux & Wright, 2010; McCarthy & Walker, 2004). For example, one study of nonprofit survival notes, “Ecological factors (such as organizational density, and the size and age of the organization) and environmental factors (demographic, economic, and regulatory changes) determine the life chances of organizations” (Wollebaek, 2009, p. 268). The deterministic nature of these explanations emphasizes that environmental factors are outside of the control of organizational decision makers. This approach minimizes the understanding and agency of organizational members in accounting for organizational decisions and instead emphasizes the deterministic role of external and internal factors (Donaldson, 2001).
In contrast to the focus on how the environment influences organizational practices and outcomes, the strategic choice perspective underscores the importance of leaders’ perceptions, agency, and interests in organizational decision-making processes. According to this framework, a deterministic view fails “to give due attention to the agency of choice by whoever has the power to direct the organization” (Child, 1972, p. 2). Organizations’ structures follow from stakeholders’ (e.g., managers and/or the board of directors) voluntaristic choices, based on their perceptions of the environment and their organizations (Bradshaw, 2009; Campbell, 2008; Delfin & Tang, 2008; Slessarev-Jamir, 2004; Weick, 1995; Yaghi, 2009). Brown and Iverson (2004, p. 377), for example, state that “managers of nonprofit organizations make choices that seek to improve the performance of their organization. These decisions involve interpreting and framing the environment, developing and implementing programs and services, and creating processes and structures to monitor and control resources for optimal impact.”
Recent studies on the nonprofit sector demonstrate the significance of organizational members’ subjectivity for understanding organizational outcomes. A multiplicity of perspectives can exist across stakeholders in a single organization, who often disagree in their understanding of an organization’s mission, behavior, and effectiveness (Brown & Iverson, 2004; DuBois et al., 2009; Green, Madjidi, Dudley, & Gehlen, 2001). Some scholars focus on the perspective of nonprofit managers/leaders as prominent stakeholders who influence organizational processes, such as relationships with funders and interpretation of organizational mission. Delfin and Tang (2008), for example, conducted a survey of environmental nonprofits in order to assess managers’ impressions of philanthropic foundations. Contrary to literature that emphasizes the negative impact of foundation funding on recipients’ autonomy, respondents held largely positive views of foundations and strategically sought out their support. Similarly, Slessarev-Jamir (2004) shows that variation in churches’ engagement with faith-based community organizing can be explained by examining the geographical orientation of pastors’ concerns. Congregations led by pastors with local priorities were more likely to engage in such political activity, regardless of differences in organizations’ structural or environmental characteristics. This research underscores the importance of paying attention to how managers’ perceptions influence decisions that impact a wide range of organizational processes and outcomes.
Factors Associated With Variation in Nonprofit Implementation of Outcome Measurement
In this article, we examine how nonprofit managers’ explanation of the decision to implement OM affects a multidimensional measure of OM use. Literature from nonprofit and organizational studies suggests three sets of influences that might influence managers’ decisions concerning the implementation of OM: resource providers, organizational networks, and internal stakeholders.
The Role of Resource Providers
Many studies of OM posit that nonprofits adopt this evaluative practice when compelled to do so by resource providers (Carman, 2009; Hatry, 2002). For example, the national United Way encourages local United Ways to require sponsored nonprofits to measure program impact (Julian & Kombarakaran, 2006; Plantz et al., 1997). Philanthropic foundations also are increasingly asking recipients to engage in OM (Benjamin, 2010; Carson, 2000; Frumkin, 2004).
The Role of Organizational Networks
In addition to their relationships with funders, nonprofits are located in a web of connections to other organizations. These networks, including partnerships or collaborations with other nonprofits, serve as the means by which norms and expectations of good governance are diffused (DiMaggio & Powell, 1983; Hwang & Powell, 2009). National associations in the nonprofit sector, including Big Brothers Big Sisters of America and the Child Welfare League of America, provide training and resources to their member organizations (Plantz et al., 1997; Zimmerman & Stevens, 2006). Likewise, nonprofits that receive an accreditation from industry or third-party intermediaries may be more likely to adopt OM given the requirements involved in securing such approval (Packard, 2010).
The Role of Internal Stakeholders
Finally, previous studies have established that nonprofit organizations are accountable to multiple constituencies, including external parties such as funders and organizational collaborators, as well as internal organizational stakeholders such as staff, volunteers, the board of directors, and clients (Hatry, 2006; Ospina, Diaz, & O’Sullivan, 2002; Speckbacker, 2003). Nonprofits face pressures from members of boards of directors to implement performance measurement as a sign of professionalism (Newcomer, 2008). Similarly, staff are members of professional communities, subject to distinct sets of norms, values, and expertise, often involving techniques of evaluation (Powell, Gammal, & Simard, 2006).
Outcome Measurement: From a Dichotomous to a Multidimensional Measure
To investigate the role of manager’s perceptions of the relative importance of a range of organizational stakeholders on the use of OM in nonprofit organizations, we introduce a multidimensional measure of OM. The growing body of research examining OM in the nonprofit sector typically operationalizes the adoption of OM as a dichotomous concept. Nonprofits are treated as if they implement OM in a consistent and universal manner across all mission-related programs or as if they do not implement OM at all. Studies that speak to the extent of use of OM typically report on nonprofits’ use of OM with respect to whether one or more recent program evaluations have included outcome measures. If a nonprofit reports using outcome measures in one or more program evaluations or performance measurement practices, then the nonprofit organization is counted as using OM. For example, Fine, Thayer, and Coghlan asked whether nonprofits “had completed at least one program evaluation in the past three years” (2000, p. 332). They found that 56% of nonprofits’ recent evaluations were conducted to measure outcomes. Several studies based on state-level samples of nonprofits found similar levels of OM usage: 65% in South Carolina (Zimmerman & Stevens, 2006), 62% in Indiana (Carman & Fredericks, 2008), and 60% of nonprofits in New York (Carman, 2009) used OM. Despite the growing body of research that examines OM in the nonprofit sector, Thomson notes that “structured inquiry into the extent of nonprofit outcome measurement is a relatively recent phenomena” (2010, p. 612, emphasis added).
One exception to this dichotomous approach is Thomson’s (2010) investigation of how funders’ reporting mandates influenced the extent of OM use before and after the imposition of new reporting requirements for a sample of government-funded nonprofits in Detroit. He develops an index to measure the extent of OM based on the number of outcomes a nonprofit identified, the time horizon for the outcomes, and whether outcomes were supported by an assessment tool. Whereas 64% of nonprofits identified outcomes, Thomson found that slightly more than half (55%) measured outcomes across all identified categories.
Thomson’s (2010) index contributes a more nuanced measure to assess the extent of OM, one based on the number, complexity, and collection of outcome measures. We build on this approach by introducing a multidimensional measure of OM implementation in order to better facilitate comparison of this practice across nonprofit organizations. Our multidimensional measure (see Figure 1) incorporates two important dimensions of variation in OM: the extent of use of OM to evaluate a nonprofit’s programs, and whether resources are earmarked for OM. Figure 1 specifies four patterns that characterize variation in OM implementation: (A) low program use, low resources; (B) high program use, low resources; (C) low program use, high resources; and (D) high program use, high resources. Although this two-by-two typology does not capture the full range of variation in how nonprofits might implement OM, it offers an important first step beyond the simple dichotomous conceptualizations dominant in existing research.
Multidimensional Measure of Outcome Measurement Implementation.
This multidimensional measure of OM, as opposed to a simple dichotomous (yes/no) measure, advances our understanding of how nonprofits implement OM in two important ways. First, we know that nonprofits—like other organizations—are subject to pressures from funders and regulators to comply with accountability-based expectations. Institutional theory (DiMaggio & Powell, 1983; Meyer & Rowan, 1977) has shown that organizations’ adoption of a new practice may be “ceremonial” and may not substantively alter actual practices. Nonprofits that adopt OM in response to external pressure might do so in name only in order to assert legitimacy with funders and other stakeholders. It is important that studies of OM consider the extent to which nonprofits are actually employing OM in their day-to-day provision of programs and services. Measuring the extent of use of OM across all of a nonprofit’s programs and services directs attention to an important aspect of variation that a simple dichotomous measure, which classifies a nonprofit as using OM if “one or more” programs is subject to OM, does not capture.
Second, our multidimensional measure of OM directs attention to an important factor—lack of resources—that previous research has cited as an obstacle to implementation of this practice. Although funders may require grantees to incorporate OM into their work, these same funders often do not provide funding for the training or implementation of this evaluative practice (Reed & Morariu, 2010; Stone & Cutcher-Gershenfeld, 2002). As Buckmaster notes, “The greatest impediment to measuring outcomes is one of resource availability” (1999, p. 186). As a result of a lack of resources and training, nonprofits may lack the internal capacity to actually employ OM in their everyday practices. For example, one study of Dallas nonprofits found almost half of organizations that had not evaluated their programs reported “there was not enough money” to do so (Hoefer, 2000, p. 171). Likewise, a study of northern Californian nonprofits found that “most agencies lacked the resources for systematic implementation” of OM (Botcheva, White, & Huffman, 2002, p. 421). When budgetary resources are allocated for the practice of OM, a nonprofit may be better positioned (than a nonprofit that lacks such resources) to train staff in the use of OM, to hire outside consultants to collect data, or to develop a systematic process by which OM can be used to inform future service delivery. The presence (or absence) of resources allocated for OM undoubtedly matters for our understanding of nonprofits’ implementation of this evaluative practice.
Data and Methods
To analyze variation in OM implementation by service-providing nonprofits, we rely on a new survey of executive directors of 600 nonprofits in the greater Boston area. Respondents were asked about a range of organizational attributes and practices, including whether and in what ways their nonprofit used OM. The survey also collected data on a number of organizational characteristics, including age, size, and mission. The survey sample was drawn from the Business Master File maintained by the National Center for Charitable Statistics (NCCS) at the Urban Institute. The sample was stratified by organization type/industry, size, and geographic location. The University of Chicago Survey Lab administered the online survey between September 2008 and February 2009 and achieved a 63% response rate (n = 379).
The sample included service-providing organizations from across all nonprofit subsectors. This means that all nonprofits in the study could have potentially employed OM to assess the impact of the services they provide. Nonprofit organizations without a primary focus on providing services, such as philanthropic foundations, were excluded. The sample also excluded religious organizations that are not required to register with the Internal Revenue Service (IRS) and are, therefore, not fully represented in the NCCS data, as well as smaller nonprofits with less than US$25,000 in annual income that are not required to file with the IRS. Finally, the sample excluded universities and hospitals since these organizations are subject to accreditation guidelines that require systematic implementation of performance metrics (Nolan & Berwick, 2006; Simpson, 1985). Aside from the exclusions noted above, the distribution of nonprofit industry type and organizational size in the sample is comparable to nonprofits across Massachusetts (MacIndoe & Barman, 2009).
A comparison of the nonprofits in our sample that report use of OM (n = 272) reveals some differences from those nonprofits that do not report using OM. Human service and education nonprofits are more likely to use OM than are nonprofits with other missions (e.g., arts or environmental organizations). Organizational size is an important correlate of the use of OM. Nonprofits with less than US$50,000 in annual revenue are less likely to report the use of OM than nonprofits with larger revenues. Finally, no age differences exist between nonprofits that use OM and those that do not (MacIndoe & Barman, 2009).
Dependent Variable: Variation in OM Implementation
The purpose of this study is to develop an understanding of how nonprofit managers’ identification of stakeholder influences impacts the pattern of OM implementation in nonprofit organizations. In the survey, we defined OM as “any systematic attempt by your organization to assess the impact of your organization’s activities or programs.” The majority of respondents, 72% (n = 272), indicated that their nonprofit used OM to evaluate services. The dependent variable in the analysis is a categorical variable that results from the juxtaposition of two dimensions of variation in nonprofit use of OM: its extent of program use and the allocation of resources. We operationalize the first dimension, extent of OM use (Figure 1, vertical axis), using responses to the survey question: “In the last fiscal year, what percentage of your organization’s programs/services were subject to outcome measurement?” For practical purposes, we choose to compare nonprofits that used OM to evaluate a majority of their programs/services with those organizations that used OM to assess less than half of their programs. Approximately 42% of nonprofits stated they used OM to evaluate less than half of their services. For 58% of the sample, OM was used to evaluate the majority of their nonprofit’s services.
We operationalize the second dimension in the typology, the allocation of resources (Figure 1, horizontal axis), using responses to the survey question: “In the last fiscal year, did your organization budget funds to implement outcome measurement?” About 44% of nonprofits in the sample budgeted funds for OM. Although there are other organizational resources that might be allocated to the practice of OM, such as a dedicated staff person, including a line item in a nonprofit’s budget is a clear demonstration of organizational commitment to the practice. Our survey instrument did not collect the percent of a nonprofit’s budget allocated to OM.
Table 1 shows the distribution of the sample across the dependent variable, which follows the multidimensional measure of OM implementation presented in Figure 1. The dependent variable takes on four values, which characterize different patterns of OM implementation: (A) Low program use/low resources (25% of sample). Nonprofits in this group evidence low levels of OM implementation with less than half of their services being subject to OM, and no funds budgeted for the practice. (B) High program use/low resources (31% of sample). These nonprofits use OM to evaluate a majority of their services but do so without organizational funds budgeted for OM. (C) Low program use/high resources (17% of sample). These nonprofits employ OM to evaluate less than half of their services, but they also have a line item in their budget for OM. (D) High program use/high resources (27% of sample). These nonprofits evaluate a majority of their services using OM and also budget for the practice.
Distribution of Dependent Variable.
Control Variables
Our study includes three critical control variables incorporated in previous studies of organizational practices in the nonprofit sector: organizational age, size, and mission. Age, for example, has been proven to influence organizations’ ability to respond to changing environmental demands. Older organizations may be less likely to adapt to external expectations given that their organizational blueprint, generated by founders, circumscribes their capacity for change (Stinchcombe, 1965). We expect that older organizations might be more likely to implement OM in a ceremonial fashion. Other research has emphasized size, arguing that larger organizations have the “slack” needed for substantive implementation of practices (Dobbin, Edelman, Meyer, Scott, & Swidler, 1988; Edelman, 1992). In addition, the use of OM is likely to vary with nonprofit industry since some types of programs and services are more amenable to this evaluative technique than others. For example, educational organizations can assess an increase in rates of literacy with relative ease, whereas an environmental organization faces greater difficulty in measuring progress in global warming.
We collected data about organizational age (based on year of IRS registration) and size (expenses in 2009) from the National Center for Charitable Statistics. We gathered data on nonprofit mission with our organizational survey, using categories from the National Taxonomy of Exempt Entities. Mission categories included health and human services, public benefit, education, arts, and the environment. Table 2 shows the distribution of the control variables across the dependent variable.
Distribution of Control Variables.
Managers’ Accounts of Influences on Outcome Measurement
To learn more about why and how nonprofit organizations implement OM, we examine managers’ responses to survey questions about which influences led to their organization’s decision to use OM. We asked executive directors who reported that their nonprofits utilized OM to identify the influences leading to the adoption of OM by their organization via a series of survey questions. First, we asked executive directors to specify all the influences that led to their adoption of OM, drawing from a list of 10 different stakeholders drawn from the nonprofit literature. The list of stakeholders included resource providers, network participants, and internal organizational stakeholders.
To assess the impact of resource providers on OM implementation, we examine managers’ response to a survey question that asked them whether one or more of the following influenced their organizational decision to implement OM: foundations (noncorporate), corporate donors, individuals, and/or the United Way. To assess the effect of networks, we consider managers’ responses to survey questions that asked whether one or more of the following influenced their organizational decision to implement OM: organizational partners, accrediting organizations, and/or a national headquarters organization. To assess the importance of internal stakeholder on OM implementation, we look to managers’ responses to survey questions that asked whether one or more of the following influenced their organizational decision to implement OM: board of directors, staff, and/or clients. Table 3 shows the distribution of responses across the 10 influences in our survey question.
Percent of Managers Selecting Influence on Decision to Use Outcome Measurement.
Some observers have argued that the turn to OM is a result of pressures within nonprofits’ wider environment (Hatry, 2002; Samples & Austin, 2009; Zimmerman & Stevens, 2006). In contrast, we find that the most frequently selected factors identified by managers—boards of directors and foundations—include both internal and external organizational stakeholders. A majority of all managers (57.5%) identified foundations as one of the key influences on their organization’s decision to use OM. This finding reflects the recent move by institutional funders to tie their resources to accountability requirements (Frumkin, 2004). Interestingly, managers did not commonly select other supposedly critical external drivers of OM, including the United Way (16.2%), as important for understanding the move to OM. Surprisingly, the largest percentage of managers (61.7%) identified the board of directors as a central influence and slightly less than half (48.5%) specified that staff drove the move to OM. Perhaps even more unexpectedly, a quarter of respondents selected clients/program participants as an important influence on the decision to implement OM, suggesting that nonprofits’ internal constituents are also key proponents of the move toward OM by charities. Means, standard deviations, and correlations for the independent variables are shown in Table 4.
Means, Standard Deviations, and Correlations for Independent Variables.
p < .05. **p < .01.
Findings and Discussion
Research on OM in the nonprofit sector emphasizes the socially constructed nature of evaluation in organizational settings (Forbes, 1998; Herman & Renz, 1999; Packard, 2010). These studies, drawing from a strategic choice perspective, highlight the importance of examining how nonprofit managers perceive and act on influences in their environment. In order to examine the significance of the factors identified by nonprofit managers for understanding a multidimensional measure of OM, we estimate a multinomial logistic regression model. The categorical dependent variable specifies four patterns of OM implementation, as summarized in Table 1. There is no natural ordering among the values of the dependent variable, so multinomial logistic regression is an appropriate estimation method. The results of the regression model are shown in Table 5. A significant positive logistic coefficient means that the independent variable increases the odds of being in the nonreference category versus the reference category. The reference group (Table 1, Pattern A) consists of nonprofits with low program use of OM and no budget line item to support its use. Since all nonprofits in the analysis use OM (to some extent) to evaluate the services they provide, the results of this model apply only to nonprofits that use OM.
Multinomial Logistic Regression Model of Outcome Measurement (OM) Implementation.
p < .05. **p < .01.
Dependent variable category (A): Low program use, low resources, is the reference variable.
High program use: Nonprofit uses OM to evaluate more than 50% of their services.
Low program use: Nonprofit uses OM to evaluate less than 50% of their services.
Low resources: Nonprofit does not budget for OM.
High resources: Nonprofit budgets for OM.
Human service nonprofits are the reference category.
The independent variables included in the model are variables capturing the stakeholders that nonprofit managers identified as important to their decision to use OM. These influences are grouped in three categories: resource providers, networks, and internal stakeholders. The model includes controls for organizational age, size, and mission. 2 With one exception, the control variables are not statistically significant. 3 Organizational age was a significant predictor of a nonprofit being in the low program use/budget category (Pattern C), as opposed to the reference group of low program use/no budget category (Pattern A). This means that, among nonprofits that evaluate less than 50% of their services, older organizations are more likely to budget for the practice of OM.
The analysis, which shows that influences shaping the extent of use of OM to evaluate programs are different from those that impact the allocation of resources for OM, highlights the value of a multidimensional measure of OM. First, the influence of resource providers, notably philanthropic foundations, is a significant positive predictor of whether a nonprofit budgets for OM. 4 If an executive director perceives foundations to be an important driver of the adoption of OM, this increases the probability that a nonprofit will be in the low program use/budget or the high program use/budget categories (Patterns C and D) versus the low program use/no budget category.
Interestingly, and in contrast to other research (Carman, 2009), managers’ perceptions of the importance of the United Way are not a significant predictor of membership in any category of OM implementation, despite efforts by the United Way of America to promote the use of OM by affiliated organizations. It is important to note that pressure from funders to implement OM affects the allocation of resources for OM, but does not differentially impact the extent of use of OM to evaluate programs. While foundation influence is a significant predictor of OM patterns regarding budgeted resources (Patterns C and D), these two patterns of implementation evidence both high and low program use. Allocating resources for OM might result in substantive employment of OM, or it might result in limited or ceremonial usage in order to satisfy funders. This finding complicates research that concludes that resource scarcity prevents the use of OM, despite pressures toward its diffusion. Instead, our data indicate that some nonprofits budget for OM but only implement it sparingly (Pattern C), whereas other nonprofits employ OM to evaluate their programs, even without specific funds budgeted for this purpose (Patterns A and B).
Second, we find that network factors, such as accrediting organizations, organizational partners, and national headquarter organizations, do not impact either dimension of OM implementation (program use or resource allocation), with one important exception. The influence of a national headquarter organization increases the probability that a nonprofit organization will be in the low program use/high resource category (Pattern C), instead of the low program use/low resource category (Pattern A). This means that, among nonprofits with low implementation of OM (less than 50% of services subject to OM), organizations in which the executive director perceives the national headquarters organization to be an important influence will be more likely to budget for OM. However, the influence of the national headquarters does not extend to substantive employment of OM to assess a majority of a nonprofits’ programs.
Finally, our analysis shows that the variables capturing the influence of internal stakeholders have significant positive coefficients across all patterns of OM implementation. In particular, the variable indicating that an executive director perceives the board of directors to be an important influence on the adoption of OM increases the probability of a nonprofit being in an implementation pattern that allocates resources for OM (Patterns C and D) as compared to membership in the reference category (low program use/no budget). Clearly, nonprofit managers feel that their board of directors must support the practice of OM in order to budget funds for it.
The analysis also highlights the importance of a second group of internal stakeholders, the nonprofit staff members who are often charged with implementation of OM, even in the face of low resources indicated by no explicit budgeting for the practice. One example of how a nonprofit organization can employ OM without a budget line item for it is by employing low-cost approaches to collecting outcome data (Campbell, 2002). For example, an after-school tutoring program can ask children and/or their parents to fill out a survey to assess how participating in the program improved students’ knowledge and skills. Our expectation that nonprofits can perform OM without budgeting for the practice by relying on the talents of existing staff (and the labor of clients) appears to be supported by our analysis. We find that managers’ perception that nonprofit staff is an important influence on the use of OM increases the likelihood that a nonprofit organization will be in a high program use category (Patterns B or D) versus the low program use /no budget category (Pattern A the reference category). This indicates the critical role that organizational staff play in the implementation of OM in nonprofits that evaluate more than 50% of their services, particularly in situations of low resources (Pattern B).
A multidimensional approach to OM suggests two innovative findings concerning why and how nonprofits employ OM. First, following from previous research, we might expect OM use by nonprofits to be clustered into patterns of “high use and high resources” and “low use and low resources.” However, we find considerable variability in implementation patterns of OM (Table 1), with more than half of the nonprofits in our study implementing OM without specific resources budgeted for its use. In addition, less than half of nonprofits implement OM across a majority of their services or programs. For nonprofit observers concerned with questions of ceremonial versus substantive employment of this practice, our study also sheds some light on the different factors (internal vs. external stakeholders) that impact patterns of OM use in nonprofit organizations.
Second, we find that two different sets of actors or pressures drive the two different dimensions of OM use—influences driving the extent of OM use to evaluate programs/services are distinct from those that impact resource allocation for OM. The expectations of internal stakeholders (the board of directors and staff) drive patterns of high program use of OM, regardless of resource allocation. The extent of use of OM is shaped by nonprofit managers’ perception of the importance of internal actors. In contrast, resource allocation for OM implementation appears to be driven by funders, specifically foundations, regardless of the extent of OM program use. The allotment of funds for OM is influenced by nonprofit managers’ perception of the importance of external actors.
Conclusions
Making sense of how organizations come to adopt specific practices is of perennial concern to nonprofit scholars and practitioners. It is well established that nonprofits do not operate in a vacuum but are influenced by a range of external and internal influences. However, scholars disagree about how to measure those factors—whereas an environmental determinism perspective emphasizes the critical role of structural and environmental factors, the strategic choice approach suggests that those forces matter most when organizational members perceive them to have an important impact on organizational practices.
In this article, we applied the premises of the strategic choice framework to the study of how nonprofits implement OM. For those organizations that already employ OM, we sought to determine which set of manager-identified influences—resource providers, networks, and internal stakeholders—explained patterns of implementation, as indicated by usage across programs and resource allocation for OM. Our study shows that external stakeholders, such as funders and national headquarters, drive the adoption of a budget line for OM, but nonprofits only substantively implement OM to evaluate their programs when internal stakeholders also support its use. Although nonprofits may be budgeting funds for OM, the commitment of resources does not necessarily translate into the actual employment of OM to evaluate programs. This finding accords with research on the use of performance measurement systems by state and local governments. For example, de Lancer Julnes and Holzer argue that “rational and technocratic” factors drive the adoption of performance measures, whereas “political and cultural” factors impact its implementation (2001, p. 702).
Our research suggests several implications for the implementation of OM across the nonprofit sector. First, the factors that previous research highlights as leading to the adoption of OM do not also explain the pattern of implementation of OM by organizations. Observers have emphasized the role of external funders in the diffusion of OM, while also suggesting that internal constituents rarely initiate the use of OM. As Hatry (2002, p. 352) notes, “The impetus for performance measurement has typically come from external funders seeking accountability, not from public managers themselves seeking the information to help them improve their programs.” Instead, our study indicates that analyses of the impact of OM use for nonprofits’ effectiveness (e.g., LeRoux & Wright, 2010) should distinguish between the forces leading to its adoption versus those determining its implementation. Second, this research might be relevant for actors in the nonprofit sector who seek to promote the use of OM by nonprofits. We find that the provision of resources for the employment of OM does not result in the widespread employment of OM across a nonprofit’s programs. External actors must ensure that internal constituents are brought on board with the idea of OM in order to achieve substantive implementation across programs. One way to encourage such organizational buy-in would be for funders to explicitly provide support for staff training to implement OM in their program delivery.
Future scholarship could test the validity of our findings for the implementation of other types of organizational practices in the nonprofit sector, such as partnerships (Guo & Acar, 2005), advocacy (Schmid, Bar, & Nirel, 2008), or organizational restructuring (Campbell, 2008). Given our findings, research on a specific practice should address how external and internal influences affect not only the adoption of that activity but also how it is implemented. Future investigation of the use of OM in the nonprofit sector could also examine the effect of accounts by managers on the uses to which OM is put in nonprofits. For example, Carman and Fredericks (2008) find that organizations perceive evaluative techniques in three distinct ways: as a resource drain and distraction, as an external promotional tool, and/or as a strategic management tool. Subsequent studies might investigate how organizational leaders’ perceptions of how OM is used affect its program use and resource allocation. Finally, although our analysis shows that budgeting for OM influences its implementation, future research could investigate whether and how the amount of money budgeted for OM impacts nonprofits’ use of OM.
Our analysis should be framed by some caveats. First, our study is based on a sample of nonprofits located in the greater Boston area. Much research on the nonprofit sector has been grounded in analyses of specific geographical regions (e.g., Gronbjerg & Paarlberg, 2001; Joassart-Marcelli & Wolch, 2003); nonetheless our study’s findings are limited to this particular city and should be generalized with care to larger populations of service-providing nonprofits. In addition, this research relies on survey results to examine a multidimensional measure of OM. Other research might use qualitative interviews to explore the dynamics of this growing organizational practice.
Despite these limitations, this study advances a new multidimensional measure that is useful for comparing implementation patterns for OM. The analysis also improves our understanding of the factors that influence the employment of OM by focusing on managers’ perceptions of their environments. Most critically, our finding that different influences drive the extent of use versus the allocation of resources improves our understanding of this increasingly widespread practice in the nonprofit sector.
Footnotes
Acknowledgements
The authors would like to acknowledge three anonymous reviewers for their constructive comments.
Authors’ Note
An earlier version of this article was presented at the 2010 Association for Research on Nonprofit Organizations and Voluntary Associations.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors would like to acknowledge funding from The Boston Foundation, University of Massachusetts Boston (McCormack Graduate School of Policy and Global Studies), Boston University, the Kennedy School of Government (Hauser Center for Nonprofit Organizations and Rappaport Institute for Greater Boston), and the ASA Fund for the Advancement of the Discipline.