The Impact of Capacity-Building Programs on Nonprofits

Abstract

Nonprofit organizations (NPOs) play an important role in delivering social services to those in need. Capacity-building efforts for NPOs derive from funders’ desire to increase NPOs’ effectiveness and redress problems associated with limited administrative and organizational capacity. Through technical assistance, training, and grant funds, funders aim to enhance NPO functioning and ultimately improve client outcomes. Despite a general consensus about the importance of capacity building, little high-quality evidence exists on the impact of capacity-building investments. This article presents the findings from the first random assignment evaluation to be conducted in the field of nonprofit capacity building. The subject of the evaluation was one of country’s largest organizational capacity-building initiatives, the federal government’s Compassion Capital Fund (CCF) Demonstration Program. Findings from the evaluation provide clear evidence that capacity-building efforts increase capacity in each of five critical areas of capacity.

Keywords

randomized control trial nonprofit capacity building evaluation

Overview

Nonprofit organizations (NPOs) play an important role in delivering social services to those in need. Foundations, as well as local, state, and federal governments fund NPOs to deliver a wide range of services, typically in community-based settings accessible to those who need assistance most. While working through NPOs often allows funders to reach deeper into communities and reach more of those in need; some of these same NPOs tend to experience capacity limitations that hinder their growth, sustainability, and effectiveness. These capacity gaps may include incomplete recordkeeping, limited financial monitoring, and/or unstructured volunteer management. Since the mid-1990s, these concerns have led funders to invest in NPOs, providing technical assistance, training, and grant funds to improve the ways organizations function, typically with the goal of ultimately improving client outcomes.

Despite 20 years of investment in organizational capacity building, rigorous empirical evidence of capacity-building programs’ effectiveness is limited. Prior to the study described here there have been no random assignment evaluations. Instead, the evaluations of capacity-building programs in the current literature have used a variety of less rigorous methods such as prepost, cross-sectional analysis (regression adjusted), or quasi-experimental difference in differences (Abt Associates, 2007; Abt Associates & Branch Associates, 2008; Brown, 2008; Doherty & Mayer, 2003; Kapucu, Healy, & Arslan, 2011; Leake et al., 2007; Leviton, Herrera, Pepper, Fishman, & Racine, 2006; Markovitz & Magged, 2008; Millese & Bies, 2007; Patrizi, Gross, & Freedman, 2006; Popescu & Dewan, 2009; Sobeck, 2008; Sobeck & Agius, 2007). Specifically, without random assignment, concern remains that any estimated impacts are due to imperfect controls for preexisting differences between those that did and did not receive capacity building, and are therefore spurious. Random assignment solves the problem of imperfect controls by forming program and control groups by randomization, such that the two groups do not differ systematically prior to the intervention. Thus, any statistically significant differences between the program and control groups observed at follow-up can be attributed to the intervention or chance (and we can use conventional statistical methods to bound the impact of chance).

The study described in this article was designed to evaluate one of the nation’s largest capacity-building initiatives, the Compassion Capital Fund Demonstration Program (CCF), using a random assignment design. The evaluation’s funder, the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services, sought answers to two primary research questions: How effective is the capacity-building assistance in increasing organizational capacity among NPOs, and in what areas of organizational capacity have NPOs shown the greatest improvement? ACF contracted with Abt Associates to design and complete the evaluation.

The results reported here show that the CCF Demonstration Program had significant impact in each of the five critical areas of organizational capacity examined. The largest improvements in capacity were also spread across all five capacity areas. Not only does this random assignment study provide methodologically rigorous evidence about the effectiveness of capacity-building assistance provided through the CCF Demonstration Program; more broadly, it also provides the first random assignment evidence that capacity-building assistance can be used to improve NPOs’ organizational capacity.

The remainder of this article proceeds in five sections. The first two sections provide additional detail on the context and program design of the CCF Demonstration Program. The third section describes our evaluation design and data methods. The final two sections present the evaluation results and discuss their implications.

Historical Context

Over the past two decades, there has been an increased focus on the role faith-based and community nonprofit organizations (NPOs) play in the country’s social service network (Jolley & Meissen, 2009). NPOs help to fill gaps in the social safety net by providing an array of services such as child care, adult education and job training, mental health counseling, and substance abuse treatment (Allard, 2009a, 2009b; Sobeck, 2008). NPOs are often perceived as more accessible to individuals reluctant to seek help from large, unfamiliar public agencies. NPOs tend to be seen as trusted and respected community institutions, which increases their approachability to individuals with specific needs, such as victims of domestic violence, people with addiction issues, and troubled youth. For these reasons, there is an ongoing interest among local, state, and federal governments in partnering with NPOs to deliver services (Sherman & Stanakis, 2002).

Logic of Capacity-Building Assistance

While NPOs have vital attributes that increase their accessibility within and to the community, some NPOs lack the organizational capacity critical to their survival and effectiveness (Blumenthal, 2003; De Vita, Fleming, & Twombly, 2001; Letts, Ryan, & Grossman, 1999; Light, 2004). Organizational deficiencies have been found to include inadequate financial tracking systems, limited fundraising experience, and/or ineffective Boards of Directors. In addition, small, frontline organizations with limited administrative and organizational capacities and resources may have difficulty complying with the rigors of state or federal reporting and funds management requirements. They thus cannot effectively partner with the state or federal government. Because of the unique role that NPOs play and the capacity gaps that many face, funders, including the federal government, believe it is worthwhile to help NPOs increase their potential by assisting them in strengthening their organizational capacity (De Vita et al., 2001; Letts et al., 1999; Light, 2004; McKinsey & Company, 2001).

The capacity-building logic model assumes that areas of limited organizational capacity inhibit the degree to which NPOs can facilitate program and participant outcomes. Technical assistance and supplemental funds provided through capacity-building assistance are expected to lead to improvements in business operations (short-term outcomes), more effective service delivery and, ultimately, to better outcomes for service recipients and their communities (a nuanced version of this logic model for the CCF Demonstration Program is presented in Exhibit 1.1).

Exhibit 1:

Exhibit 1: Logic Model - Compassion Capital Fund Demonstration Program.

In the nonprofit field, the specific definition of “organizational capacity” varies. Definitions generally comprise the skills, practices, and systems that allow NPOs to operate more effectively and sustainably. Key components of capacity building often cited in the literature include the implementation of financial controls and clear operating oversight, as well as policies and procedures related to staffing, governance, finances, long-term sustainability, and expansion of services (Brown, 2007; Connolly & York, 2002; De Vita et al., 2001, Glickman & Servon, 1998; Rodriguez & Herzog, 2003).

Capacity-building programs proliferated in the late 1990s and early 2000s, including the investment of government funds in programs to build the capacity of potential NPO partners. While considerable resources have been devoted to capacity-building programs, we have limited rigorous evidence about the extent to which such programs actually build capacity, and if they do, whether nonprofits with greater capacity actually have larger impacts on clients and constituents (Herman & Renz, 1998; Light, 2004; Linnell, 2003; Sobeck & Agius, 2007; Wing, 2004). While there are a few evaluations providing nonexperimental evidence that capacity building does increase NPOs’ capacity, and therefore may make them more effective at serving their clients (Abt Associates, 2007; Abt Associates & Branch Associates, 2008;¹ Brown, 2008; Doherty & Mayer, 2003; Elliott, 2002; Glickman & Servon, 2003; Kapucu et al., 2011; Klein, et al., 2009; Leake et al., 2007; Markovitz & Magged, 2008; Millese & Bies, 2007; Patrizi et al., 2006; Popescu & Dewan, 2009; Sobeck, 2008; Sobeck & Agius, 2007), these studies are open to concerns about internal validity. Furthermore, these studies almost exclusively consider the extent to which NPO capacity-building programs actually increase the capacity of NPOs (i.e., columns 1 through 3 on the logic model presented in Exhibit 1). There has been virtually no work on the key follow-on question: the extent to which NPOs with greater capacity are more effective at helping clients achieve their desired outcomes (i.e., columns 4 and 5 on the logic model presented in Exhibit 1). The current study focused on studying the former question, using rigorous methods to further the field; the latter question was beyond the scope of the study and will require future research.

Capacity-Building Program Description

The federal government has funded several programs designed to increase NPOs’ organizational capacity. Those programs included the CCF, administered by the Administration for Children and Families (ACF) within the Department of Health and Human Services, the Department of Justice’s Prisoner Reentry program and the Department of Labor’s Ready4Work program. Each of these programs provides funds for faith-based and community NPOs to access capacity-building services (e.g., workshops, one-on-one training).

This article reports on the random assignment evaluation of CCF. Specifically, the ACF-funded evaluation focused on CCF’s Demonstration Program, one of CCF’s principal and largest components. The CCF program focused on organizations’ management capacity and was based implicitly on what is perceived to be important for nonprofit organizations’ effectiveness, including such things as a formal mission statement, a strategic plan, human resource systems, an independent audit, and an information technology system or systems (Sowa, Selden, & Sandfort, 2004).

Specifically, the Demonstration Program funded intermediary organizations to deliver capacity-building training, technical assistance, and subawards (financial assistance) to NPOs in five critical areas:

Organizational Development: nonprofit incorporation; board governance; existence or adoption of systems related to human resources, finances, information technology; existence or adoption of specific policies and procedures; and fiscal controls.

Program development: expansion or enhancement of program or service delivery; program monitoring; and evaluating program outcomes.

Revenue development: diversified funding sources; financial sustainability; and donor development.

Leadership development: leadership professional and career development; staff professional and career development; and volunteer training and development.

Community engagement: community asset mapping; level of community participation; use of community needs assessments; and leveraging resources/collaborative relationships.

Through CCF’scapacity-building assistance in some or all of these areas, NPOs were expected to improve their effectiveness and overall management, for example, their ability to develop and train staff, expand the types and reach of their programs, and improve their access to and management of funds from diverse sources, including federal funds (Administration for Children and Families, 2009). The CCF Logic Model, illustrated in Exhibit 1, further illustrates the connection between the services delivered by intermediaries and the outcomes anticipated by program designers.

Services Delivered

In the target year for the impact evaluation (2006), 10 intermediary grantees were selected, each receiving grants of approximately US$500,000 to deliver capacity-building services to NPOs over a 17-month period. These services included three types of activities: group training, customized technical assistance (TA), and financial subawards. At least 40% of the intermediary’s grant funds were to be used for subawards to NPOs that received either TA or group training (i.e., not subawards, but no other capacity-building services). The one-on-one technical assistance was individualized based on NPO needs and was provided by either intermediary staff or consultants. The subawards were “working capital” grants and were typically used to purchase equipment and software, to update marketing materials such as brochures and websites, and to hire consultants for specific capacity needs. However, program rules stated that subawards could not be used by NPOs to fund direct client services.

Findings from the evaluation’s process study (described below in the Data and Measures section) indicated that topics commonly addressed through both group training and one-on-one technical assistance included: grant writing and fundraising, strategic planning, board development, community outreach, and financial management. Subaward funds were used to purchase equipment, typically but not exclusively computers and financial management and database management software, to update marketing materials such as brochures and websites, and to hire consultants for capacity building in areas such as board development, strategic planning, and obtaining 501(c)(3) status.

NPO Characteristics

The NPOs included in the CCF impact study were a diverse group. The organizations varied in years of operation and base revenue. Organizations were approximately evenly split between faith-based and secular. Thirty-four percent of the NPOs had existed for less than 5 years, 32% had existed for 5 to 15 years, and 34% had existed for 16 years or more. In terms of their annual revenue, one quarter of the organizations reported revenues under US$25,000 while 40% had annual revenues over US$150,000; the remaining 34% reported revenues between US$25,000 and US$150,000.

At the time they applied for capacity-building assistance, the majority of the NPOs were 501(c)(3)s (68%), had Boards of Directors (80%), and were engaged in community partnerships (88%). However, over half of the NPOs had a part-time or unpaid executive director; most other staff were unpaid and had no job description or performance reviews. Many lacked standard best practices, such as written strategic plans, regular organizational assessments, and a system for tracking participant outcomes.

Limitations of Study

Even though the study used a random assignment design, there are some limitations that should be considered when contemplating the results.

Results are self-reported. The survey and key informant interview data were entirely self-reported; there was no independent verification.

Unvalidated measurement instrument. Because a study of this size and depth had not previously been conducted, it was not possible to use a previously validated instrument for measuring capacity building. Instead, the evaluation developed a measurement tool based on the existing literature and validated through an expert review process.

Measuring capacity building. Since the study examined changes in capacity over a short timeframe, it was only possible to measure capacity built during that period, but not changes in services delivered and improvements in client outcomes. Given the scale of funders’ and intermediary agencies’ investment, it is possible that observed changes may lead to increases in services delivered and improvements in client outcomes. However, this study was not designed to measure these future outcomes, nor was it designed to identify links between individual capacity-building activities and specific capacity-building outcomes (e.g., how increases in IT capacity alone might lead to more clients served).

Limits to generalizability. It is difficult to generalize results about the impact of capacity building because: (a) organizations vary at the outset in their size and sophistication; and (b) they vary in their goals for capacity building. For example, we learned from the process evaluation that some NPOs sought technical assistance to increase their ability to serve more clients. In contrast, other NPOs sought to operate more effectively and efficiently at their current scale. As a result, in our instruments and analysis we have made efforts to detect movement from a number of starting points toward a variety of capacity-building goals, needs, and priorities (e.g., by including two types of outcome questions, as well as measuring capacity changes across a variety of capacity domains). However, results are based on the average change, so they may not generalize to all similar organizations.

Randomization Design

The study sample included the entire population of NPOs that applied for assistance and were determined eligible by the 10 intermediary organizations awarded CCF Demonstration Program grants in the 2006 grant cycle (September, 2006). The NPOs that met the intermediaries’ self-defined eligibility criteria were randomly assigned either to the program group that was to receive capacity-building services (i.e., a subaward and/or technical assistance) from the intermediary or to a control group that would not receive intensive capacity-building assistance for the duration of the study (an embargo period of 15 months). During the application period, 454 NPOs were enrolled in the Impact Study (237 assigned to the program group and 217 to the control group).

Prior to conducting our analysis, we tested whether program-group members reported receiving significantly more capacity-building assistance than control-group members, a key assumption in the study design. This was particularly important in this study because control-group members were not barred from attending group trainings offered by the intermediaries, nor from receiving intensive services from other capacity-building providers. This test was conducted using data from the retrospective follow-up survey that all study members were asked to complete (see the Data and Measures section for a description of the follow-up survey). Analysis showed that 87% of the program-group members received capacity-building assistance during the study period as compared to 52% of the control group, which is a difference significant at the 0.01 level. In addition, Exhibit 2 illustrates that in all topic areas program-group members were significantly more likely to receive assistance than were organizations in the control group.

Exhibit 2:

Assistance Received.

	% of NPOs that Received Assistance in Topic Area
Topics of Assistance	Program Group	Control Group	Level of Significance
Resource Development, Fundraising, & Grant Writing	77%	36%	**
Strategic Planning	69%	24%	**
Board Development	68%	27%	**
Networking, Collaboration, Partnerships	61%	26%	**
Human Resources & Volunteer Management	57%	20%	**
Evaluation & Outcome Measurement	56%	16%	**
Program Design	52%	16%	**
Financial Management	49%	19%	**
Becoming a 501(c)(3)	23%	7%	*

All data presented in this paper have been weighted.

p-value < 0.05; ** p-value <0.01

Source: CCF Demonstration Evaluation NPO Survey

Data and Measures

As part of the application process and before random assignment, NPOs completed a self-administered baseline survey which was developed for the study (see the following section for additional detail). This timing (during application, before randomization) ensured that the baseline survey was completed by organizations in both the program and control groups. The survey was typically completed by the organization’s executive director or a knowledgeable program staff member, who consulted with colleagues when they needed additional information.

To assess change and improvement in various areas of organizational capacity, 15 months after random assignment the research team administered a follow-up survey to both program and control group NPOs. A hard copy of the follow-up questionnaire was mailed to the individual that completed the baseline survey; after about 4 weeks of follow-up, a shortened version of the questionnaire was sent to the remaining nonrespondents. The follow-up survey response rate was 87%—76% of the organizations completed the full survey and 11% completed a shortened version of the survey.

In addition to obtaining information directly from NPOs, intermediaries completed a survey on their technical and financial assistance processes (February, 2008) and participated in in-person, 2- to 3-day site visits. During these site visits, senior evaluation team members gathered information about the intermediaries’ missions and histories, the services they delivered, and NPOs’ receptiveness to the CCF services provided. This information was collected through semistructured individual and small group interviews with the CCF program director, key intermediary staff, and, when applicable, program volunteers and partners. The evaluation team used data from the surveys together with data collected during site visits to understand and document intermediaries’ approaches to carrying out the CCF-supported activities.

NPO Survey Instrument Characteristics

The research team surveyed the literature and found that funders and capacity-building providers were using a variety of different tools to assess organizational capacity, but there did not appear to be an accepted standard for measuring, or even describing, organizational capacity. Because no existing instrument was found to meet all study needs, the team chose to create a new instrument, drawing relevant questions from a variety of instruments. Knowing that intermediaries were going to customize their capacity-building assistance to meet the individual needs of NPOs across all five domains specified by the CCF funding guidelines—organizational development, program development, revenue development, leadership development, and community engagement—the survey had to be designed to measure a wide range of capacity-building outcomes, both in terms of content (e.g., board development) and depth of capacity changes.²

To measure the depth of an organization’s change in capacity, the CCF survey asked two different types of questions.³ The first type of question asked about measurable actions taken by organizations to demonstrate growth in capacity (concrete changes), such as whether the organization implemented financial management procedures or whether its revenue increased. Most of these questions were yes/no questions, though there were a few continuous variables as well (e.g., number of staff, total revenue). The second type of question measured incremental steps toward capacity (continuum of focus). This second set of questions captured organizations’ perception of their progress in addressing a specific capacity area. NPOs rated their status on a scale ranging from “know we should work on this, but we lack the time or resources” to “not a focus because we are satisfied with our achievement in this area.” NPOs’ progress along this scale was taken as indicating their progress toward building specific capacities. By measuring NPOs’ movement along this continuum it was possible to better understand the CCF Demonstration Program’s success in helping organizations meaningfully engage in the capacity-building process.⁴

Analysis Methods

Our analysis methods included the assignment of proper weights, joint testing to address the problem of multiple comparisons, and individual item and subgroup analysis using a regression model.

To estimate program impacts, we first weighted the sample to adjust for nonresponse to the follow-up survey; these weighted data were used to estimate program impacts on individual outcomes as well as to conduct joint tests for significance. Impacts were estimated using regression-adjusted treatment/control differences in each outcome measured at follow-up. The regression models controlled for the baseline value of the outcome being tested and various NPO characteristics.

We also completed an additional level of analysis because there were 241 outcomes, which raises concerns about multiple comparisons; that is, when there are that many measures, a few items will often appear (using uncorrected tests) to be statistically significant merely by chance (Schochet, 2008). Following Schochet, for each of the five critical areas we performed a joint test to determine whether there was any evidence that the intervention collectively affected all of the outcomes in each critical area (e.g., Organizational Development).

Finally, we examined the impacts within subgroups of NPOs to determine whether the services work better for some types of NPOs than others. These subgroups included organization type (faith-based or secular); organization age; paid executive director; organization revenue level; and intermediary’s previous experience providing technical and financial assistance.⁵ To test for differences in impacts across subgroups, we augmented the basic impact model with an interaction between treatment and a dummy variable for one of the values of the subgroup.

Results

The Impact Evaluation showed that NPOs in the program group—those eligible to receive capacity-building services from CCF intermediary grantees—demonstrated significantly higher levels of capacity at follow-up in each of the five critical areas (Exhibit 3).⁶ The program group also had significantly higher capacity growth than the control group on several individual outcome measures in each of the five areas (Exhibit 4). Specifically, NPOs receiving capacity-building services from CCF intermediary grantees reported significant (at 5%) improvement on several measures of organizational development (20 of 64), program development (20 of 44), revenue development (21 of 52), leadership development (seven of 17), and community engagement (five of 30).

Exhibit 3:

Joint Tests for No Impact, by Critical Area.

Critical Area^	Mean Treatment Effect	Standard Error of Treatment Effect	p-value
Leadership Development (17 measures)	0.2095	0.0408	0.000	**
Concrete Changes (1 measure)	NA	NA	NA	NA
Continuum of Focus (4 measures)	0.1465	0.0561	0.026	*
Organizational Development (67 measures)	0.1487	0.0241	0.000	**
Concrete Changes (39 measures)	0.1233	0.0255	0.001	**
Continuum of Focus (24 measure)	0.2273	0.0402	0.000	**
Program Development (44 measures)	0.2335	0.0244	0.000	**
Concrete Changes (19 measures)	0.1180	0.0471	0.031	*
Continuum of Focus (24 measures)	0.3167	0.0290	0.000	**
Revenue Development (50 measures)	0.1399	0.0254	0.000	**
Concrete Changes (32 measures)	0.1191	0.0297	0.002	**
Continuum of Focus (16 measures)	0.1881	0.0452	0.002	**
Community Engagement (30 measures)	0.1686	0.0341	0.001	**
Concrete Changes (26 measures)	0.1292	0.0415	0.011	*
Continuum of Focus (4 measures)	0.0247	0.0584	0.002	**

Notes: ^ Training-related measures were excluded from the second round of analysis.

p-value<.05

p-value<.01

Exhibit 4:

Summary of Results, by Critical Area.

	Total Outcomes
Capacity Domain	Total Number of Indicators	Number of Significant Indicators	Percent Significant
Organizational Development	64	20	31%
Program Development	43	20	47%
Revenue Development	52	21	40%
Leadership Development	17	7	41%
Community Engagement	30	5	17%
Total	206	73	35%

The next five subsections review significant outcomes in each domain. Full results are included in the online appendix (available at http://nvsq.sagepub.com/supplemental).

Organizational Development

NPOs receiving capacity-building services from CCF intermediary grantees reported significant improvement on several measures of long-term planning, human resources management, technology access and use, and financial management systems. Specifically, as a result of the CCF services more nonprofits had written strategic plans, completed organizational assessments, written job descriptions for paid staff, and a designated person responsible for financial management. NPOs also increased their level of focus on developing effective financial management systems, recruiting and managing volunteers more effectively, and recruiting and developing board members with a wide range of expertise and community backgrounds. There was no impact on staff size, 501(c)(3) status, board composition, use of staff performance reviews, or the existence of financial management procedures.

Program Development

The CCF services also improved NPOs’ service delivery capacity. In the areas of recordkeeping and evaluation, the number of NPOs in the CCF program that were keeping systematic records on services provided to clients and developing or implementing plans to improve outcome measurement systems increased significantly. In the areas of service delivery and program design, significantly more NPOs in the CCF program developed or implemented plans to increase the number of clients served, expanded services to include new types of clients and/or geographic areas, added new types of services, and used new approaches to improve the quality of existing services. There was no impact on the actual number of clients served, the number of NPOs that added or expanded program areas, or the number of NPOs that measured client outcomes.

Revenue Development

NPOs that received CCF services increased their capacity in several areas of revenue development. Specifically, significantly more program-group organizations had written revenue development plans, hired grant writers to train staff, participated in development training, and increased their overall organizational focus on revenue development planning. The number of CCF program-group NPOs that applied for or received grants from any source in the prior 12 months increased significantly. Moreover, organizations that had never applied for federal funding were more likely to have applied for and received federal funding as a result of the CCF. While their total number of funding sources remained the same between baseline and follow-up, program-group NPOs were more likely to have obtained funding from new, additional sources during the follow-up period, which indicates that organizations may have been diversifying their funding sources. There was no impact on the total number of grant applications submitted, pending, or approved, or on total grant funds received.

Leadership Development

The CCF program also improved NPOs’ leadership development capacity. In particular, their executive directors completed a greater number of leadership training sessions, and they increased their level of focus on providing leadership and job skill development opportunities for staff. In the area of board development, CCF services increased the percentage of NPOs that sent board members to training about their roles and responsibilities, and strengthened the overall focus of the organizations on improving board performance. There was no impact on the number of organizations whose directors or staff met regularly with a mentor.

Community Engagement

Within this domain, the percentage of NPOs receiving CCF services that engaged in partnerships in their communities (especially with businesses and schools) increased significantly. There was also a significant increase in the percentage of NPOs that used both paid and unpaid advertisements to expand awareness of their organizations among potential partners and funders. There was no impact on the use of websites, written materials, or presentations to expand awareness of potential partners and funders, or on efforts to expand awareness in the general community.

Areas of Greatest Organizational Capacity Improvement

While NPOs in the program group experienced significant improvements in capacity in each critical area of organizational capacity, certain outcomes showed greater magnitudes of impact. Using effect size—that is, impact divided by standard error of outcome in the control group—as the measure of magnitude of improvement, the outcomes with the greatest improvement are presented in two exhibits. As seen in Exhibit 5, relatively large effects (0.40 or larger) were found on several “concrete change” measures. These are important foundational activities for organizations. Within the “continuum of focus” measures (Exhibit 6), there were large effects related to training, expanding program services, increasing evaluation expertise, and identifying new funding sources.

Exhibit 5:

Areas of Greatest Capacity Improvement—Concrete Changes.

Survey Item	Effect Size	p-value (Adjusted)
Organizational Development
In the past 12 months, organization conducted or participated in an assessment of organizational strengths/needs	0.54	**
In the past 12 months, number of staff that participated in training related to: Management & Administration	1.79	*
The number of functioning computers that the organization owns is sufficient for organization/staff needs	0.43	*
Leadership and Staff Development
Number of types of training in which head organization participated in the past 12 months	0.59	**
In the past 12 months, any Board member participated in training/learning opportunities to learn more about governance/roles and responsibilities of Board members	0.50	*
Revenue Development
In the past 12 months, the head of the organization participated in training related to fundraising	0.44	**
In the past 12 months, number of staff that participated in training related to: Fundraising	1.05	**
In the past 12 months, number of applications submitted for funding from federal government agencies	0.47	*
Total number of sources of revenue/funding received over the past 12 months	0.40	**
Among organizations that had never applied for or received federal funding at the time of the baseline survey, the number that had received federal funding at the time of the follow-up survey	0.45	*

p-value < .05; ** p-value < .01

Exhibit 6:

Areas of Greatest Capacity Improvement—Continuum of Change.

Level of focus on:	At a minimum, knows it should work on this, but lacks the time or resources		At a minimum, has developed plans to work on this, but has not implemented them yet		At a minimum, has implemented steps to address focus area
Organizational Development
Recruiting, developing, and managing volunteers more effectively:			0.45	**
Developing a Board with ties to different constituencies:			0.48	**
Leadership and Staff Development
Providing staff with professional development and training to enhance skills in service delivery or skills in administration and management			0.47	**
Program Development
Increasing the number of clients served by the organization:			0.43	**
Increasing the number or scope of services offered to clients:			0.47	**
Level of focus on incorporating a new approach to services to improve quality/ effectiveness:			0.49	**
Strengthening the organization’s ability to evaluate its overall effectiveness:			0.55	**	0.41	**
Developing a way to collect more information about clients, including the number and characteristics of clients as well as how they are helped by the programs:	0.42	**
Revenue Development
Identifying and pursuing new sources of in-kind donations					0.56	**
Developing a fund-development plan (including setting fundraising goals					0.41	*

p-value < .05; ** p-value < .01.

In reviewing the results, we noticed that the largest effect sizes were related either to training or to small incremental steps measured by the continuum questions. This observation led us to question whether these more modest capacity changes were driving the overall results of the study, rather than the potentially more meaningful concrete capacity outcomes. To answer this question, we performed a second round of joint tests, removing all training-related variables (e.g., “In the past 12 months, number of staff that participated in training related to: service delivery”) and separating the “concrete change” measures and the “continuum of focus” measures into two separate groups. The results of the additional analysis were consistent with the initial findings: the program group organizations experienced significantly larger increases in capacity whether measured on the continuum of focus or in terms of actual capacity outcomes (Exhibit 3).

Subgroup Differences

To determine whether the services work better for some types of NPOs than for others, outcomes for five subgroups were analyzed, as measured at baseline: (a) organization type: faith-based or secular organization; (b) age of the organization; (c) size (based on annual expenditures); (d) pay status of the executive director at baseline; and (e) whether the intermediary had prior CCF grant experience.

Our results showed that there was not a consistent pattern of findings to suggest that one type of organization is more receptive to capacity-building assistance. In fact, very few differences in outcomes between faith-based and secular organizations across the five domains of organizational capacity were found. Similarly, there were no differences in the effect of the services based on organization age or prior experience of the intermediary. There were differences on some measures based on the size of the organization and those with paid and unpaid executive directors but no consistent pattern could be identified. These results suggest that the range and types of services provided through CCF intermediary grantees are relevant and useful to many different types of organizations.

Discussion

Nonprofit organizations play a critical role in addressing the needs of local communities and our nation as a whole. The federal government, through the CCF and other initiatives, has invested in building the organizational capacity of NPOs with the expectation that this investment will result in improvements in organizational structure and internal operation, more effective and efficient service delivery and, ultimately, better outcomes for service recipients. The results of this random assignment evaluation show that initial expectations were met for the CCF program. To the limited extent that the evaluation was able to measure changes in internal operations and service delivery, positive changes were also observed. Expectations that capacity building would lead to more effective and efficient service delivery and to better outcomes for service recipients were not measured because they were beyond the evaluation’s scope and design.

In the following section we discuss how the CCF results fit into the broader context of the nonprofit capacity-building field, noting areas where additional research might improve knowledge about nonprofit capacity building.

Implications for the Field

Because of its random assignment methods, this study represents a critical advance in the evidence base for the effectiveness of nonprofit capacity building. The results of this CCF evaluation support previous findings form other observational method studies. Like Glickman and Servon (2003), Sobeck and Agius (2007), and Leake et al. (2007), the CCF evaluation found that capacity building increased organizational capacity across the range of capacity areas measured, though not on all capacity indicators.

Having said that, it is important to note that previous studies relied on cross-section or prepost designs. Compared to these studies, this random assignment study provided strong evidence of a causal link between capacity-building activities and the observed changes in capacity. However because these findings cannot be generalized beyond the CCF program and the organizations studied, more rigorous studies are needed to determine whether capacity can be increased through a range of intentional capacity-building activities.⁷

These findings also raise additional questions for future research.

What is the Relative Effectiveness of Different Types of Capacity-Building Assistance?

While the CCF Impact Evaluation found positive impacts on organizational capacity, the study was not designed to provide insights about which types (e.g., training, individual technical assistance, and financial assistance) of capacity-building assistance or combinations of types contributed to the impacts. Because we did not randomly assign specific treatments we were unable to assess the impact of specific treatments in the random assignment framework. Future studies might adopt this more complicated design in order to help the field better understand whether all three types of assistance are critical to producing impacts or whether only one of them or two in combination produce impacts. Knowing this might help funders and others to develop streamlined capacity-building programs focusing on the methods of delivery that provide the greatest contribution to nonprofit capacity change. Similarly, understanding whether building capacity in a given content area (e.g., financial management) leads to broader increases in capacity might also increase the efficiency of capacity-building programs. Future research could address this question through studies that randomly assign NPOs to different models and/or areas of assistance.

What Are the Intermediate and Long-Term Effects of Capacity-Building Assistance on Organizational Capacity?

The Impact Evaluation was designed to assess short-term (within 15 months) outcomes anticipated as a result of capacity-building efforts. While a number of improvements were reported within the 15-month follow-up period, greater gains across more areas of capacity at later points in time are plausible though not guaranteed. Campbell (2011) found similar short-term increases in capacity as a result of California’s Community and Faith-based Initiative; however, few of the impacts remained at the 3-year follow-up. Future studies should include follow-up surveys and interviews with the same organizations 2 to 4 years after receiving services to improve our understanding of how organizational change occurs—for instance, the timing of different outcomes (short-term vs. long-term), whether gains achieved early are sustainable over time, and whether incremental steps measured through the continuum of change measures lead to concrete changes. Related to the timing of different outcomes, it would also be useful to identify which outcomes are achievable in the short term and which can only be expected to be achieved over longer periods of time. By better understanding the progression of capacity building, future researchers can more accurately draw conclusions as to whether organizations are following a trajectory to greater capacity changes later.

What Are the Effects of Capacity Building on Service Delivery and Client Outcomes?

The CCF Impact Evaluation was able to document improvements in short-term organizational capacity. It was not, however, able to measure whether these capacity changes resulted in improved client outcomes, the ultimate goal of investments in capacity building. To estimate impact on these ultimate outcomes, longer term random assignment studies of programs with shared goals would be needed. In practice, the type of program that might lend itself to this next phase of capacity-building evaluation would be a capacity-building effort with a large number of homogeneous NPOs with existing client outcome data. There would also need to be a sufficient lag between the intervention and the outcomes such that it was plausible that capacity building would lead to capacity changes, which would in turn lead to client outcomes (Gan & Klerman, 2011). Finally, future studies would benefit from building upon the literature related to measuring organizational effectiveness. Authors such as Bagnoli and Megali (2011), Sowa, Selden, and Sandfort (2004), Shea (2011) have wrestled with these measurement issues and proposed various theoretical and quantitative models for linking capacity and effectiveness. A rigorous evaluation could incorporate some combination of these theories and test their validity.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Administration for Children and Families, U.S. Department of Health and Human Services, under Contract No. 233-02-0088, Task Order No. 12, “Compassion Capital Fund Evaluation.” An extended version of this article is available from the authors upon request. This research does not necessarily represent the official positions of Abt Associates Inc. or its research sponsors.

Notes

Author Biographies

Amy Minzner is a senior associate with Abt Associates Inc. in Cambridge, Massachusetts. Her research addresses urban development, workforce policy, nonprofit capacity building, and community revitalization.

Jacob A. Klerman is a principal associate and Abt Fellow with Abt Associates in Cambridge, Massachusetts. His research expertise includes evaluation methodology, social welfare policy, and American labor markets.

Carrie E. Markovitz is a statistician at NORC at the University of Chicago who specializes in both qualitative and quantitative evaluations of youth and community programs. Dr. Markovitz has contributed to several evaluations of small faith-based and community organizations that work with disadvantaged youth, minority populations, and victims of crime.

Barbara Fink is a senior research associate at Branch Associates, Inc. Her research addresses a variety of topics relating to low-income families and communities, including studies of capacity building in the nonprofit sector.

References

Abt Associates, Inc. (2007). Findings from a retrospective survey of faith-based and community organizations (FBCOs) (ACF-OPRE Report). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.

Abt Associates, Inc., & Branch Associates, Inc. (2008). An assessment of the Compassion Capital Fund Demonstration Program: Outcome study final report (ACF-OPRE Report). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.

Abt Associates, Inc., & Branch Associates, Inc. (2010). Improving Capacity among nonprofits—Random assignment impact study of the Compassion Capital Fund Demonstration Program (ACF-OPRE Report). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.

Administration for Children and Families. (2009). Compassion capital fund fact sheet. Retrieved from http://www.acf.hhs.gov/programs/ocs/ccf/about_ccf/facts.html

Allard

(2008). Accessibility and stability of nonprofit service providers: Faith-based and community-based organizations in urban and rural America. In Joshi

Hawkins

Novey

(Eds.), Compendium of research papers presented at the faith-based and community initiatives conference on research, outcomes, and evaluation. Washington, DC. Retrieved from http://aspe.hhs.gov/fbci/comp08/

Allard

(2009a). Out of reach: Place, poverty, and the new American welfare state. New Haven, CT: Yale University Press.

Allard

(2009b, November). The overlooked role of faith-based organizations and secular nonprofits in the modern American welfare state. Paper presented at the Annual Meetings of the Association for Public Policy Analysis and Management, Washington, DC.

Bagnoli

Megali

(2011). Measuring performance in social enterprises. Nonprofit and Voluntary Sector Quarterly, 40, 149-165.

Blumenthal

(2003). Investing in capacity building: A guide to high-impact approaches. New York, NY: The Foundation Center.

10.

Brown

(2008). Improving organizational capacity of faith-based and community-based domestic violence service providers. In Joshi

Hawkins

Novey

11.

Brown

(2007). Board development practices and competent board members: Implication for performance. Nonprofit Management & Leadership, 17, 301-317.

12.

Campbell

(2011). Reconsidering the implementation strategy in faith-based policy initiatives. Nonprofit and Voluntary Sector Quarterly, 40, 130-148.

13.

Cnaan

Boddie

(2006). Setting the context. Journal of Religion & Spirituality in Social Work: Social Thought, 25, 5-18.

14.

Connolly

York

(2002). Evaluating capacity building efforts for nonprofit organizations. OD Practitioner, 34, 33-39.

15.

Cornforth

Simpson

(2003). The changing face of charity governance—The impact of organizational size. In Cornforth

(Ed.), The governance of public and non-profit organizations (pp. 187-206). New York, NY: Routledge.

16.

Dart

Bradshaw

Murray

Wolpin

(1996). Boards of directors in nonprofit organizations: Do they follow a life-cycle model? Nonprofit Management and Leadership, 6, 367-379.

17.

De Vita

C. J.

Fleming

Twombly

E. C.

(2001). Building nonprofit capacity: A framework for addressing the problem. In De Vita

C. J.

Fleming

(Eds.), Building capacity in nonprofit organizations (pp. 5-32). Washington, DC: Urban Institute.

18.

Diiulio

(2004). Getting faith-based programs right. Public Interest, 155, 75-88.

19.

Doherty

Mayer

(2003). Results of an inquiry into capacity building programs for nonprofit programs. Minneapolis, MN: Effective Communities.

20.

Eisinger

(2002). Organizational capacity and organizational effectiveness among street-level food assistance programs. Nonprofit and Voluntary Sector Quarterly, 31, 115-130.

21.

Elliott

G. L.

(2002). Capacity as a fundamental objective: Definition and measurement in the AmeriCorps*VISTA—Habitat for Humanity International Affiliate Partnerships. Washington, DC: Corporation for National & Community Service, National & Community Service Fellowship Program.

22.

Gan

Klerman

J.A.

(2011). Designing the next generation of capacity building evaluations. Cambridge, MA: Abt Associates.

23.

Glickman

N. J.

Servon

L. J.

(1998). More than bricks: Five components of community development corporation capacity. Housing Policy Debate, 9, 497-539.

24.

Glickman

N. J.

Servon

L. J.

(2003). By the numbers: Measuring community development corporations’ capacity. Journal of Planning Education and Research, 22, 240-256.

25.

Herman

R. D.

Renz

D. O.

(1998). Nonprofit organizational effectiveness: Contrasts between especially effective and less effective organizations. Nonprofit Management & Leadership, 9, 23-38.

26.

Jolley

Meissen

(2009). Capacity building with faith-based and community organizations. Paper presented at the biannual meeting of the Society for Community Research and Action, Montclair State University, Montclair, NJ. Retrieved from http://www.allacademic.com/meta/p301770_index.html

27.

Kapucu

Healy

Arslan

(2011). Survival of the fittest: Capacity building for small nonprofit organizations. Evaluation and Program Planning, 34, 236-245.

28.

Klein

Brown

Small

Tucker

Fischer

(2009). Evaluation of the rural domestic violence and child victimization grant program special initiative: Faith-based and community organization pilot program. Sudbury, MA: Advocates for Human Potential.

29.

Leake

Green

Marquez

Vanderburg

Guillaume

Gardner

V. A.

(2007). Evaluating the capacity of faith-based programs in Colorado. Research on Social Work Practice, 17, 216-228.

30.

Letts

C. W.

Ryan

W. P.

Grossman

(1999). High performance nonprofit organizations: Managing upstream for greater impact. New York, NY: Wiley.

31.

Leviton

L. C.

Herrera

Pepper

S. K.

Fishman

Racine

(2006). Faith in action: Capacity and sustainability of volunteer organizations. Evaluation and Program Planning, 29, 201-207.

32.

Light

P. C.

(2004). Sustaining nonprofit performance: The case for capacity building and the evidence to support it. Washington, DC: Brookings Institution Press.

33.

Linnell

(2003). Evaluation of capacity building: Lessons from the field. Washington, DC: Alliance for Nonprofit Management.

34.

Markovitz

Magged

(2008). HOPE II: Faith-based and community organization program evaluation study—Summary report. Cambridge, MA: Abt Associates Inc.

35.

McKinsey & Company. (2001). Effective capacity building in nonprofit organizations. Reston, VA: Venture Philanthropy Partners.

36.

Millese

Bies

(2007). Nonprofit capacity building: Who is doing what for whom and to what end? Journal of Nonprofit Management, 11, 1-10.

37.

Patrizi

P. A.

Gross

E. K.

Freedman

(2006). Strength in flexibility: Lessons from a cluster of capacity building grants in the juvenile justice field. Evaluation and Program Planning, 29, 162-170.

38.

Popescu

Dewan

(2009). Esperanza Trabajando gains, losses, lessons learned and steps forward: An evaluation report. International NGO Journal, 4, 403-420.

39.

Rochester

(2003). The role of boards in small voluntary organisations. In Cornforth

(Ed.), The governance of public and non-profit organizations (pp. 115-130). New York, NY: Routledge.

40.

Rodriguez

Herzog

(2003). Replacing passionate leaders. Paper presented at the U.S. Department of Housing and Urban Development Capacity-Building Conference, Milano Graduate School of Management and Urban Policy, New School University, New York, NY.

41.

Schochet

P. Z.

(2008). Technical methods report: Guidelines for multiple testing in impact evaluations. Washington, DC: Institute for Education Sciences.

42.

Schuh

Leviton

(2006). A framework to assess the development and capacity of non-profit agencies. Evaluation and Program Planning, 29, 171-179.

43.

Shea

(2011). Taking nonprofit intermediaries seriously: A middle-range theory for implementation research. Public Administration Review, 71, 57-66.

44.

Sherman

A. L.

Stanakis

(2002). Building fruitful collaboration between Florida’s faith communities and one stop career centers. Washington, DC: Hudson Institute.

45.

Smith

S. R.

Bartkowski

J. P.

Grettenberger

(2005). Comparative views on the role and effect of faith in social services. Albany: SUNY, Albany Rockefeller Institute of Government, The Roundtable on Religion and Social Welfare Policy.

46.

Sobeck

(2008). How cost-effective is capacity building in grassroots organizations? Administration in Social Work, 32, 49-68.

47.

Sobeck

Agius

(2007). Organizational capacity building: Addressing a research and practice gap. Evaluation and Program Planning, 30, 237-246.

48.

Sowa

Coleman

Sandfort

(2004). No longer unmeasurable? A multidimensional integrated model of nonprofit organizational effectiveness. Nonprofit and Voluntary Sector Quarterly, 33, 711-728.

49.

Williams

Imam

(2007). Systems concepts in evaluation: An expert reader. Point Reyes, CA: EdgePress/ American Evaluation Association.

50.

Wing

(2004). Assessing the effectiveness of capacity-building Initiatives: Seven issues for the field. Nonprofit and Voluntary Sector Quarterly, 33, 153-160.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.56 MB