Managerial Strategies to Influence Frontline Worker Understanding of Performance Measures in Nonprofit Child Welfare Agencies

Abstract

In response to demands of funders and interorganizational competition, nonprofit human service organizations have invested in performance measurement to demonstrate the effectiveness and efficiency of internal operations. Literature suggests that frontline workers’ involvement in performance measurement is critical in supporting organizational efforts to improve performance. Yet, we lack information on how nonprofit managers convey performance standards to frontline workers and promote their engagement in performance measurement. This study draws on data from the 2011 National Survey of Private Child and Family Serving Agencies to identify strategies nonprofit managers used to engage frontline workers in performance measurement. Findings indicate that less than half of managers reported that their workers had a strong or very strong understanding of the agency’s performance measures. Managerial communication and board involvement in performance measurement were associated with greater worker understanding of performance measures. These managerial approaches may be key factors in frontline understanding of performance measures.

Keywords

public and nonprofit sectors performance measurement managerial strategies frontline engagement

Introduction

Even before the Great Recession, state and county-level governmental entities responsible for administering and delivering human services (hereafter referred to as public HSOs) have been asked to adapt to dwindling funding while also facing increasing demands for accountability and higher performance (Graaf, Hengeveld-Bidmon, Carnochan, Radu, & Austin, 2016; Hatry, 2002). In response, public HSOs have increasingly contracted out services to the private nonprofit sector to reduce costs and increase service effectiveness (Collins-Camargo, McBeath, & Ensign, 2011; Wells, Jolles, Chuang, McBeath, & Collins-Camargo, 2014). Against this backdrop of higher accountability with fewer financial resources, private nonprofit agencies have invested in performance measurement systems to comply with contracting expectations and to demonstrate the effectiveness of their internal operations. Although the literature suggests that frontline worker understanding and engagement in program monitoring is critical for program improvement, we lack information on how managers convey performance measurement standards to their workers and promote their engagement (B. D. Smith, 2009).

Most of public HSOs’ contracts are steered toward nonprofits, as these agencies’ mission and commitment to social values are valued by funders and communities (Hasenfeld, 2010). These contractual relationships also reflect the historic evolution of the U.S. human service sector, which originated in charitable organizations receiving funding from public and private sources (Salamon, 1995; S. R. Smith & Grønbjerg, 2006). As a result of these evolving relationships, public HSOs’ accountability demands have led to increased requirements of nonprofit providers to report on programmatic outputs and results (Jos & Tompkins, 2004; Tuan, 2004). From a nonprofit organizational perspective, performance expectations should drive agencies to develop administrative infrastructures to support performance monitoring via the use of in-house data to track frontline and program practices (Vaughan, 2010). In fact, funders and policymakers often use performance measurement—as expressed through contract-based reporting requirements—as a formal governance tool to ensure that publicly funded programs are in compliance with expectations and demands (Austin, 2008).

From an institutional perspective, nonprofit provider engagement in performance measurement can confer legitimacy in the eyes of public HSOs and may provide nonprofits an edge in competitive markets (DiMaggio & Powell, 1983). Legitimacy gains among nonprofit providers are particularly relevant as these organizations are driven by a mission of social responsibility, rely heavily on voluntary work and donations, and may be challenged to calculate the societal benefits of their programs (Hasenfeld & Garrow, 2012). Performance measurement information can also be used by nonprofit providers to demonstrate responsiveness to community demands (Alexander, Brudney, & Yang, 2010; Döring, Downe, & Martin, 2015).

Performance Measurement in the Context of the Child Welfare System

The child welfare system is a highly institutionalized field in which performance measurement requirements are required by federal, state, and county statutes (Carnochan, Samples, Myers, & Austin, 2014; Lindsey & Shlonsky, 2008). Beginning in 2001, the federal government established the Child and Family Services Review (CFSR) process to assess the performance of state child welfare agencies regarding federal performance standards and capacity to provide an array of programs and organizational activities such as quality assurance systems and workers training (Government Accountability Office, 2004). The CFSR process results in the development of state-specific dashboards, which are reviewed federally to determine the extent to which states are in compliance with federal performance targets. Within these performance measurement standards, the public child welfare system has relied historically on contracted nonprofit agencies to deliver family preservation, family support, and in-home child and family services and to provide foster care placements and residential treatment services to maltreated children (Frumkin & Andre-Clark, 2000).

Performance measurement requirements are promoted by federal, state, and county governments (Carnochan et al., 2014; Lindsey & Shlonsky, 2008). In response to these institutional demands, these contracted nonprofit child welfare agencies have invested in performance measurement to track the effectiveness and efficiency of their internal operations, in an effort to compete for public contracts and private donations (S. R. Smith, 2010). Although performance measurement has become an area of managerial emphasis (Weigensberg, 2009; Meezan & McBeath, 2011), nonprofit child welfare agencies may differ in their capacity to invest in performance measurement systems.

Therefore, in states where child welfare services are delivered by nonprofit child welfare agencies through contracts with public HSOs, these nonprofit agencies are required to contribute to program performance monitoring. In response to these institutional demands and to compete for public contracts and private donations, nonprofit child welfare agencies have invested in performance measurement to demonstrate the effectiveness and efficiency of their internal operations (S. R. Smith, 2010). For these reasons, performance measurement has become an area of managerial emphasis for nonprofit child welfare agencies (Weigensberg, 2009; Meezan & McBeath, 2011). However, organizational investments in performance measurement are unlikely to yield benefits without the engagement of frontline workers because they are responsible for entering information on program indicators to demonstrate organizational accountability (Lipsky, 2010). Yet, workers are unlikely to accurately and consistently report performance information if they do not understand performance measures, or believe they are not meaningful (Pulakos, 2009). In short, managing frontline workers’ engagement in performance measurement is critical to success.

We can better understand how nonprofit child welfare agencies meet performance measurement standards by examining strategies employed by their managers to engage frontline workers in performance measurement. The purpose of this article is to identify managerial strategies associated with frontline workers’ understanding of performance measurement among nonprofit child welfare agencies. Identifying strategies associated with frontline understanding of and engagement in performance measurement should inform nonprofit managerial practices and foster adaptation to external performance standards.

Factors Associated With Frontline Engagement in Performance Measurement in Nonprofit Child Welfare Agencies

The following sections present a review of the management literatures in HSOs and nonprofit agencies and pertaining specifically to nonprofit child welfare agencies. The reviewed literature suggests that promoting frontline engagement in performance measurement depends on various organizational characteristics and resources as well as efforts of managers.

Organizational Characteristics

Management research has explored organizational characteristics for promoting an evidence-informed approach to human service delivery. Nonprofit agency success in implementing performance measurement systems may depend on the availability of multiple and overlapping factors. These factors include the various supports and resources that public HSOs provide to nonprofits to influence the larger organizational context. These public-based supports ultimately play a role in frontline engagement with and the success of performance measurement efforts. As the nature of the relationship has shifted in recent years to higher public HSO reliance on nonprofits to carry out services, public HSOs have retained a role in guiding and evaluating the nonprofit agency performance (Collins-Camargo, Hollie, & McBeath, 2014). Incentives provided by public HSOs, such as technical assistance and performance-based contracting, may ultimately increase attention to the need for nonprofit infrastructure and capacity needed to support performance measurement. In fact, it is often the case that public HSOs assist nonprofits in understanding and implementing new fiscal requirements and information management systems (Austin, 2003).

Public HSOs also guide and evaluate nonprofits through the use of performance-based contracting practices that reimburse nonprofit agencies for services that meet contractually specified objectives (Martin & Kettner, 2010). Performance-based contracting, although supporting nonprofit agency entrance to the service market, increases the risk that these providers bear in delivering services (Collins-Camargo, McBeath, & Ensign, 2011). As a result, nonprofit program performance tracking has emerged as an essential managerial task (Kaplan, 2001). Performance measures are intended to focus frontline and managerial attention on the tasks involved in meeting contractual expectations with public HSOs.

Organizational factors related to nonprofit agency capacity for performance measurement include the development and implementation of concrete performance measurement policies (Thomson, 2010), and organizational resources such as funding, administrative workers, and physical infrastructure to support measurement efforts (Berman, 2002; Campbell, 2002; Sawhill & Williamson, 2001). Research has highlighted the importance of access to information technology to accurately measure performance (Hendricks, Plantz, & Pritchard, 2008). Successful engagement in performance measurement may be in great part achieved by nonprofit agencies that have built the resources to support high performance (Letts, Ryan, & Grossman, 1999). In situations where nonprofits have insufficient organizational and professional capacity, they may not be able to implement performance measurement systems effectively. Letts and colleagues (1999) have argued that nonprofit adaptation to performance expectations may depend on the ability to build organizational capacity to track program outcomes. For example, nonprofit agencies may invest in Human Resource Management (HRM) systems designed to better align worker performance with organizational objectives or in research/evaluation units and/or data reporting platforms like performance dashboards (Barratt, 2003; Packard, 2009).

This increased organizational capacity to engage in research and evaluation efforts can support nonprofits engagement in data-driven decision making and increased frontline involvement in performance improvement initiatives (Hoole & Patterson, 2008; Torres & Preskill, 2001). Executive leaders are in a unique position to monitor and balance external demands with internal resources to inform and build the organization’s capacity to engage in performance measurement (Alaimo, 2008).

In addition, performance measurement efforts at the executive leadership level do not often translate into frontline practice given the wide range of professional credentials and administrative tasks among caseworkers, a complex child welfare data system and high personnel turnover (Lee, Bright, & Berlin, 2013; Parrish & Rubin, 2012). Despite training and technical assistance, a lack of managerial emphasis on program improvement may inhibit use of existing data systems to promote performance (Carrilio, Packard, & Clapp, 2004). Supervisory support and an outcome-oriented climate may support the use of data to promote outcome achievement when working with families with complex needs and living in crisis-ridden environments such as is typical in the child welfare system (Collins-Camargo & Royse, 2010).

In nonprofit child welfare agencies, managers thus rely on frontline workers’ understanding of their agency’s performance measurement indicators to accurately gather this data. And in order for agency-wide performance measurement data to be viewed as reliable, frontline workers should be actively engaged in these activities and knowledgeable about the efforts to which they are contributing. Gruman and Saks (2011) argued that a key contributing factor to effective performance measurement is to “focus on fostering employee engagement as a driver of increased performance” (p. 1).

However, research on the implementation of performance measurement systems among nonprofit child welfare agencies suggests that frontline workers lack understanding of performance measures (Campbell & Lambright, 2017). One study of child welfare frontline workers in one state found that 20% did not know the outcomes upon which agency performance is measured, and only one third noted they had access to the data they needed to understand the impact of their work on clients. Even fewer thought data collected by the child welfare agency was adequate for the purpose (Collins-Camargo, Sullivan, & Murphy, 2011). Carnochan and colleagues (2014) found that incorporating user perspectives into performance measurement systems and providing adequate access to data supported worker understanding and implementation of performance measurement. Yet, Devaney (2004) noted that agencies often measure what is available rather than what is perceived by frontline workers to be useful. Thus, critical to the engagement of frontline workers in performance measurement is their belief that performance indicators are valid and useful. Lower engagement in the process of inputting performance data may, in turn, limit the overall robustness of performance measurement.

In sum, frontline engagement in performance measurement may depend on a comprehensive array of support strategies (Manuel, Mullen, Fang, Bellamy, & Bledsoe, 2009; Scurlock-Evans & Upton, 2015). We propose that, to gain a deeper understanding of performance measurement engagement, we must distinguish between organizational factors and strategies used by nonprofit managers to promote frontline worker engagement in performance measurement. Unfortunately, the question of how nonprofit managers incorporate performance measurement into organizational practices has received less attention (Lynch-Cerullo & Cooney, 2011). The following section reviews literature concerning managerial efforts to promote worker engagement in performance measurement systems.

Managerial Efforts to Engage Frontline Workers in Performance Measurement

The successful use of performance measurement relies on coordinated managerial and frontline action: Managerial efforts support the development of well-functioning service programs and can improve organizational performance; yet their implementation and in practice occurs at the level of the frontline worker. Situated at the nexus of these organizational and frontline levels is the nonprofit manager, who is often tasked with the development of the performance measurement system, its implementation, and the analysis and reporting of performance information.

Managers play a key role in influencing frontline worker appraisal of and engagement in performance measurement (Döring et al., 2015) such as top and middle management support to frontline practice, time, training, and access to evidence itself (Birken, Lee, Weiner, Chin, & Schaefer, 2013; Packard, 2009; Scurlock-Evans & Upton, 2015). The existing literature has identified strategies used to shape social interactions between leaders (e.g., administration and board of directors) and workers to increase frontline knowledge of performance measurement and engagement in it.

In particular, the integration of policy expectations into direct service provision is essential for increased program performance. One way for managers to link the policy and frontline levels is via strategies that promote knowledge gain, social interactions, and governance tools at the frontline level as well as data gathering systems that are perceived to be relevant by workers (Austin, 2008; Mone & London, 2014). These strategies also allow nonprofits to link “knowledge sharing and organizational goals and outcomes” (Austin, 2008, p. 584). First, strategies promoting social interactions (e.g., regular discussions with supervisors) through social support and coaching (Gruman & Saks, 2011) shape workers’ understanding of performance indicators and enforce organizational expectations (Sandfort, 2000). Second, strategies promoting knowledge gain may engage workers through managerial use of data systems to assess worker effectiveness (Carnochan et al., 2014) as well as worker training to support and sustain evidence-based practices (Johnson & Austin, 2006). Worker training is essential in this context as performance measurement systems are often sophisticated and require specific information and skillset.

Finally, worker engagement may be reinforced through the consistent involvement of formal organizational authorities such as administrators and agency boards (LeRoux & Wright, 2010). The role of boards is underscored in performance measurement given their influence in policy making and oversight in nonprofit organizations with broad influence in decision making (LeRoux & Wright, 2010). These strategies are not necessarily as mutually exclusive; managers may employ more than one strategy to promote a comprehensive approach to worker engagement in performance measurement.

Overall, the reviewed literature focuses on the organizational factors that promote the use of performance measurement and their impact on expected outcomes. The literature also shows a strong consensus that individual worker understanding and engagement in performance standards process are likely to increase buy-in and adherence to external demands. Yet, there is a lack of empirical research on the influence of managerial strategies on frontline worker understanding of performance measurement within nonprofits. Nor has there been research evaluating the potential benefits of individual versus cumulative strategies, in which managers adopt multiple strategies to promote workers performance measurement engagement (B. D. Smith, 2009). This lack of information limits the development of evidence-informed managerial approaches to performance measurement implementation.

The Current Study

The central research question we address is as follows:

Research Question 1: Within nonprofit private child welfare agencies, what strategies do managers use to increase the understanding of their frontline workers of required performance measures?

We examined the relative contribution of different managerial strategies—including the development of policies, leadership involvement, and training—to support frontline worker understanding of agency performance measures, and the relationship of managerial strategy selection to related factors such as confidence in the agency outcome measurement system and the agency’s participation in performance-based contracts. Empirically, we analyzed survey data gathered from the first multistate study of private nonprofit child welfare agencies.

Our logic for analysis is as follows: We first identify the prevalence of specific strategies that agency managers reported employing to promote agency-wide engagement in performance measurement. We then report on multivariate regression analyses in which we evaluate the influence of these managerial strategies on manager-perceived levels of understanding of frontline workers of the performance measures they are required to use. Our statistical models operationalize this relationship between managerial strategies and frontline worker understanding of performance measures in two ways: via an individual effect, in which the influence of each managerial strategy is estimated; and cumulatively, in which we test for the potentially cumulative influence of different managerial strategies in use.

Method

Sample and Data Collection

The current study draws upon quantitative data from the 2011 National Survey of Private Child and Family Serving Agencies (NSPCFSA). NSPCFSA is the first national survey of private child and family serving agencies across state child welfare systems. The survey was developed by the National Quality Improvement Center on the Privatization of Child Welfare Services in collaboration with the two largest U.S. membership associations for private child welfare agencies, the Child Welfare League of America (CWLA) and the Alliance for Strong Families and Communities (the Alliance). The NSPCFSA sampling frame reflected a purposive, nonprobabilistic sampling model that included all nonprofit and for-profit child and family serving agencies that were members of CWLA, the Alliance, or a state-based private child welfare agency membership association involved in the National Organization of State Associations for Children.

The survey was administered online to preserve the anonymity and confidentiality of agencies and their directors. Invitations to participate in the survey were sent through listservs managed by these national and state membership associations. Questions were asked in seven domains: interorganizational collaboration and competition, internal and external pressures on organizational maintenance, respondent characteristics, organizational demographics, funding levels, service array, and performance measurement and organizational maintenance.

A total of 446¹ agency directors from private child welfare agencies in 38 states participated in the survey over May 1-June 30, 2011. The majority of NSPCFSA agencies (64%) were located in large states with significant private sector involvement in child welfare service delivery and large consumer populations (e.g., California, Texas). Taking into consideration overlap in agency memberships (e.g., in both Alliance and CWLA), we had an average estimated response rate of 59%. More detailed information about the NSPCFSA survey is provided elsewhere (Chuang, Collins-Camargo, McBeath, Wells, & Bunger, 2014).

Given that most public HSO contracts are with nonprofit agencies (S. R. Smith & Grønbjerg, 2006), the current study sample was restricted to only those private child welfare agencies reporting a nonprofit status at the time of the survey. The final analytical sample after accounting for sample restrictions and for cases with full information on the dependent variable was 332 nonprofit child welfare agencies.

Measures

Frontline worker understanding of performance measurement (dependent variable)

Agency administrators were asked to rate their perceived level of worker understanding via the following question: “Do frontline workers in your agency understand the performance measures upon which your agency’s performance is evaluated?” Likert-based response categories for each survey question included 1 = no understanding, 2 = very little understanding, 3 = some understanding, 4 = strong understanding, and 5 = very strong understanding. This question assessed managers’ perceptions of their frontline workers’ understanding of agency performance measurement.

Organizational characteristics associated with frontline worker understanding of performance measurement

Public influence on agency operations

Managers were asked to what extent their relationship with the public child welfare agency affected their ability to meet key agency performance outcomes. Response options ranged from 1 = it’s made it much worse to 5 = it’s made it much better. Due to small number of observations on the lower categories, a binary variable was constructed where answers of “It’s made it somewhat better” or “It’s made it much better” were coded 1 to indicate a positive public agency influence on the nonprofit agency’s ability to meet key agency performance outcomes. The remaining answer options were coded 0.

Agency has a performance-based contract

Managers were asked to report (yes/no) whether each of their current child welfare services were being delivered through performance contracts. These contracts were defined in the survey question as those in which agency revenue was dependent on the achievement of agreed upon performance milestones (Collins-Camargo, McBeath, & Ensign, 2011). A binary variable was constructed and coded as 1 if the agency was delivering any of its services through a performance-based contract.

Infrastructural capacity for performance measurement

Managers were asked to report whether they had the following resources (yes/no): an established program evaluation unit, a quality assurance/quality improvement unit, and an information technology department. These three resources serve a complementary function within a learning organizational environment (Hoole & Patterson, 2008; King & Volkov, 2005), and all three may foster performance measurement effort at an organizational level; thus, infrastructural capacity was operationalized as a binary variable was coded as 1 if the agency reported having all three of these resources in place.

Managerial efforts to engage frontline workers in performance measurement

Managerial strategies

Respondents were asked to report whether they or other agency administrators used any of the following strategies to articulate performance measurement outcomes to frontline workers: (a) written policy, (b) formal training, (c) discussion in workers’ meetings, (d) formal communication by agency, (e) frontline workers discussion with supervisor, and/or (f) review or discussion by agency board of directors. Respondents were able to select more than one strategy.

Managerial Strategy Index

An additional variable was constructed to indicate the total number of the aforementioned strategies used by each agency to foster performance measure uptake among employees. This was a count variable ranging from zero strategies (referent) to five strategies. This variable was included as a way to capture the comprehensiveness of the agency approach to frontline worker uptake of performance measures.

Quality of performance measurement data collection

Respondents were asked to report on the extent to which their agency was gathering adequate data to assess performance regarding its services to children and families. Answers were reported in the following scale: 1 = not at all, 2 = a little, 3 = somewhat, 4 = a lot, 5 = very much. Due to the low frequency on the three lowest categories for this survey question, this measure was recoded into a binary variable where 1 = the director reported their agency collects adequate performance measure data “a lot” or “very much” and 0 = all the other categories. This binary variable allowed us to capture more versus less positive perceptions of the relative quality of agency performance measure data collection efforts.

Covariates

Additional factors potentially associated with managerial action to influence worker engagement in performance measurement were included. We included a system integration measure through a binary variable denoting whether the agency was part of a larger network or organization, in which agencies shared a common name, affiliation, and organizational history but in all other regards operated independently (Wells et al., 2014). The number of employees in the most recent fiscal year was included as a proxy for organizational size. This variable ranged from fewer than 10 Full-time Employees or FTEs (i.e., Category 1) to over 1,000 FTEs (i.e., Category 9). There is evidence that resource strain limiting nonprofit growth negatively influences accountability efforts (Bargerstock, 2000). An additional continuous variable pertaining to organizational age was included as agency experience in service delivery has been associated with the ability to engage in performance measurement (Carman, 2009).

Analytic Methods

Descriptive and bivariate analyses were used to describe the nonprofit organizational sample. Proposed associations between variables were examined through logistic regression models. For the outcome variable, the number of responses in the lowest (n = 22; 6.4%) and highest (n = 36; 10.4%) categories were very low. We merged response categories and recoded this variable from a 5-point to a binary variable (set = 1 if an agency director reported a perceived “very strong” or “strong” understanding of their frontline workforce to the survey question, and 0 = if the respondent reported any one of the other categories). For the individual evaluation of managerial strategies, we did not include those strategies that were highly correlated (i.e., use of written policy and formal training strategies had a correlation above 70%) in the regression models. With such highly correlated measures, it not possible to assess the individual role of these strategies. In a separate regression model, we included the count variable pertaining to the overall number of strategies to account for high correlation among these variables and to explore the association between cumulative managerial approaches and frontline worker understanding of performance measures.

Results

Descriptive Statistics

As shown in Table 1, less than half (48%) of agency managers reported that their workers held a strong or very strong understanding of the agency’s performance measures. With respect to organizational characteristics, 30% of managers noted having a strong influence of the public agency sector on their agency operations, a quarter (25%) of agencies had one or more performance-based contracts and just over a third (38%) reported adequate tools for performance management. Additional descriptive analyses included in the analytical sample showed that on average respondents had been in their current position for 10 years, and had an average of 24 years working in child welfare agencies and less than half had a non-MSW degree (40%).

Table 1.

Descriptive Characteristics for Nonprofit Agencies.

	M/%	SD	Minimum	Maximum
Dependent variable
Frontline worker understanding of performance measures	48%		0	1
Organizational characteristics
Public influence on agency operations	30%		0	1
Agency has a performance-based contract	25%		0	1
Capacity for performance measurement and management	38%		0	1
Managerial efforts to engage frontline in performance measurement
Managerial strategies			0	1
Discussion in workers meetings (item used to filter the sample)	100%		0	1
Frontline workers discussions with supervisor	82%		0	1
Formal communication by agency leadership	75%		0	1
Formal training	72%		0	1
Written policy	66%		0	1
Review or discussion by agency board of directors	32%		0	1
Managerial Strategy Index (i.e., number of managerial strategies used)	3.34	1.51	0	5
Quality of performance measurement data collection	66%		0	1
Covariates
Agency is part of a larger network organization	12%		0	1
Number of Full-time Employees in the most recent fiscal year	4.58	1.77	1	9
Agency age	24.66	10.64	1	47

Note. The sample size for variables ranged from 332 to 346.

Regarding managerial strategies for increasing frontline engagement in performance measurement, besides discussions in workers’ meetings, most agencies implemented strategies related to frontline workers’ interactions with supervisors (82%) and formal communications by the agency leadership (75%). The least frequently reported strategy was review or discussion of performance measures by the board of directors (32%). On average, agencies used between three and four strategies, out of five, to convey performance measures to workers (M = 3.34; SD = 1.51). Multiple-response distribution analyses showed that the top two combination strategies used by agencies, besides discussions in workers’ meetings, related to (a) using formal communication by agency leadership (88% of the cases) and (b) frontline workers’ discussions with supervisors (81% of the cases).

Multivariate Results

As shown in Table 2, frontline worker understanding of performance measurement was positively associated with organizational factors related to performance-based contracting (odds ratio [OR] = 1.89; 95% confidence interval [CI] = [1.02, 3.48]) and with several managerial strategies. That is, besides discussions in workers’ meetings, formal communication by agency leadership (OR = 2.51; 95% CI = [1.25, 5.01]) and review or discussion by agency board of directors (OR = 2.15, 95% CI = [1.26, 3.68]) increased the odds of frontline worker understanding of agency performance measures. In addition, agency size (OR = 0.73; 95% CI = [0.61, 0.87]) was negatively associated, and agency age was positively associated (OR = 1.03; 95% CI = [1.01, 1.06]) with our dependent variable.

Table 2.

Multivariate Logistic Regression Models Predicting Frontline Performance Measure Understanding—Examination of Individual Nonprofit Managerial Strategies (N = 332).

	OR	SE	P > \|t\|	95% CI
Organizational characteristics
Public influence on agency operations	1.09	0.30		[0.63, 1.86]
Agency has a performance-based contract	1.89	0.59	*	[1.02, 3.48]
Capacity for performance measurement and management	1.41	0.44		[0.77, 2.60]
Managerial efforts to engage frontline in performance measurement
Managerial strategies (besides discussions in workers meetings)
Frontline workers discussions with supervisor	2.07	0.82		[0.96, 4.48
Formal communication by agency leadership	2.51	0.89	**	[1.25, 5.01
Review or discussion by agency board of directors	2.15	0.59	**	[1.26, 3.68
Quality of performance measure data collection	3.18	0.96	***	[1.76, 5.76
Covariates
Agency is part of a larger network or organization	2.18	0.93		[0.95, 5.02]
Number of Full-time Employees in the most recent fiscal year	0.73	0.66	**	[0.61, 0.87]
Agency age	1.03	0.01	**	[1.01, 1.06]

Note. CI = confidence interval.

p < .05. **p < .01. ***p < .001.

We also explored the influence of cumulative strategies to enhancing frontline worker understand of performance measurement. Table 3 presents the results of these analyses. The use of a greater number of managerial strategies was positively associated with perceived worker understanding of performance measurement (OR = 1.51, 95% CI = [1.27, 1.83]). Additional post hoc analyses suggested that the use of at least four of seven strategies was significantly associated with the outcome measure (p < .05).

Table 3.

Multivariate Logistic Regression Models Predicting Frontline Performance Measure Understanding—Testing of the Managerial Strategy Index (N = 332).

	OR	SE	P > \|t\|	95% CI
Organizational characteristics
Public influence on agency operations	1.80	0.55		[1.00, 3.28]
Agency has a performance-based contract	1.44	0.45		[0.79, 2.65]
Capacity for performance measurement and management	1.06	0.28		[0.64, 1.81]
Managerial efforts to engage frontline in performance measurement
Managerial Strategy Index	1.51	0.14	***	[1.27, 1.83]
Quality of performance measure data collection	3.61	1.04	***	[1.83, 5.77]
Covariates
Agency is part of a larger network or organization	2.16	0.88		[0.97, 4.80]
Number of Full-time Employees in the most recent fiscal year	0.75	0.07	**	[0.63, 0.89]
Agency age	1.03	0.01	**	[1.01, 1.06]

Note. CI = confidence interval.

p < .05. **p < .01. ***p < .001.

Across all regression models and while maintaining other factors constant, results showed that higher perceived quality of performance measurement data collection efforts increased the odds of frontline worker understanding as reported by administrators.

Sensitivity Analyses

A linktest was implemented to evaluate the overall robustness of model specification by computing a new regression using as predictors the estimated values of the original model and their squares. If the quadratic term is significant in this new regression, there is evidence of model misspecification (Cameron & Trivedi, 2009). In the current study, this quadratic term was not significant, suggesting no major specification errors in the multivariate models. Survey item nonresponse did not exceed 3% for any given variable and all analyses were conducted using Stata 13.0 (StataCorp, 2011). Sensitivity analyses (not shown but available from authors upon request) involving regression models testing the functional form of the infrastructural capacity variable as a categorical variable measuring whether the organization reported zero, one, two, or all three units (i.e., program evaluation unit, quality assurance, and IT department), and also adding two additional variables measuring the directors’ number of work years on his or her current position and educational degree did not change the main results. These results are also consistent with the results provided by the specification diagnostic test (i.e., linktest) that did not detect model specification problems (p < .05) as reported in the analysis section. Thus, only the former regression models are reported here.

Discussion

Over the past three decades, policymakers and funders have viewed performance measurement as a central tool for ensuring accountability for publicly funded programs delivered by nonprofit child welfare agencies. The child welfare field in particular has seen increased emphasis on accountability (Lindsey & Shlonsky, 2008), in part driven by state and federal regulations (Carnochan et al., 2014), influencing requirements on private agencies (Austin, 2008) and in turn promoting nonprofit investment in performance measurement (S. R. Smith, 2010). Success in implementing performance measurement can be influenced by nonprofit managers’ access to organizational resources to support their efforts to ensure that their frontline workers understand the value of performance measurement systems. The literature to date has emphasized the challenges facing nonprofit child welfare service agencies in developing effective performance measurement systems, the fiscal and administrative infrastructural resources needed to do so, and the importance of obtaining stakeholders buy-in throughout this process. However, empirical research has not clarified how nonprofit managers achieve this, which specific strategies nonprofit managers use to navigate multiple challenges, or whether the use of these strategies enhances the engagement of frontline workers with performance measurement.

The current study provides the first empirical estimates on these questions, and contributes to the nonprofit performance measurement literature by using a unique national study of nonprofit child welfare agencies that captures the perspective of nonprofit executives concerning the strategies that they used in an effort to foster performance management understanding among frontline workers. In addition, the response rate for this study’s survey is quite high as compared with other large-scale studies of child welfare agencies and even the lower bound of this range is considered robust for organizational surveys (Baruch & Holtom, 2008). The literature has emphasized the importance of workers understanding and appreciating the value of performance indicators (Pulakos, 2009). Surprisingly, we found that less than half of managers perceived their frontline workers as understanding their agency’s performance measurement and found that managers often reported using strategies related to facilitating face-to-face interactions with frontline workers to convey performance measurement expectations through workers’ meetings, discussions with supervisors, and communication by agency leadership. The managerial strategy used least by managers related to the involvement of the board of directors in the process. On average, nonprofit managers used over three performance measurement strategies, out of seven possible, suggesting that managers were using a comprehensive approach involving multiple strategies to convey the importance of performance measurement and promote its use at the frontline worker level.

These findings are consistent with prior research in nonprofit child welfare agencies, particularly on frontline worker implementation of evidence-based practices. For example, Aarons and Palinkas (2007) reported a lack of understanding of and organizational support for performance measurement among child welfare workers. Low understanding of performance indicators, and limited engagement in performance measurement, may be exacerbated by high workers turnover and a disconnect between frontline workers and executive leadership (Lee et al., 2013; Parrish & Rubin, 2012). The use of multiple communication strategies reported in the current study may be in part an effort to counter these characteristics among nonprofit child welfare agencies.

Regression analyses determined that managers in agencies that had one or more performance-based contracts, in older agencies, and in smaller agencies perceived higher levels of frontline understanding of required performance measures. The first of these results accords with the nonprofit child welfare service agencies literature, in that agencies in high-performance environments may be more likely to rely on performance monitoring to support the attainment of performance milestones, and thus to convey the importance of performance measurement to frontline workers. The latter result may suggest that larger nonprofits may struggle due to greater overall organizational complexity and bureaucratization to engage frontline workers in the process of performance measurement. Also, frontline workers in smaller nonprofits might have a stronger understanding of performance measures as agency leaders have fewer people to engage, communicate with, and train. These results are consistent with at least one statewide study in which stronger understanding and engagement in performance measurement emerged among private nonprofit agencies than the much larger public child welfare agency (Collins-Camargo, McBeath, & Ensign, 2011). It seems, however, that infrastructural capacity for performance measurement (in regard to having a greater number of quality assurance and evaluation personnel) did not predict frontline worker understanding of performance measurement. This finding sits in contrast to previous research (Letts et al., 1999), and suggest a need for further research into the link between nonprofit administrative capacity and frontline worker engagement in performance measurement.

Importantly, two specific strategies were associated with frontline workers’ understanding of performance measurement (i.e., formal communication by leadership to frontline workers and involving the board of directors in review and dialogue concerning performance measurement). These results are in support of research suggesting an organizational culture promoting performance measurement and learning is associated with frontline worker engagement in performance measurement (O’Toole, 2002; Torres & Preskill, 2001). These findings also imply that performance measurement is facilitated when agency leadership discusses (separate from their conversations with frontline workers) how performance measurement may facilitate organizational performance.

This finding highlights the administrative and board of director roles for their importance in facilitating performance measurement engagement (Alaimo, 2008; Newcomer, 2008). That is, executives who are more actively involved in shaping how their agencies respond to funder requirements for performance reporting may also be those who are invested in developing informal organizational cultures and formal internal governance systems in which performance monitoring is taken seriously. This may be particularly important in crisis-driven nonprofit child welfare agencies in which frontline workers’ involvement in performance measurement may interfere with high caseloads and the demands associated with serving families with complex needs. Lack of an organizational culture supporting performance measurement efforts has been suggested to impede frontline worker engagement (Carrilio et al., 2004). We also found that the overall number of strategies used by managers was positively associated with this outcome. This finding implies that the use of more than one strategy (i.e., a comprehensive strategic approach) may be more successful in influencing frontline worker understanding of organizational performance measurement within nonprofit child welfare agencies.

Study results also suggest that frontline worker understanding of performance measurement was associated with the degree to which managers believed their agency was gathering adequate information to assess service effectiveness. This result supports the findings of other studies that have found that nonprofit frontline workers may struggle to engage in performance measurement if they view performance measures as irrelevant, unhelpful for service and program improvement, and imposed upon them by funders and/or other authorities (Kukla-Acevedo, Hodges, Ferreira, & ôcMazza, 2008). This evidence also implies that even in situations where administrators consider their performance measurement process to be well-designed, they may struggle to convey the importance of engagement in performance measurement to their frontline workers. This in part may explain the low percentage of frontline worker understanding of performance measurement as reported by managers.

In this study, a substantial proportion of managers indicated their frontline workers did not understand the measures upon which performance would be evaluated. This finding raises additional questions such as, Is this because managers believe they have not been successful in communicating them? Or because managers don’t believe it is appropriate altogether? In this regard, future research may elucidate what motivates managers to rely on agency leadership, such as boards, to promote worker performance measurement uptake, and the extent to which managers believe it is important for workers to be fully aware of their agency performance measurement process.

These findings should be understood in relation to a number of methodological limitations. First, our measures rely on self-reports from nonprofit executives, whose perspectives on frontline workers’ understanding of performance measures may have been limited or biased. Corroborating data from frontline workers were not available from the overall NSPCFSA study. In addition, our measure of number of FTEs counts is based on ordinal categories to reduce burden among survey respondents as recommended during the instrument piloting. Yet, we recognize that ordinal categories are not as precise as count variables. Second, our main predictors concerned the presence or absence of different managerial strategies; more in-depth measures pertaining to the specific manner in which each strategy was used to support performance measurement among frontline workers were not available. Third, this is a cross-sectional survey and it is not possible to infer causality. There was, as a result, the potential for endogeneity given lack of data on whether the use of specific managerial strategies was later associated with changes in frontline worker understanding of performance measurement, or the potential influence of performance measurement characteristics (e.g., task complexity, time burden for frontline workers). Fourth, this national survey of nonprofit child welfare agencies may not be generalizable to nonprofit organizations in other human service fields. Last, we acknowledge that our sample is probably overrepresented by high-performing agencies (that might be most likely to participate in large state and/or national membership organizations). We were not able to evaluate potential sample selection bias in the NSPCFSA survey because identifying contact information was not collected.

Despite these limitations, the current study provides the first national estimates of nonprofit managerial strategies to promote frontline workers’ engagement in performance measurement, a topic of strong and growing interest in the child welfare literature. Our study also benefits from a large dataset of nonprofit child welfare agencies, which allowed us to examine the influence of organizational and managerial factors (shown in prior research to facilitate performance measurement). In addition, the survey’s multistate research design allowed for the examination of proposed research question across jurisdictions with potentially different performance measurement requirements and differences in underlying client populations and thus nonprofit service delivery.

In relation to these contributions to the nonprofit child welfare literature, our study highlights the need for additional research on how nonprofit managers effectively convey the importance of performance measurement to their frontline workers. Case-based studies may help unpack the manner in which managerial strategies are selected and used, and how they inform frontline worker involvement in program monitoring. We also see the need for research on the outcomes of worker involvement in performance measurement. Not all performance measurement systems are equal. Of particular interest are those systems that allow for significant participation by frontline workers that combine the use of mandated performance indicators with metrics that agency stakeholders believe to have greater relevance and value, and that support an organizational commitment to evidence-informed practice and quality services (Pulakos, 2009). Research might determine how nonprofit managers collaborate with workers to develop and implement performance measurement systems (Carnochan et al., 2014) and future studies might examine how frontline workers at multiple levels of the organization use program data to enhance service effectiveness and organizational performance.

Conclusion

These results have implications for nonprofit management practice as well. As nonprofit managers strive to meet demands for accountability and outcome achievement, consideration of not only the type but also the comprehensiveness of performance measurement strategies for promoting worker uptake may be important as a multifaceted approach may be associated with greater worker uptake. These results also provide some indication that strategies used by managers to engage frontline workers in performance measurement and management may have the purpose of promoting an outcomes-focused organizational culture, in response to contractual requirements and/or to seek improved outcomes. These managerial efforts may be particularly important when nonprofit child welfare agencies are required by federal regulations to document outcomes and discern practices associated with their achievement (Meezan & McBeath, 2011), and may contribute to building the research case for evidence-based practices in a field which has been noted as lagging in this area (Barth et al., 2012). Nonprofit human service managers are faced with daily decisions in regard to where resources and managerial efforts are invested. These decisions require the input from performance information by frontline workers who may have little interest in performance measurement or in the use of data to increase program performance. Insight into how such managerial decisions can support organizational and frontline worker efforts focused on performance has practical value.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Monica Perez Jolles has a PhD in health policy and management and is currently an assistant professor at the Suzanne Dworak-Peck School of Social Work, University of Southern California, Los Angeles. A core research area of her work relates to the examination of multiple drivers of evidence use, patient/consumer engagement, and organizational change within health and human service agencies.

Crystal Collins-Camargo is an associate professor at the University of Louisville Kent School of Social Work. Much of her research has focused on public–private partnerships in child welfare and strategies within these sectors to enhanced achievement of organizational, practice, and client outcomes.

Bowen McBeath is a professor of social work and public administration at Portland State University. His research concerns the institutional political economic context of human service organizations. He received his PhD from the University of Michigan in social work and political science.

Alicia C. Bunger is an assistant professor at the College of Social Work at Ohio State University. Her research focuses on interorganizational relationships, innovation implementation, and frontline practices within human service organizations and systems.

Emmeline Chuang is an assistant professor in the Fielding School of Public Health at the University of California, Los Angeles. Her research interests focuses on how health and human service organizations can improve service access and quality of care for low-income women and children. She received her PhD from the University of North Carolina Chapel Hill in Health Policy and Management, with a specialization in Sociology and Organizational Studies.

References

Aarons

G. A.

Palinkas

L. A.

(2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 411-419.

Alaimo

S. P.

(2008). Nonprofits and evaluation: Managing expectations from the leader’s perspective. New Directions for Evaluation, 2008(119), 73-92.

Alexander

Brudney

J. L.

Yang

(2010). Introduction to the symposium: Accountability and performance measurement: The evolving role of nonprofits in the hollow state. Nonprofit and Voluntary Sector Quarterly, 39, 565-570.

Austin

M. J.

(2003). The changing relationship between nonprofit organizations and public social service agencies in the era of welfare reform. Nonprofit and Voluntary Sector Quarterly, 32, 97-114.

Austin

M. J.

(2008). Strategies for transforming human service organizations into learning organizations: Knowledge management and the transfer of learning. Journal of Evidence-Based Social Work, 5, 569-596.

Bargerstock

A. S.

(2000). The HRM effectiveness audit: A tool for managing accountability in HRM. Public Personnel Management, 29, 517-527.

Barratt

(2003). Organizational support for evidence-based practice within child and family social work: A collaborative study. Child & Family Social Work, 8, 143-150.

Barth

R. P.

Lee

B. R.

Lindsey

M. A.

Collins

K. S.

Strieder

Chorpita

B. F.

. . . Sparks

J. A.

(2012). Evidence-based practice at a crossroads: The timely emergence of common elements and common factors. Research on Social Work Practice, 22(1), 108-119.

Baruch

Holtom

B. C.

(2008). Survey response rate levels and trends in organizational research. Human Relations, 61, 1139-1160.

10.

Berman

(2002). How useful is performance measurement. Public Performance & Management Review, 25, 348-351.

11.

Birken

S. A.

Lee

S.-Y. D.

Weiner

B. J.

Chin

M. H.

Schaefer

C. T.

(2013). Improving the effectiveness of health care innovation implementation middle managers as change agents. Medical Care Research and Review, 70, 29-45.

12.

Cameron

C. A.

Trivedi

P. K.

(2009). Microeconometrics using Stata. College Station, TX: Stata Press.

13.

Campbell

D. A.

(2002). Outcomes assessment and the paradox of nonprofit accountability. Nonprofit Management & Leadership, 12, 243-259.

14.

Campbell

D. A.

Lambright

K. T.

(2017). Struggling to get it right. Nonprofit Management & Leadership, 27, 335-351. doi:10.1002/nml.21249

15.

Carman

J. G.

(2009). Nonprofits, funders, and evaluation: Accountability in action. The American Review of Public Administration, 39, 374-390. doi:10.1177/0275074008320190

16.

Carnochan

Samples

Myers

Austin

M. J.

(2014). Performance measurement challenges in nonprofit human service organizations. Nonprofit and Voluntary Sector Quarterly, 43, 1014-1032.

17.

Carrilio

T. E.

Packard

Clapp

J. D.

(2004). Nothing in—Nothing out: Barriers to the use of performance data in social service programs. Administration in Social Work, 27, 61-75.

18.

Chuang

Collins-Camargo

McBeath

Wells

Bunger

(2014). An empirical typology of private child and family serving agencies. Children and Youth Services Review, 38, 101-112.

19.

Collins-Camargo

Hollie

McBeath

(2014). Private child and family serving agencies: Implications of national survey results for policy and managerial practice. Children and Youth Services Review, 38, 142-149.

20.

Collins-Camargo

McBeath

Ensign

(2011). Privatization and performance-based contracting in child welfare: Recent trends and implications for social service administrators. Administration in Social Work, 35, 494-516.

21.

Collins-Camargo

Royse

(2010). A study of the relationships among effective supervision, organizational culture promoting evidence-based practice, and worker self-efficacy in public child welfare. Journal of Public Child Welfare, 4, 1-24.

22.

Collins-Camargo

Sullivan

Murphy

(2011). Use of data to assess performance and promote outcome achievement by public and private child welfare agency staff. Children and Youth Services Review, 33, 330-339.

23.

Devaney

(2004). Relating outcomes to objectives in child protection. Child & Family Social Work, 9, 27-38.

24.

DiMaggio

Powell

W. W.

(1983). The iron cage revisited: Collective rationality and institutional isomorphism in organizational fields. American Sociological Review, 48, 147-160.

25.

Döring

Downe

Martin

(2015). Regulating public services: How public managers respond to external performance assessment. Public Administration Review, 75, 867-877.

26.

Frumkin

Andre-Clark

(2000). When missions, markets, and politics collide: Values and strategy in the nonprofit human services. Nonprofit and Voluntary Sector Quarterly, 29(Suppl. 1), 141-163.

27.

Government Accountability Office. (2004). Child and family service reviews: Better use of data and improved guidance could enhance HHS’s oversight of state performance. Retrieved from http://www.gao.gov/new.items/d04333.pdf

28.

Graaf

Hengeveld-Bidmon

Carnochan

Radu

Austin

M. J.

(2016). The impact of the great recession on county human service organizations: A cross case analysis. Human Service Organizations: Management, Leadership & Governance, 40, 152-169.

29.

Gruman

J. A.

Saks

A. M.

(2011). Performance management and employee engagement. Human Resource Management Review, 21, 123-136.

30.

Hasenfeld

(2010). Organizational responses to social policy: The case of welfare reform. Administration in Social Work, 34, 148-167.

31.

Hasenfeld

Garrow

E. E.

(2012). Nonprofit human-service organizations, social rights, and advocacy in a neoliberal welfare state. Social Service Review, 86, 295-322.

32.

Hatry

H. P.

(2002). Performance measurement: Fashions and fallacies. Public Performance & Management Review, 25, 352-358.

33.

Hendricks

Plantz

M. C.

Pritchard

K. J.

(2008). Measuring outcomes of United Way–funded programs: Expectations and reality. New Directions for Evaluation, 2008(119), 13-35.

34.

Hoole

Patterson

T. E.

(2008). Voices from the field: Evaluation as part of a learning culture. New Directions for Evaluation, 2008(119), 93-113.

35.

Johnson

Austin

M. J.

(2006). Evidence-based practice in the social services: Implications for organizational change. Administration in Social Work, 30, 75-104.

36.

Jos

P. H.

Tompkins

M. E.

(2004). The accountability paradox in an age of reinvention: The perennial problem of preserving character and judgment. Administration & Society, 36, 255-281.

37.

Kaplan

R. S.

(2001). Strategic performance measurement and management in nonprofit organizations. Nonprofit Management & Leadership, 11, 353-370.

38.

King

J. A.

Volkov

(2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35(3), 10-16.

39.

Kukla-Acevedo

Hodges

Ferreira

ôcMazza

(2008). System of care implementation issue brief# 6-Quality improvement strategies that work. Retrieved from http://www.rosied.org/Resources/Documents/Motion%20for%20Clarification.Sherwood_aff.Ex.%20D.pdf

40.

Lee

S. J.

Bright

C. L.

Berlin

L. J.

(2013). Organizational influences on data use among child welfare workers. Child Welfare, 92, 97-118.

41.

LeRoux

Wright

N. S.

(2010). Does performance measurement improve strategic decision making? Findings from a national survey of nonprofit social service agencies. Nonprofit and Voluntary Sector Quarterly, 39(4), 571-587.

42.

Letts

C. W.

Ryan

W. P.

Grossman

(Ed.). (1999). High performance nonprofit organizations: Managing upstream for greater impact. New York, NY: John Wiley.

43.

Lindsey

Shlonsky

(2008). Child welfare research: Advances for practice and policy. New York, NY: Oxford University Press.

44.

Lipsky

(2010). Street-level bureaucracy, 30th ann. Ed.: Dilemmas of the individual in public service. New York, NY: Russell Sage Foundation.

45.

Lynch-Cerullo

Cooney

(2011). Moving from outputs to outcomes: A review of the evolution of performance measurement in the human service nonprofit sector. Administration in Social Work, 35, 364-388.

46.

Manuel

J. I.

Mullen

E. J.

Fang

Bellamy

J. L.

Bledsoe

S. E.

(2009). Preparing social work practitioners to use evidence-based practice: A comparison of experiences from an implementation project. Research on Social Work Practice, 19, 613-627.

47.

Martin

L. L.

Kettner

P. M.

(Eds.). (2010). Measuring the performance of human service programs (2nd ed.). Los Angeles, CA: SAGE.

48.

Meezan

McBeath

(2011). Moving toward performance-based, managed care contracting in child welfare: Perspectives on staffing, financial management, and information technology. Administration in Social Work, 35(2), 180-206.

49.

Mone

E. M.

London

(2014). Employee engagement through effective performance management: A practical guide for managers. Stony Brook, NY: Routledge.

50.

Newcomer

(2008). Assessing performance in nonprofit service agencies. In DeLancer Julnes

Berry

F.S.

Aristigueta

M.P.

Yang

(Eds.), International handbook of practice-based performance management (pp. 25-44). Thousand Oaks, CA: SAGE.

51.

O’Toole

(2002). The relationship between employees’ perceptions of safety and organizational culture. Journal of Safety Research, 33, 231-243.

52.

Packard

(2009). Staff perceptions of variables affecting performance in human service organizations. Nonprofit and Voluntary Sector Quarterly, 39, 971-990.

53.

Parrish

D. E.

Rubin

(2012). Social workers’ orientations toward the evidence-based practice process: A comparison with psychologists and licensed marriage and family therapists. Social Work, 57, 201-210.

54.

Pulakos

E. D.

(2009). Performance management: A new approach for driving business results. Malden, MA: John Wiley.

55.

Salamon

L. M.

(1995). Partners in public service: Government-nonprofit relations in the modern welfare state. Baltimore, MD: JHU Press.

56.

Sandfort

J. R.

(2000). Moving beyond discretion and outcomes: Examining public management from the front lines of the welfare system. Journal of Public Administration Research and Theory, 10, 729-756.

57.

Sawhill

J. C.

Williamson

(2001). Mission impossible? Measuring success in nonprofit organizations. Nonprofit Management & Leadership, 11, 371-386.

58.

Scurlock-Evans

Upton

(2015). The role and nature of evidence: A systematic review of social workers’ evidence-based practice orientation, attitudes, and implementation. Journal of Evidence-Informed Social Work, 12, 369-399.

59.

Smith

B. D.

(2009). Service technologies and the conditions of work in child welfare. In Hasenfeld

(Ed.), Human services as complex organizations (pp. 253-267). Thousand Oaks, CA: SAGE.

60.

Smith

S. R.

(2010). The political economy of contracting and competition. In Hasenfeld

(Ed.), Human services as complex organizations (pp. 139-160). Thousand Oaks, CA: SAGE.

61.

Smith

S. R.

Grønbjerg

K. A.

(Eds.). (2006). Scope and theory of government-nonprofit relations. New Heaven, CT: Yale University Press.

62.

StataCorp. (Ed.). (2011). Stata statistical software: Release 12. College Station, TX: Author.

63.

Thomson

D. E.

(2010). Exploring the role of funders’ performance reporting mandates in nonprofit performance measurement. Nonprofit and Voluntary Sector Quarterly, 39, 611-629.

64.

Torres

R. T.

Preskill

(2001). Evaluation and organizational learning: Past, present, and future. The American Journal of Evaluation, 22, 387-395.

65.

Tuan

(2004). Cultivating a culture of measurement. In Kibbe

(Ed.). Funding effectiveness: Lessons in building nonprofit capacity (pp. 105-142). San Franscisco, CA: Jossey-Bass.

66.

Vaughan

S. K.

(2010). The importance of performance assessment in local government decisions to fund health and human services nonprofit organizations. Journal of Health and Human Services Administration, 32, 486-512.

67.

Weigensberg

E. C.

(2009). Child welfare agency performance: How are child, agency, and county factors related to achieving timely permanency outcomes for children in foster care? (Doctoral dissertation). The University of North Carolina at Chapel Hill.

68.

Wells

Jolles

M. P.

Chuang

McBeath

Collins-Camargo

(2014). Trends in local public child welfare agencies 1999-2009. Children and Youth Services Review, 38, 93-100.