Abstract
How do accountability policies affect failing organizations? Are additional interventions used to improve underperforming agencies effective in raising performance outputs? This article investigates the effectiveness of turnaround policies in organizations that persistently fail to meet accountability standards. Using Performance-Based Monitoring Analysis System data from169 school districts in Texas, this article shows that turnaround interventions have only limited success. While monitoring strategies work for the most salient performance indicator in the short term, improvements quickly dissipate following an intervention. Supporting the notion that management matters, results also show that the type of monitor assigned to a failing school can affect the extent of improvement in performance.
The performance of public organizations has been the subject of much attention following recent demands for a more efficient and effective system of governance. In response, elected officials have adopted a range of incentive policies aimed to increase the performance of the bureaucracy. These incentive policies assume that performance can be improved through changes in management strategies so that one-size-fits-all rewards and penalties will be adequate best practices for increasing performance in all types of organizations. As a result, these efforts have produced improvements in performance for some organizations, whereas little change is seen in others. Many scholars have examined whether these broad-based accountability policies have produced performance gains, and others have documented evidence of unintended consequences and organizational dysfunction stemming from pressures to raise performance (Moynihan, 2008; Rainey, 2003; Radin, 2006).
Although this literature has largely established that performance incentives do not always contribute positively to organizational performance in general, little attention has been given to how these accountability efforts affect underperforming organizations. In these organizations, penalties for failing to meet performance standards include additional interventions, such as site visits, audits, or the replacement of management, aimed at turning around the organization. Although these interventions may provide additional resources, they also incur substantial costs to failing organizations (i.e., payment for services, goal displacement, decreased morale). Empirical work has yet to determine whether turnaround strategies lead to higher performance or if failing organizations continue to perform below expectations despite interventions.
Existing research on the effectiveness of turnaround efforts consists largely of small-n case studies for which mixed findings are incapable of leading to broader generalizations (Beeri, 2009; Eitel, 2004; Turner, Skelcher, Whiteman, Hughes, & Jas, 2004). This study aims to expand this research through the use of a large-N data set to examine both short- and long-term effects of monitoring interventions in the context of K-12 education, as public education has been the subject of highly salient accountability policies and turnaround strategies that have received mixed reviews.
I begin by considering how elected officials have responded to calls for greater accountability in the public sector. I then connect this discussion of policy implementation to theoretical propositions of whether we should expect accountability policies and turnaround mechanisms to be effective in organizations that consistently underperform. Next, I introduce turnaround policies in the realm of public education and focus on the strategy of performance monitoring. The core of the analysis focuses on the performance impact of monitors in both the short and long term. Findings indicate substantively interesting relationships between monitoring and performance over time that have important implications for the development of turnaround policies in the public sector.
Public Sector Performance
Existing research on public organization performance often grapples with the question, “Why do government organizations seem to constantly underperform?” (Moynihan, 2008; Rainey, 2003). Unlike private organizations, where performance and survival are generally tied to measureable profits (Arogyaswamy, Barker, & Yasak-Ardekani, 1995; Cameron, Sutton, & Whetten, 1988; Mellahi & Wilkinson, 2004), public organizations may be evaluated on multiple dimensions. Public organizations are often expected to pursue a number of goals for different stakeholder groups, and each goal may be met with a different degree of success. However, with a safety net of public funding, specialized policy expertise, and near monopoly status, public agencies are often irreplaceable even if they fail to adequately meet performance goals (Meier & Bohte, 2003). With no replacement readily available to supply a public good or service, suboptimal levels of performance by agencies are generally tolerated for substantial periods of time (Paton & Mordaunt, 2004).
Despite common perceptions that public organizations are immortal (Kaufman, 1976), they are not immune to decline and failure (Lewis, 2002). Threats to agency life generally include policy changes, mission completion, and market competition (Jas & Skelcher, 2005). In the presence of policy change, political leaders may have priorities that are vastly different than previous administrations. Thus, political power may be used to change the purpose of an agency so that it becomes ineffective by design. Following mission completion, an organization may no longer be seen as necessary for providing a public good. For instance, though the Works Progress Administration was once viewed as vital for reinvigorating the economy, it eventually lost relevance and was ended. Finally, for organizations such as the postal service or public education, increased competition has threatened agency life. However, these organizations are often permitted to continue with little retribution for low levels of performance, contributing to the view of bureaucracy as inefficient.
With the spreading popularity of accountability mechanisms, public organizations have faced greater penalties for persistently underperforming. Performance initiatives—often identified as performance management, pay-for-performance, performance planning, managing for results, total quality management, or contracting out—have challenged public organizations to account for organizational outputs and outcomes through a variety of reporting standards. In some cases, these efforts have produced improvements in performance of organizations, but in others, these incentives fail to result in positive performance gains. Many scholars have examined the inability of these broad-based accountability policies to produce performance gains, and others have documented evidence of unintended consequences and organizational dysfunction stemming from pressures to raise performance (Meyer & Zucker, 1989; Moynihan, 2008; Radin, 2006). Yet few have moved beyond general outcomes of accountability policies to focus on the additional interventions needed for poorly performing organizations. Do failing organizations perform better or worse following the implementation of accountability policies as compared with more successful organizations? Are additional interventions effective in improving performance of these organizations? In other words, can failing public organizations be turned around successfully, or do they continue to fail despite additional intervention mechanisms?
Responding to Turnaround
Although the discussion of organization turnaround is widely lacking from public organization theory, scholars of private management have contributed much more time to developing a stage model to describe turnaround processes. In this literature, turnaround mechanisms are traditionally categorized as either strategic or operating (Chowdhury, 2002; Hambrick, 1985; Hofer, 1980). Strategic turnarounds emphasize changing the business the firm is engaged in and include actions such as developing new markets, divestment, or vertical integration. Operating turnarounds reassess the way the organization conducts business and include short-run tactics such as revenue generation and cost cutting. These scholars generally agree that declines caused by the external environment should be addressed with strategic turnaround strategies, whereas internal threats should be addressed with operating turnaround mechanisms (Chowdhury, 2002).
As costs associated with turnaround are generally believed to be less than the cost incurred through an agency closure (i.e., finding a suitable replacement, training new personnel, etc.), underperforming public organizations are often exposed to turnaround strategies. Analyzing whether strategic or operating strategies for turnaround are feasible for underperforming public organizations, Boyne (2003) categorizes turnaround policies as retrenchment, repositioning, or reorganization. Retrenchment consists of focused downsizing of the scope or size of an organization in efforts to increase efficiency. Though this drastic form of turnaround may be feasible for private firms, it is less feasible for public organizations because of legal constraints and obligations. Still possibilities for retrenchment may lie in outsourcing nonprimary duties to third parties, allowing organizations to cut costs and apply extra resources to core responsibilities (Meier & O’Toole, 2007).
Under repositioning, new efforts toward growth and innovation are expected to jump start organizations with new target audiences. Similar to retrenchment, this strategy can prove quite difficult for public organizations because of statutory constraints. For example, K-12 schools cannot provide services for college students. Nevertheless, instances of repositioning can be found in public agencies. Although schools may not be able to provide services for college students, they can provide college-level courses for current students. As another example, the post office has expanded service options and offers mail delivery to a much larger geography as compared with the pony express. Strategies may also include increasing service options and improving the internal and external reputation of the organization (Boyne, 2003).
Third, reorganization is similar to the concept of operating strategies in private management literature in that it focuses on internal changes. Boyne (2003) argues that this approach is most similar to the replacement of personnel in struggling public organizations, though it may also include developing new budgetary or planning processes. Leadership change as a turnaround strategy has been increasingly used among public agencies, but strong empirical evidence is lacking as to whether this change results in improved performance in the short and long term (Hill, 2005).
Testing the Theory: Performance Monitoring
This study will seek to contribute to existing knowledge of turnaround by providing a large-N analysis of the effect of monitoring on performance. Monitoring, a technique similar to inspections and audits, is used by a superior organization to regulate smaller entities. Monitoring includes site visits and face-to-face meetings that are used to complement other sources of performance reporting. Although monitoring, as defined here, may relate to audits through a review of financial health of an organization, it also entails a review of the competence of personnel, compliance with standards, and success in meeting outcome goals (Boyne, Day, & Walker, 2002). For the context of this analysis, monitoring is most closely related to reorganization strategies that attempt to correct for deficiencies internal to the organization by either influencing managerial decisions or replacing top level managers. Boyne et al. (2002) provide theoretical explorations concerning the potential for inspection to improve the performance of an organization. They argue that inspections are associated with both costs and benefits for public organizations. Benefits include the provision of a safety net to help organizations cope with failure, an increase in across-the-board standards among agencies, and a symbolic gesture that provides assurance to multiple groups of agency stakeholders. These benefits, however, do not come without substantial costs. Costs to the organization include the direct costs of funding and operating an inspection system, indirect compliance costs, and goal displacement costs as organizations attempt to meet multiple, and sometimes competing, standards. Boyne et al. (2002) further argue that both benefits and costs are largely dependent on the expertise and judgment of the inspector. An individual inspector must possess greater knowledge than the manager of an organization in order to add value to an organization’s outputs. The inspector must also be able to apply the interpretation of standards evenly and consistently across organizations.
This theoretical discussion of inspection provides multiple testable hypotheses, two of which can be tested here for failing organizations. Given the benefits that may be associated with monitoring by an expert, the first hypothesis can be stated as: Monitoring interventions will lead to an increase in performance for failing organizations. However, given the costs associated with monitoring and the problems associated with one-size-fits-all policies, a second hypothesis is warranted: Monitoring interventions will have no effect on the performance of failing organizations. 1
Turnaround Mechanisms in Public Education
School districts provide an ideal setting to test the effectiveness of turnaround strategies, as schools have been the center of much discussion of performance accountability and policy interventions for turnaround throughout the last decade. With the rise in demands for greater accountability and an increase in high-stakes testing, many schools have been widely criticized for producing consistently poor results. As schools traditionally have operated as virtual monopolies (Chubb & Moe, 1990, Meier & Bohte 2003), recent policy changes through initiatives such as No Child Left Behind have focused on setting new performance and accountability standards for education across all states. Implementation of these policies, however, makes multiple assumptions that may not always hold. First, policies assume that all schools can succeed, but that certain elements for success are missing (Brady, 2003). Thus, policymakers believe the solution to failing schools can generally be addressed by applying a set of standards with proper management skills. This implies that all schools are capable of improving but that some are simply choosing not to because of a lack of will or misplaced priorities by those at the top of the organization (Brady, 2003, Fryar & Rabovsky, 2012). Little consideration is given to why schools may not be able to adjust to new standards quickly and easily or what turnaround mechanisms may be best suited for different types of school districts. For instance, some schools may be facing issues of financial mismanagement while others are combating dropout rates and still others are just learning how to comply with new special education rules.
Following the assumption that performance can improve for all schools, elected officials often presume that new policies can be implemented through a one-size-fits-all approach. As performance levels are considered to be linked to internal mechanisms, larger environmental factors that may limit desired performance outcomes are often ignored. Thus, the question of relative starting points for different organizations is ignored (Jas & Skelcher, 2005). In addition to these assumptions regarding performance, the measurement of performance in schools is complex and needs to include multiple goals of education. The failure to consider multiple performance indicators can lead to negative unintended consequences. Often the most salient performance outputs, state standardized test scores, may be achieved at the detriment of larger, more important outcomes such as learning and college readiness that are more difficult to quantify (McDermott, 2011; McNeil, 2000).
As dissatisfaction of school performance continues to accelerate, a variety of turnaround strategies have been developed at the federal, state, and local levels. Though not an exhaustive list, these interventions include school improvement plans, the provision of choice, the provision of supplemental services, reconstitution, and monitoring by outside experts (Willms, 2000). School improvement plans are mandated by No Child Left Behind for Title 1 schools failing to make adequate yearly progress in two consecutive years (Mazzeo & Berman, 2003). School improvement plans are intended to bring teams together to create unified strategies to raise performance. Provision of choice mechanisms allow guardians of students in schools identified for improvement to transfer their student(s) to another public school that is not underperforming. This strategy was available in 13 states prior to No Child Left Behind and is now required by federal mandate (McDermott, 2011). Likewise, schools that fail to meet adequate yearly progress for three consecutive years are now required to offer supplemental educational services, often in the form of free tutoring. The number of students eligible for these services has increased on a yearly basis, signaling that schools are still struggling to meet performance criteria (Peterson, 2005). Under more extreme circumstances, school reconstitution may involve removing a large portion of school administrators and teachers and replacing them with individuals deemed to be more qualified (Rice & Malen, 2003). This type of reorganization can be controversial and, to date, has produced mixed anecdotal evidence of success (Rudo, 2001).
The turnaround intervention analyzed in this study is monitoring by third party experts. Used at varying levels across states, monitoring techniques consist of assigning former educators, often former superintendents with a high levels of experience, to oversee school districts that have been identified as failing in regard to at least one performance standard. Individual monitors not only observe actions of school administrators, but they often assume management of the school district. Monitoring may also be used as a threat, encouraging schools to improve performance before a monitor assumes leadership in the district. Though school monitoring has been available as a turnaround mechanism by multiple states since the early 1980s, some states use this intervention more than others. Furthermore, very little is known about the extent to which monitoring is successful as a turnaround strategy for failing schools.
Data and Measures
Data on monitoring in failing organizations come from a set of Texas schools districts. Between 1993 and 2011, data were collected for 169 school districts that were subject to monitoring by former educators appointed by the Texas Education Agency (TEA). Monitoring data are combined with 17 years (1993-2010) of pooled data on school performance collected by the TEA. Texas schools are evaluated yearly on a range of both absolute and relative standards as defined by the TEA. Absolute standards include set passage rates for each portion of the state standardized test (65% on math and 70% on reading), as well as for graduation rates (75%). Performance levels are assigned yearly to each district, and an increase in performance level assignment for a given performance indicator is possible through “adequate yearly progress” for the given indicator (TEA, 2011). If a school fails to meet set performance requirements in one or more reporting years, the district is subject to monitoring intervention. Though monitoring is represented here by a dummy variable indicating whether a monitor was assigned to a district for a given year, monitor responsibilities, as defined by the Texas Education Code (TEC), Chapter 39, Subchapter C, include but are not limited to the following steps:
Conduct, with the involvement and advice of the school community partnership team, if needed (a) an on-site needs assessment relevant to the area of insufficient performance or (b) a comprehensive on-site needs assessment
Recommend appropriate actions relating to any area of insufficient performance (may include reallocation of resources, staff development, waivers from state rules, etc.)
Assist in the development of a targeted improvement plan
Assist the campus in submitting the targeted improvement plan to the board of trustees for approval and presenting the plan in a public hearing
Assist the commissioner in monitoring the progress of the campus in implementing the targeted improvement plan.
Furthermore, monitors may request an action taken by a school board, district superintendent, or school principal, such that the individual becomes a manager of the underperforming district. The monitors’ actions are limited, however, from changing board elections or setting a district tax rate (TEC §39.111; TEC §39.106).
Although monitoring may bring the benefits of expert advice to failing school districts, costs associated with this intervention strategy may offset any gains. For underperforming districts in Texas, direct costs largely consist of payment to the monitor, conservator, or management team acting on behalf of the state agency. Indirect costs may be even more threatening and include the reallocating of resources to comply with monitor recommendations, the embarrassment of being identified as a failure by local media, and goal displacement to meet performance directives from the monitors. Though costs can be identified, they can be difficult to measure. Additionally, little is known regarding whether this type of intervention provides any type of payoff by improving performance for failing schools. An analysis of district performance data across time should provide substantial evidence regarding the success of this type of turnaround strategy. Findings of positive or negative impacts may have important implications not only for the development of a more generalizable turnaround theory but for the decision making process of policy makers.
Dependent Variables
Definitions of school success are likely to vary across stakeholder groups (politicians, parents, students, local community members) and across different types of environments. While more affluent schools may prioritize college readiness, inter-city schools may be focused on increasing attendance rates. Accordingly, five outcome measures will be used in this analysis in an effort to determine if organizational monitoring affects a variety or organizational processes and goals. The most salient performance measures in the political environment of Texas school districts is the overall student pass rate on the state standardized test (for Texas, this is the Texas Assessment of Knowledge and Skills [TAKS]). The test is administered to student in Grades 3 through 11, and the final test must be passed for a student to receive a high school diploma from the State of Texas. Related to passage rates, a second measure of performance includes graduation rates. Students who graduate are those who have passed the final high-stakes TAKS exam and have met all other requirements for graduation. Third, a growing concern of parents, members of an active stakeholder group, is college readiness. This is often measured in Texas by calculating the percentage of students who score at least a 1,110 SAT (or an equivalent ACT score) in each school district.
Finally, two bottom-end indicators of performance, dropout rates and attendance rates, are included in this assessment. High attendance rates are valued, as students cannot learn and often cannot graduate if they are not in class on a daily basis. Furthermore, district state aid is allocated based on average daily attendance, giving schools an incentive to keep students in the classroom. Dropout rates, though believed to include downward bias when reported by districts not under supervision, are included in performance assessments of school districts. The state measure is defined as the annual percentage of students who leave school between Grades 7 and 12.
Using multiple dependent variables will test whether monitors influence some types of performance more than others. The use of multiple dependent variables will also affect the number of cases reported across model estimations, as not all schools have reported each outcome across time (graduation rates, e.g., are only relevant for high schools). For the present analysis, determining effects across outcomes takes priority over dropping cases that may not report all five measures. 2
Many studies assess absolute gains or losses of similar performance indicators in school districts. However, this measurement approach will not capture state-level trends that are important for identifying low performing schools. For instance, if the state changes the structure of a standardized test or adjusts the calculation for dropout rate, performance levels may shift for all districts in the state. To account for these state level trends, performance indicators will be measured as the difference in value of the performance indicator for each monitored school compared with the overall state average for each year. Mathematically, the dependent variable tested here is calculated by the equation District Performance Differenceit = District Performanceit − Average District Performancet. For underperforming schools, the dependent variable is generally negative, as the state average is greater than performance in failing schools. If monitors lead to improvements in school performance, the difference between underperforming schools and the schools at the mean should become more positive following the intervention. As modeled, a positive coefficient will be related to an increase in performance, whereas a negative coefficient would indicate a decrease in performance as compared with the average for the state. 3
Control Variables
Though not reflected in the assumptions of legislators in developing many performance expectations placed on public agencies such as school districts, the external environment is believed by scholars to have a substantial effect on the organizational outcomes. For education, a set of education production functions have been well developed to control for resources and constraints that vary by organization (Hanushek, 1996).
Empirical research on school resources provides substantial evidence that schools with greater resources face a less challenging task in educating students. Measures of resources include expenditure per pupil, revenue per pupil, school enrollment, student–teacher ratio, and percent central administration. With the exception of student–teacher ratio, each should be positively correlated with school performance indicators.
Though the availability of resources may decrease school task difficulty, the presence of a variety of constraints may also limit a school’s ability to educate students. As both poverty and race are correlated with constraints such as family income and education (Jencks & Phillips 1998), measures of constraints include the percentage of students who are eligible for free or reduced price school lunches, the percentage of students classified as special education, the percentage of African American students, and the percentage of Latino students. The greater the population of these student groups, the more difficult it may be for schools to meet performance expectations.
Of 169 districts included in this dataset, 78 (46%) are charter schools. As public schools and charter schools are likely to differ in age, resources, and task difficulty levels (Hanushek, Kain, Rivkin, & Branch, 2007; Sass, 2006), a dummy variable is included in the model to test for differences between these two types of underperforming schools. Scholarship presents mixed findings on the quality of charter schools, as nonrandom selection of students into charters presents methodological challenges (Booker, Gilpatric, Gronberg, & Jansen, 2007; Hoxby & Rockoff, 2004; Sass, 2006; Zimmer & Buddin, 2006). Previous research indicates that charter schools may experience difficult start-up periods in a struggle to attract and retain students (Hanushek et al., 2007), and this perceived failure may become apparent for charter schools in this data set.
Finally, a control must be included for the previous performance as well as for effects of the monitor after intervention. A lagged dependent variable is included in each model, as performance is likely to be autoregressive. Including this lag corrects for any threats of autocorrelation present in the model. 4 Furthermore, errors are clustered by school district or charter code to correct for variance in error across groups. 5 Additionally, schools administrators should develop better strategies for improving performance over time through experience and learning. Although both controls for previous performance and postmonitor effects should be positively related to performance, the structure of the later makes assumptions about the functional form of the model as following a pattern of trend improvement or shift improvement. Trend improvement assumes a positively sloped linear relationship between performance and effects of the intervention over time. Shift improvement assumes a shift from one performance level to the next as a result of an intervention, with a general slope of zero over time. In testing the overall model with each assumption, trend improvement is insignificant and adds little to the model (see Table A.1 in Appendix A). Including a dummy variable to control for a shift change, however, proves significant in explaining the relationship between monitoring and performance.
Table 1 provides a description of variables for all districts in Texas as compared with failing districts. From these data, it is clear that monitored districts are performing well below the state average for all dependent variables in this study. Notably, underperforming districts also appear to face greater levels of task difficulty because of an increased presence of low income and minority students. School enrollment sizes are noticeably higher for monitored school districts, though this is largely affected by cases of monitoring for Houston and Dallas ISDs. Finally, student–teacher ratios and the percentage of central administration are largely similar for monitored schools compared with state averages. These factors may present great constraints on districts’ ability to meet performance expectations as compared with more advantaged districts so that they are targeted with interventions as a consequence of underperformance.
Mean Variable Comparisons.
Note. TAKS = Texas Assessment of Knowledge and Skills.
Findings
To compare the short-term and long-term effects of monitor interventions, more recent data (2007-2011) will be first analyzed and then compared with schools with a monitor prior to the start of the 2005 school year to test for both short- and long-term effects in monitoring intervention. 6 Data will then be combined to examine the overall effect of monitors on school performance across 17 years.
The results linking monitors to short-term school performance are shown in Table 2. Though data includes monitors present at schools in 2011, most of these are removed because of missing performance data due to the recency of the monitor in the district. Although monitors have no effect on attendance rates, graduation rates, or college readiness, the impact of this intervention shows a strong positive relationship with the percent of students passing the TAKS exam while the monitor is assigned to the school. Under the direction of the monitor, the TAKS passage rate for a district generally increases by 4.21 percentage points. This improvement, while positive, constitutes just half of the difference between the average failing school passage rate (62.05%) and that of the average school in Texas (71.03%). As these test passage rates are the most salient of school performances indicators and often create front-page news, it is rational to expect both school districts and monitors to prioritize this outcome over other performance indicators. However, while this improvement is important, it may be short-lived. As coefficients for the shift improvement and time are negative, the model indicates that the monitor’s positive effect on performance will decline over time. In addition to the decline following the exit of the monitor, passage rates steadily decrease each year so that the impact of the monitor on performance disappears within 4 to 5 years. Thus, schools will return to performance levels previously identified as failing in the long term.
Impact of Monitor Presence, 2007-2010.
Note. TAKS = Texas Assessment of Knowledge and Skills.
p < .01. **p < .05. *p < .10, two-tailed test.
Monitoring is also correlated with an increase in dropout rates for monitored schools compared with the state average. Though findings for dropout rates are counter to expectations, this may be explained by the low validity of the variable, as dropout rates are notoriously miscalculated and underreported. Monitors are likely forcing schools to document actual dropout rates in analyses required by state such that no difference is actually realized. This may be further supported by the large decrease in dropout rates once the monitor leaves the school district.
Results also indicate stark differences between public school districts and charter schools. For three of the five performance variables, failing charter schools perform far worse than failing public schools. As charter schools are a new addition in a market dominated by school districts that have existed for much longer periods of time, these schools may be facing a number of challenges in recruiting and retaining students (Hanushek et al., 2007). Additionally, charter schools often take advantage of financial incentives to recruit at-risk students, further increasing the level of task difficulty faced in meeting performance goals.
Overall, monitors appear to have the greatest effect on test performance, but evidence also suggests that this performance may fade over extended periods of time so that the school returns to its original state of underperformance. Theoretically, we should expect that monitoring should focus on improvements that persist through time. Monitoring is costly to the state in terms of planning, training, and resources. The agency is unlikely to want to reassign monitors to schools multiple times if performance improvements dissipate. Likewise, districts have multiple incentives to improve over time. Districts must pay for the activities of monitors and can face additional oversights if performance fails to meet goals across multiple performance assessments. If districts do not display suitable levels of performance across longer periods, they are highly likely to be ordered to close its doors by the state.
Is the short-term performance seen in Table 2 sustainable in the long run or can monitors improve performance only temporarily? Table 3 presents an analysis of long-term performance in underperforming schools that received a monitor prior to the start of the 2005 school year. Controls perform as expected, and charter schools again appear to face larger performance problems as compared with public schools. Charter schools have significantly higher dropout rates, lower attendance rates, and lower rates of college-ready students. With the exception of dropout rates, monitor interventions are not related to improvements for any of the five performance indicators. Similar to short-term performance, monitors appear to increase dropout rates across a larger number of time intervals. Even though the accuracy of this performance indicator is questionable, evidence here suggests at worse a decrease in performance for one measure and at best no change in performance for failing schools.
Impact of Monitor Presence, 1993-2005.
Note. TAKS = Texas Assessment of Knowledge and Skills.
p < .01. **p < .05. *p < .10, two-tailed test.
Moreover, these findings suggest that the use of monitoring as a turnaround strategy is not effective in the long run. When a monitor is assigned to a school, administrators may respond to additional pressure so that performance experiences a slight increase to keep the school from facing additional negative publicity or the threat of closure. However, once the monitor leaves, short-term impacts fade as the school returns to strategies used prior to intervention. Performance remains relatively stable at a point below expectations across a number of years. Unlike the model presented in Table 2, the variance explained in Table 3 is far from impressive. This may indicate that while one-size-fits-all strategies may be the preferred way to approach underperforming organizations, problems leading to failing are unique to each school environment so that a one-size-fits-all strategy is inadequate to fix complex challenges.
Next, data from Tables 2 and 3 are combined to determine the overall effect of monitors on failing school districts. The combined impact of performance-based monitoring across all monitored districts between 1993 and 2010 is modeled in Table 4. On an initial glance, monitors appear to be significantly related to improving college readiness, a task that can be quite difficult to influence. The presence of the monitor generally leads to a 1.45 percentage point increase in the number of students who score above an 1,110 on the SAT (or the ACT equivalent). However, further investigation shows that this increase quickly dissipates following the intervention, such that schools return to their former level of performance for this variable. Furthermore, monitors do not improve any other performance variable. This may be explained in part by persistently failing schools that have received monitors at multiple points in time. However, controlling for these districts does not affect findings or alter model fit in explaining performance. Interestingly, monitored charter schools perform at a lower level than monitored public schools across all performance indicators, suggesting intriguing policy questions for future analysis. 7
Impact of Monitor Presence, 1993-2010.
Note. TAKS = Texas Assessment of Knowledge and Skills.
p < .01. **p < .05. *p < .10, two-tailed test.
Finally, the time required for effects of monitoring to appear must be explored. Monitors may not result in immediate performance improvements but may be more closely related to performance in future years. Tests for gains in performance at years (t + 1) and (t + 2), however, show little evidence to support this possibility (see Appendix A, Tables A.3 and A.4). 8 The findings above are clear: The use of monitoring as a turnaround strategy in schools is an ineffective tool for improving performance. Though positive gains are seen for the most salient performance indicator in the short term, these gains quickly dissipate over time such that the organizations are not better off in the long term.
In sum, the empirical analysis presented above provides a glimpse at the effects of monitoring across time. Although it remains uncertain exactly why monitors may or may not be effective, it is clear that monitoring for failing organizations in this setting does not substantively improve performance over time. For earlier periods, monitoring affects nothing other than dropout rates, which is likely a function of ensuring reporting compliance. In more recent years, monitors are also able to affect TAKS passage rates. This may be because of either the presence of a new set of monitors or organizational learning over time (so that they have a better idea of what strategies are more effective in increasing TAKS performance). In the larger picture, findings suggest that high cost investments for turnaround do not lead to substantial gains in performance.
Why are scarce resources spent on this process if little is to be achieved in helping schools increase performance? Of possible explanations that can be considered, one unintended consequence of the ineffectiveness of monitoring is perhaps most alarming. Of the schools monitored since 1993, 17% (28 charter schools or school districts) have been permanently closed, supporting the notion that efforts to turnaround schools through monitoring may be a case of “too little and too late.” Monitors are often assigned to a school after it has been underperforming for multiple years. This intervention delay may lead to monitors who are simply tasked with closing a school instead of exerting effort to turnaround the school. When these schools close, little is gained. Students in public districts are added to neighboring districts, and those in charter schools are often sent back to the public schools they previously attended, creating negative shocks in stable districts. In sum, neither the intended nor unintended consequences of monitoring provide positive outcomes for all parties involved. School administrators are embarrassed, students and parents must cope with a large amount of change, and monitors must try to understand the school district and make management decisions within a short amount of time.
These patterns of relationships merits additional analysis beyond the use of a dummy variable for the presence of a monitor. As public management literature has widely illustrated that management matters, successful changes in failing organizations may be contingent on the quality of the monitor. Under the PBMAS infrastructure, some monitors may be better managers than others through tangible means such as frequency of meetings, background experience, and networking and quality measures. Nontangible differences may also include the ability to correctly assess problems and make sound decisions as well as the ability to communicate well during a time of crisis (Meier & O’Toole, 2007). Using data for 2007 to 2011, monitor names were tracked, and human capital information for each individual was coded to test whether the type of monitor matters in revitalizing a district. Dummy variables were coded for whether the monitor was affiliated with a private consulting firm or a university as well as if the monitor has previously held a position of principal or superintendent. Quality of monitoring should increase with greater experience (Meier & O’Toole, 2002), but few have tested whether differences in experience types substantially affect district outcomes. Additionally, dummy variables were included for educational attainment and gender. 9 Most former principals and superintendents have a master’s degree, so analysis focuses on the attainment of a doctorate, which may act as a signal for better quality. Controlling for gender is relevant, as this may be related to different effects on organizational outcomes (Jacobson, Palus, & Bowling, 2010).
Results from this analysis, displayed in Table 5, reveal interesting patterns. Though no group of monitors was related to graduation or attendance rates, groups varied in effects on TAKS passage, dropout, and college readiness rates. Monitors with experience in consulting were able to raise passage rates by more than 3% in the short term while no other trait or experience appeared significant. It is likely that these individual are most suited to the environment of monitoring due to experience in consulting in a number of other environments. These monitors may be able to prioritize turnaround goals and effectively communicate to reach the expectations of multiple stakeholder groups. On the other hand, those affiliated with a postsecondary institution were significantly related to an increase in dropout rates for failing schools districts. As previously mentioned, this is likely due to the experiences of these monitors in working with systems of large bureaucracy in higher education. These individuals may be trained to enforce the accurate reporting of performance data, as they may comply with similar rules and regulations in their postsecondary classrooms. Most notably, females and previous superintendents were related to increases in college readiness while those with the highest level of education and former principals were tied to decreases in this measure. Female monitors increased the percentage of students scoring an 1,110 on the SAT by nearly 10 percentage points, and former superintendents were related to a similar increase of more than 20%. Though at first extreme, the later relationships is likely tempered, as virtually all former superintendents were also former principals, resulting in a net increase of 12.89% in college readiness.
Impact of Monitor Management Experience on School Performance, 2007-2010.
Note. TAKS = Texas Assessment of Knowledge and Skills.
Percent tested on SAT in Model 4 not shown.
p < .01. **p < .05. *p < .10, two-tailed test.
Implications
This investigation demonstrates that organizational turnaround is a complex process worth additional attention from empirical researchers and theorists alike. Especially in light of increasing demands for accountability, turnaround strategies for failing organizations are both politically and theoretically relevant. Using monitoring interventions in underperforming schools districts, this analysis shows only short-term performance improvements for the most politically salient measure that are likely to dissipate quickly over time. For some types of organizations, short-term performance improvements may be adequate. However, for many public organizations, performance improvements are needed to last for much longer periods of time to avoid incurring greater costs or agency closure. Here, monitoring proves to be a quick fix that is not suitable to address larger problems faced by public organizations. Longer term strategies are needed if improvements are to be gained and maintained across all outcome measures.
Findings also suggest that managers matter, bolstering the arguments that both internal and external mechanisms must be considered when attempting to find solutions for failing organizations. Individual monitors are significantly related to the turnaround of failing schools between 2007 and 2011: women and former superintendents increased college readiness rates while only professional education consultants were able to increase TAKS passage rates. These relationships may lead to several questions in the development of intervention policies. If state agencies continue to prioritize standardized high stakes tests, then consultants may be best suited to help school identified as poorly performing. However, if the state wishes to prioritize college readiness, perhaps a different type of monitor is best. These contextual differences illustrate that the causes of low performance may be linked to a variety of mechanisms that cannot be treated by a one-size-fits-all intervention policy. It may not make sense to have the same sanction for a range of problems. Schools targeted for poor performance generally had higher levels of disadvantaged students and lower levels of resources. These schools may not be able to address performance issues in a manner comparable with schools that serve advantaged students. This argument is further supported by the clear difference between failing public schools and failing charter schools. As charter schools likely face difficulties that are different from their public school counterparts, school-specific strategies may be necessary to help establish and increase performance in these organizations.
A number of items remain to be explored to develop knowledge of turnaround that can be applied to multiple types of public organizations. Although a turnaround strategy, such as monitoring, may prove unfruitful for one type of organization, findings may vary across organizations outside of K-12 education. Additional research is needed to determine when monitoring is more or less likely to work. Furthermore, there exists a need for policy makers to consider the external constraints on organizations as accountability policies are implemented. More effort may be needed to select turnaround strategies on an organization-specific basis to gain maximum utility from interventions.
Finally, other types of strategies should be similarly analyzed across a number of organizations to better determine matches between turnaround strategy, internal management, and external resources and constraints. Time dimensions may prove to be especially important in the likelihood of an organization to improve performance. If poor performance can be targeted earlier, chances for survival may increase. However, if performance is allowed to slide for a number of years, turnaround may become too costly. In sum, further exploration of public organization failure and turnaround will prove particularly helpful in pushing forward knowledge of public policy and performance accountability. Replication of this empirical approach through large-N studies may prove to result in generalizable findings that are important for both the short- and long-term success of public organizations.
Footnotes
Appendix A
Testing for Lagged Effects of Monitors (t + 2).
| TAKS Passage Rate (SE) | Dropout Rate (SE) | Graduation Rate (SE) | 1,110 SAT Percent (SE) | Attendance Rate (SE) | |
|---|---|---|---|---|---|
| Presence of monitor | 1.22 (1.56) | −0.37 (0.42) | 4.38* (2.43) | −0.56 (1.01) | 0.15 (0.17) |
| Charter school | −2.23 (1.52) | 1.96*** (0.61) | −9.86*** (2.91) | −1.04 (1.00) | −0.36** (0.15) |
| Postintervention trend | −2.00* (1.12) | −0.30 (0.37) | 0.10 (1.77) | −2.22** (0.88) | −0.19* (0.11) |
| Year | 0.22* (0.12) | 0.12** (0.05) | −0.48** (0.21) | 0.12 (0.11) | −0.05*** (0.02) |
| One-year performance lag | 0.21*** (0.05) | 0.43*** (0.15) | 0.61*** (0.07) | 0.34*** (0.06) | 0.69*** (0.15) |
| Constant | 0.35 (6.93) | −0.84 (3.22) | 4.86 (8.14) | −2.43 (5.50) | −0.24 (0.83) |
| R 2 | 0.13 | 0.20 | 0.58 | 0.21 | 0.42 |
| Number of observations | 1,072 | 999 | 430 | 819 | 995 |
Note. TAKS = Texas Assessment of Knowledge and Skills. Controls not shown: expenditure/pupil, revenue/pupil, percent Black, percent Hispanic, percent low income, percent special education, school enrollment, student–teacher ratio, percent central administration, and percent tested on SAT.
p < .01. **p < .05. *p < .10, two-tailed test.
Acknowledgements
I wish to thank Guy Whitten, Kim Hill, and Erik Godwin for their comments on earlier drafts of this article. I would like to give special thanks to Megan Mumford for her assistance in data collection and Ken Meier for his comments and suggestions throughout the preparation of this article. I am also grateful for feedback received from three anonymous reviewers and participants of the 2012 Annual Meeting of the Southern Political Science Association.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
