Abstract
In this case, Sharon Rowley, a veteran principal, volunteers to participate in a new community-driven accountability initiative and encounters dilemmas about what it means to be a “data-driven” instructional leader. This case provides an opportunity for aspiring school leaders to explore and apply data-use theory to the work of leading for instructional improvement, with a specific focus on using new accountability metrics intended to leverage broader conceptions of supporting students’ academic and social well-being. The teaching notes at the end of the case guide readers to consider the leadership demands of responding to the professional and market challenges that emerge from using and communicating about accountability metrics and data-use interventions more generally.
Case Narrative
For Sharon Rowley 1 , the principal of Rosemont High School, the fact that every Rosemont student who applied for college over the past 3 years had been accepted was a source of great pride. She constantly referenced these data when she talked to prospective parents and community members about the successes Rosemont students were now seeing under her leadership. Sharon felt strongly that students and parents saw her public high school’s “100 percent acceptance rate” as a compelling reason to enroll at Rosemont—as opposed to nearby charter schools, more-established private schools, and other high schools in her district, Greater Oakdale Unified. But as more families faced tougher economic conditions and moved away from the region and advocacy groups publicly critiqued district leadership for not prioritizing students’ safety and well-being, Sharon had work to do to promote positive perceptions about Rosemont in the community and keep her school’s doors open.
To address budget shortfalls, lower enrollment, and community criticisms, the Greater Oakdale school board had charged the superintendent with closing 10% of the district’s nearly 300 schools over 5 years. The superintendent’s team had announced that one criterion they would use to guide these controversial decisions would be school performance measured by the tests monitored under the state accountability system. The state used the ACT College Ready exam and merit exams in high school literacy, mathematics, and science. Rosemont’s 11th graders rarely received praise or comment for their performances on these exams. Greater emphasis on the tests threatened to overshadow the positive point that a number of Rosemont students went on to be first-generation college students. For Sharon, the students’ past test scores had served as indicators that Rosemont needed to do more to prepare students for postsecondary education. Yet, Sharon felt these numbers did not begin to capture the work that she had led over the past 2 years to increase students’ academic engagement and keep more students in school. She and her staff had invested heavily in writing across the curriculum as a tool for supporting their students’ academic learning needs. Over the past 2 years, her 11th graders’ reading scores on the state’s merit exam had jumped from 38% scoring at proficiency to 50%.
Despite this progress, Rosemont still showed up as a poor performer on state accountability metrics heavily promoted by the state education agency, property management companies, and the press. Sharon worried about a number of potential consequences. She and many of her staff might be asked to step down or Rosemont may become one of next year’s turnaround schools. She also worried about the impact of the kinds of changes she had seen other high schools pursue—mandated double doses of core subjects for struggling students and reduced opportunities for students to enroll in the kinds of courses that broadened their horizons. And, because of the ways in which these data were talked about publicly, Sharon increasingly felt pushed to use these same data to validate budget and hiring decisions instead of focusing on what really mattered for school quality: good teaching. She also worried, if she was not out in front often enough, explaining what these data indicated about the school’s success and how they would be used to guide the school’s forward movement, that prospective parents and staff might rely on rumors, or look to her competitors (that is, neighboring schools) in choosing whether to enroll and stay at Rosemont, or go elsewhere.
Improving “School Quality” at Rosemont: Sharon’s Approach
Sharon had moved into educational leadership from being a teacher because she wanted to improve students’ outcomes beyond her classroom walls. She believed that success for Rosemont would come from adopting a more comprehensive approach to collecting and analyzing information about teaching and learning and using it for improvement. As a school leader, Sharon constantly sought to model how to problem-solve issues related to curriculum, content, and pedagogy by looking at data-based evidence in ways that went beyond looking at students’ proficiency levels on state tests once or twice per school year. She also focused considerable time and energy on developing an organizational culture in which protocols and routines systematically guided how teachers interacted around formative and summative assessment data. Sharon felt that using protocols allowed for a safer space for learning, which aligned with her belief that schools should be places where students and teachers felt cared about. Her latest such effort, a standards-based tool that she used to collect observational data when she visited classrooms, had led some members of her staff to joke that she was officially protocol-obsessed. Despite the teasing, Sharon saw real growth among some of the teachers. Yet, a handful of teachers participated reluctantly in school-wide efforts to use evidence or research to improve instruction; there remained pockets of an old culture that resisted change and refused to see evidence as relevant to planning for instruction.
From administrative meetings held at the Greater Oakdale central office, Sharon knew of a few fellow principals who instrumentally used the state accountability formula’s student subgroups and proficiency categories to focus additional interventions and other instructional resources on key “bubble” students. Sharon saw this “gaming” approach as unfairly “rationing” students’ access to the responsive instruction all students deserved. Rather, she preferred to develop a comprehensive strategy and used accountability policy as only one of several frames for how she and Rosemont teachers should interact with data. Still, at times it very much felt like decisions about the future of schools in the Greater Oakdale District privileged this calculated approach.
An Alternative, Community-Driven Accountability Metric: The Great Schools Oakdale Scorecard
It was during a district-wide principals meeting that Sharon learned about a new, alternative, community-based accountability system being designed by Great Schools Oakdale (GSO), a local coalition of education, civic, community, and philanthropic leaders 2 . This group believed that standardized state test scores did not provide good enough or useful information for parents concerned about school quality. In response, the GSO created a “Scorecard” that would measure school climate and student’ academic growth in ways that could be compared across regular public, charter, and private schools alike. Unlike reporting under the state accountability system, the new system would rate schools on an A to F scale based on three elements:
Academic status (50%): Students’ proficiency rates on the most recent state standardized assessments
Progress (20%): Students’ proficiency rates over the past 3 years on state standardized assessments
School climate (30%): Site visits (conducted by community members and assessed using a school climate rubric) and results from the 5Essentials survey from UChicago Impact that measured school capacity along five dimensions (a) effective leaders, (b) collaborative teachers, (c) involved families, (d) supportive environment, and (e) ambitious instruction.
GSO would then distribute annual Scorecards to educators and parents alike using traditional and social media outlets, including a phone app designed specifically for searching the new Scorecards (see the appendix).
Principals were given the choice to participate in this program during its first year. It was clear that more than 70% of schools were going to do it; Sharon volunteered Rosemont because she wanted to help increase access to this alternative, community-driven way of gauging school quality in Greater Oakdale. In a city with regular, charter, and private options that made available three seats for every student, Sharon anticipated that the Scorecard could powerfully shape parents’ decisions about where to send their children. Sharon was enthusiastic about the Scorecard’s potential to reframe school quality as more than test scores, but she was also curious about the metrics’ validity and how they might be used.
As Sharon saw it, the school climate metric offered the most promise to help direct change at Rosemont. This metric would summarize data relevant to dynamics she saw as central to keeping her students engaged and in school including high expectations and safe and caring learning environments. For the first time, she would formally be held accountable for this aspect of her leadership work. She also hoped to use these data to reinforce the importance of Rosemont’s positive and much improved school climate to current and prospective parents as well as district leaders. Sharon volunteered to have Rosemont participate in the process. She had questions about how consumers of the new Scorecard would make sense of the quality of a school that received disparate grades across the core elements. She was not sure how she herself would judge a school receiving an A for strong academic progress and an F for weak school climate. But nevertheless, she was confident that the new, community-driven accountability metrics would help make Rosemont’s strengths more visible to parents as well as to the Greater Oakdale school board and leadership team members charged with closing schools.
Rosemont’s Results
Rosemont’s overall Scorecard grade was a C+, which placed it in a small group of schools that the GSO “recommended” to parents. Despite Sharon’s reservations about the rubric and 2-hr walkthrough process, Rosemont received an A on the school climate element. Qualitative feedback from the community reviewers suggested that they had been particularly impressed by the college tours that the school hosted for its juniors, the level of students’ engagement in classrooms, and the school’s student-operated recycling program. They also recommended that Rosemont work to foster stronger school-community ties. Sharon felt this recommendation could easily be addressed by making some of their ongoing partnerships more transparent to the site reviewers in the future. But the school had, after all, received an A.
Rosemont received low scores on the other two elements: an F for academic progress and a D for academic status. When Sharon looked closely to understand how academic status was calculated on the Scorecard, she saw that her 11th graders’ poor performance on the ACT College Ready exam had been reported as 0% proficient. Sharon’s recruitment line that Rosemont students were accepted to college seemed less compelling when juxtaposed with this evidence that highlighted that they would likely need remedial coursework once they arrived on respective college campuses. Looking at the Scorecard, Sharon felt unsure about how to build on the work she and her staff had done to increase student engagement and improve campus safety into across-the-board improvement in students’ academic outcomes. Protocols would be insufficient. She was reluctant to go adopt the “rationing” approach of some of her colleagues, but also knew she needed to take steps that more directly improved how the school measured up against its competitors on the new Scorecard.
Using the New Accountability Metrics
For Sharon, using and communicating the new Scorecard presented a leadership challenge that felt different from the traditional state accountability system. Explaining to parents and staff that Rosemont had earned a C+ posed possibly more complicated conversations than advertising 100% college acceptance on the school’s website. She had already answered a few calls from parents who saw the school’s D for academic status and planned to pull their children from Rosemont and enroll them elsewhere. Sharon needed to come up with a plan to use the Scorecard in a way that supported her focus on teaching and learning while also addressing the public’s concerns. She remained convinced that the Scorecard process offered a useful alternative to data traditionally provided by the state and district, and that it was a good way for her and the staff to learn more about Rosemont. But it also demonstrated that too many students still did not reach proficiency expectations. With the Scorecard, leading school improvement efforts and communicating about school quality with teachers and parents had become more clear and more complicated.
Teaching Notes
The narrative outlines a set of dilemmas that arose for one school leader confronted with the implementation of an alternative accountability model in her district. As questions about the efficacy of current forms of test-based accountability to engage deep changes in practice continue to mount, several states and districts are experimenting with alternative accountability models with the expectation that school leaders will be able to translate new metrics into useful knowledge needed to guide organizational and instructional changes in their schools.
Instructors may use this case to motivate aspiring school leaders to explore the nature of accountability metrics as they relate to various roles leaders may play and develop frameworks and collaborative skills needed to navigate seemingly unchartered waters of alternative accountability arrangements. The suggested activities listed below are designed for these possible uses.
Activity 1: Exploring Data-Use Theory
The purpose of this activity is to provide students an opportunity to broaden their understanding of data-use theory and data-driven leadership. The activity provides students with a supplemental reading list as a resource for developing background knowledge with which they can situate Sharon’s dilemmas with the GSO Scorecard and her role as school leader at Rosemont.
Students are to read three of the five articles below.
Firestone, W. A., & Shipps, D. (2005). How do leaders interpret conflicting accountabilities to improve student learning? In W. A. Firestone & C. Riehl (Eds.), A new agenda for research in educational leadership (pp. 81–100). New York, NY: Teachers College Press.
Goren, P. (2012). Data, data, and more data—What’s an educator to do? American Journal of Education, 118(2), 233-237.
Ikemoto, G. S., & Marsh, J. A. (2007). Cutting through the “data-driven” mantra: Different conceptions of data-driven decision making. Yearbook of the National Society for the Study of Education, 106(1), 105-131.
Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Making sense of data-driven decision making in education. RAND Corporation. Retrieved from http://www.rand.org/content/dam/rand/pubs/occasional_papers/2006/RAND_OP170.pdf
Supovitz, J. A., & Klein, V. (2003). Mapping a course for improved student learning: How innovative schools systematically use student performance data to guide improvement. Philadelphia, PA: Consortium for Policy Research in Education.
Recommended discussion and/or writing prompts:
What do the authors suggest are the essential components of data-use theory and data-driven leadership?
What have been the conditions in which data-driven leadership has been effective? What conditions hinder its effectiveness?
What problems are solved by data-driven leadership? What problems are created?
Based on your interpretation of Sharon’s situation and your answers to the questions above, what recommendations would you make to Sharon, GSO, and the Greater Oakdale superintendent?
Activity 2: Dimensions of Data-Use Interventions and Their Relations to Outcomes
The purpose of this activity is for students to consider critically key dimensions of data-use interventions and their implications for the work of school leaders and improved outcomes. Drawing on a framework proposed by Cynthia Coburn and Erica Turner (2011), students are asked to consider lessons from research as they relate to different actors in the case. This activity is useful for developing foundational knowledge on how the connections between school context, interpretive processes, and outcomes relate to one another (Coburn & Turner, 2011). This activity will spotlight for graduate students the complexity of the linkages between interventions and outcomes.
Students are to read the following:
Coburn, C. E., Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9(4), 173-206.
Discussion and/or writing prompts that could be assigned to small groups or individual students include
How would you characterize the data-use processes at Rosemont? Given your description and the lessons that Coburn & Turner (2011) outline, what do you believe should be Sharon’s priorities in responding to the Scorecard? Using these priorities as a starting point, what resources are needed to actualize these priorities at Rosemont? Which resources currently reside at Rosemont, and which would need to be acquired? Explain how you anticipate this strategy will influence outcomes.
How would you characterize the aims of the Great Schools Oakdale (GSO) Scorecard initiative? What outcomes does the intervention seek? What mechanisms (i.e., incentives) does it offer to reach those outcomes?
Drawing on the framework presented by Coburn and Turner (2011) and what you know of Sharon’s experience with the Scorecard, what recommendations would you make to GSO to revise the Scorecard for Year 2 of implementation? Provide support for your recommendations by drawing on Sharon’s experience and the framework.
Extension: Building on the responses students provide to Questions 1 and 2, the following questions could be used as discussion or writing prompts:
If the Scorecard were revised using the suggestions presented in response to #2, how would the revised Scorecard influence Sharon’s priorities?
What linkages would exist between the intervention and possible outcomes?
What scenarios lead to the best outcomes for the students are Rosemont?
Given emerging understandings from the activity, what can we conclude about the conditions that strengthen the causal links between an intervention and the outcomes it seeks, and what poses potential threats?
Activity 3: Discussing Community-Driven Accountability Metrics at Rosemont
This activity is a role-play in which students take on the role of Sharon as she talks with her leadership team about how to speak to her faculty and staff about the Scorecard and its implications for their work together at Rosemont. After reading the case narrative, students can begin the role-play directly, or they may read the supplemental reading materials to support them as they design an agenda for a discussion with Rosemont’s faculty and staff. As part of this activity, students should collaboratively prepare a slide deck that outlines what Sharon will discuss with her staff.
Supplemental readings:
Wager, T., & Kegan, R. (2006). Change leadership: A practical guide to transforming our schools. San Francisco, CA: Jossey-Bass.
McDonald, J. P., Mohr, N., Dichter, A., & McDonald, E. C. (2003). The power of protocols: An educator’s guide to better practice. New York, NY: Teachers College Press.
Footnotes
Appendix
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.