Abstract
High-quality teachers are a key ingredient for school effectiveness, and effective hiring practices are an important avenue for ensuring schools are staffed with high-quality teachers. The recognition that teachers are the most important school factor in explaining student learning gains has led to interest in whether principals can identify effective teachers in the hiring process, how principals make hiring decisions, and the strategic management of human capital (Harris, Rutledge, Ingle, & Thompson, 2010; Liu, Rosenstein, Swan, & Khalil, 2008; Odden & Kelley, 2008; Rockoff, Jacob, Kane, & Staiger, 2011; Rutledge, Harris, Thompson, & Ingle, 2008). Schools that are more successful at attracting and hiring effective teachers have steeper trajectories of school-wide achievement growth (Loeb, Kalogrides, & Béteille, 2012). Principals who spend more time on and are more effective at organizational management tasks—including hiring teachers—see greater gains in student achievement and more satisfied teachers (Grissom & Loeb, 2011; Horng, Klasik, & Loeb, 2010). This focus on helping principals identify effective teachers is perhaps best exemplified in recent efforts to develop new measures of teacher effectiveness—such as teacher value-added or growth scores and evidence-based observations (Measures of Effective Teaching Project, 2013). These new measures have the potential to influence how and why principals hire teachers as they provide new and richer information about candidates to a traditionally information-poor process (Liu & Johnson, 2006).
This article uses data from eight school systems that are early adopters of teacher effectiveness measures and enhanced data systems to examine hiring processes. This article addresses the following research questions:
How do principals use new teacher effectiveness data in hiring teachers for their schools?
How are district/charter management organization (CMO) central offices organizing themselves to support information-rich hiring?
What supports and constraints do principals face in using teacher effectiveness data in hiring decisions?
The article begins with outlining what we know about how principals make teacher hiring decisions and the potential role of the central office in mediating how new effectiveness measures may influence hiring. After describing the data and methods, we present the main findings that there is variation in how and the extent to which principals use these measures in hiring. First, we describe the patterns of principal data use in hiring by presenting three ideal types of high, moderate, and low data-use principals. Second, we describe central office practices that mediated how principals approached teacher effectiveness data in the hiring process and how principal use of data varied by the structures put in place by the central office. Third, we describe individual sources of variation among principals’ data use, such as principal knowledge and skills, perceived validity of data, and amount of social capital. Throughout this article, we use the term teacher effectiveness data to include multiple measures of teacher performance, including value-added or other growth scores, teacher observation ratings using an evidence-based rubric, and student surveys. All the school systems in this study have teacher evaluation systems that include multiple measures.
What We Know About How Principals Make Hiring Decisions
Despite the recent attention to the strategic management of human capital, research on how principals make hiring decisions is still accumulating. Previous research suggests that principals focus heavily on personality and behaviors when selecting teachers and want teachers who are caring, enthusiastic, motivated, honest, and emotionally stable (Cain-Caston, 1999; Cannata & Engel, 2012; Dunton, 2001; Engel, 2013; Harris et al., 2010; Place & Kowalski, 1993). Furthermore, districts rate teacher applicants’ human relations skills more important than their teaching skills (Ralph, Kesten, Lang, & Smith, 1998). Indeed, even when districts focus on teaching skills, they consider the ability to facilitate relationships more important than instructional planning and strategies (Ralph et al., 1998).
There is not much evidence on the importance of more direct indicators of teaching performance, probably because historically these types of data have been limited. There is some evidence on principal preferences for particular teacher characteristics that are tied to teacher performance, even if they are not a direct measure of effectiveness. Principals appear to highly value evaluations from applicants’ prior teaching experiences (including student teaching), knowledge of teaching strategies, and certification path (Abernathy, Forsyth, & Mitchell, 2001; Boyd, Lankford, Loeb, Ronfeldt, & Wyckoff, 2010; Cain-Caston, 1999; Dunton, 2001; Papa & Baxter, 2008). For example, teachers are highly likely to have their first teaching job in the district where they student taught (Krieg, Theobald, & Goldhaber, 2016). There is some emerging evidence that principals prefer teachers who demonstrate evidence of improving student achievement (Boyd et al., 2010; Cannata & Engel, 2012), although other research suggests that few principals report looking at test scores of teacher applicants’ students (Rutledge, Harris, & Ingle, 2010). In general, teacher qualifications such as certification and academic credentials appear less important than personal characteristics (Harris et al., 2010; Ralph et al., 1998). For example, principals do not prefer teachers from highly competitive colleges (Ballou, 1996; Boyd et al., 2010), possibly because they assume that all college graduates meet a minimum threshold of intelligence (Harris et al., 2010). Although the relatively weaker emphasis on teacher performance indicators may seem troubling, the complex nature of teaching focuses principals on identifying candidates who have a mix of personal and professional characteristics that fit well into the existing context of their schools (Harris et al., 2010; Papa & Baxter, 2008).
Although there is less evidence regarding information sources or tools that principals use to make hiring decisions, evidence suggests that it is a rushed and information-poor process, giving principals little information on which to base their decisions (Levin & Quinn, 2003; Liu & Johnson, 2006). For example, Liu and Johnson (2006) report that some new teachers were offered jobs immediately after interviewing with the principal, with only a paper review of their credentials. Furthermore, this study found that while nearly all candidates submitted a resume and references, less than a quarter of candidates had to submit more direct evidence of teacher effectiveness, such as standardized test scores, lesson plans, or sample lessons. Other studies support this finding, suggesting that most districts spend less than 2 hours with applicants before hiring them (Strauss, Bowes, Marks, & Plesko, 2000).
The information-poor nature of the hiring process is likely due to constraints faced by principals when hiring teachers. Many applicants are new teachers and have not accumulated any evidence of their teaching performance. Demonstration lessons or videos could help fill this gap, although they come with substantial costs and other barriers (Rutledge et al., 2008). With limited tools available, principals must rely on qualities that are easily assessed in an interview or paper credentials (Delli & Vera, 2003; Engel, 2013; Liu & Johnson, 2006; Rutledge et al., 2008). Consequently, while principals may want to assess teacher effectiveness directly, it is hard to assess knowledge or pedagogical skills in an interview (Engel, 2013). Commercial teacher screening instruments such as the TeacherInsight and Star Teacher screener, are tools used to identify teachers likely to be successful (Haberman & Post, 1998; Rutledge et al., 2008). However, there is little independent empirical research on how well these tools predict teacher effectiveness, with some evidence the screening tool is only weakly related to performance (Rockoff et al., 2011).
Another key constraint faced by principals in hiring is the context of their school in the local teacher labor market (Donaldson, 2013). DeArmond, Gross, and Goldhaber (2010) found that hiring outcomes were more closely related to location, student poverty, and financial resources available to the school than how actively the principal recruited teachers or implemented strong hiring practices. With substantial research indicating that teachers care about the location and student population served by the school, principal hiring decisions are constrained by teachers’ decisions about where they will apply (Boyd, Lankford, Loeb, & Wyckoff, 2005; Cannata, 2010; Engel, Jacob, & Curran, 2013). Other constraints faced by principals include processes established by the district around internal transfers, hiring windows, late vacancy notifications, and budgetary cuts (Levin, Mulhern, & Schunck, 2005; Levin & Quinn, 2003; Liu et al., 2008).
Responsibility for decision making in hiring is traditionally shared by individuals at the district and school levels (Engel & Cannata, 2015; Liu et al., 2008). Many human capital decisions—such as recruitment and basic screening—are centralized by the district and often driven by collective bargaining agreements (Cohen-Vogel & Osborne-Lampkin, 2007; Liu & Johnson, 2006; Strauss et al., 2000; Strunk & Grissom, 2010). Principals perceive the excessive centralization of the hiring process and bureaucratic requirements—such as needing to interview all candidates who fall into a particular eligibility pool or rules around seniority and internal transfers—as significant barriers to their hiring (Donaldson, 2013). One study of hiring in both districts and CMOs found that the central offices across these organizations both supported and constrained principal autonomy (Jabbar, 2016). Teacher hiring is thus an activity that has many functions centralized by the district office, although the degree of centralization can vary by district (Donaldson, 2013; Liu & Johnson, 2006) and there is a trend toward greater decentralization and principal autonomy in hiring, particularly for urban districts and those with collective bargaining agreements (Engel, Cannata, & Curran, 2015). This centralization in teacher hiring has important implications for how principals hire teachers as the organizational and situational context of the school—including district hiring procedures—were relatively large predictors of how principals made hiring decisions (Papa & Baxter, 2008). Indeed, the social dynamics of districts, including how human resource policies are aligned and teacher quality is defined, influence hiring decisions (Heneman & Milanowski, 2004; Pogodzinski, 2016).
Schools Are Becoming Increasingly Data Rich
That the teacher hiring process is information-poor stands in notable contrast to the increasingly data-rich environments of schools (Anderson, Leithwood, & Strauss, 2010; Cohen-Vogel & Harrison, 2013). With accountability rewards and/or sanctions tied to student performance on state assessments, educators have incentives to engage in data-driven decision making. Indeed, principals are charged with creating a “culture of data use” in which educators value data and can explain decisions using data (Halverson, Grigg, Prichett, & Thomas, 2007; Sutherland, 2004). Many instructional decisions—such as which instructional materials to purchase, assignment of students to various instructional interventions, and how teachers allocate instructional time—are increasingly informed by various types of data (Diamond, Randolph, & Spillane, 2004; Guskey, 2003; Halverson et al., 2007; Lyons & Algozzine, 2006).
More recently, the types of data available for hiring decisions have changed significantly. With federal policy incentivizing the creation of comprehensive teacher evaluation systems, and greater recognition of the challenges of traditional teacher evaluation, the nature of data that are available has become more teacher-focused and more detailed with observation scores, measures of teacher impact on student growth, and other teacher-level indicators (Donaldson & Papay, 2015). The data available in new teacher evaluation systems, and the common language provided by the evaluations themselves, have the potential to improve teacher performance (Donaldson & Papay, 2014; Kraft & Gilmour, 2016).
Although these reforms are relatively new, there is emerging evidence that hiring decisions are increasingly informed by student test scores (Cannata & Engel, 2012; Cohen-Vogel, 2011; Cohen-Vogel, Little, & Fierro, 2015). For example, one study found that half of the principals were either asking experienced teaching applicants to provide data from their past students or asked about student results when checking references (Cohen-Vogel, 2011). Two studies found that about 40% of principals weigh student achievement data or the ability to produce gains in student achievement heavily in their hiring decisions (Cannata & Engel, 2012; Cohen-Vogel et al., 2015). These studies examine how principals use student achievement data, however, not teacher evaluation data, which is the focus of the current study. While there is no current research on how principals use teacher evaluation in hiring, evidence suggests that teacher evaluation data are used in other personnel decisions, such as teacher assignment, support and professional development, dismissal, and promotion (Drake et al., 2016; Goldring et al., 2015; Master, 2014)
The existing literature suggests that teacher hiring generally remains an information-poor process, with little evidence on whether principals use direct evidence of teacher performance, even as schools become more data-rich and new measures of teacher effectiveness are developed. It is within this context of schools as increasingly data-rich environments and greater availability of teacher effectiveness data that this study is situated. By examining how principals use new teacher effectiveness measures in hiring, this article contributes to the research base on the extent to which principals use evidence in teacher hiring, and the roles of central office and individual characteristics that shape hiring practices.
Conceptual Framework: Social Construction and Sensemaking
The related traditions of social construction and sensemaking offer a framework for considering individual and system-level variation in whether and how principals use teacher effectiveness data in hiring. Focusing first on individual characteristics, principals bring personal and professional experiences and expertise, which mediate their actions. Social constructionists emphasize that the social world is the byproduct of social interaction (Searle, 1995). Reality emerges through shared interactions and negotiated definitions (Berger & Luckmann, 1966), and this reality is often mediated through social institutions (Gamson, Croteau, Hoynes, & Sasson, 1992; Gamson & Modigliani, 1989; Searle, 1995). Merton (1948, 1995) emphasized that people’s interpretations of situations are shaped by experience and such beliefs shape actions. Furthermore, an individual’s identity, both in how they see themselves and the world around them, is also a product of the social world. Individuals adjust their behaviors according to institutional expectations and rules placed on them, as well as the responses they received from others (Goffman, 1961). Factors such as principals’ social capital (Coleman, 1988) and their perceived validity of data and its appropriate use are mediated by these organizational and policy environments. Furthermore, principals’ positions within social networks, shaped and bounded by organizations, can also mediate their actions (Borgatti & Ofem, 2010). Prior research has found that formal and informal connections within networks shape how principals use data and their ideas about instructional leadership (Daly, 2012; Rigby, 2016).
Work on sensemaking builds on these traditions, arguing that the meaning individuals attribute to actions, messages, and their environments is negotiated according to experiences, attributions of motivation, perceived legitimacy of requests made on them, and so forth. For example, scholars have applied the sensemaking framework to teachers in order to understand their willingness to implement and adhere to programs with fidelity (Coburn, 2006). Teachers are social actors constantly interpreting and reinterpreting their environment and what they are asked to do (Coburn, 2001). They engage in processes whereby they interpret the merits of programs through different lenses colored by aspects such as past experiences and notions of the meaning of good teaching (Allen & Penuel, 2015; Coburn, 2001, 2005, 2006). Importantly, sensemaking is central to the process by which individuals try to organize the unknown (Waterman, 1992). For example, the availability of new forms of data, some of which are in the form of complicated value-added scores, asks principals to work in new and unfamiliar ways. Principals must balance professional judgment, unclear and at times conflicting data, or even a lack of data in the case of new teachers to make hiring decisions.
Sensemaking processes are also influenced by organizational environments. Faced with uncertainty, principals’ actions can both influence and be influenced by their organizational milieu (Weick, 1995). We examine the district environment in which principals operate, and whether districts influence principals’ use of measures of teacher effectiveness in the hiring process. This is especially important as research most commonly focuses on schools as the locus of change; less attention has been paid to district-level influences (Honig, 2012; Honig & Venkateswaran, 2012; Rorrer, Skrla, & Scheurich, 2008). However, just as principals operate within the organizational confines of the school environment, they also are influenced by district context. Districts can help disentangle the unknown, guide the sensemaking process, and the construction of principals’ understanding of the world around them. This in turn can influence principals’ actions and help them learn to disentangle the confusing web of information newly available to them (Coburn & Talbert, 2006; Spillane & Burch, 2006; Spillane et al., 2002).
The lack of attention afforded to the mediating roles of districts likely obscures factors influencing the implementation of policies and programs. For instance, when districts lack a coherent vision of what reform should entail, they often have difficulty bringing such efforts to scale (Corcoran, Fuhrman, & Belcher, 2001). In a review of the literature on the district role in reform efforts, Rorrer et al. (2008) identified four overarching functions: (1) the provision of instructional leadership, (2) reorientation of the organization to align with desired changes, (3) the creation of overall policy coherence, and (4) maintaining a focus on equity. Specific examples demonstrating the influence of district context can be found in efforts such as the development of small autonomous schools (Honig, 2009), the implementation of professional learning communities (Honig & Rainey, 2014), whole-school reform efforts (Berends, Bodilly, & Kirby, 2002), as well as school restructuring efforts (Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010).
Like in other efforts to foster changes in schools, considerable research emphasizes the role districts have in shaping how principals interpret and use data (Farrell, 2015; Marsh, 2012; Moss, 2012). Based on an extensive review of research on the influence districts have with regard to what they term evidence-use processes, Honig and Venkateswaren (2012) argue that focusing only on school-level (i.e., principal-level) data use excludes an important component when trying to understand data-driven decision making. For instance, districts can set organizational routines and expectations around data use (Grissom et al., 2015; Spillane, 2012; Wohlstetter, Datnow, & Park, 2008), and carve out time for leaders to learn to make sense of and apply data (Halverson et al., 2007; Marsh, 2012; Park & Datnow, 2009).
Honig and Venkateswaren (2012) categorize the influence of districts in school-based evidence use into four overarching categories. First, districts can be conduits in the flow of information to schools. Second, they facilitate sensemaking processes surrounding data use among various school stakeholders. Third, they create and communicate expectations, processes, and procedures around data use, and, fourth, they are instrumental in providing necessary professional development for the use of such data. Honig and Venkateswaren (2012) conclude with a call for future research to account for and try to disentangle the complexities involved in this interplay between school- and district-level contexts in the use of data, noting that different decisions might involve different logics and procedures. This article answers this call by situating patterns of individual-level data use in the hiring process within the context of school district factors that shape data use.
In addition to shaping data-use practices, research on principal human resource decisions suggests that the social and organizational dynamics of districts shape principal decisions and enactment of teacher evaluation. For example, district leaders play a role in shaping how principals interpret state policy, such as whether the purpose of the teacher evaluation system is framed as accountability or improvement (Woulfin, Donaldson, & Gonzales, 2016). District leaders also influence how principals define teacher quality when making hiring and other human resource decisions (Heneman & Milanowski, 2004; Rutledge et al., 2010). Furthermore, social relationships within a school and social dynamics within a district influence human resource decisions (Pogodzinski, 2016).
This article contributes to this research by exploring how central office expectations for hiring mediate how principals make sense of new teacher effectiveness data and use them in hiring new teachers. In addition, it highlights individual-level mediators that influence variability in data use, calling attention to the importance of skills and expertise, past experiences, perceived validity, and social interaction, all of which influence principals’ actions as they actively make meaning of the new data sources and reconstruct their hiring practices to incorporate new information available to them.
Data and Methods
As part of a larger project examining principal data use for human capital decision making, we conducted semistructured interviews of central office personnel and principals in six urban school districts and two CMOs during the 2012-2013 school year. The systems had all been implementing a new system for collecting multiple measures of teacher effectiveness for at least 1 year at the time data collection began, establishing them as early adopters of comprehensive teacher evaluation systems. These new comprehensive teacher evaluation systems include teacher observations, value-added or similar growth measures, and student perception surveys. Some of the systems also had peer surveys from other teachers, parent surveys, and other measures of student learning. The systems varied in how these components were weighted, but the observation measures had weights from 25% to 50% of the total evaluation composite rating. All the systems devoted time and resources to implementing their evaluation system and developing data systems that incorporated these measures of teacher effectiveness. The systems varied in the length of time they had been engaged in these efforts and whether they received substantial philanthropic or other external support, such as teacher incentive funds. The eight systems are located in six states, four of which received Race to the Top funds. Principals across all systems reported being held accountable for the performance of their schools on state assessments. The systems varied in size; the two CMOs both served around 5,000 students and the districts ranged from 75,000 to 200,000 students.
Participating school systems had a variety of collective bargaining arrangements, although they were not selected on that basis. Three of the school districts were in states where collective bargaining is illegal. These districts did have teacher associations that were affiliated with unions, but they did not have collective bargaining power. Two districts were in states where collective bargaining was permitted and one district was in a state where collective bargaining was a right guaranteed by the state constitution. These districts had collective bargaining agreements. Among the two CMOs, one had a collective bargaining agreement with a teacher union and one did not. Thus, across the eight systems, there was variation in how strong the union presence was felt.
The collection of interview data occurred in two stages. First, we conducted semistructured interviews with central office personnel in each system. Key system personnel, including the superintendent/president, director of human resources, director of research and accountability, and director of curriculum and/or professional development, were interviewed to examine the types of teacher quality and effectiveness data available to principals, as well as system-level expectations for data utilization for teacher human capital decisions. Other system personnel identified by individuals within each district/CMO were also interviewed. In total, we performed more than 100 interviews with central office staff across the eight systems. Interview guides asked about the district culture around data use, supports to principals for using data, and the formats or tools by which data were accessed. In addition, questions focused on specific human capital decisions, including hiring, evaluation, teacher support and professional development, compensation, assignment, teacher leadership, and contract renewal/dismissal. Because of the length of the interview guide and different roles held by central office personnel, not all participants were asked about all human capital areas. All participants were asked about district culture and support for data use. Job description and self-identified responsibilities guided the specific human capital areas we focused on with each respondent. The central office interview guide, and all other instruments, can be found in the online Appendix.
In the second stage, schools within each system were stratified by level (e.g., elementary, middle, high) and achievement (e.g., low, high), with four elementary, three middle, and three high school principals randomly selected for interview from within achievement stratum. 1 Interviews with principals were semistructured and reflected the insights gained from the central office interviews. A total of 76 principals were interviewed across the eight systems. Principals were asked about all human capital decisions and questions were focused on how they used data, who else was involved in their use of data, the driving factors behind how they used data, the barriers they encountered in using data for human capital decisions, the training and support received for using data, timing of data availability, the role of the central office in their data use, and perceptions of what distinguished the most and least effective users of teacher effectiveness data.
All interviews were audio recorded and transcribed verbatim. The analysis protocol, which stemmed from the original research proposal, guided development of an initial coding scheme. 2 All data were coded by human capital decision and the type of data used. The process, however, was also iterative in nature (Corbin & Strauss, 2008; Le Compte & Schensul, 1999); members of the research team coded a sample of central office and principal interviews, and then revised the coding scheme to address questions and concerns that emerged. Researchers compared their coding to ensure consistency in application of codes. Throughout the coding process, researchers wrote in-depth memos to capture nuance as well as ideas that were not always captured in the coding scheme. These memos were considered alongside the coded data during synthesis of findings (Corbin & Strauss, 2008).
Thematic coding led to the emergence of patterns within and across districts. For example, themes such as barriers to and facilitators of data use, and beliefs about the validity and appropriateness of data for particular decisions emerged as important topics. We used these data to construct the three ideal types of data users. That is, usage patterns and practices emerged among principals across the districts, which were rather distinct and led to the construction of these typologies. We provide quotes from participants in order to provide rich descriptions of themes related to teacher hiring that emerged across all systems to varying degrees. To explore how central office practices around hiring influence how principals use data, we used the interview data to categorize the systems into one of two categories: “high structure” and “low structure” in terms of the practices established which structured the hiring process.
First, we wrote comprehensive analyses for each system that summarized how principals used data for each human capital area and the supports and barriers present in each system. Second, using these reports, we summarized what each system did to structure the hiring process and incorporate teacher effectiveness data into hiring. Working inductively from the rich descriptions in the comprehensive reports, the central office practices clustered into three domain areas: recruitment/screening, expectations around how principals hire, and accountability for hiring. When reviewing the system summaries outlined in those three domain areas, there was a clear pattern of systems that were actively creating structures for incorporating teacher effectiveness data into hiring across each of those areas and those that were not. Four of the six systems that participated in the survey engaged in significant screening processes, provided strong messages to principals about how they should hire teachers, and held principals accountable for hiring decisions. They were classified as “high structure” systems. The other two systems were categorized as “low structure” systems. This categorical variable was used to explore differences in patterns of principal data use in hiring across different types of school systems. We note that this categorization of school systems into “high structure” and “low structure” does not necessarily reflect the degree of overall centralization in the hiring process. For example, one of the systems worked with principals to post school-specific vacancies, and teachers were not offered jobs until hired by a particular school. Thus, from the teacher’s point of view, hiring was highly decentralized as they applied to a particular school and were offered a position by that school (Liu & Johnson, 2006). Yet this system was considered “high structure” because applications were rated according a rubric aligned with the teacher evaluation system before principals interviewed them, principals were given clear expectations on how to incorporate teacher effectiveness data into hiring, and principals reported being held accountable for their hiring decisions. Likewise, one of the “low structure” systems was categorized as such because they had few practices that encouraged principals to use data in hiring, yet they still had centralized rules around internal transfers and hiring windows.
In addition to the fieldwork, we also conducted principal surveys in six of the eight systems. Online surveys were distributed to all principals by email between September and November 2013. Principal emails were obtained from systems’ central offices. Email reminders to complete the survey were sent once per week to nonrespondents for 3 to 4 weeks after the initial survey invitation was sent. Hard copies of the survey were mailed to nonrespondents approximately 1 to 2 weeks after they received their last email reminder. A total of 795 principals responded to the survey, with an overall response rate of 85%. Response rates by system ranged from 73% to 92%. The systems varied in size from 19 schools to 268 schools. The survey data presented here are summary statistics of the percentage of principals who used various forms of data for hiring. The categorization of school systems into high-structure and low-structure systems from the qualitative data was added to the survey data and used to explore the relationship between district processes related to using teacher effectiveness in hiring and principal use of data. Specifically, chi-square tests were used to test for differences between the two categories of school systems.
Findings
How Are Principals Using Data?
To answer the first research question that focuses on how principals use data for hiring, the survey data provide evidence about the extent to which principals access and use different forms of teacher effectiveness data in their hiring decisions. The survey asked about use of data in hiring three ways. First, we asked if principals had access to three forms of teacher effectiveness data (observation scores, measures of student achievement growth, and overall evaluation scores) for candidates applying to transfer into their school. Second, we asked if they used these measures when hiring transfer applicants, either via access through the central office or by requiring candidates to provide them. Third, we asked principals to weight the importance of various factors in their hiring decisions, assuming they had access to all forms of data. The factors included: teacher observation scores, the overall evaluation score, measures of student achievement growth, direct observation of instruction, their own professional judgment on the candidate’s effectiveness, their own professional judgment on the candidate’s ability to improve, and recommendations.
Table 1 provides evidence on whether principals had access to teacher effectiveness measures and whether they used them in hiring transfer applicants. About one third of principals across the six systems indicated that they had access to each of these forms of teacher effectiveness data from their central office. It is notable that even within a system, principals did not agree that they had access to these data for transfer applicants. Furthermore, about two thirds of principals reported using each of these forms of data for hiring transfer applicants. The fact that more principals reported using the data than reported having access to it from the central office suggests that about a third of principals were requesting candidates provide these data. Principals were slightly more likely to report using teacher observation scores (71%) than measures of achievement growth (62%).
Principal Access to and Use of Teacher Effectiveness Measures in Hiring Transfer Applicants.
Note. N = 685 for whether they have or use data on the teacher’s observation rating and measure of achievement growth of the teacher’s students in prior years. N = 682 for whether they have or use data on the teacher’s overall evaluation rating.
To further explore how principals use data, the first column of Table 2 provides evidence on how much weight principals put in various factors when hiring teachers. The table distinguishes between traditional measures available in hiring, such as professional judgement, recommendations, and direct observations, from newer teacher effectiveness measures. The evidence indicates that principals were most likely to rate measures of student achievement growth and their professional judgement of the teacher’s potential effectiveness as very important when hiring teachers (72% and 70%, respectively). Comparing Table 1 with the first column of Table 2, more principals reported that measures of student achievement growth were very important in hiring than those who said they used it, which may reflect difficulties in accessing such data for all teachers.
Percentage of Principals Who Rated Various Factors as Very Important to Their Hiring Decisions.
Note. The number of observations varies slightly due to nonresponse. N = 578 to 582 for high-structure systems and N = 119 to 121 for low-structure systems.
p < .05. **p < .01. ***p < .001 (p values at which low-structure systems are significantly different from high-structure systems).
With this broad understanding of the extent to which principals use data in hiring, the qualitative data uncovered substantial variation among principals in how they use data in hiring.
To describe how principals use teacher effectiveness data in hiring, we use the heuristic tool of ideal types (Weber, Mills, & Gerth, 1947; Weber, Shils, & Finch, 1949), useful for making comparisons between categories. The ideal types, which emerged from the data allow for the depiction of principals’ various uses of teacher effectiveness data. The word ideal does not connote a value judgment. Rather, ideal types are tools to articulate pure forms of any given phenomena, even though it is unlikely such a pure form could ever exist in the social world. Therefore, we did not categorize every principal into one of these ideal types; these categories illustrate the differences in types of data users that were present in the data. That said, most of the evidence on how principals use data in hiring reflected the categories of moderate- to low-use data users in hiring—relatively less evidence reflected the high-end user.
There are certain practices that are common among all groups, and serve as the backdrop for traditional hiring—for example, reference checks, interviews with the principal or a hiring committee, and reliance on the professional judgment of the principal. We acknowledge that all these practices provide principals with information that can help guide successful hiring decisions. The common practices, however, are interwoven with use of other forms of data to varying degrees, and these differences are what differentiate principals in terms of their level of data use, and ultimately distinguished high-use principals making data-rich decisions from others. The quotes we provide when we describe each of these three categories of data users are illustrative of the characteristics we ascribe to each of the ideal types.
High-use principals
High-use principals consistently incorporate teacher effectiveness data into their hiring processes. When they are able to access data such as student-growth indicators or composite teacher evaluation scores they do so. For principals considering within-district transfers, they reach out to their central offices when the data are not available to them via a data dashboard. As one principal described, “if I’m looking for a particular data point, it’s a phone call away. I can talk to [the central office] and have that within, you know, a couple days at the most” (District G). When these data are not available by either method, or the applicant is new to the system, these principals ask applicants to bring prior observations and/or evidence of student achievement with them to interviews, or to submit such information beforehand, so they may review data prior to the actual interview. One principal explained,
I have one teacher who chose my school in the [hiring] pool, and when he came to see me, I said, “Oh, here, sit down at my computer and pull up your observations,” because I can’t do that unless they’re at my school. He said, “Oh, I don’t think I can do that.” I said, “Oh, yes, you can. I’ll help you.” So he did . . . I printed out his summary report. (District A)
Ultimately, while principals may rely on different types of data, or give more weight to some forms over others, high-use principals consistently engage in the process of gathering and reviewing teacher effectiveness data as they consider hiring decisions. Take for example this principal:
Now that we have the whole [evaluation] rubric and data report, I require teachers to bring their [evaluation] data reports to the interview. And I also ask them to bring [value added] data if they’re teachers in such a grade. . . . I ask teachers to bring their writing scores. And I ask the teachers to bring me a copy of their [students’ state test] scores from the previous year so I can see what their track records are. (District B)
High-use principals are able to weigh different sources of information and understand the nuances of data. They balance data use with professional judgment and make decisions based on the needs of their schools. The principal quoted below demonstrated a keen understanding of the differences between the evaluation scores in their system and engaged in a conversation with the candidates about their value-added scores, as well as how the candidates used data in their own teaching practice. Furthermore, this principal examined strengths and weaknesses of the potential hire according to their evaluations, and weighed those against overall school context:
I ask teachers to bring a resume, and then I also ask them to bring their test scores, because a lot of people can say what they did, but I want to see it on paper, and then I want them to tell me what they did because the new evaluation if you look at it . . . sometimes your scores are good based on someone else’s. . . . I talk to them to ask them to share with me what makes you the perfect person for this job, and then show me your data based on what you did. . . . I don’t only just look at their data. . . . I also look at background information . . . And sometimes I just say how do you use data? I look at where my students are, where [the candidate’s] needs are . . . So I ask them all that to help me with my teacher hiring . . . And that gives me a perspective when I’m hiring. (District C)
Furthermore, high-use principals make use of tools deployed within their own school systems to assess potential hires. This is most evident through the demonstration lesson process. High-end users evaluate demonstration lessons as they would conduct formal evaluations of teachers in their schools—even using the same observation rubric. They follow the observation process, which usually involves a brief conference prior to the lesson, observation of the lesson, a ranking of the lesson according to the domains of the official observation rubric, and then a postlesson conference. We learned that when demonstration lessons were conducted in front of a hiring committee, high-use principals ensured that committee members were familiar with the rubric as well, so that they too could judge lessons using the mechanism by which teachers within the system were evaluated:
There’s a rubric associated to this evaluation when teachers are interviewing . . . you calibrate with your hiring panel . . . we calibrate on our scores . . . So here at the school site, the data that we collect is around different indicators within our [teacher observation] rubric . . . within the 20-minute demo lesson that they do, so checking for understanding, lesson preparation, questioning, teacher feedback, and grouping. So these are different areas that are used within that demo lesson that we’re looking at. So that data helps to identify how well a teacher is preparing the lesson, and how well they actually deliver the lesson and implement strategies for effective teaching with the lesson. (District G)
In the postobservation conference the lesson is discussed, the teacher is asked to reflect on its strengths and weaknesses, and the principal offers feedback. The postobservation conference becomes another mechanism by which the principal evaluates a potential hire’s ability to mesh with school/system culture, and to take advice necessary for growth and success should they be hired. In the following excerpt, a principal summarized their hiring approach, and emphasized the importance of debriefing after the lesson to candidates:
[The demonstration lesson] goes on for an hour. Then we debrief about that and even if the demo lesson goes well, it could still kind of lead to non-hire depending on how the debrief goes. We like to test that too, to see, “Okay, I’ve got to give them some feedback that’s not all positive, and see how they can handle it.” You know, and in doing that, I go like, “I would’ve have liked to have seen this. You did great X, Y, and Z. I would have loved to have seen this part like this.” So it comes out I think kind of nice. I’m not trying to be too critical, but if they’re very combative right then and there, I go, “okay, maybe this isn’t a good fit because we’re going to be doing a lot of this throughout the course of the school year.” (District H)
The demonstration lesson was more than a performance of teaching, and instead offered important information such as planning abilities, pedagogical approaches, interpersonal skills, and the ability to relate to students, as well as the ability to be a reflective practitioner, and to be coached and grow within the structure and culture of the school. What may have been left to professional judgment of whether a candidate was successful in a demonstration lesson became more transparent and open for systematic evaluation.
High-end users continue to rely on standard hiring practices like checking references, but they do this to fill in gaps. The principal quoted below described the data they relied on, but explained that data do not always tell the full story:
As far as hiring is concerned, if I’m interviewing some candidate that is within our county then I certainly will pull up and look at their evaluations or observations and evaluations, just to see . . . what their strengths are, and what they have to bring. Of course, we also want to call and talk to their principal. . . . [There are] a lot of things that the evaluation might not tell us like attendance and their participation in school activities . . . (District A)
Principals still rely on traditional hiring practices, but this is supplemental, rather than the foundation of the hiring decisions.
Moderate-use principals
Moderate-use principals rely on many of the same general practices as high-end users (e.g., interviewing, demonstration lessons), but teacher effectiveness data are infused into the process inconsistently. These principals do not discount this information entirely, but do not seek it out on their own, nor do they require applicants to provide it. When teachers provide data on their own volition, midrange users might use it as an additional piece of information as they judge whether a teacher is an appropriate hire for the school.
While it is common for all types of principals to have candidates perform demonstration lessons if possible, midrange users differ from high-end users, in so far as they do not evaluate these lessons according to their school system’s observation rubrics. One respondent who reflects the moderate-use principal stated,
During the school year we have invited teachers here to teach in a classroom for about an hour, and so typically the assistant principal, myself, and that teacher of that classroom observe them . . . [During observations] I usually look for three broad areas. One would be classroom management, relationships with students. So we look at that both when we observe . . . [another] area would be instructional or curricular knowledge . . . we get a feel for both instructional knowledge, how they teach a lesson beginning to end, as well as understanding literacy skills and strategies, and we ask questions about that as well. And then the other piece would be collaboration, teamwork, how they interact with their team of teachers. (District E)
To learn more about the hiring procedure described above, we asked the principal whether the team used “some kind of tool” to help track a candidate’s performance. The principal responded, “I would say we’re on the same page. Have we had that explicit discussion that these are the three things we are looking for? I don’t know.” Medium-use principals rely on professional judgment, and “knowing what good teaching looks like” to judge demonstration lessons. The evaluation of the observation is far less systematic than in the case of the high–data use principal.
Low-use principals
Though working in districts that have adopted multimeasure evaluation systems, low-use principals have not incorporated teacher effectiveness data into their hiring practices. One principal who we asked about data use in hiring responded by saying,
I have not asked for [data on previous performance]. That would be one that we could possibly use, because that would tell how well they’ve done with the group of students that they had previously. We haven’t used that, but that’s a good one. (District G).
In these instances, the hiring process is dominated by reference checks, interviews, and occasionally, demonstration lessons. When low-end users require demonstration lessons it is because they are most interested in observing how applicants respond to the school’s students and the school environment. They are interested in whether the prospective hire will be able to communicate with, relate to, and/or manage the student population he or she will teach. As this principal explained,
Ideally, I think it would be great if you had a teacher come in and interview in May, and they could go in and teach a class for you. I mean that would be wonderful. But we don’t do that . . . of course you call the principals if they worked. You call their professors if they’re right out of school. You look at their grades. You look at their specialty. . . . I think a big part of being a successful teacher is relating to the kids, so you look at that relationship piece. (District A)
While all three types of principals rely on professional judgment, this becomes the primary source on which principals draw when they do not consult formal data in their hiring practice. The following principal discussed the importance of evaluating candidates’ values as a driver in the hiring process. According to this principal, however, one never really knows if a teacher will be a good hire until they are “in the building.” Observation only happens once a teacher is hired, as opposed to the high-end user who not only observes a demonstration lesson but systematically evaluates it:
One of the first things I look at when I am hiring a teacher is their love of children and love of teaching, and their values, their core values and their beliefs in terms of education, particularly urban education and dealing with a population of students in [this district] . . . So that’s one piece, cause you want to get a good fit. You want someone that believes that all children can learn, and we know they all learn at different rates. But that someone that’s going to set high expectations, expect those students to rise to those expectations, and have a variety of strategies and interventions that they can use to meet those students’ needs. . . . But you never know until you actually have the person here in the building and they actually fulfill the practice of it, and you get an opportunity to observe, coach, and work with those teachers. (District D)
While the principal above strived to hire teachers who “have a variety of strategies and interventions that they can use,” they were unable to evaluate teachers’ actual practice until they were “in the building.” The high-end user has a mechanism, a systematic process of evaluating demonstration lessons, which would help gauge just this.
What is the Central Office Role in Hiring?
The second research question focuses on how district/CMO central offices were organized to support information-rich hiring. Overall, our findings suggest that the variation in how and the extent to which principals use teacher effectiveness measures in hiring is partly explained by central office practices as they mediated how principals approached teacher effectiveness data in the hiring process. This is consistent with a sensemaking framework that emphasizes the importance of institutional contexts and organizational expectations as mediators of individuals’ actions (Goffman, 1961; Honig & Venkateswaran, 2012; Rorrer et al., 2008). This section first uses the interview data to describe three system-wide practices related to teacher hiring that central offices put in place, including how candidates are recruited and screened, expectations for how principals should make hiring decisions, and processes for holding principals accountable for hiring or analyzing hiring decisions. We then analyze the survey data to examine how these institutional arrangements are associated with whether and how principals use data in their hiring processes.
As illustrated below, these practices are related to the overall degree of centralization in hiring, but we saw important distinctions between overall centralization in hiring and central office practices that led to greater use of teacher effectiveness data by principals in their teacher hiring decisions. While none of the systems in this study completely centralized hiring, discussions of centralization in the hiring literature typically focus on how much authority principals have; in completely centralized districts, teachers are assigned to schools with little to no principal input. The practices described here that central offices used to structure principal hiring decisions did not remove their authority over hiring, even as they shaped how those decisions were made.
Recruiting and screening candidates
The eight school systems that were part of this study varied in how they supported schools in recruiting and screening potential teachers. Although recruiting and screening are different processes, they work together to define the applicant pool. For example, one central office hiring official noted,
We go out there and recruit during job fairs. We do the job postings for them. So in essence, we create a pool of candidates for them. They take over at that time for the interview and selection process, and then they come back to us for the onboarding process. (District H)
As reflected in this quote, this system did not take an active role in recruiting teachers other than posting vacancies in a centralized location and representing the system at job fairs. Nor did they do much screening of applicants beyond a criminal background check and basic credential verification. On the other end of the spectrum, one system actively managed its recruitment pipeline and had identified where they were most likely to find teachers who would be effective in their system. For example, one central office hiring official described an active approach to developing their pipeline of teachers:
Are we getting a more effective, higher quality teacher from those pipeline programs? And then are we strategically placing them? So, as we look at [a] teacher’s [evaluation composite] score . . . in using that defined definition which is in the [evaluation composite], what are our partnerships and pipelines looking like, and how do they distribute along when we look at our [evaluation composite] one through five teachers? And where do we need, you know, what subject areas do they teach? How many years have they been working in the district? Where do we need our pipelines to help and to support us? (District B)
More common was for systems to play some role in recruiting candidates, even if they were not as systematic about their recruitment processes as this district.
There was also substantial variation in how central offices screened candidates before hiring moved to the school site. While two systems (systems D and H) conducted very little screening beyond asking for credentials, other systems had extensive screening at the central office level to assess qualifications and fit with the system before candidates could be interviewed at the school. For example, one system established a process of five selection stages—four of which occurred by central office staff. These stages include a commercial screening instrument, credential check, a phone interview and lesson plan that assessed competencies on their teacher evaluations, and another phone interview that assessed their mind-set to work in their urban setting. Only those candidates who successfully passed each stage could be interviewed by a principal; in a recent year, about 20% of all applicants made it to the final stage. As a hiring official noted, “We have a very rigorous hiring process. Candidates are vetted at several stages. Once they pass the vetting from the home office . . . the names are given to the school sites, and each school site forms hiring committees” [District G]. Another district developed a rubric aligned with their teacher evaluation system; all candidates were “prescreened” according to this rubric using traditional application materials. All candidates who passed the prescreen were then rated using a phone interview. Only candidates who did well on these two processes made it into the applicant pool, where principals access candidates’ application materials and the 1 to 5 rating assessed by the central office. One hiring official noted,
We’ve aligned our selection model with teacher evaluation, the teacher effectiveness measure . . . when you apply to the district you have to get through a screen, a prescreen as we call it. So you basically will submit your application, your resume, your references, all of those things, and then we screen that on a scale of 1 to 5 based on a rubric that we’ve created that is aligned with the [evaluation instrument]. . . . After that, it goes to the phone screen. The phone screen is about a 30-minute conversation with candidates to talk, to dive in more about their application and also ask them some scenario questions and also questions around their experience. We’ve found that the most important or the best indicator of teacher effectiveness is past performance, and so we will drill down on that. . . . They get a score on both of those. It’s put together and it’s on a scale of 1 to 5, and then into our applicant [pool]. (District B)
Other systems reported being in the initial stages of developing some type of centralized screening process. In systems with more extensive screening practices, like District B, above, there were processes in place to evaluate new hires who did not have evaluation data. Materials they submitted were still ranked in accordance with the systems’ teacher evaluation frameworks.
These screening processes established by the central offices shaped how principals used data in their hiring decisions. For example, a principal in District B referenced the way that candidates in the applicant pool are ranked and said,
they’re ranked 1,2,3,4, or 5 by the pool that interviewed them to get them into the pool. . . . This person has a rating of a 2, a 4, so if I were looking for a new person, I would choose these 4’s and not the 1’s and 2’s.
Similarly, a principal from District G described how he uses data generated from their central office screening process, by saying,
Some teachers are not greenlighted because of their results. Some teachers are but barely. So we look at the data that comes from home office, and then we adjust our questions and things we’re looking for in the classroom accordingly.
Consistent with a sensemaking framework, structures established by the central office began to shape principal hiring process; since some systems used a more data-driven screening process in the central office, we saw evidence that principals themselves saw a role for data in hiring.
Expectations for principals in teacher hiring
Another key domain that influenced variation in principals’ use of data in teacher hiring was the presence of central office expectations for principals. Systems varied in the extent to which the central office provided clear and consistent messages to principals about how they should make hiring decisions. Some systems provided very little guidance to principals regarding hiring, resulting in substantial variation in how principals within that system made decisions—in terms of both the processes used and criteria considered. For example, when asked about whether the district provided any guidance on how to hire teachers, one principal responded, “It’s pretty much up to my discretion” (District D). A member of the central office from that system confirmed,
in terms of the systemic data . . . schools use whatever they want . . . we don’t centrally say please think about test scores, or please look at your trends over the past four years in terms of new hires. (District D)
Yet other systems had very structured processes, with a common set of interview questions and a rubric for a demonstration lesson—all tied to the indicators on the evaluation system—that principals and their school hiring committees were expected to use. For example, one central office personnel member noted that they put together a “toolkit that walks the principal through the entire process. The most exciting part of it includes examples of a very structured interview process . . . all the questions are competency-based and linked to the [evaluation system]” (District E). This is an example of an organizational routine by which districts can influence principal behavior (Honig & Venkateswaran, 2012; Spillane, 2012).
Another way which some systems set expectations for how principals should use effectiveness data in hiring was by creating a culture of data use and emphasis on teacher effectiveness data. For example, District F did not give principals any explicit directions on how to make hiring decisions, but through the system-wide culture of valuing teacher effectiveness measures—particularly measures of student growth—principals began considering these data when hiring transfer applicants. For example, one principal in this system said, “For teacher hiring we do have the [value-added] data that we can pull on teachers or that are provided to us, or scores on their previous observations . . . so I do look at that information” (District F). Later, this principal reflected on the multiple sources of data she can access and said,
I think it’s more of a maybe too much to access. So I mean there’s a lot of pieces of data. . . . you tend to want to think that if the data is provided to me, it means I’m supposed to do something with it or about it.
One principal, who was a first-year principal and had thus not yet hired anyone, was asked about challenges to using data in hiring and said,
I do not foresee many barriers for teachers that want to transfer in, because I know I have access to that data . . . [this district] is very data-friendly and just open, so barriers with that, I don’t see it being an issue, perhaps with the teacher who comes from a district that has no idea that this is going on, they have to know what they’re coming into . . . and if you don’t like it, please don’t interview at my school because this is what it’s about. (District G)
Accountability in teacher hiring
The final domain in which school systems shaped how principals used teacher effectiveness data in their hiring decisions is the way in which central office personnel held either principals or the hiring process itself accountable for quality decisions. The system that provided the most structure to their hiring process, with an extensive centralized screening process and clear expectations for how principals should review candidates, exemplifies both these ways in which the central office used accountability in teacher hiring. First, this system reviewed principals’ hiring decisions and added an outside member to the school hiring committee if they were concerned the principal was not making quality hires. One central office leader said,
There are a couple of schools that I have required them to make me a part of that process because they just haven’t consistently hired well over the past few years, and so I’m concerned about the decisions they’re making. We’re hiring teachers and then having to put them on plans within a couple of months, and clearly there’s a breakdown in practice somewhere. (District G)
The superintendent in District A reported calling principals to ask about how they hired particular teachers.
Another method of holding principals accountable for hiring decisions was that systems conducted analyses on components of their hiring process to improve the overall process. For example, one system that used a commercial screening instrument for all candidates conducted an analysis that compared teachers’ scores on the commercial screener with their teacher effectiveness data and found that there was not a strong correlation between the two indicators (District C). Other systems with more extensive screening processes also reported initial efforts to examine whether the screening rubrics were related to teacher effectiveness once on the job. This principal described being aware of the findings from these types of central office analyses and adjusting their hiring process by saying,
We’re looking at the bigger trends in like how our newer teacher are successful or not, and kind of how that connects to what skill set we’re trying to identify in that process early on, like if we know that management is the biggest challenge for our newest teachers and that they’re actually relatively adept at lesson planning and developing a lesson cycle and that’s not their area of focus, then that tells me, okay, I want to look for people who have that foundation in classroom management, and we can catch them up a little bit on the instructional. (District G)
Thus, the ways in which central offices use data to examine their hiring processes can shape how principals use data in school-level hiring decisions.
One concern about a more active central office in recruitment and screening, setting expectations for principals, and accountability in teacher hiring is the potential effect on principal autonomy. School systems that took less active roles in these areas noted that they valued principal autonomy in hiring and did not want to create centralized processes that limited principal autonomy. For example, a recommendation by the human capital office in one system for a more active role in candidate selection was rejected by senior leadership because it “clashed with the idea of autonomy, that like centrally we shouldn’t require all candidates to do anything besides be certified and stuff like that. But they should not have to get rated on the same rubric” (District D). Yet systems with more centralized screening processes saw their role not as removing principal autonomy, but as supporting principals in making hiring decisions efficiently. One central office personnel noted,
We get thousands of applicants, and we want to narrow down to really the best ones and then let our principals spend their time on it. It’s not a really great use of their time to be separating the wheat from the chaff. . . . This is a time consuming process per candidate for the principals in the school. I call it expensive in terms of time and personnel hours. (District G)
A principal in this system confirmed that he or she saw the role of the central office as valuable,
I put a lot of trust in the consultants who do interviews on the phone. They had notes. . . . I definitely reviewed it first, saw if there was anything that I saw that could be a red flag for me, chose not to interview certain candidates for specific reasons because their answers weren’t very clear. (District G)
The data also point to distinctions between centralized practices that limit principal autonomy more generally, and practices that facilitate or encourage the use of data in hiring. In the interview data, most statements about lack of autonomy over hiring by principals were related to district practices about surplus pools and other contractual provisions. “I just think the hiring process is what’s limiting us; it’s because of those transfer periods, those pools, those hiring freezes, that’s what more limits us” (District A). Centralized rules that restrict which candidates principals can consider, or force them to hire particular teachers, were criticized for reducing principal autonomy. That is, the practices that principals found most limiting to their autonomy in hiring were less about how the central office screened candidates, or the expectations about how to use teacher effectiveness data in hiring, but about mandates on hiring from a surplus pool or waiting until after a particular deadline for considering external applicants. Consistent with prior literature on principal hiring, these centralized rules prompted principals to try to manipulate the system so they could have more autonomy in hiring. The central office structures that encouraged the use of data for teacher hiring, however, were, in general, seen by principals as not restricting their autonomy, but providing information in a complex hiring process.
Variation in principal data use by central office structures
To further understand how the organizational arrangements created by the central office influenced the extent to which the principals used teacher effectiveness data for hiring, we returned to the survey data. According to the conceptual framework we have laid out, principals in high-structure systems should weigh teacher effectiveness data more heavily than those in low-structure systems. For example, districts have been found to play an important role as change agents by ensuring processes and procedures that are coherently aligned with desired changes—in this case principals’ use of data in the hiring process (Rorrer et al., 2008). As the second and third columns of Table 2 indicate, there are notable differences between high- and low-structure systems in how principals rated the importance of various factors in their hiring decisions. Principals in high-structure systems were more likely to rate the three direct measures of teacher effectiveness (measures of student achievement or growth, teacher observation scores, and overall evaluation scores) as very important in hiring compared to low-structure systems. Conversely, there is no difference between system type in principals’ rating of more traditional hiring factors, such as their own professional judgment and direct observation of instruction. Principals in high-structure systems were also more likely to rate recommendations by others and their own professional judgment of the teacher’s ability to improve as very important in hiring, which are more traditional hiring factors, although the differences between system types were smaller than those for the new measures of teacher effectiveness.
Table 3 provides additional evidence on differences between high- and low-structure systems on principal use of data for hiring transfer applicants. The findings indicate that principals were more likely to report using all three forms of teacher effectiveness data if they were in a high-structure system. For example, 75% of principals in high-structure systems reported using teacher observation scores when hiring transfer applicants, compared with 53% of principals in low-structure systems. Again, this aligns with our conceptual framework. High-structure systems placed expectations on individuals that, in fact, influenced their hiring behaviors. This was especially evident in the differences surrounding newer and more complex measures of teacher effectiveness data. These findings demonstrate how districts can assist principals in understanding the unknown and ambiguous (Waterman, 1992), guide the sensemaking process, and construct principals’ understanding of the world around them. Given the newness and complexity of new forms of data such as value-added measures, and the differences observed between principals in high-structure versus low-structure districts, we believe that this point is particularly important. Overall, the relationship between data-use prevalence, its perceived importance, and district structure, leads us to conclude that districts that proactively support the use of data in the hiring process encourage the use of these measures when principals engage in the practice of hiring.
Percentage of Principals Who Reported Using Various Teacher Effectiveness Measures in Hiring Transfer Applicants, by Degree of Structure Provided by the Central Office.
Note. The number of observations varies slightly due to nonresponse. N = 567 to 570 for high-structure systems and N = 115 for low-structure systems.
p < .001 (p value at which low-structure systems are significantly different from high-structure systems).
Individual-Level Contributors to Observed Variability
While central offices mediate principals’ use of teacher effectiveness data in hiring, we also found that how principals used data in the hiring process was not purely a byproduct of district expectations, but was shaped by principals’ own background and social position in the district. In this section, we answer the third research question about additional supports and constraints principals face in using teacher effectiveness data in hiring by describing the ways sensemaking and social construction can be leveraged to understand individual-level variation beyond the purview of district-level influence. Individual-level supports and constraints included: knowledge and skills, perceived validity of data, and amount of social capital.
Skills and knowledge
Principals’ skills and knowledge (or lack thereof) were important drivers of data use. Some principals expressed a desire to use data in the hiring process, but believed that they did not have access to these data. While this may at times have been true, it was not always; many principals were not aware of the resources available to them. In the following quote, a principal provided an example of how a lack of knowledge about what is available impeded the data-use process.
If we had something to know about who was coming to the school, who we were interviewing so that we could make a better decision of whether this person would be a good fit for our school or not, and that takes knowing the strengths and weaknesses of our school. So, just like a basketball team, if you have some, a three-point shooter, and my school is low in that, then that’s someone that I want to hire. So, if I have more access of their strengths and weaknesses, it would be great to help me make a better decision about who to bring on board here. That would be great just to have that access. (District B)
In other interviews with principals from the same district, we encountered principals who required candidates to provide observation and value-added data, suggesting that this is an individual-level constraint, rather than a systemic barrier.
Even when principals were aware of the data available to them, they did not necessarily know how to access the data. This was complicated by an absence of integrated data systems in many districts. Principals often had to consult numerous data platforms to gather teacher effectiveness data. This meant that they needed to know how to use each program, and which platform offered the type of data they sought:
I still don’t know the ins and outs . . . so I’m tripping over trying to get a report . . . that takes a whole lot for somebody walking into a leadership role to do all that backtracking . . . So for me to access data, it’s taken me a while to figure out where I can simply get something . . . and then part of it is I don’t even know necessarily what’s out there. (District D)
Another impediment to data use under this category was whether principals were able to analyze data:
My main challenge is not the use of the data, it’s me having to analyze, dissect, crunch the data . . . some principals may not be as comfortable in doing that. They struggle . . . for data to be effective it should be readily accessible and available, so you don’t have to spend time to get it all together . . . there’s just not enough hours in the day to do all of that . . . [some principals] may not know how to do it. You know I’m really comfortable in it, but you have some principals that’s not. (District B)
In sum, our data suggest that principals faced three major barriers to data use under the umbrella of skills and knowledge. The first, was basic awareness of data available to them, the second was the ability to access the data via the data systems, and the third was the lack of skills needed to analyze the data so they could draw meaningful results and apply them to hiring decisions. These hurdles again point to the important role districts can play in disentangling the complexity of data for principals. That districts have the ability to encourage data use and fluency by devoting time for school leaders to learn to make sense of and apply data speaks directly to the hurdles which impeded principals’ data use in the hiring process (Halverson et al., 2007; Park & Datnow, 2009).
Perceived validity of data
Perceived validity of the various components of teacher effectiveness data (e.g., value-added measures, observation ratings, student surveys, etc.) was another constraint in how principals used teacher effectiveness data. From a sensemaking perspective, perceived legitimacy influences individuals’ subsequent actions. Similar to Coburn’s (2006) study that found teachers more likely to implement practices when they felt requests made of them were legitimate, we similarly attribute some of principals’ willingness, or lack thereof, to infuse new forms of data in their work processes to the legitimacy they attribute to such data. In the following interview excerpt the principal expressed multiple concerns about the quality of data, whether the data told a true story, and apprehension about using data that might be flawed:
The challenge is reliability of the data . . . So, is the data really telling us what we need to know, or what we need? . . . How practical is it?. . . I want to believe in that data, because we’re making decisions based on data, right? Data gotta be good data, right? It’s gotta be analyzed correctly and appropriately, right? And once again, once you make these decisions based on data, you can always say hey, the data [were] solid, it was flawed, okay, but we believed in data . . .
Is there one set of data that you feel more concerned about than the others in terms of reliability . . .?
It’s all connected. See, the observation gives you one set of data. Then who’s doing the observation, right? (District B)
The principal quoted above expressed a broad concern about the data, noting that it was all connected. By asking, “who’s doing the observation, right?” the principal expressed concern that observers’ ratings might not reflect teacher quality. We encountered some instances when principals conveyed the belief that classroom observations were not consistent across evaluators, and therefore could be very different from their own appraisal of the teacher. This was especially true with candidates transferring from other districts, but at times even within the same school system—particularly in systems with less rigorous calibration procedures around the observation rubric.
Principals also raised concerns about the validity of value-added and other student growth measures. 3 The following quote exemplifies one typical concern related to the validity of value-added measures in evaluating teachers: “I rarely use the [value-added] data in hiring just because I think that there’s so many factors involved in [value-added] data that it’s hard to just look at that individually without knowing a person and watching them teach” (District F). Another principal concisely expressed a similar sentiment: “In some cases the teacher got the school-wide [value-added score], and how do I use that data to make good hiring decisions?” (District B).
Finally some principals believed that much of what teachers do does not appear in data. This principal told the story of a teacher working with a student who was particularly alienated from school. The principal explains that the work teachers must do to reach students in this or similar situations would not be reflected in data:
At some point in time folks are going to have to understand . . . data don’t do it justice, because when you get to a point where this young man, he’s in jail, [you] invested some time and energy into this young man . . . You got him. So, now instead of you going off and threatening or even bringing a pistol here, you’re not going to do that because [of this teacher] . . . So, you’re dealing with things like that. That’s not going to show up on data. (District D)
Social capital
A third category of support or constraint involved principals’ social capital, and reflected their unique position in the district’s informal social structure. As posited by network theorists (Daly, 2012; Rigby, 2016), networks shaped by the organization in which they reside serve as structures that mediate action. We found that some principals relied on informal networks of principals, or relationships with central office personnel they had cultivated to access teacher effectiveness data. These connections served as sources of information regarding how to access data that were not easily available, or how to interpret and make use of various data elements. Although some of the difficulty in knowing how to access data was likely due to central office data systems and training, the evidence also suggests that there was considerable variation within districts on whether principals knew who to contact for support. For example, one principal told us, “It’s not like I have a magic number I can call and go, ‘hey, can you get me . . . ’ No, that’s not how it works, and I wish it did” (District D). On the other hand, a principal from the same district told us about numerous support systems on which she could rely:
So sometimes even if it’s a report that we don’t see, we don’t know how to run, we can ask these people [from the central office]. . . . We also have a data specialist. . . . I can as a principal pretty much get whatever it is that I’m looking for . . . Just an email. (District D)
While there was variation in the degree to which principals knew who to contact at the central office, principals also varied in the degree to which they relied on one another. One principal who struggled with navigating the numerous data systems he needed to access told us he felt he had no formal support. When we asked him to explain how he managed, he responded, “I’ll call one of my principal friends and say . . . ‘what do I do, how do I get this?’” (District D). This was not the case for all principals, however, especially those who were new to the system. Principals also mentioned relying on principals who had served as mentors, even when those principals worked in other systems. Some principals made use of connections from training programs, or residency programs, while others depended on principals under whom they worked when they themselves were either teachers or assistant principals. It is through shared social interactions in instances of uncertainty that principals construct subsequent actions. Their interactions with others directly influenced their ability to access data and make use of it in the hiring process. The importance of social capital, social networks, and social interactions for some principals suggests a powerful mechanism by which districts could encourage data use in the process of teacher hiring.
Conclusion
This study focused on school systems that are at the forefront of developing and using new measures of teacher effectiveness. We found that principals were beginning to use these measures for teacher hiring, but the extent and ways in which principals were using the data varied based on principal skills and knowledge, perceived validity of the data, their own social capital, and the ways in which the central office structured, supported, and created expectations for how principals make hiring decisions. Hiring practices are key components of how districts and schools manage their human resources. Helping school systems and principals to make efficient and informed hiring decisions is of utmost importance given the fact that teachers are integral to student learning.
Our findings indicate that much of the evidence on principal data use in hiring fell into the categories of moderate to low use—relatively few represented the high-end user. The findings from this study are consistent with prior research on principal hiring decisions that find a mix of individual principal characteristics and district practices mediate how they make hiring decisions (Donaldson, 2013; Papa & Baxter, 2008), although they seem to contradict the emerging evidence on use of teacher evaluation data in personnel decisions (Drake et al., 2016; Goldring et al., 2015; Master, 2014). The relative scarcity of high data use in hiring decisions reflects much prior literature on hiring that principals are constrained by central office processes (Levin et al., 2005; Levin & Quinn, 2003; Liu et al., 2008). Principals in hard-to-staff schools are constrained even further by the preferences of teachers (Cannata, 2010; Engel et al., 2013).
Still, our findings suggest that, beyond formal centralized hiring procedures, the central offices played key roles in mediating how principals approached using teacher effectiveness data in their hiring decisions. This is consistent with the notion of central office personnel as mediators and brokers in how principals interpret improvement efforts, particularly those that involve data use (Farrell, 2015; Honig & Venkateswaran, 2012; Marsh, 2012). We found that in school systems that provided clearer expectations for principals and held principals accountable for their hiring decisions, principals tended to use more information-rich processes when they hired teachers. The important role of the central office reflects other research on how the social dynamics of school systems shape human resource management in schools (Pogodzinski, 2016).
In addition to the central office, several characteristics of the principals themselves, such as their knowledge and skills to analyze data, how they perceived its validity, and social capital influenced the extent to which they used teacher effectiveness data for hiring. Using data effectively requires certain analytic skills that not all educators have (Wayman & Stringfield, 2006). Social networks have also been shown to influence how educators make sense of reforms and use data (Daly, 2012; Frank, Zhao, & Borman, 2004; Rigby, 2016) and thus it is not surprising that principals’ social networks influenced how they used data in hiring. Finally, principals have concerns about the validity and usefulness of value-added measures that shape how much emphasis they give to such measures (Goldring et al., 2015). Together, the attention to both the organizational structures and individual differences, suggest that researchers need to pay attention to how these different forces work together to shape individual behavior (Marsh, 2012; Moss, 2012; Spillane, 2012).
As one of the first explorations of the role of teacher effectiveness measures in teacher hiring, this study has a number of limitations. First, the school systems were selected because they were at the forefront of developing teacher effectiveness measures and had devoted considerable resources to doing so. Thus, it is unclear if our findings generalize to systems without such a strong commitment to teacher effectiveness measures. Second, the interview sample does not allow for us to examine how the pattern of high–, medium–, and low–data use principals are represented across high- and low-structure districts, limiting the ability to make similar comparisons across the qualitative and quantitative data. Relatedly, the data are limited in the ability to directly connect individual principal characteristics and district characteristics to determine how much statistical variation in principal use of data in teacher hiring is explained by district or individual characteristics, even as our findings suggest both are important.
Despite these limitations, the article makes a substantial contribution to understanding the role of emerging teacher effectiveness measures in hiring and has implications for policy and practice. Efforts to increase principal use of teacher effectiveness measures need to face the challenges in supporting principals to gain the necessary skills and knowledge. As leadership preparation and professional development programs prepare principals for new expectations around data use, it is important to focus on these types of job-embedded analytic skills. Principals need additional support in identifying specific data that could help them answer particular questions required of their job, such as which job applicants have evidence of excellent teaching performance. Relatedly, principals, and school systems more generally, need more specific evidence to help them make informed decisions when hiring first-year teachers, such as prehire indicators that are predictive of later effectiveness. Our findings also suggest that central office personnel who want to shape how principals make hiring decisions can do so by how they recruit and screen candidates, establish expectations for principals, and hold principals accountable for hiring decisions.
Several areas of additional research are warranted to further understand the impact of teacher effectiveness measures on hiring. First, this study focused on school systems that were at the forefront of developing teacher effectiveness measures; additional research that includes a broader array of school systems with varying commitment to teacher effectiveness measures can add to our knowledge of how these measures are influencing hiring more generally. Second, our findings highlight a distinction between centralized hiring procedures more generally that principals report as constraining their autonomy, and more targeted hiring structures that some school systems used to encourage the use of teacher effectiveness measures. Alongside other research that principals report increasing autonomy over the teachers hired in their schools (Engel et al., 2015), this may be an indication that the role of the central office in hiring is changing. Future research should explore this evolving role of the central office in new teacher hiring. Finally, while we found that most principals were moderate to low users of teacher effectiveness data in hiring, there is little evidence that being a high data user results in better hiring outcomes. That is, future research on teacher hiring needs to extend past the point of hire to identify hiring behaviors that predict later outcomes.
Using effectiveness data for hiring teachers is challenging as candidates may be new college graduates, experienced teachers from another school in the system, or experienced teachers from another system or even state. This means that, in addition to more traditional barriers to data use such as data accessibility and data analysis skills (Honig & Coburn, 2008; Wayman & Stringfield, 2006), principals have to grapple with the fact that different candidates will have different types of data available, depending on their prior experience. This creates particular challenges for principals in high-poverty schools, who often have sparser applicant pools with more inexperienced teachers than principals in more advantaged schools (Engel et al., 2013). Still, our results demonstrate ways in which certain school systems and principals are incorporating teacher effectiveness data, and the teacher evaluation process more generally, into the process of teacher hiring. In particular, high-use principals were distinguished in the way they used the system’s evaluation rubric to structure how they observed candidates’ demonstration lessons and focused questions during interviews. These principals were often in the high-structure systems that actively screened candidates using indicators from the evaluation system and set clear expectations for principals. Indeed, this use of the teacher evaluation rubric in hiring is consistent with research that the presence of a comprehensive evaluation system itself (in addition to the data produced by it) can aid principals in their work (Rubin et al., 2015). This incorporation of new teacher evaluation systems into teacher hiring is noteworthy not only as a way to use new forms of data themselves but also as a potential mechanism for signaling to applicants what the system considers to be an effective teacher and providing a common language for quality teaching (Kraft & Gilmour, 2016). This is important since the hiring process is the first stage in establishing the work environment for new teachers (Johnson, Kardos, Kauffman, Liu, & Donaldson, 2004). Teacher evaluation systems can signal districts’ definitions of high-quality teaching, and incorporating that language into the hiring process can help induct new teachers into the work of the school and the system.
Footnotes
Acknowledgements
We acknowledge Bill and Melinda Gates Foundation’s generous support for this project.
Authors’ Note
The opinions expressed are those of the authors and do not necessarily represent the views of the sponsor.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was conducted with funding from the Bill and Melinda Gates Foundation.
Notes
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
