Investigating principals’ data use in school: The impact of evidence-oriented attitudes and epistemological beliefs

Abstract

In this study, we examine the relationships between principals’ evidence-oriented attitudes and their data use in school leadership practices. Based on theoretical assumptions and existing empirical research, we develop hypotheses on the correlation of evidence-oriented attitudes, epistemological beliefs and principals’ data use. We test these hypotheses based on data collected from a sample of N = 297 principals at 153 schools of different types using structural equation modelling. The findings show that the examined attitudes and epistemological beliefs have a slight, yet significant influence on principals’ data use. In particular, principals’ skepticism towards knowledge or academia influences data use. We conclude by addressing the limitations of our study and the challenges they pose for future research as well as implications for evidence-based school improvement.

Keywords

Evidence-based school management evidence-based school leadership structural equation modeling principals’ data use

Introduction

In Germany, as in many other countries (see, e.g., Geijsel et al., 2010; Levin and Datnow, 2012), the implementation of the new governance model (NGM) brought about an increasing decentralization of the school system. Schools have more responsibilities, but also more freedom of action in order to comply by the centrally dictated educational standards—a process that is controlled through external (and internal) evaluations. In this context, evidence-oriented school and curriculum development is becoming more and more important,¹ as it is assumed that school agents will use the provided empirical knowledge from these internal and external evaluations (e.g., comparisons, etc.) as a starting point for their actions and development measures (see also Zlatkin-Troitschanskaia, 2016; for US context, see, e.g., Anderson et al., 2010; Black and William, 1998).² However, current findings show that schools have substantial deficits and teachers’ actions often are hardly evidence-based at all (see the according articles in Zlatkin-Troitschanskaia, 2016; Zlatkin-Troitschanskaia et al., 2013; see also Schildkamp and Kuiper, 2010; Schildkamp and Teddlie, 2008).

The principal plays an essential role in evidence-based school development (see, e.g., Anderson et al., 2010; Hoogland et al., 2016; Levin and Datnow, 2012; Park and Datnow, 2009; Wayman and Springfield, 2006; Wayman et al., 2012). Research findings from different countries show that teachers’ data use is influenced by the principal’s leadership actions (Stump et al., 2016). To be more precise, there is a positive correlation between transformational leadership and teachers’ motivation as well as changes in teaching practice (Leithwood and Jantzi, 2006), transformational leadership and a supportive and active school environment (McCarley et al., 2016), transformational leadership and teachers’ attitudes and actions (Leithwood et al., 1996) as well as the influence of leadership on the use of data in data teams (Schildkamp and Poortman, 2015).

To date, principals’ data use as a part of principals’ leadership actions and its influence factors have hardly been researched (see also the next section; Demski and Racherbäumer, 2015). It is evident that attitudes in general have an influence on actions. In the context of evidence-based school management, Dormann et al. (2016) differentiate between different types of evidence orientation and empirically examine this modelling and operating approach in the school context. The aim of the following study is to identify principals’ attitudes towards knowledge and academia, including their epistemological beliefs, and to examine the correlations with principals’ data use based on this model.

Research context: state of research on principals’ data use

Principals’ data use is mostly examined with regard to the type, form and extent of data used, as well as the aims and areas of data use in different countries (Brooks, 2012; Goldring et al., 2015; Henry, 2011; Williams, 2011). Regarding the degree to which data use occurs, findings on the use of different data sources from a German study show that principals’ use of internal data is more pronounced than the use of external data (Demski and Racherbäumer, 2015). Principals tend to use data originating from their own school and reflecting their own school’s circumstances for their leadership actions, and are less likely to use externally generated data (such as comparison tests, scientific findings) for this purpose (see also Demski et al., 2012).

Wurster et al. (2013: 39) compare the use of two different external evaluation procedures (data from school inspection and from state-wide comparison tests in Germany (VERA)) on a school level by surveying principals in two German states and show that such data is mostly used in the context of curriculum development. The findings from comparisons in particular are used more often for curriculum-related development measures than for school development measures such as school programs. The perceived usefulness of internal evaluation findings is examined as a potential influence factor on data use. The usefulness of internal evaluations is rated highest. Results from the VERA comparison, however, are perceived as the least useful (Wurster et al., 2013: 34, 41–43). This shows a discrepancy between perceived usefulness and reported actual use. In the study by Bach et al. (2014: 73), however, the findings indicate that among the examined influence factors, only the principals’ perceived usefulness of an evaluation instrument has a significant influence on the derivation of staff development measures. Almost half of the surveyed principals claimed to have derived staff development measures from VERA findings, 73% of responses confirm the use of the findings for the area of further training planning, 82% of principals claim to use the findings for curriculum development measures and 42% claim to initiate further work on the school program based on these results (Bach et al., 2014: 73).

Studies examining supporting and hindering influence factors on principals’ data use analyze—following the cycle model by Helmke and Hosenfeld (2005)—mainly external and in-school circumstances. According to the School Performance Feedback System framework model by Visscher and Coe (2003), external circumstances mainly encompass features of the testing system and of implementation such as accessibility, availability and timelessness of data, the perceived quality of data, tools for data collection and interpretation and support through federal or local policies (Anderson et al., 2010; Henry, 2011; Ikemoto and Marsh, 2007).

Regarding the individual conditions of principals and their data use, the focus is in particular on the principal’s abilities and (further) training³ (Henry, 2011: 135; Martinez, 2010: 94; see also Verhaeghe et al., 2010). In the study comparing countries which was carried out by Schildkamp et al. (2014: 19–20), factors other than those mentioned above were also listed. These included organizational characteristics (teacher collaboration, colleague support, data expertise, vision and goals for data use, training and support in both data use and time and—in some countries—school leaders), data and data system characteristics (primarily the availability of and access to different types of data and the quality of the data) as well as user characteristics. Among these, factors were presented such as knowledge and skills for data use and attitudes toward data as a user characteristic that influences data use. Vanhoof et al. (2014) have shown that especially the affective aspects of principals’ attitudes are strongly related to their data use. The influence of the cognitive dimension, however, is of minimal impact. The use of data in a deliberate and target-oriented fashion is linked to an increase in the significance of attitudes and beliefs of principals as predictors. Thanks to this, Martinez (2010: 94) was able to determine a connection between the attitudes of principals and their use of data. For data-driven decision-making practices in the context of leadership in school instruction, Mingchu (2008) determined a direct significant positive effect of principals’ perception of data quality (2008: 621). Principals’ perception of the extent to which data is believable, accurate, reliable and comes from good sources (2008: 612) therefore plays an important role for data-driven decision-making; in general, attitudes towards data could consequently influence the use of data (Mingchu, 2008: 626). Saunders (2000: 252–253) demonstrated that school leaders differ greatly in their attitude towards (value added) data. She identified four groups (unengaged and resistant, enthusiastic reliance, skeptical, resistance to literal use of data (rather than per se), accepted and valued performance). This classification suggests that beliefs and values about teaching and learning are closely linked to the use of data within and by a school department. In order to promote data-based school improvement, a combination of enthusiasm and the ability to put things into perspective appears to be of great importance (Geijsel et al., 2010: 63). However, non-cognitive factors such as the attitudes and beliefs of principals have so far been subject to little research, and current findings are not consistent (Martinez, 2010; Vanhoof et al., 2014).

Research focus

In light of the current lack of research, this article aims to examine attitudes towards evidence as well as epistemological beliefs as a possible way of predicting the data use of principals. In accordance with the model presented by Dorman et al. (2016), this study will focus on examining the difference between using internal sources of data (internal data use) and external sources of data (external data use). The use of data sources within these two categories will not be subject to examination. On the one hand, this overarching perspective enables (a) statements to be made regarding the use of types of data (internal/external), (b) a comparison of the use of these two types of data (internal/external) by principals and (c) the inclusion of numerous types of data in each group (internal/external). On the other hand, this perspective follows the idea that principals’ actions are not limited to using a certain type of data, but that it is more a case of using data within the framework of their leadership at school and in the context of a data-based school development. Anderson et al. (2010: 323) realized that

the broader concern has to do more with principal leadership for data use in schools, than with the specific uses of data by principals themselves. School data use was a collective activity involving principals working with their teachers in those schools where we observed what seemed to be the most intensive and potentially productive patterns of data use for improvement in student learning.

Conceptual and theoretical background and hypothesis

Principals’ data use

Data use apparently forms the basis of principals’ actions. Boudett and Steele (2007) speak of data-wise leadership, which encompasses the inclusion of both internal as well as external scientifically generated information in principal leadership. With regard to the use of data, a reference is made to the concept of data according to Schildkamp and Lai (2013), who view data as “systematically collected information on students, schools, […] which can be obtained from qualitative […] and quantitative […] methods” (177). Here, data is understood as a general concept made up of collected information. According to Dormann et al. (2016), as far as the sources of the generated data are concerned, there is a difference between the use of internal and external data. The use of internal data refers to the use of empirically proven knowledge (scientific data) acquired at a specific school (see Van Ackeren et al., 2013: 53). Internal data sums up findings on class and school activities of a specific school that, for example, are the result of internal school evaluations.⁴ According to Demski et al. (2012: 146), internal data originates from process-oriented and school-specific data sources. In Germany, internal evaluations are regulated by law, their form, process, instruments, arrangement and timing, however, remain free and vary from school to school. It is not mandatory in all German federal states that the results count towards the external evaluations (European Commission/EACEA/Eurydice, 2015: 78–79).⁵

External data refers to the use of empirically proven knowledge (scientific data) which was generated externally (e.g., in comparative tests or school inspections; see Dormann et al., 2016: 5). External data can be divided into two groups: the first group represents, for example, results from school inspections, comparative tests and statewide learning assessments—initiated and implemented externally, yet concerning the school in question. The second group is characterized by general results (e.g., from PISA (Programme for International Student Assessment)) and recommendations that do not stem from (external) evaluations within the school in question (see Table 1).

Table 1.

Internal and external data use—classification of the data basis and initiators (original representation).

Initiator/producer of the generation of data	Data basis
Initiator/producer of the generation of data	Individual school	Outside school
Individual school	Findings from internal evaluations (students’ feedback on lessons, collegial visits, school-level parallel exams within grades)		Internal data use
External institution	Findings from external evaluations (school inspection report, nationwide learning assessments, comparison tests (regional, school catchment area, national))	Research findings from pilot projects, recommendations based on international comparative studies, etc.	External data use

Principals’ evidence-oriented attitudes and epistemological beliefs

In the context of data-based school and classroom development, recommendations based on scientifically generated knowledge are of great importance in order to improve educational practice. According to Earl and Katz (2002), there is tension between a general interest in data and the actual use of data in the context of school improvement. The authors derive school leader capacities “in a data-rich world,” which are required for successful school improvement (see also Earl and Katz, 2006). Accordingly, school leaders need a positive orientation as well as a broad range of knowledge and abilities in order to understand, interpret and use data. Overall, three requirements for “leading schools in a data-rich world” can be identified. Ranked first is the requirement to “develop an inquiry habit of mind” (Earl and Katz, 2002: 4); this means that school leaders should think critically and open-mindedly; they should have a dynamic and interactive way of thinking, which approaches problems using decision and feedback loops. School leaders should not presume an outcome, but allow for a range of outcomes and assume a range of perspectives (Earl and Katz, 2002: 5). Thus, school leaders must develop a “research minded way of working” (Geijsel et al., 2010: 62)—a “‘mind shift’ among school leaders is necessary” (63).

When it comes to data use in schools, it is particularly significant how principals evaluate the significance and usefulness of scientific findings and statements, and whether and to what extent this correlates with the principals’ actual use of data. Generally, it is evident that attitudes and beliefs can influence actions. According to Fishbein (1963: 233), attitudes can be defined as follows: “an individual’s attitude toward any object is a function of his beliefs about the object […] and the evaluative aspect of those beliefs.” To date, there are only few findings on principals’ attitudes towards knowledge and scientific recommendations. It is yet to be examined whether evidence-oriented attitudes influence the actions of principals in terms of their leadership actions and data use.

The concept of capable and confident data users also means that their way of thinking encompasses the understanding of the nature of evidence, “from its definition and collection to its interpretation and presentation” (Earl and Katz, 2002: 3). The capacities of leaders in a data-rich world, therefore, also involve the leaders becoming data literate (see also Earl and Katz, 2006). Data literacy comprises, among other things, an understanding of statistical and measurement concepts in order to realize whether data are reliable and valid, or whether the information has perhaps been manipulated or whether the source of information is unreliable and should not be trusted (Earl and Katz, 2002: 5–6). School leaders should also recognize that it is important to understand and interpret data, and that data as such do not provide quick results, but rather “offer decision-makers an opportunity to view a phenomenon through a number of different lenses” (2002: 6). These aspects constitute a part of epistemological beliefs.

Aside from assumptions and definitions, epistemological beliefs as “beliefs around a concept” are also a part of the concept of mental models (Jimerson, 2014). In her framework for the development of mental models for data use among educators, the author applies the concept of mental models to the concept of data-driven decision-making and assumes that “actors possess mental models […] and that subsequent actions or perceptions proceed from this framing” (Jimerson, 2014: 6). Empirically, she highlights different mental models for data use (2014: 13): while with the goal of accountability, some school leaders focused strongly on the test results but found this obstructive to their everyday work in practice, others considered the data an opportunity to focus more strongly on the students’ needs. Other groups of school leaders were classified as skeptical or very open-minded and welcoming.

On the basis of previous research, a focus should be placed on epistemological beliefs (Hofer, 2002), which are essential especially for information intake and knowledge acquisition (Wilkinson and Schwarz, 1987, quoted in Köller et al., 2000: 231). In the context of beliefs, epistemological beliefs are of particular importance for the successful implementation of knowledge in real-life situations (Urhahne, 2006: 189). In contrast to attitudes, which are considered affective, beliefs can be considered cognitive (for an overview, see Weschenfelder, 2014: 82). Epistemological beliefs are basic views and attitudes towards the sciences (Priemer, 2006: 160). They can be understood as domain-specific (i.e., only regarding individual academic disciplines) or as non-domain-specific (domain generality). The non-domain-specific perspective places the individual and the individual views in the focus of the examination of knowledge and science, while the domain-specific perspective focuses on views prevailing in the specific domains (Priemer, 2006: 160). When examining influence factors on principals’ data use, epistemological beliefs in particular are relevant in the context of the aspired evidence-based school and classroom development.

In literature, it is often assumed that certain functions influence actions (for an overview, see Müller et al., 2008). Epistemological beliefs as ideas regarding the structure of knowledge and knowledge acquisition (Urhahne and Hopf, 2004: 71) comprise different aspects (Hofer and Pintrich, 1997; Schommer, 1990; Schommer-Aikins et al., 2003). According to Hofer and Pintrich (1997: 118–120), epistemological beliefs can be differentiated into ideas about the nature of knowing on the one hand and ideas about the nature of knowledge on the other hand. In view of the expectations placed on the individual schools regarding knowledge-based school and classroom development, which follows up on collecting and providing different types of data, the nature of knowing in particular plays an important role. In order to promote the reception, processing and use of the data provided, for instance, by school inspections and external evaluation of teachers or independent generation of data as part of internal evaluations, principals’ ideas about the nature of knowing are also important in the context of transformational school leadership.

According to Hofer and Pintrich (1997), the nature of knowing can be divided into two dimensions: source of knowledge and justification for knowing. The dimension justification for knowing aims to explore to what extent learners lean towards one-sided, either-or or right-or-wrong judgements, or whether they tolerate diverse opinions. It provides information on how learners deal with claims, evidence and statements (for an overview, see Urhahne and Hopf, 2004: 74). Principals’ ideas could be particularly significant for the area source of knowledge, when knowledge is meant to be systematically developed and expanded in the individual schools. The aspect source of knowledge concerns the idea that “knowledge originates outside the self and resides in external authority, from whom it may be transmitted” (Hofer and Pintrich, 1997: 120). Depending on the manifestation of epistemological beliefs, there is the naive view of an omniscient authority on the one hand, where knowledge is transmitted to learners from the outside, and a differentiated view on the other hand, where knowledge is acquired by means of subjective and objective experiences as well as empirical evidence and logical thinking (see Hofer and Pintrich, 1997: 120; Schommer, 1990; Urhahne and Hopf, 2004: 73–74). In contrast to the aforementioned perspective (authorities), knowledge therefore builds on the learners’ ability to construct knowledge in interaction with others (Urhahne and Hopf, 2004: 74).

Transferred to the context of data-based school and classroom development, the focus, therefore, lies on ideas as to the extent to which empirical findings from, for example, external evaluations of the principals’ schools as well as overarching scientific findings in the field of school development or even results from evaluations within their own school are fully acknowledged, considered to be true and considered credible. It will be significant to what extent the principals tend to (blindly) trust such findings in the sense of an omniscient authority or whether they take a critical stance and construct their own knowledge from a variety of sources, including exchange with others. For the principals, the latter, more mature epistemological beliefs can be assumed due to their career path and their prior qualifications.

Hypothesis

Based on the generally action-influencing function of attitudes and epistemological beliefs on the one hand, and the specific characteristics of internal and external data use on the other hand, the following hypotheses can be derived.

When principals do not trust an authority, which could be understood as “immature” epistemological beliefs towards the source of knowledge, but rather consider knowledge acquired by means of empirical evidence and logical thinking, it can be assumed that principals are more likely to consult external data for further school and classroom development. Thus, principals’ more mature epistemological beliefs could be connected to a greater likelihood of using external data. The first hypothesis can be summarized as follows:

Hypothesis 1. The more pronounced (“more mature”) the principals’ epistemological beliefs (regarding the source of knowledge for the area of school development) are, the more pronounced the principals’ use of external data is.

Likewise, it can be assumed that principals with more mature epistemological beliefs are also more likely to use internal data. If principals have differentiated views and assume that knowledge is acquired by means of subjective and objective experiences as well as empirical evidence and logical thinking, it can be assumed that they are also more likely to use internal data in order to construct a more objective reflection of reality, using these findings as well as other data. Thus, a second hypothesis can be formulated:

Hypothesis 2. The more pronounced (“more mature”) the principals’ epistemological beliefs (regarding the source of knowledge for the area of school development) are, the more pronounced the principals’ use of internal data is.

When examining the principals’ attitudes towards academia as well as considering scientific recommendations to be a result of interpreted scientific data and findings from scientific research, it can be assumed that principals who have a skeptical attitude towards scientific recommendations and scientific statements have a tendency to not consider external data and will, therefore, not use it later on. If the principals are generally skeptical towards academia, it can be assumed that they are not very likely to turn to scientific findings or new scientific research in their decision-making or when looking for support in their actions. Thus, a third hypothesis can be derived:

Hypothesis 3. The more skeptical principals are towards academia and scientific recommendations, the less pronounced their use of external data is.

Another aspect concerns the skepticism towards evidence and knowledge in general. A very skeptical attitude towards knowledge and evidence in general on the part of the principals could lead to internal—that is, self-acquired—knowledge not being used, or the process of data acquisition not even being initiated. While external data is usually (a) gathered or provided to the school by other institutions (school inspections, etc.) or (b) consists of overarching studies and findings from scientific research, gathering internal data requires a high level of personal initiative. Thus, a fourth hypothesis can be postulated:

Hypothesis 4. The more skeptical principals are towards knowledge, the less pronounced their use of internal data is.

Methods

Sample and procedure

The empirical analysis is based on a cross-sectional study conducted in the German federal state of Rhineland-Palatinate in 2012. In the standardized written paper–pencil survey, 297 principals of all school types of 153 were surveyed.⁶ The goal was to perform a census of all teachers and school leaders at these schools in Rhineland-Palatinate. In a first step, the schools were sent an email inviting them to participate. When necessary, they were also contacted by phone in a second step.

Measures and instruments

The questions covered (a) school leadership, including external and internal data use (Stumm et al., 2010); (b) principals’ attitudes towards knowledge, academia and scientific recommendations (Schiefele et al., 2002); and (c) principals’ epistemological beliefs (following Urhahne and Hopf, 2004: 87). Participants were asked to respond to items on a five-point scale from “not at all” (1) to “to a large extent” (5).

The dependent variable data use was measured using the scales use of internal data and use of external data (Stumm et al., 2010). With regard to the independent variable principals’ epistemological beliefs, the section “source of knowledge” was included (following Urhahne and Hopf, 2004: 87; see also Hofer and Pintrich, 1997; Urhahne, 2006: 192 for the background of the test instrument). The items were adapted in accordance with the project context. In line with the subject matter they cover, they explicitly aim towards, for example, empirical findings or evaluation results. One example is: “An empirical database on school development and leadership is essential to discovering the truth” (see Table 2). All items on the scale epistemological beliefs/source of knowledge were inverted, which means that higher scale values point to “more mature” beliefs. As the scale only depicts a section of epistemological beliefs and refers only to empirical findings and evaluation results in the context of school development, it was titled trust in findings on school development.

Table 2.

Sample items.

Scales	Sample items
Trust in findings regarding school development	An empirical database on school development and leadership is essential to discovering the truth.
Skepticism towards academia and scientific recommendations	I question scientific recommendations on the development of schools and teaching.
Skepticism towards knowledge	I am confident that investing in knowledge still yields the best interest. (Recoded)
External data use	Before implementing major innovations, I explicitly search for reliable (e.g., scientific) information on their effectiveness.
Internal data use	Before adopting a procedure from another school, I examine to what extend the schools’ framework conditions are similar.

Statistical analysis

The assumed relations between the principal’s data use on the one hand and the principal’s evidence-oriented attitudes and epistemological beliefs on the other hand as potential influence factors thereof were analyzed on the basis of structure equation models. The data were evaluated using the software MPlus 7.3 (Muthén and Muthén, 2010) for the covariance-based approach.

Results

The five latent constructs trust, skepticism towards academia, skepticism towards knowledge, external data use and internal data use had satisfactory loadings. These factor loadings met the threshold value in the measurement model of the covariance-based structural equation modeling (CVSEM) analyses. Considering the structure model to examine correlations, the model fit of analyses using the covariance-analytical approach using MPlus appears satisfactory. The root mean square error of approximation (RMSEA) is lower than 0.05, and both comparative fit index (CFI) and Tucker-Lewis Index (TLI) have values larger than 0.95 or 0.94 (see summary by Förster, 2015). The characteristic value for the standardized root mean square residual (SRMR) is acceptable and regular with a value of 0.051 between 0.05 and 0.1 (see summary by Förster, 2015). The explained variance of the variables internal data use and external data use can be rated as insufficient with values of R ² = 0.086 and R ² = 0.109. Considering the correlations of interest, only rather small path coefficients of the influence factors point to the two dependent variables internal data use and external data use. The influence directions are always in conformance with expectations.

Among the influence factors on internal data use, the MPlus analyses show a significant negative effect (Table 3) of skepticism towards knowledge on internal data use (ß = –0.242). A skeptical attitude of the principal towards knowledge in general tends to impede the use of internal data. Principals’ skepticism towards academia also slows down internal data use (ß = –0.126). This influence, however, is relatively weak and not significant. The section of epistemological beliefs, trust in findings on school development, is slightly positive (ß = 0.074), but also not significant (Figure 1).

Table 3.

Coefficients of the covariance-based structural equation modeling analysis.

	CVSEM coefficients
	ß path coefficient	T-statistic
	R ² = 0.086
Trust → internal data use	0.074	0.807
Skepticism academia → internal data use	–0.126	–1.438
Skepticism knowledge → internal data use	–0.242	–2.894
	R ² = 0.109
Trust → external data use	0.128	1.471
Skepticism academia → external data use	–0.221	–2.653
Skepticism knowledge → external data use	–0.134	–1.620

Figure 1.

Results of the SGM analysis using MPlus; standardized regression coefficients and standardized factor loadings; significant coefficients are identified.

For external data use, negative influences of skepticism towards knowledge and academia can also be noted. Here, path coefficients are overall higher than for internal data use, but remain on a low level (ß _{skepticism knowledge} = –0.134, ß _{skepticism academia} = –0.221). The negative influence of skepticism towards knowledge does not appear to be significant. The positive influence of trust on external data use is weak as well (ß = 0.128), but not significant.

Summarizing these results, it can be noted that the use of internal data is significantly and substantially negatively influenced by an existing skepticism towards knowledge. If principals are skeptical and doubtful towards (new) knowledge overall and, therefore, also towards generating new knowledge, they will not consult internal data generated by the school itself. Their critical attitude leads to them not using such self-generated internal data, to not even generating such data in the first place in their role as principal and giver of impulses, or to not supporting it. In this case, it would be obvious that measures such as student feedback or joint measures to improve teaching would be even less likely to be initiated, supported and supervised.

Skepticism towards academia also significantly and substantially negatively influences the use of external data. The more principals exhibit a doubtful and skeptical attitude towards academia—and, therefore, also towards scientists and derivative work as well as scientific recommendations—the more they refuse to use such external data and, consequently, do not (voluntarily) include it in their own work. In this respect, the findings are in conformance with expectations; thus, hypotheses H3 and H4 cannot be rejected.

For the relations assumed in hypothesis H1, no significant,⁷ but nonetheless substantially positive, effects of trust on the use of external data can be noted. The path coefficient indicates a medium effect (Hair et al., 2013: 7). The more strongly principals consider knowledge as acquired by means of empirical evidence and logical thinking, the more likely they are to consult external data for the further development of school and teaching practices; for example, consider scientific studies or comparisons of school performances and include such findings in their work. They have a lot of trust in findings on school development—however, not in the sense of non-critical examination—and therefore contradict the rival belief that knowledge stems from omniscient authority.

The more developed the epistemological beliefs are, the more it will be possible to support knowledge sharing and data use in schools, leading back to the position of the principal. Overall, the results regarding the influence direction are in conformance to expectations. In several cases, however, they turn out to be non-significant influences, which may be due to the comparatively small sample.

Discussion

Teachers’ attitudes towards data use (Groß Ophoff, 2013; Hosenfeld, 2010; Schneewind, 2006) or concerning the introduction of changes at the school or in teaching (Dedering, 2007; Holtappels, 2007; Yilmaz and Bayraktar, 2014) are established areas of research. However, school principals’ attitudes and beliefs as individual conditions for their data use in school and classroom development in the context of their leadership practices remain largely under-researched. This paper examines evidence-oriented attitudes as well as epistemological beliefs of school principals as possible factors of influence on data use.

The analyses presented here aim to advance the research of data use at schools by observing the principal as the central controlling and managing agent of the school. The research gave first indications as to the extent to which the selected individual characteristics of school principals can be relevant to the process of data use. As the analyses addressed both the group of internal and the group of external data, conclusions can be drawn with regard to the characteristics of these groups rather than being restricted to certain types of data such as, say, school inspection.

Overall, the influences are significant as expected, albeit rather weak. From the selected characteristics, particularly skepticism towards academia and skepticism towards knowledge, were shown to have significantly substantial influence on data use. Therefore, two of the four hypotheses (H3, H4) cannot be discarded. Principals’ skepticism towards academia and skepticism towards knowledge affect both internal and external data use.

The items for the assessment of skepticism towards science and scientific recommendations are formulated in a way that refers to science in general. Skepticism towards science and scientific recommendations comprise the implementation of scientific studies as well as explicitly the interpretation of findings and recommendations derived from them. Thus, they refer mainly to external data use (see Table 1).

Additional findings on school leaders’ skepticism cannot be derived from this. To what extent skepticism, for example, has an influence that is concretely based on a pseudo-scientific presentation of scientific findings cannot be determined. This aspect could be included in skepticism towards science. The presentation or representation of data plays a significant role in literature on the topic of unintended effects (e.g., Ball, 2003: 224–225). So-called “re-presentation” (Ball, 2003: 3; Hardy and Lewis, 2016: 3) can make people or institutions appear in a better light. Thiel et al. (2017) verify this for no- or low-stakes contexts such as Germany as well, and Pennickx et al. (2015) for the Flemish education context, a “low-stakes” inspection context. This means that side effects are systematic effects that must also be considered. In this context, side effects referring to teachers’ attitudes such as the erosion of trust (e.g., vote of no confidence by educational policy and administration, distrust of educational policy and administration) could be of importance (Thiel et al., 2017: 17).

In the present study, the additionally conducted partial least squares (PLS) analyses showed a significant positive, yet only slight, influence of trust in school development findings as a part of epistemological beliefs on the use of external data. However, further research is required in this regard. Data use cannot be understood as a linear, solely rational process. The use of internal and external data can be affected by numerous conditions, processes and contexts, which are interlinked in various and complex ways. According to Schildkamp et al. (2014: 21), users’ characteristics, stakeholder learning and the data itself are all interlinked in this process: “Data use involves an interpretative process, in which data has to be identified, collected, analyzed and interpreted to become meaningful and useful for actions” (21).

While the variable skeptical attitude focused on science and scientific recommendations in schools and teaching, the variable trust focuses concretely on findings on school development. A lack of trust towards comparative studies, for example, could lead to the interpretation that school leaders assume other schools act in an unsportsmanlike manner (e.g., by modifying their data representation, teaching to the test) and, therefore, assume that the data basis is manipulated (Ball, 2003).

It remains unclear, for example, to what extent the school leaders’ trust in findings on school development can (also) be shaped by a (pseudo-)scientific presentation of the findings. With regard to the pressure to perform that teachers and school leaders experience in a competitive environment, inspections can bring phenomena such as gamesmanship, inauthentic practice as well as misrepresentation or cheating to light in the presentation of findings (see Ball, 2003: 222–225 for UK context). Hardy and Lewis (2016) verified the phenomenon of the “doublethink of data” of those involved in school education; in other words, teachers, for example, engage in these processes without seeing any real point in their actions.

By focusing on attitudes and epistemological beliefs, the study of factors influencing data use is limited in two ways: firstly, other factors of influence, which could not be considered in the analyses at hand, must be taken into account. The organizational and structural characteristics of schools mentioned in the section ‘Research context: state of research on principals’ data use’ should be further addressed in future studies (Zlatkin-Troitschanskaia et al., 2016). Secondly, the selected characteristics only concern principals’ “voluntary” use of data by the principals’ own account. More in-depth research on principals’ data use must include further individual factors (such as qualifications, previous experiences). As far as Germany is concerned, considering the insufficient research of principals’ data use considering qualifications, it is necessary to conduct further analyses based on normative guidelines in the form of school laws, guidelines and recommendations. Particularly regarding the use of external data in school leadership practices, it can be expected that guidelines and regulation by authorities can have a significant effect.

When drawing implications for evidence-based school improvement and considering that a school’s principal has an essential role within the school—for example, through collective processes—their epistemological beliefs and views as to how knowledge is generated and how it is to be understood are important for data use at schools. In this context, the principal can also be understood as the “enabler of data use by teachers” (Anderson et al., 2010: 319).

Footnotes

Funding

The author disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was supported by the Federal Ministry of Education and Research (grant numbers 01JG1303A and 01JG1303B).

Notes

References

Anderson

Leithwood

Strauss

(2010) Leading data use in schools – Organizational conditions and practices at the school and district levels. Leadership and Policy in Schools 9(3): 292–327.

Apple

(2009) Politics, theory, and reality in critical pedagogy. In: Cowen

Kazamias

(eds) International Handbook of Comparative Education. Dordrecht: Springer, pp.991–1007.

Bach

Wurster

Thillmann

, et al. (2014) Vergleichsarbeiten und schulische Personalentwicklung – Ausmaß und Voraussetzungen der Datennutzung. Zeitschrift für Erziehungswissenschaft 17(1): 61–84.

Ball

S J

(2003) The teacher’s soul and the terrors of performativity. Education Policy 18(2): 215–228.

Berkemeyer

Müller

Van Holt

(2015) Schulinterne Evaluation – Nur ein Instrument zur Selbststeuerung von Schulen? In: Altrichter

Maag Merki

(eds) Handbuch Neue Steuerung. Wiesbaden: VS Verlag, pp.209–234.

Black

William

(1998) Inside the black box: Raising standards through classroom assessment. Phi Delta Kappa 80(2): 139–148.

Boudett

Steele

(2007) Data Wise in Action - Stories of Schools using Data to Improve Teaching and Learning. Cambridge, MA: Harvard Education Press.

Brooks

(2012) South Carolina middle school principals’ use of data-driven decision making. Dissertation, University of South Carolina, SC, USA.

Bush

(2016) Preparing new principals: Professional and organisational socialisation. Educational Management Administration & Leadership 44(1): 3–5.

10.

Dedering

(2007) Change Management durch Projekte zur Qualitätsentwicklung? Eine Analyse am Beispiel eines internationalen Netzwerkprojekts [Change Management Through Quality Development Projects; an Analysis based on an International Network Project]. In: Berkemeyer

Holtappels

(eds) Schulische Steuergruppen und Change Management: Theoretische Ansätze und empirische Befunde zur schulinternen Schulentwicklung. Weinheim: Juventa, pp.177–200.

11.

Demski

Racherbäumer

(2015) Principals’ evidence-based practice – Findings from German schools. In: Brauckmann

Thiel

Kuper

, et al. (eds) School Leadership in Germany. Special Issue, International Journal of Educational Management 29(6): 735–748.

12.

Demski

Rosenbusch

Van Ackeren

, et al. (2012) Steuerung von Schule durch evidenzbasierte Einsicht? In: Hornberg

Parreira do Amaral

(eds) Deregulierung im Bildungswesen. Münster: Waxmann, pp.131–150.

13.

Dormann

Binnewies

Koch

, et al. (2016) Transferring best evidence into practice: Assessment of evidence-based school management. Journal for Educational Research 8(3): 14–38.

14.

Earl

Katz

(2002) Leading schools in a data-rich world. Aporia Consulting Ltd. OISE/ UT. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.501.7711&rep=rep1&type=pdf (accessed 19 November 2017)

15.

Earl

Katz

(2006) Leading Schools in a Data-Rich World: Harnessing Data for School Improvement. Thousand Oaks, CA: SAGE Publications Inc.

16.

Eger

Egerová

(2016) Project risk management in educational organizations. Educational Management Administration & Leadership 44(4): 578–598.

17.

European Commission/EACEA/Eurydice (2015) Qualitätssicherung im Bildungswesen: Strategien und Konzepte der Schulevaluation in Europa. Eurydice-Bericht. Luxemburg: Amt für Veröffentlichungen der Europäischen Union.

18.

Fishbein

(1963) An investigation of the relationship between beliefs about an object and the attitude toward that object. Human Relations 16(3): 233–239.

19.

Förster

(2015) Organisationale und motivationale Einflussfaktoren auf das Reformhandeln von Lehrkräften – Eine empirische Untersuchung mittels Strukturgleichungs- und Mixture-Analysen. Landau: Empirische Pädagogik.

20.

Geijsel

Krüger

Sleegers

PJC

(2010) Data feedback for school improvement: The role of researchers and school leaders. The Australian Educational Researcher 37(2): 59–75.

21.

Goldring

Grissom

Rubin

, et al. (2015) Make room value added – principals’ human capital decisions and the emergence of teacher observation data. Educational Researcher 44 (2): 96–104.

22.

Groß Ophoff

(2013) Lernstandserhebungen – Reflexion und Nutzung (Pädagogische Psychologie und Entwicklungspsychologie, Band 85). Münster: Waxmann.

23.

Hair

Ringle

Sarstedt

(2013) Partial least squares structural equation modeling: Rigorous applications, better results and higher acceptance. Long Range Planning 46(1-2): 1–12.

24.

Hardy

Lewis

(2016) The ‘doublethink’ of data: Educational performativity and the field of schooling practices. British Journal of Sociology of Education 38(5): 671–685.

25.

Helmke

Hosenfeld

(2005) Ergebnisorientierte Unterrichtsevaluation. In: Interkantonale Arbeitsgemeinschaft Externe Evaluation von Schulen (eds) Schlüsselfragen zur externen Schulevaluation. Bern: h.e.p.-Verlag, pp.127–151.

26.

Henry

(2011) Principals’ use of assessment data to drive student academic achievement. Dissertation, California State University, CA, USA.

27.

Hofer

(2002) Personal epistemology as a psychological and educational construct: An introduction. In: Hofer

Pintrich

(eds) Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing. Mahwah, NJ: Erlbaum, pp.3–15.

28.

Hofer

Pintrich

(1997) The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research 67(1): 88–140.

29.

Holtappels

(2007) Schulentwicklungsprozesse und Change Management: Innovationstheoretische Reflexionen und Forschungsbefunde über Steuergruppen [School development processes and change management. Innovation-theoretical reflections and research findings on steering groups]. In: Berkemeyer

Holtappels

(eds) Schulische Steuergruppen und Change Management - Theoretische Ansätze und empirische Befunde zur schulinternen Schulentwicklung. Weinheim: Juventa, pp.11–40.

30.

Hoogland

Schildkamp

van der Kleij

, et al. (2016) Prerequisites for data-based decision making in the classroom: Research evidence and practical illustrations. Teaching and Teacher Education 60(2016): 377–386.

31.

Hosenfeld

(2010) Führt Unterrichtsrückmeldung zu Unterrichtsentwicklung? Die Wirkung von videographischer und schriftlicher Rückmeldung bei Lehrkräften der vierten Jahrgangsstufe (Empirische Erziehungswissenschaft, Vol. 22). Münster: Waxmann.

32.

Ikemoto

Marsh

(2007) Cutting through the “data-driven” mantra - Different conceptions of data-driven decision making. In: Moss

(ed) Evidence and Decision Making. Malden, MA: Blackwell, pp.105–131.

33.

Jimerson

(2014) Thinking about data: Exploring the development of mental models for “data use” among teachers and school leaders. Studies in Educational Evaluation 42(2014): 5–14.

34.

Köller

Baumert

Neubrand

(2000) Epistemologische Überzeugungen und Fachverständnis im Mathematik- und Physikunterricht. In: Baumert

Bos

Lehmann

(eds) TIMSS/III. 2 - Mathematische und physikalische Kompetenzen am Ende der Gymnasialen Oberstufe. Opladen: Leske + Budrich, pp.229–269.

35.

Leithwood

Jantzi

(2006) Transformational school leadership for large-scale reform: Effects on students, teachers, and their classroom practices. School Effectiveness and School Improvement 17(2): 201–227.

36.

Leithwood

Tomlinson

Genge

(1996) Transformational school leadership. In: Leithwood

Chapman

Corson

, et al. (eds) International Handbook of Educational Leadership and Administration. Dordrecht: Kluwer Academic Publishers, pp.785–840.

37.

Levin

Datnow

(2012) The principal role in data driven decision making - using case study data to develop multimediator models of educational reform. School Effectiveness and School Improvement 23(2): 179–201.

38.

Lewis

(2017) Communities of practice and PISA for schools: Comparative learning or a mode of educational governance? Education Policy Analysis Archives 25(92): 1–25.

39.

McCarley

Peters

Decman

(2016) Transformational leadership related to school climate – a multi-level analysis. Educational Management Administration & Leadership 44(2): 322–342.

40.

Martinez

(2010) Investigating school principals’ personal variables related to the use of data: Implications for professional development. Dissertation, New Mexico State University, NM, USA.

41.

Mingchu

(2008) Structural equation modeling for high school principals’ data-driven decision making: An analysis of information use environments. Educational Administration Quarterly 44(5): 603–634.

42.

Müller

Paechter

Rebmann

(2008) Aktuelle Befunde zur Lehr-Lernforschung - Epistemologische Überzeugungen zu Wissen und Wissenserwerb. bwp@ 14: 1–16.

43.

Muthén

(2010) Mplus User’s Guide. Sixth Edition. Los Angeles, CA: Muthén & Muthén.

44.

Szeto

(2016) Preparing school leaders – The professional development needs of newly appointed principals. Educational Management Administration Leadership 44(4): 540–557.

45.

Park

Datnow

(2009) Co-constructing distributed leadership: District and school connections in data-driven decision-making. School Leadership and Management 29(5): 477–494.

46.

Pennickx

Vanhoof

De Maeyer

, et al. (2015) Effects and side effects of Flemish school inspection. Educational Management Administration & Leadership 44(5): 728–744.

47.

Pietsch

Janke

Mohr

(2014) Führt Schulinspektion zu besseren Schülerleistungen? Difference-in-Differences-Studien zu Effekten der Schulinspektion Hamburg auf Lernzuwächse und Leistungstrends? [Does school inspections lead to better student performance?] Zeitschrift für Pädagogik 60(3): 446–470.

48.

Priemer

(2006) Deutschsprachige Verfahren der Erfassung von epistemologischen Überzeugungen. Zeitschrift für Didaktik der Naturwissenschaften 12: 159–175.

49.

Saunders

(2000) Understanding schools’ use of ‘value added’ data: The psychology and sociology of numbers. Research Papers in Education 15(3): 241–258.

50.

Schiefele

Moschner

Husstegge

(2002) Skalenhandbuch SMILE-Projekt. Bielefeld: Universität Bielefeld, Abteilung für Psychologie, Unveröffentlichtes Manuskript.

51.

Schildkamp

Kuiper

(2010) Data-informed curriculum reform: Which data, what purposes, and promoting and hindering factors. Teaching and Teacher Education 26(3): 482–496.

52.

Schildkamp

Lai

(2013) Conclusions and a data use framework. In: Schildkamp

Lai

Earl

(eds) Data-based Decision Making in Education - Challenges and Opportunities. Dordrecht: Springer, pp.177–191.

53.

Schildkamp

Poortman

(2015) Factors influencing the functioning of data teams. Teachers College Record 117(4): 1–42.

54.

Schildkamp

Teddlie

(2008) School performance feedback systems in the USA and in the Netherlands: A comparison. Educational Research and Evaluation 14(3): 255–282.

55.

Schildkamp

Karbautzki

Vanhoof

(2014) Exploring data use practices around Europe – Identifying enablers and barriers. Studies in Educational Evaluation 42: 15–24.

56.

Schneewind

(2006) Wie Lehrkräfte mit Ergebnisrückmeldungen aus Schulleistungsstudien umgehen – Ergebnisse aus Befragungen von Berliner Grundschullehrerinnen. Dissertation, Freie Universität Berlin, Germany.

57.

Schommer

(1990) Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology 82(3): 498–504.

58.

Schommer-Aikins

Duell

Barker

(2003) Epistemological beliefs across domains using Biglan’s classification of academic disciplines. Research in Higher Education 44(3): 347–366.

59.

Singh

(2017) Performativity, affectivity and pedagogic identities. European Educational Research Journal. Epub ahead of print 31 August 2017. DOI:10.1177/1474904117726181

60.

Stumm

Dormann

Mohr

(2010) Evidence based management – An effective approach for public agencies? In: 27th International Congress of Applied Psychology, Melbourne, Australia, 11 – 16 July 2010, p. 1147. Melbourne: Australian Psychological Society.

61.

Stump

Zlatkin-Troitschanskaia

Mater

(2016) The effects of transformational leadership on teachers’ data use. Journal for Educational Research Online 8(3): 80–99. Available at: http://www.j-e-r-o.com/index.php/jero/article/view/705 (accessed 26 January 2018).

62.

Thiel

Schweizer

Bellmann

(2017) Rethinking side effects of accountability in education: Insights from a multiple methods study in four German school systems. Education Policy Analysis Archives 25(93): 1–32.

63.

Urhahne

(2006) Die Bedeutung domänenspezifischer epistemologischer Überzeugungen für Motivation, Selbstkonzept und Lernstrategien von Studierenden. Zeitschrift für Pädagogische Psychologie 20(3): 189–198.

64.

Urhahne

Hopf

(2004) Epistemologische Überzeugungen in den Naturwissenschaften und ihre Zusammenhänge mit Motivation, Selbstkonzept und Lernstrategien. Zeitschrift für Didaktik der Naturwissenschaften 10: 71–87.

65.

Van Ackeren

Binnewies

Clausen

, et al. (Autorengruppe EviS) (2013) Welche Wissensbestände nutzen Schulen im Kontext von Schulentwicklung? Theoretische Konzepte und erste deskriptive Befunde des EviS-Verbundprojektes im Überblick. Die Deutsche Schule 12: 51–73.

66.

Vanhoof

Vanlommel

Thijs

, et al. (2014) Data use by Flemish school principals: impact of attitude, self-efficacy and external expectations. Educational Studies 40(1): 48–62.

67.

Verhaeghe

Vanhoof

Valcke

, et al. (2010) Using school performance feedback: perceptions of primary school principals. School Effectiveness and School Improvement 21(2): 167–188.

68.

Visscher

Coe

(2003) School performance feedback systems - Conceptualisation, analysis, and reflection. School Effectiveness and School Improvement 14(3): 321–349.

69.

Wayman

Springfield

(2006) Technology-supported involvement of entire faculties in examination of student data for instructional improvement. American Journal of Education 112(August): 549–571.

70.

Wayman

Jimerson

Cho

(2012) Organizational considerations in establishing the data-informed district. School Effectiveness and School Improvement 23(2): 159–178.

71.

Weschenfelder

(2014) Professionelle Kompetenz von Politiklehrkräften - Eine Studie zu Wissen und Überzeugungen. Wiesbaden: Springer.

72.

Wilkinson

Schwartz

(1987) The epistemological orientation of gifted adolescents: An empirical test of Perry’s model. Psychological Reports 61(3): 976–978.

73.

Williams

(2011) A comparison of secondary principals’ use of data systems to increase student achievement in mathematics as measured by standardized assessment. Dissertation, Indiana University of Pennsylvania, PA, USA.

74.

Wurster

Richter

Schliesing

, et al. (2013) Nutzung unterschiedlicher Evaluationsdaten an Berliner und Brandenburger Schulen. Rezeption und Nutzung von Ergebnissen aus Schulinspektion, Vergleichsarbeiten und interner Evaluation im Vergleich. Die Deutsche Schule Beiheft 12: 19–50.

75.

Yilmaz

Bayraktar

(2014) Teachers’ attitudes towards the use of educational technologies and their individual innovativeness categories. Procedia – Social and Behavioral Sciences 116: 3458–3461.

76.

Zlatkin-Troitschanskaia

(2016) Evidence-based actions within the multilevel system of schools - Requirements, processes, and effects (EviS). Journal for Educational Research Online 8(3): 5–13. Available at: http://www.j-e-r-o.com/index.php/jero/article/view/701 (accessed 26 January 2018).

77.

Zlatkin-Troitschanskaia

Seidel

Stump

(2013) Teachers' evidence-based actions – A comparison of different school types. In: Beck

Zlatkin-Troitschanskaia

(eds) From Diagnostics to Learning Success. Proceedings in Vocational Education and Training. Rotterdam: Sense Publishers, pp. 281–924.

78.

Zlatkin-Troitschanskaia

Förster

Preuße

, et al. (2016) The relationship between teachers’ evidence-based actions and communication, cooperation, and participation structures at schools. Journal For Educational Research Online, 8(3), 59–79. Available at: http://www.j-e-r-o.com/index.php/jero/article/view/704