Evaluating the measuring properties of the principal instructional management rating scale in the Chinese educational system

Abstract

During the last 25 years researchers have proposed a number of conceptual frameworks to measure the various functions of instructional leadership. One of the most frequently used frameworks is the Principal Instructional Management Rating Scale (PIMRS). Despite the great number of studies employing the PIMRS, evidence for its reliability and validity is relatively limited. In addition, we still don’t know much about the extent to which this instrument could be used in diverse demographic and cultural educational settings. This study explores the content, face, and construct validity, reliability and internal consistency of the PIMRS in the Chinese educational system. A total number of 311 teachers from five middle schools in the Haidian District of Beijing participated in the study. The data were analysed using Confirmatory Factor Analysis (CFA). On an overall basis the results provided support for the face, content, and construct validity, reliability and internal consistency of the PIMRS. However, six out of the 50 items had to be removed to reach satisfactory fit indices. Implications of the findings in relation to the importance of evaluating the measuring properties of research instruments are discussed and, finally, suggestions for future studies are provided.

Keywords

Instructional leadership PIMRS school leadership in China evaluation of measuring properties Confirmatory Factor Analysis

Introduction

Instructional leadership is identified in the literature as a factor associated with effective schools which improves the quality of teaching. This, in turn, enhances student learning (Antoniou, 2013b; Bush, 2015). Instructional leadership has been defined in a number of different ways; some of these are directly related to activities and others are indirectly related to the processes of teaching and learning (Marks and Printy, 2003; Shatzer et al., 2014). According to Portin et al. (2003: 18), instructional leadership is the process of ‘assuring quality of instruction, modeling teaching practice, supervising curriculum, and assuring quality of teaching resources’. It involves a number of functions, such as coaching, critical reflection, teacher collaboration, teachers as action researchers and generally collaborative and critical thinking on quality of teaching (Glanz and Neville, 1997).

Many studies have been conducted during the last few years to enhance our understanding and importance of instructional leadership (Bush and Glover, 2014; Hallinger and Lee, 2014). According to Leithwood et al. (2004), instructional leadership was seen as having an indirect impact on student outcomes through improving organizational learning culture and staff performance. Likewise, Marks and Printy (2003) found that school effectiveness could be improved by adopting instructional leadership. Robinson et al. (2008) focused on another point, concluding that transformational leadership associated with instructional leadership could have a significant influence on student outcomes and achievements. Lee et al. (2012) have summarized the two main conclusions that stem from all previous studies on the concept. The first was that instructional leadership affected the performance of learning organizations in a positive way. The second was that instructional leadership is a multifaceted structure that adjusts itself in different ways depending on the context, which is of great importance to this study.

During the last 25 years researchers have proposed a number of conceptual frameworks that have aimed to describe the various functions of instructional leadership (e.g. Bossert et al., 1982; Hallinger and Murphy, 1985; Leithwood and Stager, 1989; Ogawa and Bossert, 1995; Pitner, 1988). In addition to the conceptual progress, methodological progress has also been made through the development of improved data collection instruments for the measurement of instructional leadership (e.g. Leithwood and Steinbach, 1991). One of the most renowned and frequently used conceptual frameworks of instructional leadership is the Principal Instructional Management Rating Scale (PIMRS) (Hallinger, 1983; Hallinger and Murphy, 1985). During the last few years, the PIMRS has been used extensively in more than 25 countries and in more than 200 empirical studies (Hallinger and Wang, 2013). According to Hallinger et al. (2013), the PIMRS has maintained a consistent record of yielding reliable and valid data. In particular, and in relation to the PIMRS reliability, Hallinger (2008: 24) states that ‘while relatively few researchers using the instrument sought to replicate the initial findings, several did. The replication studies of reliability and validity included Howe (1995), Jones (1987), Nogay (1995), Sawyer (1997), Taraseina (1993) and Wotany (1999)’.

However, the earlier review by Hallinger (2011) examined broadly the utilization of the PIMRS by various methodologies. As a result, very little information was provided in relation to the reliability and construct validity of the PIMRS based on the results obtained in previous studies. A number of researchers (e.g. Krug, 1990) have discussed some of the main issues with regard to the quality of the PIMRS; for example, one criticism is about the length and complexity of the items. The main criticism is in relation to the lack of supportive evidence with regard to its reliability and validity, which are sometimes hard to assess mainly because of insufficient size of research samples. Most importantly, we still don’t know much about the extent to which the PIMRS is sensitive and appropriate to use in schools having different and diverse student populations, of different sizes and at different levels of education (from primary to secondary), etc. In addition, it is still not clear whether the instrument is sensitive to diverse contextual factors that could modify the interpretation of the items included in the PIMRS (Condon and Matthews, 2010). The above issues stress the importance of an updated evaluation of the validity and reliability of the PIMRS through appropriate statistical approaches, such as Confirmatory Factor Analysis (CFA).

In addition, very little information is available in relation to the measuring properties, and especially about the construct validity, of the PIMRS in the Chinese educational system. As Hallinger et al. (2013) acknowledged, from the rather limited number of studies evaluating the validity of the PIMRS, three were conducted in the United States (Hallinger, 1983; Howe, 1995; Jones, 1987), one in Thailand (Taraseina, 1993) and one in Cameroon (Wotany, 1999). The overall conclusion from the literature is that there is a general shortage of research on leadership in the Chinese context (Walker and Dimmock, 2002). Despite the growing interest in and writings on school leadership in the Chinese context (Sun, 2014), most studies have explored the links between headteachers’ leadership and student outcomes in a Western context, and school leadership has been mainly constructed and developed theoretically by Western scholars (Bush, 2014). Thus, we could claim that a greater focus on a cultural approach to exploring leadership must be undertaken, because leadership in the West and in China possess different cultural roots (Chen and Lee, 2008; Edwards and Turnbull, 2013). Leadership is a socially constructed process, the essences of which are culturally affected. This is important, as not only does the conceptualization of leadership vary, but also the ways in which it is exercised across societal cultures (Dimmock and Walker, 2005; Yukl, 2006). As Dimmock (2011: 321) argues ‘it is timely for educational researchers in Asia to generate cultural- and empirical-based knowledge in school leadership that will speak to the specific interests of Asian students, educators and practitioners’.

Research aims

Exploring the measuring properties of research tools is important and, in addition, has important implications for the research design, the quality of the data and the conclusions drawn. Considering the importance attributed to measuring school leadership and identifying its relation to student outcomes internationally, a detailed and systematic evaluation of the measuring properties of widespread research instruments and tools is necessary for further improvements in this field. This is even more important in the case of the PIMRS, as no systematic attempt to evaluate its measuring properties has been undertaken since 1985 (Hallinger and Murphy, 1985).

In addition, the use of CFA approaches to evaluate the construct validity and internal consistency of the PIMRS has very rarely been reported. For these reasons, and also taking into consideration the contextual and cultural characteristics of the Chinese educational system, the purpose of this study is to explore the face, content, and construct validity, reliability and internal consistency of the PIMRS research tool and provide suggestions to researchers, headteachers and educators in relation to the extent to which it could be used in the Chinese educational system to measure instructional leadership. In doing so, the importance of considering contextual and cultural factors in choosing appropriate methodological tools to measure instructional leadership is also discussed.

Framework of the study – PIMRS

The PIMRS was designed by Hallinger and Murphy (1985) and consists of three dimensions: (a) defining the school mission; (b) managing the instructional program; and (c) developing a positive school learning climate. Each dimension is further analysed into ten instructional leadership functions. The respondents are requested to indicate the frequency of a headteacher’s actions on a scale ranging from 1 (almost never) to 5 (almost always). In particular, the ‘defining the school mission’ dimension consists of two functions: it both frames and communicates the school’s goals. Both of these functions are related to the extent to which a headteacher works with other teachers to develop the school mission and the extent to which this mission is focused on student academic progress. The second dimension relates to the extent to which a headteacher coordinates the school instructional program. This incorporates three leadership functions: (a) supervising and evaluating instruction; (b) coordinating the curriculum; and (c) monitoring student progress. Finally, the third dimension of the PIMRS consists of several functions, such as protecting teaching time, promoting teacher professional development, maintaining high visibility, providing incentives for teachers and providing incentives for learning. This dimension has a broader focus and, to a certain extent, overlaps with factors related to transformational leadership (e.g. Leithwood et al., 2006; Marks and Printy, 2003).

Research methods

Employing a survey research design, five middle (lower secondary) schools have been selected from the Haidian District of Beijing. From a total number of 544 teachers who were working at the five schools in our sample, we distributed 492 questionnaires. A total number of 311 questionnaires were returned (a response rate of 63.2%). Information about the procedure employed in order to translate the PIMRS, and on the participants, is provided below.

Translating the PIMRS

The questionnaire was translated into Chinese following the approach proposed by Beaton et al. (2000). In particular, for the forward translation step, two translations of the original questionnaire were generated by two independent translators, both native speakers of the Chinese language. Following the comparison between the two independent translations, a reconciled language version was developed along with a report elaborating on the reconciliation rationale. Then, for the backward translation step, the reconciled questionnaire in the Chinese language was translated into English by one professional translator, who was a native speaker of the Chinese language and also fluent in English. The backward translation version and the original questionnaire were then compared. Some minor discrepancies that were encountered were resolved.

Sampling

Due to the size and complexity of the educational system in China, it was not possible to gain a representative sample of the whole country. For practical and accessibility reasons, it was decided to focus on the Haidian District of Beijing, which is the second largest district in Beijing. It lies towards the north-western part of the urban core and it is where most universities are located. Schools in the Haidian District are considered as being among the best schools and are usually found at the top of the education league in China.

The sampling approach was based on a random stratified approach, drawing from a list of the Haidian District records where all private schools are assigned to one of three strata/clusters: (a) top-performing schools; (b) average schools; and (c) poor-performing schools, according to their students’ results in the National High School Entrance Examination. From a total number of 66 private schools, 5 schools were selected. Table 1 provides some basic information in relation to the characteristics of the schools in our sample.

As we can observe from Table 1, two of the schools included in the sample were from the 20 top-performing schools, two from the 20 poor-performing schools and the last one from the average schools cluster. The average ‘years of teaching’ was 12.81, and the average ‘years of working with the current headteacher’ was 4.1. These figures support the fact that most of the participants were experienced teachers who were familiar with the school and actions and behaviors of their respective headteachers and, therefore, able to make credible judgments in relation to the items included in the PIMRS.

Table 1.

Characteristics of the schools included in the sample.

	Cluster based on National High School Entrance Examinations	Number of staff	Number of students	Years established	Headteacher’s years at this school
School 1	Top performing	205	1802	15	4
School 2	Top performing	138	1100	16	5
School 3	Average	78	647	12	4
School 4	Poor performing	55	580	21	4
School 5	Poor performing	68	710	21	3

Data analysis

The data analysis was initially conducted for each one of the ten functions of the PIMRS. Then, the extent to which those functions could be incorporated into one of the three dimensions: (a) defining the school mission; (b) managing the instructional program; and (c) developing a positive school learning climate (Hallinger and Murphy, 1985), has been evaluated. In particular, the analysis evaluated the construct validity, the reliability and the internal consistency of each function and dimension. The data were analysed using the SPSS 22, SPSS Amos and EQS software programs.

Content and face validity of the PIMRS

As the PIMRS has very rarely been used in the Chinese context, we considered it important to explore the content and face validity of the questionnaire. The content validity was evaluated in collaboration with two faculty members from the Department of Education in a Chinese university and three experienced headteachers from schools not included in the research sample, and refers to the extent to which the content of the measuring instrument is appropriate and relevant to the research purpose. Content validity indicates whether the content reflects the complete range of the attributes under study and is usually undertaken by a number of experts (Antoniou, 2012; DeVon et al., 2007; Pilot and Hunger, 1999). On an overall basis, the conceptual framework of the PIMRS was found to be comprehensive and satisfactory in relation to contemporary research findings on school leadership research and in relation to leadership practices in Chinese middle schools. However, doubts have been raised in relation to the suitability of some items as far as the Chinese education system is concerned. Such items were mainly related to the flexibility headteachers had with regard to being able to take decisions on issues prescribed by the Ministry of Education (MoE) in China; for example, the school curriculum and dealing directly with students and parents. Some suggestions relating to the extent to which some of the PIMRS items could be expanded so as to capture more accurately issues such as student and teacher behavior outside of the classroom (during breaktime, for example) were provided, as part of the individual school learning climate. At this stage, we decided to keep the PIMRS in its formal and complete version and explore these issues empirically via the results of the CFA analyses.

Second, the face validity of the survey was examined. Face validity indicates the extent to which questionnaires appear to be suitable to a study’s purpose and is considered as the weakest form of validity (DeVon et al., 2007; Haladyna, 1999; Trochim, 2001). The three headteachers, the two faculty members mentioned above and five other teachers, who did not work in any of the schools sampled in the main data collection phase, were asked to evaluate the face validity of the PIMRS. All provided positive comments in terms of how the instrument appeared on an overall basis, and with regard to particular issues related to questionnaire readability, style, language used and formatting.

Internal consistency/reliability of the PIMRS

To evaluate the internal consistency of the PIMRS, Cronbach’s Alpha (Cronbach, 1990) was calculated for each of its ten functions. The results were particularly satisfactory (α > 0.82) for six out of the ten subscales. Further examination of ‘Cronbach’s Alpha if item deleted’ suggested that all items should be retained for these six subscales. However, the ‘Cronbach’s Alpha if item deleted’ for the remaining four functions indicated that the reliability could reach particularly satisfactory results by deleting a number of items. Interestingly, almost all of these items had been identified during the evaluation of the PIMRS content validity by our team of experts as not being particularly applicable to the Chinese context. Based on the ‘Cronbach’s Alpha if item deleted’ results, the six items, presented in Table 2 below, have been removed to reach satisfactory reliability indices.

Table 2.

Items removed from PIMRS to achieve satisfactory reliability scores.

Dimension	Function	N	Item
Managing the instructional program	Coordinates the curriculum	Q17	Draw upon the results of school-wide testing when making curricular decisions
	Coordinates the curriculum	Q20	Participate actively in the review of curricular materials
	Monitors student progress	Q25	Inform students of the school’s academic progress
Developing the school’s learning climate	Maintains high visibility	Q32	Visit classrooms to discuss school issues with teachers and students
	Provides incentives for learning	Q48	Recognize superior student achievement or improvement by asking students to come to the office with their work
	Provides incentives for learning	Q49	Contact parents to communicate improved or exemplary student performance or contributions

PIMRS: Principal Instructional Management Rating Scale.

Once the six items had been taken out, the reliability for each function was recalculated. As demonstrated in Table 3, all functions were found to have α scores ranging from .82 to .94, which indicates high internal consistency in all cases.

Table 3.

Reliability of the PIMRS function.

Function	Number of Items	Cronbach’s Alpha (a)
Frames the school’s goals	5	.93
Communicates the school’s goals	5	.94
Coordinates the curriculum	3	.93
Supervises and evaluates instruction	5	.94
Monitors student progress	4	.90
Protects instructional time	5	.82
Provides incentives for teachers	5	.93
Provides incentives for learning	3	.92
Promotes professional development	5	.94
Maintains high visibility	4	.93

We also followed the suggestion by Hallinger et al. (2013) and employed Ebel’s test (Ebel, 1951) to explore the reliability of the PIMRS. This test provides a reliability estimate which is based on the total aggregated teacher responses from each school. Some researchers (e.g. Howe, 1995; Taraseina, 1993) suggested that Cronbach’s test violates a basic assumption by treating each teacher’s response independently, not taking into account the fact that teachers are grouped into schools. When Ebel’s formula was applied to the data for each of the ten leadership functions, the reliability was again found to be satisfactory and higher than 0.84 for each of the ten functions.

Construct validity of the PIMRS

To evaluate the construct validity of the PIMRS, data were analysed through CFA approaches using the SPSS Amos and EQS software programs. For each of the ten leadership functions of the PIMRS, separate CFA analyses were conducted to help identify the extent to which the theoretical models described by the PIMRS came within acceptable fit indices and parameters. Here, we drew on structural equation modeling (SEM). There are two major types of variables in SEM, observed (indicator) variables and latent (construct) variables. As Schumacker and Lomax (2004: 196) argue, ‘latent variables are not directly observable and hence they are inferred constructs, based on the observed variables that were selected to define each latent variable’. So, to operationalize the latent variables, i.e. the ten functions of the PIMRS, the instrument items (predictors) were used. Missing values were less than 4%, so the typical method of listwise deletion was employed (Allison, 2002).

First-order factors: Construct validity of the PIMRS functions

Having prepared the database for the analyses, first-order CFA models were tested and compared to identify the final model with the optimum fit indices for each function included in the PIMRS. This procedure finally led to the development of ten first-order CFA models, one of each function, demonstrating the construct validity of the questionnaire items in each function. An example relating to the ‘frames the school’s goals’ function is described below.

A first-order CFA model designed to test the multidimensionality of a theoretical construct (Byrne, 1998) was used. In particular, the model aimed to evaluate the construct validity of the ‘frames the school’s goals’ function of the ‘defining the school mission’ dimension. The model hypothesized that: (a) the five variables (i.e. questionnaire items) could be explained by one factor; and (b) each variable would have a non-zero loading on the factor that it was designed to measure, and zero loadings on other factors. The findings of the first-order factor SEM analysis generally affirmed the theory upon which this function of the PIMRS was developed. Particularly, the scaled X² for the one-factor structure (X² = 5.6, df = 2, p >.05) did not reach statistical significance, the Root Mean Square Error of Approximation (RMSEA) was .012 and the Comparative fit index (CFI) was .95, all meeting the criteria for acceptable level of fit. Figure 1 presents the one-factor model and the factor parameter estimates, all of which were statistically significant (p < .001).

Figure 1.

First-order structural equation model for ‘frames the school’s goals’ function.

A similar approach was used for each of the ten functions of the PIMRS. In particular, two first-order CFA models were generated for the ‘defining the school mission’ function, three first-order CFA models were generated for the ‘managing the instructional program’ function and five first-order CFA models were generated for the ‘developing a positive school learning climate’ function. To evaluate the construct validity of the PIMRS subdomains, several fit indices were estimated for each model, such as the significance of X², the Root Mean Square Residual (RMR), Standardized Root Mean Square Residual (SRMR), Goodness of Fit Index (GFI), CFI and RMSEA. In cases where the factor loadings of questionnaire items were not found to be considerably high, alternative models that excluded the particular items were tested and compared with the original models. In all cases, however, it was found that the existing models yielded a better fit than alternative reduced models; thus, all questionnaire items were retained in the final CFA models.

We also considered it important to compare the fit indices of the CFA models with and without the six items that had previously been excluded based on the Cronbach’s Alpha reliability test results. Such comparisons were made for four functions. In particular, comparisons were made for the ‘coordinates the curriculum’ function, with and without Q17 and Q20, for the ‘monitors student progress’ function, with and without Q25, for the ‘maintains high visibility’ function, with and without Q32, and for the ‘provides incentives for learning’ function, with and without Q48 and Q49. In all cases, the results provided empirical support for the construct validity of the reduced scales of the PIMRS functions. Based on the Cronbach’s Alpha and first-order CFA modeling results, the six items have been removed from the second-order factor analyses elaborated below.

Second-order factors: Construct validity of the PIMRS dimensions

The previous section presented the analysis results and in so doing provided support for the construct validity of the ten functions of the PIMRS. Factors obtained from survey correlations are called first-order factors, irrespective of whether they are orthogonal or oblique. Due to the high positive correlation coefficients between all first-order factors, i.e. functions of each dimension, ranging from 0.83 to 0.95 (p < 0.05), it was decided to consider the development of second-order factors: i.e. factors that may be determined from the correlations of the first-order factors, in our case, the three dimensions of the PIMRS. Such factors are important for the interpretation of the correlated variables. Second-order CFA models are usually applied when a measurement instrument measures a number of constructs, each related to the other and each of which is, in turn, measured by several items. The basic assumption is that these distinct but related constructs could be represented by one or more common higher-order constructs (DeYoung et al., 2002).

For example, for the purposes of this study, we tested whether there is a second-order factor for the ‘developing the school learning climate’ dimension that underlies the five specific leadership functions, such as ‘protecting instruction time’ and ‘providing incentives for teachers’ (each assessed by multiple items) (Chen et al., 2005). The assumption is that the higher-order factor could account for the commonality among the specific issues measured by the lower-order factors.

Second-order CFA analyses were performed for each of the three PIMRS dimensions of instructional leadership. Figures 2, 3 and 4 below present the findings and the factor loadings for: (a) defining the school mission; (b) managing the instructional program; and (c) developing a positive school learning climate, respectively.

Figure 2.

Second-order structural equation model for ‘defining the school mission’ dimension.

Figure 3.

Second-order structural equation model for ‘managing the instructional program’ dimension.

Figure 4.

Second-order structural equation model for ‘developing the school learning climate’ dimension.

The following observations could be made from the above figures. First, the standardized factor loadings were all positive (higher than 0.55). The only exception found was in relation to two items from the ‘protects instructional time’ function of the ‘developing the school learning climate’ dimension, with standardized factor loadings of 0.46. The standardized path coefficients between the first- and second-order factors were all higher than .83. All parameter estimates were statistically significant (p < .001). To test the fitting of each model presented above, several fit indices were estimated, such as the significance of X², the RMR, SRMR, GFI, CFI and RMSEA. It was found that the existing models yielded a better fit than alternative reduced models. The results of the analyses provided empirical support for the construct validity of the PIMRS dimensions, and the findings of the second-order factor SEM analysis generally affirmed the theory upon which the PIMRS was developed. In all cases, although the scaled chi-square (X² = 87.1, df = 3, p < 0.001) was statistically significant, the values of RMSEA (ranging from .012 to .031) and CFI (ranging from .955 to .969) met the criteria for acceptable level of fit.

Discussion

This study has evaluated the face, content, and construct validity, reliability and internal consistency of the PIMRS in the Chinese educational system. Although the PIMRS has been utilized extensively in a number of countries (Hallinger, 2011; Hallinger and Chen, 2015; Hallinger and Wang, 2013), the instrument has scarcely been used for data collection in the Chinese educational system. This is important, because exploring the measuring properties of research tools could have important implications for the research design, the quality of the data and the conclusions drawn (Antoniou and Kyriakides, 2011, 2013; Fromm et al., 2016).

Data were collected from 311 teachers from five middle schools in the Haidian District of Beijing. The questionnaire was translated into Chinese following the approach proposed by Beaton et al. (2000), with backward and forward blind translations. The content and the face validity of the questionnaire were explored by a local Chinese team of scholars, headteachers and teachers, with satisfactory results. To evaluate the reliability of the PIMRS, Cronbach’s Alpha (Cronbach, 1990) and Ebel’s (Ebel, 1951) reliability tests were calculated. To evaluate the construct validity of the PIMRS, data were analysed through CFA approaches using the SPSS Amos and EQS software programs. For each of the ten leadership functions of the PIMRS, separate CFA analyses were conducted to help identify the extent to which the theoretical models developed and used in the study came within acceptable fitting indices and parameters. Second-order factor CFA models were also tested for each of the three PIMRS dimensions, i.e. factors that may be determined from the correlations of the first-order factors (DeYoung et al., 2002).

The use of CFA approaches to explore the construct validity of the PIMRS has very rarely been reported. This is important, because several researchers have questioned the reliability and validity of several leadership scales, including the PIMRS (e.g. Condon and Matthews, 2010). The CFA is particularly appropriate for evaluating the extent to which measures of a construct – instructional leadership in our case – are in line with our understanding of the nature of that construct (Kline, 2010). From this perspective, for the data analysis in this study, not only first-order CFA models, but also second-order CFA models were developed. As Dwyer and Sejo (1987) suggest, a second-order structural equation model can be used to combine several related first-order latent variables (i.e. functions of the PIMRS) into a single higher-order latent variable (i.e. one of the three dimensions of the PIMRS) to simplify a structural equation model and a theoretical framework.

Based on the findings of the analyses, to improve the fitting of the model in the Chinese educational system, the PIMRS was reduced from 50 to 44 items. In particular, it was found that two items under the ‘coordinates the curriculum’ function, i.e. (a) draw upon the results of school-wide testing when making curricular decisions and (b) participate actively in the review of curricular materials, one item under the ‘monitor student progress’ function, i.e. inform students of school’s academic progress, one item under the ‘maintains high visibility’ function, i.e. visit classrooms to discuss school issues with teachers and students and, finally, two items under the ‘provides incentives for learning’ function, i.e. (a) recognize superior student achievement or improvement by asking students to bring their work to the office and (b) contact parents to communicate improved or exemplary student performance or contributions, were not found to be relevant and appropriate for measuring instructional leadership in the Chinese context and had to be removed to improve both the reliability of the scales and the fitting of the CFA models.

It is also important to note that these items had also been identified by the Chinese team of local experts we collaborated with during the content validity evaluation of the PIMRS. For example, in relation to the ‘coordinates the curriculum’ function they stressed that the two items mentioned above might not be relevant in the Chinese educational system because the curriculum is strictly set by the MoE and there is very little flexibility left to the headteachers to implement modifications or improvements. Similarly, in relation to the ‘monitor student progress’ and ‘provides incentives for learning’ functions, the team of local experts explained that in China the ‘grade-centered’ administration has shifted the authority and direct responsibility of supervising teachers and students to grade administrators, rather than to headteachers directly.

The results of this study indicate that in our attempts to measure and evaluate the impact of instructional leadership we need to consider two types of leadership functions: holistic and contextual. The holistic functions refer to those functions that are found to work in a number of educational systems, irrespective of the differences in context, and the diverse cultural backgrounds (Antoniou et al., 2015; Creemers et al., 2012; Kyriakides and Creemers, 2009). For example, in this study it was found that 44 out of the 50 items included in the PIMRS are relevant for measuring instructional leadership functions in the Chinese educational system, as found in other educational systems (Hallinger et al., 2013; Heck and Hallinger, 2005).

At the same time, there are also contextual functions to be taken into consideration in our attempts to measure instructional leadership (Crow, 2001). For example, in this study it was found that not all PIMRS items were appropriate for measuring instructional leadership in the Chinese educational system, as we had to remove 6 (out of the 50 items) to reach acceptable and satisfactory fit indices in the CFA analyses. Such contextual factors are based on the assumption that no single set of administrative competences will be effective in all the different schools and social contexts (Davis et al., 2005).

We need to acknowledge that school leadership is a dynamic and multidimensional concept, in which context has an important role. As Hallinger and Heck (1998) argue, there is no universal paradigm or theory for examining organizational behavior that is valid in all social or organizational contexts. A similar argument has been made by colleagues undertaking indigenous research. Such a kind of research requires location-specific contextual factors that must be indigenous, but the theoretical lens can be borrowed (Eacott and Asuga, 2014). Leadership is a socially constructed process, the essences of which are culturally affected. This is important, because not only does the conceptualization of leadership vary, but also the ways in which it is exercised across societal cultures (Bush, 2011; Dimmock and Walker, 2005; Yukl, 2006). Leaders who grow up in different societal cultures have different internalized values and beliefs and these tend to drive them to exercise leadership in particular ways. This is also in line with the results of numerous studies conducted in China that found important variations between leadership in Chinese and Western contexts (Law, 2010).

Although a number of authors have argued about the need to conceptualize school leadership using more complex conceptualizations, studies exploring leadership and its impact are still focusing on certain activities in isolation from the complex context in which school leadership takes place (Yang, 2014). As Mulford (2008: 48) argues,

successful leaders adapt and adopt their leadership practice to meet the changing needs of circumstances in which they find themselves. As schools develop and change, different leadership approaches will inevitably be required and different sources of leadership will be needed so that development work keeps moving.

Any single one-size-fits-all or adjectival approach to leadership will eventually limit leadership effectiveness (Crawford, 2012).

In an attempt to better understand how instructional leadership is implemented and how such practices vary between different educational systems, contexts and cultures, and make an impact on the efforts for improvement, we argue in this paper that school leadership must be conceptualized as a complex system rather than as a linear series of events or actions (Collins and Clarke, 2008; Opfer and Pedder, 2011), consisting of both holistic and contextual functions.

In such complex and dynamic systems, the importance of context in our efforts to explore the impact of leadership needs to be acknowledged (Antoniou, 2013a; Kyriakides et al., 2015). This is mainly because all leadership described as successful is contingent on certain contextual circumstances, and research findings support the fact that school leaders interact and behave in a different manner, depending on the conditions they face at any given time (Creemers and Kyriakides, 2010) and on the teachers and stakeholders with whom they are interacting. As Gronn (2003) argues, research about the forms and effects of leadership is becoming increasingly sensitive to the contexts in which leaders work and how, in order to be successful, leaders need to respond flexibly to their contexts. The results of this study are also in line with the claims of Li et al. (2012) who claim that given that almost all extant theories of management are built upon the philosophies and values of the West (Leung, 2012), indigenous research in non-Western cultures, which have different intellectual and cultural traditions, has immense potential to contribute to universal theories by modifying, enriching or supplementing Western theoretical concepts of management.

On an overall basis, the results of this study provided support for the face, content, and construct validity, reliability and internal consistency of the PIMRS in the Chinese educational system. The fit indices of the CFA models with the 44 items demonstrated that the instrument holds together well as an entity and as separate factors. The findings of the second-order factor SEM analysis generally affirmed the theory upon which this function of the PIMRS was developed. At the same time, six items had to be removed, which indicates, as mentioned earlier, the important role of contextual and cultural factors in measuring instructional leadership. These results indicate the importance of evaluating the measuring properties of any research instrument, for example the face, content, and construct validity, reliability and internal consistency, especially when these are to be adopted and used in different settings that are different from the ones the instruments have been developed for. The results also stress the significance of considering contextual and cultural factors in future attempts to measure instructional leadership.

By no means could the sample of this study, drawn from one district only, represent the whole of China. Future studies could also build on the findings of this study to assess the external validity of the PIMRS by drawing on different samples and cultural contexts. In addition, the external validity of the PIMRS could be evaluated by comparing its results with the results from other instruments (Hallinger et al., 2013). In relation to the research design, we could argue that there is a need for mixed methods research design projects to measure both the generic and contextual factors of school leadership, since in the past both qualitative case studies and quantitative studies were conducted in isolation and along parallel routes. By combining both strategies in a mixed methods design, future studies could overcome problems of external validity and generalizability, which occur in case studies, and, at the same time, problems of interpretation and de-contextualization, which occur in quantitative large-sample studies. Such studies could further enhance our understanding of the suitability and external validity of the PIMRS in measuring instructional leadership, especially in diverse educational settings. This could also be useful for researchers and educators in choosing research instruments to measure instructional leadership and in making methodological choices when using the PIMRS.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Allison

(2002) Missing Data. Thousand Oaks, CA: SAGE Publications.

Antoniou

(2012) The short- and long-term effects of secondary schools upon students’ academic success and development. Educational Research and Evaluation 18(7): 621–640.

Antoniou

(2013a) A longitudinal study investigating relations between stages of effective teaching, teaching experience and teacher professional development approaches. Journal of Classroom Interaction 48(2): 25–40.

Antoniou

(2013b) Development of research on school leadership through evidence-based and theory-driven approaches: A review of school leadership effects revisited. School Effectiveness and School Improvement 24(1): 122–128.

Antoniou

Kyriakides

(2011) The impact of a dynamic approach to professional development on teacher instruction and student learning: Results from an experimental study. School Effectiveness and School Improvement 22(3): 291–311.

Antoniou

Kyriakides

(2013) A dynamic integrated approach to teacher professional development: Impact and sustainability of the effects on improving teacher behavior and student outcomes. Teaching and Teacher Education 29(1): 1–12.

Antoniou

Kyriakides

Creemers

BPM

(2015) A dynamic integrated approach to teacher professional development. Rationale and main characteristics. Teacher Development 19(4): 535–552.

Beaton

Bombardier

Guillemin

Ferraz

(2000) Guidelines for the process of cross-cultural adaptation of self-report measures. Spine 25: 3186–3191.

Bossert

Dwyer

Rowan

Lee

(1982) The instructional management role of the principal. Educational Administration Quarterly 18(3): 34–64.

10.

Bush

(2011) Theories of Educational Leadership and Management. 4th ed. London: SAGE Publications.

11.

Bush

(2014) Educational leadership and leadership development in Africa: Building the knowledge base. Educational Management Administration & Leadership 42(6): 787–791.

12.

Bush

(2015) Understanding instructional leadership. Educational Management Administration & Leadership 43(4): 487–489.

13.

Bush

Glover

(2014) School leadership models: What do we know? School Leadership and Management 34(5): 553–571.

14.

Byrne

(1998) Structural Equation Modeling with LISREL, PRELIS and SIMPLIS: Basic Concepts, Applications and Programming. Mahwah, NJ: Lawrence Erlbaum Associates.

15.

Chen

Lee

(2008) Leadership and Management in China – Philosophies, Theories, and Practices. Cambridge: Cambridge University Press.

16.

Chen

Sousa

West

(2005) Testing measurement invariance of second-order factor models. Structural Equation Modeling 12(3): 471–492.

17.

Condon

Matthews

(2010) Measuring Principal Performance: How Rigorous are Commonly used Principal Performance Assessment Instruments? A Quality School Leadership Issue Brief . Naperville, IL: Learning Point Associates.

18.

Collins

Clarke

(2008) Activity frames and complexity thinking: honouring both public and personal agendas in an emergent curriculum. Teaching and Teacher Education 24: 1003–1014.

19.

Crawford

(2012) Solo and distributed leadership: Definitions and dilemmas. Educational Management Administration & Leadership 40(5): 610–620.

20.

Creemers

BPM

Kyriakides

Antoniou

(2012) Teacher Professional Development for Improving Quality of Teaching. New York, USA: Springer Publishing.

21.

Creemers

BPM

Kyriakides

(2010) Explaining stability and changes in school effectiveness by looking at changes in the functioning of school factors. School Effectiveness and School Improvement 21(4): 409–427.

22.

Cronbach

(1990) Essentials of Psychological Testing. 3rd ed. New York: Harper and Row.

23.

Crow

(2001) School leader preparation: A short review of the knowledge base. Nottingham: National College for School Leadership. Available at: http://www.ncsl.org.uk/researchpublications (accessed 11 August 2016)

24.

Davis

Darling-Hammond

LaPointe

Myerson

(2005) Review of Research, School Leadership Study: Developing Successful Principals. Stanford, CA: Educational Leadership Institute.

25.

DeVon

Block

Moyle-Wright

Hayden

Lazzara

Savoy

Kostas-Polston

(2007) A psychometric toolbox for testing validity and reliability. Journal of Nursing Scholarship 39(2): 155–164.

26.

DeYoung

Peterson

Higgins

(2002) Higher-order factors of the big five predict conformity: Are there neuroses of health? Personality and Individual Differences 33: 533–552.

27.

Dimmock

(2011) Formulating a research agenda in school leadership and organisational change for school improvement in Singapore. School Leadership and Management 31(4): 321–338.

28.

Dimmock

Walker

(2005) Educational Leadership: Culture and Diversity. London: SAGE Publications.

29.

Dwyer

Sejo

(1987) Output sector munificence effects on the internal political economy of marketing channels. Journal of Marketing Research 24(November): 347–358.

30.

Eacott

Asuga

(2014) School leadership preparation and development in Africa: A critical insight. Educational Management Administration & Leadership 42(6): 919–934.

31.

Ebel

(1951) Estimation of the reliability of ratings. Psychometrika 16: 407–424.

32.

Edwards

Turnbull

(2013) A cultural approach to evaluating leadership development. Advances in Developing Human Resources 15(1): 46–60.

33.

Fromm

Hallinger

Volante

Wang

(2016) Validating a Spanish version of the PIMRS: Application in national and cross-national research on instructional leadership. Educational Management Administration & Leadership. Epub ahead of print 23 May 2016. DOI: 1741143215617948.

34.

Glanz

Neville

(1997) Educational Supervision: Perspectives, Issues, and Controversies. Norwood, MA: Christopher-Gordon.

35.

Gronn

(2003) The New Work of Educational Leaders: Changing Leadership Practice in an Era of School Reform. London: Paul Chapman.

36.

Haladyna

(1999) Developing and Validating Multiple-Choice Test Items. Mahwah, NJ: Lawrence Erlbaum Associates.

37.

Hallinger

(1983) Assessing the instructional management behavior of principals. PhD Thesis, Stanford University, USA.

38.

Hallinger

(2008) A review of PIMRS studies of principal instructional leadership: Assessment of progress over 25 years. Paper presented at the Annual meeting of the American Educational Research Association (AERA), New York, 14 May 2008.

39.

Hallinger

(2011) A review of three decades of doctoral studies using the principal instructional management rating scale: A lens on methodological progress in educational leadership. Educational Administration Quarterly 47(2): 271–306.

40.

Hallinger

Chen

(2015) Review of research on educational leadership and management in Asia: A comparative analysis of research topics and methods, 1995–2012. Educational Management Administration & Leadership 43(1): 5–27.

41.

Hallinger

Lee

(2014) Mapping instructional leadership in Thailand: Has education reform impacted principal practice? Educational Management Administration & Leadership 42(1): 6–29.

42.

Hallinger

Murphy

(1985) Assessing the instructional leadership behavior of principals. Elementary School Journal 86(2): 217–248.

43.

Hallinger

Wang

(2013) Measurement Properties of the Principal Instructional Management Rating Scale. Technical Report, version 4.2. Hong Kong: Hong Kong Institute of Education, Asia Pacific Centre for Leadership and Change.

44.

Hallinger

Wang

Chen

(2013) Assessing the measurement properties of the principal instructional management rating scale: A meta-analysis of reliability studies. Educational Administration Quarterly 49(2): 272–309.

45.

Heck

Hallinger

(2005) The study of educational leadership and management. Where does the field stand today? Educational Management Administration & Leadership 33(2): 229–244.

46.

Howe

(1995) Instructional leadership in catholic elementary schools: An analysis of personal, organizational, and environmental correlates. PhD Thesis, Stanford University, USA.

47.

Jones

(1987) The relationship between principal behavior and student achievement in Canadian secondary schools. PhD Thesis, Stanford University, USA.

48.

Kline

(2010) Principles and Practice of Structural Equation Modeling. 3rd ed. New York: Guilford Press.

49.

Krug

(1990) Leadership and learning: A measurement-based approach for analyzing school effectiveness and developing effective school leaders. Project Report No. BBB28334. Urbana, IL: National Center for School Leadership.

50.

Kyriakides

Creemers

BPM

(2009) The effects of teacher factors on different outcomes: Two studies testing the validity of the dynamic model. Effective Education 1(1): 61–85.

51.

Kyriakides

Creemers

BPM

Antoniou

Demetriou

Charalambous

(2015) The impact of school policy and stakeholders’ actions on student learning: a longitudinal study. Learning and Instruction 36: 113–124.

52.

Law

(2010) Culture and school leadership in China: Exploring school leaders’ views of relationship- and rule-based governance. In: Wiseman

(ed.) Educational Leadership: Global Contexts and International Comparisons. Bingley: Emerald Publishing, pp.303–341.

53.

Lee

Hallinger

Walker

(2012) A distributed perspective on instructional leadership in International Baccalaureate (IB) schools. Educational Administration Quarterly 48(4): 664–698.

54.

Leithwood

Stager

(1989) Expertise in principals’ problem solving. Educational Administration Quarterly 25(2): 126–161.

55.

Leithwood

Steinbach

(1991) Indicators of transformational leadership in everyday problem-solving of school administrators. Journal of Personnel Evaluation in Education 4(3): 221–244.

56.

Leithwood

Day

Sammons

Harris

Hopkins

(2006) Seven Strong Claims about Successful School Leadership. Nottingham: National College of School Leadership.

57.

Leithwood

Louis

Anderson

Wahlstrom

(2004) Review of Research: How Leadership Influence Student Learning? New York: Wallace Foundation.

58.

Leung

(2012) Indigenous Chinese management research: Like it or not, we need it. Management and Organizational Review 8(1): 1–5.

59.

Leung

Chen

Luo

J-D

(2012) Indigenous research on Chinese management: What and how. Management and Organizational Review 8(1): 7–24.

60.

Marks

Printy

(2003) Principal leadership and school performance: An integration of transformational and instructional leadership. Educational Administration Quarterly 39(3): 370–397.

61.

Mulford

(2008) The Leadership Challenge: Improving Learning in Schools. Victoria: Australian Council for Educational Research.

62.

Nogay

(1995) The relationship of superordinate and subordinate gender to the perceptions of leadership behaviors of female secondary principals. PhD Thesis, Youngstown State University, USA.

63.

Ogawa

Bossert

(1995) Leadership as an organizational quality. Educational Administration Quarterly 31(2): 224–243.

64.

Opfer

Pedder

(2011) Conceptualizing Teacher Professional Learning. Review of Educational Research 81(3): 376–407.

65.

Pilot

Hunger

(1999) Nursing Research: Principals and Methods. Philadelphia: Lippincott Williams & Wilkins.

66.

Pitner

(1988) The study of administrator effects and effectiveness. In: Boyan

(ed.) Handbook of Research in Educational Administration. New York: Longman, pp.106–132.

67.

Portin

DeArmond

Gundlach

Schneider

(2003) A Practical Look at School Leadership. Policy Brief . Seattle, WA: Center on Reinventing Public Education.

68.

Robinson

VMJ

Lloyd

Rowe

(2008) The impact of leadership on student outcomes: An analysis of the differential effects of leadership type. Educational Administration Quarterly 44(5): 635–674.

69.

Sawyer

(1997) The relationship between cognitive styles and instructional leadership . PhD Thesis, University of Missouri, USA.

70.

Schumacker

Lomax

(2004) A Beginner’s Guide to Structural Equation Modelling. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates.

71.

Shatzer

Caldarella

Hallam

Brown

(2014) Comparing the effects of instructional and transformational leadership on student achievement: Implications for practice. Educational Management Administration & Leadership 42(4): 445–459.

72.

Sun

(2014) 30 years of Chinese educational management: Achievements, characteristics and problems. International Journal of Educational Management 28(3): 340–348.

73.

Taraseina

(1993) Assessing instructional leadership behavior of secondary school principals in Thailand. PhD Thesis, Peabody College for Teachers of Vanderbilt University, Nashville, USA.

74.

Trochim

WMK

(2001) The Research Methods Knowledge Base. Cincinnati, OH: Atomic Dog.

75.

Walker

Dimmock

(2002) Cross-cultural and comparative insights into educational administration and leadership: An initial framework. In: Walker

Dimmock

(eds) School Leadership and Administration: Adopting a Cultural Perspective. New York: Routledge Falmer, pp.13–32.

76.

Wotany

(1999) The training, selection, and hiring practices of Cameroonian school leaders and the evaluation of the present effectiveness of their instructional leadership behaviors. PhD Thesis, University of Missouri, USA.

77.

Yang

(2014) Principals’ transformational leadership in school improvement. International Journal of Educational Management 28(3): 279–288.

78.

Yukl

(2006) Leadership in Organizations. 6th ed. Upper Saddle River, NJ: Prentice Hall.