Abstract
The purposes of this study were to report on a systematic approach to validating a Spanish version of the Principal Instructional Management Rating Scale and then to apply the scale in a cross-national comparison of principal instructional leadership. The study yielded a validated Spanish language version of the PIMRS Teacher Form and offers a model of scale validation for other scholars who are interested in preparing an instrument for use outside of its culture of origin. The researchers used a back translation and expert panel procedure combined with psychometric analysis in order to achieve cultural equivalency for the translation. Psychometric properties of reliability (G Theory test of reliability) and validity (CFA, Rasch analysis) of the new version were assessed and compared to the original version in order to underpin its validity. The validation study employed a sample of 595 teachers and 26 principals in Santiago, Chile. The study yielded both a comprehensive analytical approach to cultural adaptation of survey instruments, as well a scale that can be used in Spanish speaking contexts. The study further demonstrated variability in patterns of instructional leadership practice across several national contexts.
Keywords
Introduction
During the early years in the development of educational administration as a discipline, research reviews highlighted the lack of high quality research tools as a key obstacle holding back the accumulation of knowledge (Bridges, 1982; Erickson, 1967; Haller, 1979). In response to these critiques, scholars made a concerted effort to strengthen the research tools used in educational administration (Hallinger and Heck, 1996; Leithwood et al., 1990). Several decades hence, scholars have identified methodological progress in the methods used in research on educational administration (Goldring et al., 2012; Hallinger, 2011a; Hallinger and Heck, 1996; Heck and Hallinger, 2005; Krüger et al., 2007; Leithwood and Sun, 2012). Moreover, subsequent reviews have concluded that the field of educational administration is finally beginning to accumulate a useful body of knowledge (Hallinger, 2011a, 2011b; Hallinger and Heck, 1996; Leithwood et al., 2008; Leithwood and Sun, 2012).
One of the most widely used research tools deployed in educational leadership studies over this period has been the Principal Instructional Management Rating Scale or PIMRS (Hallinger, 1982, 1990; Hallinger, 2011a; Hallinger and Murphy, 1985; Hallinger and Wang, 2015). Since 1982, the PIMRS has been used in more than 325 published studies, master theses and doctoral dissertations conducted in over 30 countries around the world (Hallinger and Wang, 2015). Although the trend of PIMRS use in recent years has increasingly shifted towards scholars outside of the USA (Hallinger and Wang, 2015), few ‘international users’ have taken steps to validate the PIMRS for use outside the instrument’s ‘zone of origin’ (Hallinger and Wang, 2015). Instead scholars have tended to assume that ‘simple translation’ of a validated instrument was sufficient for their research. We characterize this modal approach in the global use of the PIMRS (and other scales) as a threat to research quality.
We further suggest that scholars need to understand the need and procedures for validating research instruments when used outside their ‘zone of origin’. This study addressed three main goals: to develop a Spanish language version of the PIMRS and validate its use in the assessment of principals in Chile; to develop a profile of primary school principal instructional leadership for Chile; to compare patterns of instructional leadership practiced by the Chilean principals with principals in a cross-national sample.
The validation study surveyed 595 teachers and 26 principals in Santiago de Chile using a pilot Spanish-language version of the PIMRS Teacher-Form. Advanced analytical tests were then employed in both the validation and cross-national comparative studies.
The study yielded a Spanish-language version of the PIMRS that can be employed not only for research, but also for principal needs assessment and performance evaluation (Hallinger and Wang, 2015). From a research perspective, the study responds to critiques that have highlighted an Anglo-American dominance of empirical research in our field (Bajunid, 1996; Hallinger and Leithwood, 1996; Walker and Dimmock, 2002). Finally, from a policy standpoint, the study demonstrates how the use of a validated instrument can assist in developing a multi-dimensional perspective on a nation’s principal instructional leadership capacity.
Theoretical Background
In this section of the article we discuss the relevance of instructional leadership as a focus for research and practice in the global education context. Then we introduce the conceptual framework underlying the PIMRS instrument, and provide background information about the scale. Finally, we provide an overview of how school leadership is approached in the Chilean education context as a means of illuminating the relevance of the research project.
Instructional Leadership
Principal instructional leadership has its roots in the USA (Bridges, 1967; Gross and Herriot, 1965; Lipham, 1981). As far back as the 1950s, American school principals were recognized as ‘potentially pivotal figures’ with a determining influence on teaching and learning quality in their schools (Lipham, 1981). These observations suggest that instructional leadership was born as a practice-based rather than a theory-driven construct (Bridges, 1967). Thus, scholars during the 1960s, 1970s and 1980s highlighted the need to develop better conceptualizations of ‘instructional leadership’ as well as validating the means by which this approach to leadership practice achieved its ‘effects’ on teaching and learning (Bridges, 1967, 1982; Erickson, 1967; Hallinger and Murphy, 1985).
During subsequent decades, a global community of scholars has engaged in programmatic research that has sought to understand the nature of effective school leadership and how it is linked to improvements in teaching and learning (Hallinger and Heck, 1996; Heck and Hallinger, 2014; Heck et al., 1990; Kruger et al., 2007; Leithwood et al., 2008; Leithwood and Sun, 2012; Neumerski, 2013; Rigby, 2013; Sebastian and Allensworth, 2012; Walker and Ko, 2011). Improvements in the application of theory as well as in research methods have gradually led to the growth of a knowledge base with stronger foundations (Heck and Hallinger, 2005; Leithwood et al., 2008). Thus, when compared with research reviews published in the 1960s, 1970s and 1980s, later reviews of research in educational administration have yielded a more optimistic set of findings concerning the nature of ‘school leadership that makes a diference’ (Hallinger, 2011b; Hallinger and Heck, 1996; Leithwood et al., 2008; Leithwood and Sun, 2012). Nonetheless, observations that this research remains largely concentrated in economically developed, Anglo-American contexts have led to calls for research on school leadership in a broader set of cultural settings (Hallinger and Leithwood, 1996; Walker and Dimmock, 2002).
The PIMRS Framework
The PIMRS instrument is grounded in a conceptual framework developed by Hallinger and Murphy (1985). The framework incorporates three dimensions and 10 job functions (see Figure 1).
PIMRS instructional leadership model (Hallinger and Murphy, 1985: 221).
The first dimension, Defines the School Mission, is comprised of two functions, Frames and Communicates the School Goals. This dimension recognizes the frequently discussed role of leaders in helping to define purpose and setting direction for the organization (Leithwood et al., 2008). Developing a shared vision and mission for the school is without question a requisite foundation for school improvement (Murphy and Torre, 2014).
The second dimension, Manages the Instructional Program, is comprised of three job functions concerned with ‘managing the technical core’ of the school. Relevant job functions concern the role of school leadership in developing the quality of teaching and learning through direct supervision of teachers, coordinating of curriculum activities, and monitoring progress towards shared learning goals.
The third dimension, Promotes a Positive School Learning Climate, is comprised of five job functions that together help to create an ‘academic press’ for learning in the school. Through these job functions the principal provides normative support for the school’s mission and maintains positive pressure and support for shared commitments to improve the quality of teaching and learning.
The PIMRS Instrument
Consistent with this framework, the PIMRS instrument employs five items to measure each of the 10 instructional leadership functions. Item are ‘behaviorally anchored’ in the sense that they describe critical job related behaviors (Latham and Wexley, 1981). The ratings range in a five-point Likert-type scale from (1) almost never to (5) almost always. Therefore, the PIMRS assesses the frequency with which the principal enacts a behaviour or practice associated with that particular instructional leadership function (Hallinger and Wang, 2015). Scores may be calculated for the full scale, the three dimensions, or for the 10 job functions depending upon the purpose of the assessment (for example, research, performance evaluation, needs assessment).
The PIMRS has three different ‘forms’ for assessment by teachers, supervisors and principals. The PIMRS Teacher and Principal Forms have undergone the most systematic validation (Hallinger and Wang, 2015). Although each of these forms has been documented to provide reliable, valid data for these varied purposes, prior studies have affirmed the PIMRS Teacher form as the ‘first choice instrument’ (Hallinger, 2011; Hallinger and Murphy, 1985; Hallinger and Wang, 2015; Hallinger et al., 2013). With this in mind, the current study focused solely upon validation of a Spanish language version of the PIMRS Teacher Form.
The Chilean Setting for Principal Instructional Leadership
Currently Chile’s national school system has 3.2 million students taught by 190,000 teachers. There are approximately 9000 school principals distributed in 346 municipalities. Most of the schools have a formal managerial team formed by three to four staff (MINEDUC, 2014; Volante et al., 2012).
Since 2005, policies oriented to school leadership have acquired increased relevance in Chile. Consequently, the country has allocated increasing funds for the training, development and evaluation of principals (MINEDUC, 2014). Legislative changes enacted over the past decade have increasingly required principals to take a more proactive role in exercising instructional leadership (Avalos and Assael, 2006), as well as enhancing this role among middle level leaders (Weinstein and Muñoz, 2012). Notably, however, to date Chilean policymakers and practitioners have lacked the tools necessary for properly assessing this domain of principal practice.
Methodology
In this section of the article we present the method of instrument translation, data collection and data analysis. The validation methodology applied here closely follows the steps reported by Hallinger and Wang (2015) in their revalidation study of the original PIMRS instrument. The cross-national comparison of Chilean data with data obtained from several other countries was used to provide a benchmark for gaining further insight into the particular pattern demonstrated by the Chilean results.
Content Validity
Prior to translation of the PIMRS, we assessed the dimensions and functions in terms of their suitability for the Chilean context. After analysing the conceptual definitions of the three dimensions and 10 instructional leadership functions, a team of Chilean scholars agreed that they all appeared relevant to the current Chilean education context. Although it could be argued that selected ítems could be viewed as more or less salient to instructional leadership practice in Chile, we did not undertake advance revision of ítems for two reasons. First, our conceptual assessment already offered reasonable confidence that the broader constructs were relevant. Second, we felt that the deletion or revision of PIMRS items was premature and was more suitably undertaken via empirical examination of ítem functioning.
PIMRS Translation into Spanish
Effective translation of a rating instrument represents an ‘interpretative challenge’ (Perneger et al., 1999). Indeed, the goal of instrument adaptation is to achieve cultural equivalence so that the assessment tool and its underlying constructs retain their meaning in the new cultural context. If the original concept is assumed to be universal when it is not, simple translation of items may achieve criterion equivalency but not construct equivalency (Jones and Kay, 1992; Marsh et al., 2006).
Although several approaches were considered, we settled on back translation (Brislin, 1986) complemented by psychometric assessment to achieve content equivalency. This combination of translation methods would allow us to maximize the strengths of back translation while also addressing its limitations. Back translation (Brislin, 1986) consists of using one translator to translate the selected text into a target language and another translator (blind to the original version) to translate it back.
We implemented back translation through a four step process as described below. First, a bilingual researcher began with forward translation of English version of the PIMRS into the target language, Spanish. Technical understanding of the conceptual content of the instrument, and of the target population are critical factors in maximizing the first. The resulting Spanish version was reviewed, without the English version at hand, by a panel of two limited English-speaking and two bilingual colleagues. Each panel member had expertise in the educational leadership field and practical knowledge of school leaders’ job functions. The draft PIMRS Spanish version was reviewed for incomprehensible, misleading, culturally inapproriate or ambiguous wording. Two 3 hour sessions were dedicated for this step.
An external professional translator with specific experience in school education was hired for the next step of back translation. The backward translator worked ‘blind’ to the original English version of the PIMRS instrument. This step surfaced discrepancies between the meaning of items in the translated and original versions. The researchers then compared the back-translated version of the PIMRS Teacher Form-Spanish with the original PIMRS English version for linguistic congruence and cultural relevance. Every item was analysed to identify and correct any possible discrepancies, no matter how small. It was the case that some problems originated in the forward translation and some in the backward translation. Two sessions with the expert panel settled all questionable or discrepant cases. The final version was unanimously accepted by the panel and the researchers.
Our validation process also evaluated translation equivalency through analysis of the psychometric properties of both language versions (Jones and Kay, 1992). This was used as a complementary means of checking the results of our back translation procedure. These results are described later in the article.
Data Collection
Chilean Data Employed for the Valiation Study
The PIMRS Teacher Form-Spanish was applied together with demographic questions and a teacher’s social justice orientation scale to a sample of 641 Chilean teachers. After accounting for omitted cases, we had a functional sample of 595 teacher respondents who were assessing their principals. The 595 surveys were gathered during visits to 26 schools. In each of these 26 schools, at least 10 teachers completed the survey.
The average age of the teachers was 40 years old, with 15 years of work experience and four years with their current principals. Of the 595 teachers, 72% were females and 28% male. These sample characteristics are close to Chile’s national means reported by the international TALIS 2013 study (OECD, 2013). This suggested to us that the sample was suitable for our purposes in terms of representativeness and time duration spent with their principals.
The principals assessed in this study had worked an average 4.5 years in their schools. They were equally distributed between males and females. There were three kindergarden principals, 10 primary school principals and 13 secondary school principals. After comparing our principal data with data from the recent TALIS study, we conclude that the sample is quite representive of the national population of teachers and principals by age, gender, and years of experience.
Data Employed for the Cross-national Comparative Study
Secondary data used in the cross-national comparative study was obtained by the researchers from the publisher who maintains a data bank comprised of datasets of completed PIMRS studies conducted around the world. For our analytical purposes, we requested PIMRS datasets consisting of teacher perceptions collected in primary school from several countries: USA (Dale and Phillips, 2011; Goldring et al., 2012; Greb 2011; Shatzer 2009), Turkey (Bellibas et al., in press), Malaysia (Wong, 2010) and Maldives (Shafeeu, 2011). This range of nations covered both developing and developed countries from several regions of the world.
The sample sizes were as follows: Turkey, 294 teachers and 23 principals; Chile, 595 teachers and 26 principals; Malaysia, 98 teaches and 14 principals; Maldives, 201 teachers and 14 principals; USA 1729 and 549 principals. While the samples from Malaysia and Maldives were quite small, they were still included since the goal was to conduct a ‘preliminary comparison of profiles’.
Data Analysis
Reliability
Reliability measures the consistency or accuracy of responses and is a precondition for validity. As reported by Hallinger and colleagues (2013), researchers have most frequently employed Cronbach’s (1951) alpha test of internal consistency in checking the reliability of the PIMRS. The use of ‘Cronbach’s alpha’ has become so ubiquitous in survey research that it has almost become synonymous with the concept of reliability. Use of Cronbach’s alpha for analysis of the reliability of the PIMRS Principal Form represents a correct application of the test, but not for teacher samples (Hallinger 2011a; Hallinger and Wang, 2015; Hallinger et al., 2013).
When analysing a PIMRS data set obtained from teacher respondents, Cronbach’s alpha test treats aggregates the total sample of teachers from different schools into a single group in order to arrive at an assessment of the instrument’s internal consistency. In reality, however, teachers are ‘nested’ within schools, and each school’s teachers are actually rating their own principal. Thus, the data structure violates the assumption of independent responses between samples on which Cronbach’s alpha is based.
Given the nested structure of PIMRS teacher data, Hallinger and colleagues (2013) argued that a Generalizability Theory (Gen Theory) test generates a more valid assessment of reliability (Cronbach et al., 1972; Kane et al., 1976). We used SPSS 20.0 to run the Gen Theory test of reliability on the teacher data in our study.
Construct Validation
Proper use of data collected by a survey instrument requires that the user be able to justify both the accuracy of the instrument and the inferences drawn from the scores that it yields. As Nunnally (1978: 87) stated, ‘strictly speaking, one validates not a measuring instrument but rather some use to which the instrument is put’. This study focused on the internal validity of the PIMRS teacher scale. More specifically, we targeted construct validity by examining the relationship among a set of theoretically related factors through empirical analysis at both the item level and the test level.
At the ‘test level’, we employed confirmatory factor analysis (CFA) to examine the relationship between the items and the theoretical constructs (the measurement model) comprising instructional leadership (see Figure 1). A measurement model transforms categorical item responses to a location along a construct map which describes the characteristics of constructs. As described earlier, the PIMRS framework can be characterized as a ‘two-level higher order model’ (Andrich, 1978). The ‘goodness of fit’ between this conceptual framework and the empirical data yielded from the instrument were assessed by several indices. The statistical software package MPLUS 7 was used to run CFA with a mean- and variance-adjusted weighted least squares (WLSMV) estimator.
At the ‘item level’ analyses, both classical testing theory and item response theory guided our approach to check whether the items fit the hypothesized structure of the PIMRS conceptual framework (Nunnally, 1978). The validity of an individual item is quantified by the ‘item-total correlation’, or the bivariate correlation between the item scores and test scores. Item-total correlation is often used to assess the discrimination of items.
Another index used for construct validation is ‘item difficulty’. As all items in the PIMRS use the same rating scale, the Rasch rating scale model (Andrich, 1978) was chosen to guide our analytical procedures. Its an advantage of Rasch analysis is that the quality of each item can be tested by reference to an item-fit index. In test development, we propose that items located in the same sub-scale are assessing the same latent trait or construct. This statistic measures the fit of the observed data to the expectation of the Rasch model. Finally, a RSM validity analysis was conducted to assess the goodness to fit index of every item. The index is termed outfit mean square (outfit MNSQ) and similarly relates the item to its theoretically defined trait.
Profile Development and Cross-national Comparative Analaysis
Finally, we examined the instructional leadership practices of the Chilean primary school principals in a two step process. First, we used descriptive statistics and Rasch analysis to portray their dimension- and function-level scores on the PIMRS. This revealed patterns of engagement of the Chilean principals, as a group, in different aspects of the instructional leadership role.
We used Rasch analysis to compare patterns of instructional leadership across the five countries described above (Andrich, 1978; Wright et al., 1994). Since the Since Rasch scores for the five countries were estimated under the same item parameter (for example, the scales of the instrument are constant), the comparison was meaningful. We then used ANOVA to test differences among mean Rasch scores in the five countries and post hoc analysis to test differences between pairs of countries. Finally, we employed Wright Maps (Wright and Master, 1982) to compare the pattern of teacher item responses for their principals in the five country samples.
Results
We begin this section by presenting the results of the translation procedure. This is followed by analysis of the reliability and validity of the PIMRS Teacher Form-Spanish.
Translation Results
The back translation process generated 17 identical translations of items (34%) from the different translators. Six additional items evidenced minor differences like an article or the order inversion of two terms. This meant that almost one-half of the translated items approached equivalence between the two forms. The remaining translation issues were of different types.
Some issues were due to language limitations in which there were no equivalent terms between English and Spanish. In addition, we were able to identify a number of instances in which the back translator had made translation mistakes. This more common problem was an issue of professional quality, as in all 10 cases very similar wordings with the same meanings were available. This has been referred to as an error production tendency characteristic of the back translation method. Nonetheless, the systematic cross checks incorporated into the back translation method allowed us to identify and address these inaccuracies. Last of all there is the cultural significance and daily expression of some words that produced other discrepancies.
The importance of employing systematic means for translation can be illustrated through reference to a previous study (Avilés, 2009) that employed a Spanish translation of the PIMRS instrument. In our opinion, Aviles’ translation would have been more consistent if one of the described strategies had been followed. For instance in the very first item a back translation procedure would have said: ‘Develop a broad group of yearly school goals’ (‘Desarrolla un conjunto amplio de metas escolares anualmente’) instead of ‘Develop a focused set of annual school-wide goals’ making the difference between ‘broad group’ and ‘focused set’ an evident flaw. Going further, the second item would have been: ‘Develop school goals in terms of staff responsabilities’ (‘Desarrolla objetivos escolares en términos de las responsabilidades del profesorado’) instead of ‘Frame the school’s goals in terms of staff responsibilities for meeting them’ leaving the crucial part of ‘meeting them’ out.
Reliability Analysis
Table 1 compares the raw mean scores, standard deviations and G-theory reliabilities for full scale, three dimensions and ten functions obtained from the Chilean data with results reported in a recently published meta-analysis of PIMRS reliability studies (Hallinger et al., 2013). The results for the Spanish language form of the PIMRS compare quite favourably, and generally indicate a high standard of reliability for all three levels of scale analysis (full scale, three dimensions and 10 functions).
Descriptive and gen theory reliability results for the PIMRS Teacher Form-Spanish.
Construct Validation
In this section, we begin with presentation of the ítem-level analyses. This is followed by the test-level analyses.
Item Level Analysis
The result of item-total correlation and item item difficulty is shown in Figure 2. The discrimination of all items was above 0.5, indicating that all items assessed one general latent trait (instructional leadership). The item difficulty showed that the items comprising the dimension Defines the School Mission (mean item difficulty = 3.93) were somewhat ‘easier’ for teachers to endorse than those assessing Manages the Instructional Program (mean item difficulty = 3.59) and Develops a Positive School Learning Climate (mean item difficulty = 3.43). This implies that principals in this sample generally attend to the practices involved in defining and communicating school goals more frequently than those in the other two dimensions.
Item-total correlation and item difficulty of the PIMRS Teacher Form-Spanish.
Figures 3, 4 and 5 show the person-item maps (also refered to as the Wright maps) for the three dimensions (Defines the School Mission, Manages the Instructional Program, Develops a Positive School Learning Climate) respectively. The principal and teacher distributions are shown in the first and second columns on the left side of the Rasch logit scale, and the item distribution on the right side. The higher a teacher is located along the logit scale, the higher the ratings were given to the principal by the teacher. The higher a principal is located on the scale, the higher the ratings the principal had received from his or her teachers. The higher an item is located on the scale, the more difficult it was for the item to be endorsed (that is, higher ítem difficulty).
Wright Map for Defines the School Mission (PIMRS Teacher Form-Spanish). Note: Principal’s score is the average of the teachers’ scores in the corresponding school.
Wright Map for Manages the Instructional Program (PIMRS Teacher Form-Spanish). Note: Principal’s score is the average of the teachers’ scores in the corresponding school.
Wright Map for Develops a Positive School Learning Climate (PIMRS Teacher Form-Spanish). Note: Principal’s score is the average of the teachers’ scores in the corresponding school.
The logit scale was classified into three categories: basic (< -1 logit), proficient (between -1 and 1 logit), and advanced (> 1 logit). In other words, a principal’s performance was classified as Basic if he or she was located below -1 logit, Proficient between -1 and 1 logit, and Advanced above 1 logit.
In Figure 3 (Defines the School Mission), the teacher distribution seemed to be located somewhat higher than the item distribution, indicating that teachers tended to give their principals a high score on these items (this also meant the principals received high ratings from their teachers). This finding was in line with the high mean item score of Defines the School Mission (see Table 1). A comparison of Figure 3 (the emprical person-item Wright map) and the theoretical person-item construct map for the PIMRS (Hallinger and Wang, 2015) revealed a strong similarity, further evidence of construct validity.
The empirical Wright map in Figure 4 (Manages the Instructional Program) was also consistent with the theoretical construct for this dimension (Hallinger and Wang, 2015). For example, the item ‘Coordinates the classroom objectives of teachers with those of the school and evaluate classroom instruction’ was considered ‘Proficient’ in the theoretical construct map. The corresponding item ‘Monitors the classroom curriculum to see that it covers the school’s curricular objectives’ was similarly located in the Proficient area in the empirical Wright map.
There was again a high degree of alignment between the empirical Wright map (see Figure 5) for the third dimensión, Develops a Positive School Learning Climate and the corresponding theoretical construct map reported by Hallinger and Wang (2015). Exceptions to the general trend included a few item thresholds that were classified as Basic in the theoretical construct map but were located in the Proficient area in the empirical Wright map.
For example, the ‘Seldom’ category of the item ‘Tutor students or provide direct instruction to classes’, was located in the Basic area on the theoretical construct map, but was located in the Proficient area in the Wright map. This discrepancy was also reported for the original PIMRS scale (Hallinger and Wang, 2015). This pattern suggests that the leadership practice of modeling through teaching by the principal tends to occur less frequently in practice than was hypothesized by the scale developers both in the USA and in Chile. Despite these discrepancies, the overall agreement between the theoretical and empriical maps was quite high.
The summary statistics of the person measures of teachers and principals on the three dimensions are shown in Table 2. The mean item difficulty of each dimension was constrained at zero for model identification. All the mean latent traits for the three dimensions were above zero, thereby suggesting that the teachers tended to give high ratings to their principals. Among the three dimensions, Defines the School Mission had the highest mean. This indicates that teachers rated their principals as most actively engaged in leadership practices associated with this dimension. The same dimensional pattern was also obtained from the mean item scores of the three dimensions (see Table 1).
Mean of rasch scores for the three PIMRS dimensions.
Note. The number in the brackets indicates the standard error of the mean estimates. Teacher sample size: USA = 1,729;
Maldives = 201; Malaysia = 98; Turkey = 294; Chile = 595.
Tables 3, 4 and 5 show the item fit statistics for the three dimenions, respectively. Overall, the items in PIMRS Teacher Form-Spanish fit the model acceptably. In the first dimension (Table 3), only item 9 (II_CSG_09) ‘Ensure that the school’s academic goals are reflected in highly visible displays in the school (for example, posters or bulletin boards emphasizing academic progress)’ had a poor fit with an MNSQ of 1.46.
Item fit statistics for defines the school mission (PIMRS Teacher Form-Spanish).
Item fit statistics for manages the instructional program (PIMRS Teacher Form-Spanish).
Item fit statistics for develops a positive school learning climate (PIMRS Teacher Form-Spanish).
In the second dimension (Table 4), only item 24 (V_MSP_24) ‘Inform teachers of the school’s performance results in written form (for example, in a memo or newsletter)’ had a poor fit with an MNSQ of 1.51.
In the third dimension (Table 5), items 27 (VI_PIT_27), 34 (VII_MHV_34) and 35 (VII_MHV_35) indicated a poor fit with MNSQ statistics of 1.54, 1.63 and 2.00, respectively. It is interesting to note that this pattern was also consistent with results obtained for the English language version of the PIMRS Teacher Form (Hallinger and Wang, 2015). In Halliner and Wang’s (2015) validation study of the PIMRS, items VII_MHV_34 and VII_MHV_35 also had a poor fit. This suggested that principals across the sample engaged in these behaviors too infrequently to achieve an adequate level of discrimination. In the current study, we also found that item VI_PIT_27 (‘Ensure that students are not called to the office during instructional time’) was practiced infrequently.
Despite these discrepancies between the theoretical and empirical results, the overall pattern for this dimension was quite strong. It should be noted that ítems with a ‘poor fit’ represented a small percentage of the items in this, the largest of the three PIMRS dimensions. Thus, we retained all of these items in the PIMRS Teacher Form-Spanish, while noting that one function, Protecting Instructional Time, did not appear to be well embedded into the instructional leadership practices of the Chilean simple.
Test Level Analysis
For evaluation of model-data fit, we employed several idexes: CFI, TLI, RMSEA and WRMR. Following established conventions (Browne and Cudeck, 1993), we interpreted scores on CFI and TLI greater than 0.90 as indicating an acceptable fit and greater than 0.95 indicating a good fit. Steiger (2007) claimed that RMSEA should be smaller than 0.07 for a good model fit, and Muthén and Muthén (2010) asserted that when the WRMR is closer to one, the data have a better fit to the theoretical model.
Figure 6 shows the CFA results. All items had factor loadings greater than 0.613, indicating that items in the Spanish version are highly correlated with their underlying functions (latent traits). The model fit indices, both CFI (0.964) and TLI (0.962), were greater than 0.95, thereby indicating that the data fit the theoretical model quite well. Good model-data fit was further confirmed by the RMSEA (0.050) and WRMR (1.340) statistics.
Confirmatory factor analysis of the two-level higher order PIMRS model.
As expected, inter-correlations among the three dimensions were very high (0.889, 0.912 and 0.831). This is because they are all measuring highly related underlying sub-constructs of instructional leadership. Altogether, the two-level higher order model fit the PIMRS Teacher Form-Spanish very well. This means that the theoretical framework can be applied to the PIMRS in this Spanish speaking context.
National Profile and Cross-national Comparative Analysis
The first stage statistical analyses yielded an instructional leadership profile of the Chilean primary school principals. While not generalizable to ‘all Chilean principals’ the data showed a pattern of relatively stronger performance on the dimension Defines the School Mission. At the function level, principal engagement in the two goal-related functions in the dimension Defines the School Mission were rated the highest by teachers: Frames School Goals (4.07) and Communicates School Goals (3.79). In contrast, the lowest rated functions were Provides Incentives for Teachers (3.09), Protects Instructional Time (3.39) and Maintains High Visibility (3.41). While this ‘instructional leadership profile’ provides a salient performance baseline in Chile, it does not include any benchmarks against which to compare the baseline.
Thus, the third goal of this study was to compare the Chilean profile of principal instructional leadership with profiles obtained from studies conducted in several other societies. As described earlier, we employed a variety of statistical tests for this purpose, starting with generating Wright Maps for each dimension from the cross-national data. The Wright maps in Figures 7, 8 and 9 compare the Rasch scale distribution in the five countries for each of the three dimensions. These analyses can be used to compare visually the scores from the different principal and teacher populations.
Wright map for five countries on the items in Defines the School Mission.
Wright map for five countries on the items in Manages the Instructional Program.
Wright map for the five countries to the items in Develops a Positive School Learning Climate.
Across the five countries, the highest score was consistently obtained for the first dimension, Defines the School Mission (see Figure 7). This was followed by Manages the Instructional Program and Develops a Positive School Learning Climate in order of relative strength. This indicates that, based on these data, this dimension is most actively practiced by principals across the five nations.
Table 6 shows the Rasch mean score for each country on each of the three PIMRS dimensions. The USA had the highest Rasch mean scores in two dimensions (Defines the School Mission and Manages the Instructional Program) and Malaysia in one (Develops a Positive School Learning Climate). The Maldives and Chile had similar magnitudes across the three dimensions and Turkey’s Rasch means were the lowest on each of the three dimensions.
ANOVA summary table for the cross-national comparison.
***p < .001
In order to determine if the cross-national differences indicated by the Rasch scores shown in Table 6 were statistically significant, we ran a series of ANOVA tests. These yielded the following results (not tabled): Defines the School Mission F = 169.38 (p < 0.001, η2 = 0.189); Manages the Instructional Program, F = 171.22 (p < 0.001, η2 = 0.190); Develops a Positive School Learning Climate F = 83.33 (p < 0.001, η2 = 0.103). The ANOVA results support the conclusion that differences between countries shown in Table 6 are statistically significant.
We then employed the post hoc test in ANOVA to test differences between pairs of countries. With only three exceptions, the differences between every pair of countries were significantly large. The results indicate that the US and Malaysia had similar Rasch mean scores and larger than other countries scores on all three dimensions. Chile and Maldives had similar magnitudes in Manages the Instructional Program and Develops a School Learning Climate and were located in a ‘middle level’ compared to other countries. Turkey was significantly lower than all of other countries on the three dimensions.
Taken together, the results indicate that the pattern of instructional leadership practiced by this sample of Chilean primary school principals shares a common ‘strength’ in the dimension, Defines a School Mission. With respect to the other two dimensions, Chilean principals fell into a middle band whereby performance was higher than Turkey, similar to the Maldives and lower than the USA and Malaysia. Again, given sample limitations, we proffer these findings as indicative but not conclusive results.
Discussion
The purposes of this study were to validate the Principal Instructional Management Rating Scale (Hallinger, 1982, 1990) for use in Spanish-speaking Chile and then to demonstrate its utility in generating an example of a national profile of principal instructional leadership. Subsequently, we compared the Chilean profile with profiles generated for a cross-national sample of principals. In this final section of the paper, we revisit limitations of the research method, and discuss key conclusions and implications drawn from the study.
Limitations
Several limitations related to this study should be noted. First, the extensive discussion devoted to translation issues should cause readers in other Spanish-speaking contexts to be cautious when employing the ‘Chilean version’ of the PIMRS Teacher Form-Spanish without further refinement.
Second, although our cultural adaptation did assess the suitability of the conceptual constructs comprising the PIMRS in the Chilean context, we did not explore whether there were any ‘omitted constructs’ that might characterize instructional leadership in the Chilean context. Future studies should address this issue.
Third, the ‘instructional leadership profile’ of Chilean primary school principals presented here cannot be considered nationally representative. It was limited to a relatively small sample of primary school principals from the national capital. Thus, we cannot claim that the results reflect all Chilean principals. Nonetheless, the groundwork laid in this study will enable future research capable of addressing these limitations of the current study.
Finally, we reiterate that the cross-national comparison included in this article represents a preliminary demonstration of an approach for comparing instructional leadership profiles obtained from different national contexts. Limitations in sample size and composition hampered our ability to develop generalizable results.
Conclusion and implications
Our effort to develop and validate a Spanish language version of the PIMRS for use in Chile was successful. This PIMRS Teacher Form-Spanish meets standards of reliability and internal validity required for use in research, needs assessment and performance evaluation (American Educational Research Association, 1999; Cronbach et al., 1972; Latham and Wexley, 1981; Lissitz and Samuelsen, 2007; Nunnally, 1978).
When instruments are used for research and needs assessment we generally expect reliability coefficients to exceed a standard of at least 0.70 (Cronbach, 1988; Latham and Wexley, 1981; Nunnally, 1978). As with the English forms of the PIMRS, measures for the Spanish language version exceeded this standard for the full-scale, dimension-level and function-level constructs. In combination with strong results obtained for internal validity, we conclude that the PIMRS Teacher Form-Spanish can be used with confidence by researchers and staff developers in Chile.
When employed for performance evaluation, it is common to apply a higher standard of scale reliability for measurement instruments since the results can directly impact personnel decisions. Experts have suggested a minimum reliability standard of 0.90 when data from an instrument will be used for personnel decisions (Cronbach, 1988; Nunnally, 1978). The PIMRS Teacher Form-Spanish met this standard with data reported for the full scale and three dimensions. In addition, it met this standard on seven of the 10 function-level subscales; subscales below the standard yielded coefficients of 0.88, 0.87 and 0.74 (see Table 1). Thus, this form could still be considered for use for performance appraisal, though the publisher’s current recommendation is to employ the three dimensions of the PIMRS as the optimal level for personnel evaluation (Hallinger and Wang, 2015; Hallinger et al., 2013). We note that these findings were very consistent with results recently reported by the scale’s publisher (for example, Hallinger and Wang, 2015; Hallinger et al., 2013), thereby lending additional confidence to the results of this validation study.
Unfortunately, the current study did not collect self-report data from principals. Nonetheless, we predict that future validation of a PIMRS Principal Form-Spanish will yield roughly similar results to those reported here for the PIMRS Teacher Form-Spanish. Our confidence is based on reliability and validity results recently reported by Hallinger and Wang (2015). Their comprehensive validation study, employing multiple international datasets, found that the pattern of reliability and validity statistics for the PIMRS Principal Form was consistently similar to the Teacher Form, though at a slightly lower level of magnitude. With this in mind, we expect the PIMRS Principal Form-Spanish to yield a pattern of reliability and validity results that approximate those reported in this paper.
Our attempt to develop a national profile of the instructional leadership of Chile’s primary school principals followed Hallinger and Lee (2013) who undertook a similar effort in Thailand. Their national profile was subsequently used to evaluate the impact of Thailand’s educational reform on principal instructional leadership over time. More specifically, they found no signfiicant differences in the principal instructional leadership profiles of principals generated before and after the implementation of reforms aimed at enhancing learning and teaching development in Thai schools.
The Chilean profile can be viewed as a preliminary baseline for Chilean principals. Future studies will no doubt seek to replicate and expand on this initial effort. The availability of this type of data will enable policymakers in Chile to more accurately assess the impact of current investments in principal preparation and development as well as contributions made by principals to national and local accountability targets. At the practitioner level, availability of the instrument provides Spanish-speaking principals in Chile and elsewhere with a potentially valuable tool for self-reflection and accessing reliable feedback from their teachers (Hallinger and Wang, 2015).
For researchers, the study sought to demonstrate a means of conducting cross-national research in educational leadership and management (see also Dimmock and Walker, 2000; Hallinger and Leithwood, 1996; Walker and Dimmock, 2002). To date, despite considerable commentary on its potential importance, comparative research in educational administration is most notable for its absence. Although the corpus of empirical studies conducted outside of traditional Anglo-American nations is growing – albeit slowly – a search conducted in conjunction with this article turned up startlingly few quantiative cross-national studies of educational leadership. This contrasts with the general leadership and management literature which evidences a reasonably long tradition and abundance of cross-cultural studies (Adler, 1983; Ajiferuke, and Boddewyn, 1970; House et al., 2002; Tsui et al., 2007).
Moreover the very few studies that we were able to locate examined school leadership in only two countries, or relied on large scondary data-sets where the quality of the measures is often compromised (for example, Lee and Hallinger, 2012). The current study demonstrated how data collected with an instrument commonly used in our field could be compared using cross-national data. We further suggest that it would be easy to include additional variables of interest (for example, principal self-efficacy, teacher collective efficacy, teacher commitment) along with the PIMRS in such a study. Studies such as this would represent a true advance in the scope of research conducted in our field.
We assert that the gradually improving research capacity of higher education institutions in developing nations combined with global interest in education reform will reshape the face of educational administration as a field of study over the next 20 years. Scholars in our field need only look at how research and practice in business management studies has internationalized over the past half-century in order to gain a glimpse of the future in our field. Because educational administration tends to emphasize ‘local’ concerns over the ‘cosmopolitan’, it has lagged behind business management in recognition of the importance of an international focus (Hallinger and Bryant, 2013). But we suggest that this shift is slowly but surely gathering momentum.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors wish to acknowledge the funding support of the Research Grant Council (RGC) of Hong Kong for its support through the General Research Fund (GRF 841711).