Multilevel Confirmatory Factor Analysis of the Teacher My Class Inventory–Short Form

Need for Sound Elementary-Level Psychometric Classroom Climate Measures

Bronfenbrenner’s work in the 1970s was instrumental in the setting the stage for the study of classroom climates, suggesting the power various sociocultural systems (family, teachers, school, neighborhood) can have on a person’s developmental process (Bronfenbrenner, 1979). Extensive research conducted by Rudolf Moos (1979), in particular his development of the Classroom Environment Scale, also sought to assess the classroom environment. Moos based his measurement on three dimensions: the relationship dimension, personal development dimension, and system maintenance and change dimension. The majority of climate research since that time has investigated constructs related to the physical setting, psychological environment, teacher characteristics and behaviors, and interactions among students themselves (Fraser, 1998, 2002; La Paro & Pianta, 2003). Surveys, observations, interviews, and critical data elements (i.e., grades, attendance, discipline referrals, standardized test scores) have all been used to assess positive and negative student outcomes related to classroom climate. That being said, there remains a need for more rigorous, psychometrically sound instrumentation to effectively measure the quality of classroom learning environments (La Paro, Pianta, & Stuhlman, 2004). Regrettably, informal methods of measuring classroom climate tend to be highly subjective and use less than rigorous data collection procedures. Therefore, using sound psychometric climate surveys contributes to the fairness, accuracy, and defensibility of performance evaluation systems (National School Climate Council, 2007). Moreover, instruments need to be theoretically strong, vigorously field-tested, measure target populations accurately, be easy to administer and score, and meet the needs of each particular school community (Faster & Lopez, 2013).

Patrick, Kaplan, and Ryan (2011) noted numerous classroom climate measures exist for assessing a variety of classroom dimensions perceived by students in preK–12 settings; however, very few are specifically designed for elementary level classrooms and even fewer take into account the teacher’s perception of the learning environment. One measure, TMCI-SF (Sink & Spencer, 2007) was developed to assess teachers’ perceptions of their classroom climate. Sink and Spencer (2007) conducted a study to assess the initial psychometric quality (i.e., underlying factor structure and determine reliability and validity estimates) of the TMCI-SF using data from K–6 teachers (N = 303) employed at 22 urban elementary schools across western Washington. The sample included 257 (84.8%) females, 36 (11.9%) males, and 10 (3.3%) teachers who did not report their gender. The sample also reported varying levels of teaching experience, ranging from first year (n = 30, 10%), 1.5 to 5 years (n = 98, 44%), 5.5 to 10 years (n = 42, 14%), 10 to 20 years (n = 78, 26%), to more than 20 years (n = 41, 13.5%), and 14 (4.5%) individuals who did not provide their years of experience. The majority of the teachers were Caucasian (n = 230, 75.9%), 27 (8.9%) identified as minority, and 46 (15.2%) did not report their ethnicity.

The underlying dimensionality and factorial validity of the original, 30-item TMCI-SF were initially examined through principal component analysis, using the maximum likelihood factor analysis procedure, and by a confirmatory factor analysis (CFA). The researchers reported the means, standard deviations, and interitem correlations for each TMCI-SF scale as follows: Satisfaction (M = 19.64, SD = 2.59, r_s = .39–.73), Friction (M = 15.73, SD = 3.45, r_s = .18–.52), Competition (M = 15.64, SD = 2.76, r_s = .06–.43), Difficulty (M = 10.86, SD = 2.94, r_s = .27–.53), Cohesion (M = 16.32, SD = 3.16, r_s = .14–.57), and School Counselor Impact (M = 17.07, SD = 3.82, r_s = .49–.73). Low to moderate internal consistency reflected by coefficient alphas for the six scales ranged from .57 to .88.

The initial CFA resulted in an unsatisfactory fit for the six-factor model. In addition, item analyses resulted in low item-to-scale correlations (r_s = .25); consequently, the instrument items were reduced from 30 to 24. Specifically, problematic item loadings were discovered for three of the scales (Friction, Cohesion, and Competitiveness). Therefore, the Friction and Cohesion scales were collapsed into one Peer Relations scale. The exploratory factor analysis then resulted in a better fit for a five-factor model, accounting for 59.84% of the intercorrelation matrix variance. Recomputed coefficient alphas ranged from .66 to .87 and reflected a low-moderate to moderate reliability for each scale: Satisfaction (r_s = .28–.60, α = .84), Competitiveness (r_s = .32–.47, α = .66), Difficulty (r_s = .27–.49, α = .75), Peer Relations (r_s = .30–.63, α = .80), and School Counselor Impact (r_s = .46–.73, α = .87). The researchers concluded that the revised TMCI-SF resulted in an improved fit for the model and provided some psychometric information about the utility of the TMCI-SF. However, the researchers suggested the instrument be tested using various teacher populations to increase the generalizability of the findings. Though this foundational study provided some psychometric information about the utility of the TMCI-SF, further research is needed to confirm its psychometric soundness.

Purpose of the Study

Previous research on the psychometric properties of the TMCI-SF has generated promising results. Sink and Spencer (2007), as mentioned to above, in their initial research study recommended additional psychometric work be conducted to confirm the measure’s dimensionality. Thus, the primary intent of this study was to reexamine the factorial validity of the TMCI-SF through the use of an MCFA with a new sample of teachers located in the north and southeast regions of Florida.

Method

Results

Preliminary Analyses

The descriptive statistics for survey items grouped by scales are provided in Table 1. As expected, item means and standard deviations were similar within each scale. Next, the researchers compared the results of the six scales (i.e., Satisfaction, Friction, Competitiveness, Difficulty, Cohesiveness, and School Counselor Impact) by group using a multivariate analysis of variance. There were no significant differences in responses when considering the teachers’ gender or whether the teacher was in the treatment group. However, even after controlling for the percentage of low-income and White students in each classroom and in the multivariate analysis of variance, significant differences were found between the two school districts on the Cohesion and School Counselor Impact scales. For instance, District B’s scale scores were larger for both Cohesion and School Counselor Impact by 1.5 and 2.1 points, respectively (with 95% multivariate confidence intervals [0.002, 3.03] and [0.20, 3.9]). Subsequently, CFAs were conducted on the data set to determine the measure’s factorial validity.

OPEN IN VIEWER

Table 1.

Descriptive Statistics of the Survey Items.

Dimension	Item No.	M	SD	Skewness	Kurtosis
Satisfaction	1	4.09	0.58	1.62	−0.41
	7	4.19	0.55	2.36	−0.40
	13	4.32	0.54	1.31	−0.29
	19	4.08	0.55	2.16	−0.42
	25	3.91	0.64	1.25	−0.61
Friction	2	2.93	1.14	−1.19	0.39
	8	3.04	1.17	−1.18	0.23
	14	3.97	0.88	1.06	−1.04
	20	2.30	0.92	0.38	0.87
	26	3.53	1.00	−0.56	−0.64
Competitiveness	3	2.79	1.03	−0.81	0.30
	9	3.38	0.85	−0.58	−0.61
	15	3.78	0.68	3.31	−1.59
	21	3.66	0.79	0.30	−0.98
	27	2.93	1.11	−1.28	0.01
Difficulty	4	2.58	0.93	−0.73	0.36
	10	2.33	0.99	0.18	0.92
	22	2.07	0.81	1.28	0.93
	28	2.09	0.92	1.39	1.17
Cohesiveness	5	3.12	1.01	−1.08	−0.06
	12	4.24	0.48	−0.22	0.60
	18	3.28	1.04	−1.27	−0.25
	24	2.93	0.94	−1.11	0.17
	30	3.84	0.68	1.59	−0.98
SCI on classroom environment	6	4.09	0.58	1.62	−0.41
	11	4.19	0.55	2.36	−0.40
	17	4.32	0.54	1.31	−0.29
	23	4.08	0.55	2.16	−0.42
	29	3.91	0.64	1.25	−0.61

Note. SCI = School Counselor Impact; SE kurtosis = 0.32; SE skewness = 0.16.

Confirmatory Factor Analyses

Multilevel Confirmatory Factor Analysis

An MCFA was conducted to assess the structure of the TMCI-SF. Because the teachers in this study were nested within schools, there was a hierarchical nature to the data. Therefore, an MCFA was conducted to assess the five-factor structure of the TMCI-SF based on Sink and Spencer’s (2007) initial psychometric analysis. The MCFA was conducted using maximum likelihood estimation (du Toit & du Toit, 2008) via LISREL Version 8.80 (Jöreskog & Sörbom, 2006). Again, to be consistent with the work of Sink and Spencer (2007), the Cohesion and Friction scales were combined into one factor, Peer Relations (α =.87, for 10 items). The resulting path diagrams and standardized solutions for the between and within schools results are shown in Figures 1 and 2, respectively. As expected, the chi-square test was significant, χ²(734) = 1909.11, p < .001; thus, the null hypothesis of the model exactly fitting the data was rejected. The root mean square error of approximation (RMSEA) of the five-factor multilevel model was .083 with a 90% confidence interval [0.079, 0.088], where a value of .05 can be thought of as a good fit and values from .05 to .08 indicate a moderate fit (Browne & Cudeck, 1993).

OPEN IN VIEWER

Figure 1.

Multilevel confirmatory factor analysis path diagram of the five-factor model between schools.

OPEN IN VIEWER

Figure 2.

Multilevel confirmatory factor analysis path diagram of the five-factor model within schools.

Single-Level Confirmatory Factor Analysis

Since the average number of teachers nested within each school was so small (3.9 teachers per school), fitting the MCFA was not ideal. The small cluster size did not afford adequate variability to properly estimate the multilevel model, so a follow-up single-level CFA was conducted to further assess the five-factor structure. Figure 3 presents the resulting path diagram of the five-factor model. The single-level model was examined using the robust diagonally weighted least squares technique (Jöreskog, 1990). The fit indices suggested a moderate to good fit with the hypothesized model. As expected, the chi-square test was significant, χ²(367) = 613.12, p < .001; thus, the null hypothesis of the model exactly fitting the data was not accepted. However, large sample sizes will inflate the chi-square statistic, so other fit indices were evaluated: Goodness-of-fit index (GFI = .97), comparative fit index (CFI = .97), normed fit index (NFI = .93), and RMSEA = .054, 90% confidence interval: [.046, .061]. While fit indices greater than .90 and a RMSEA less than .10 indicate an acceptable fit, values greater than .95 indicate a good fit (Browne & Cudeck, 1993; Hu & Bentler, 1999). Considering the range of the indices and the RMSEA, the five-factor model yielded an appropriate fit for an attitudinal survey.

OPEN IN VIEWER

Figure 3.

Confirmatory factor analysis path diagram of the five-factor model.

Discussion

The principal aim of this psychometric investigation was to cross-validate the TMCI-SF, a classroom climate inventory, with a sample of elementary school teachers located in the north and southeast regions of Florida. The multilevel and single-level CFA results were mixed. The CFA and MCFA involved five scales: Satisfaction (5 items), Peer Relations (a combination of 10 items from Cohesion and Friction), Difficulty (4 items), Competitiveness (4 items), and School Counseling Impact (5 items). In an attempt to account for the nested data, a five-factor multilevel CFA was conducted. The chi-square test and RMSEA suggested that the fit was less than optimal. It appears that the model fit was adversely affected by the limited sample size coupled with too few teachers nested within each school. Specifically, the 233 teachers nested within 60 schools did not allow for much variability per school, thus creating difficulties in convergence of the multilevel models. This difficulty did not allow for additional fit indices to be computed or a competing six-factor multilevel model to be fitted. In the Figure 1 and Figure 2 path diagrams, the values representing the relationship between a scale, or latent variable, and an item, or observed variable, should be interpreted as regression coefficients. As seen in Figure 1, the regression coefficients between four items—12, 14, 26, and 30—and the Peer Relations factor are small, indicating that these relationships are not consistent across the different schools. However, once the between-schools differences are accounted for, the within-schools results in Figure 2 demonstrate that the items within a given scale are similarly related and behaving relatively consistently within a school. The only exception is the Competitiveness scale, where two of the items loaded much more strongly on the factor than the other three items. This finding is consistent with the smaller coefficient alpha for the Competitiveness scale (α = .48).

The single-level CFA five-factor model was confirmed. According to the CFA, as shown in Figure 3, Peer Relations generated a positive correlation with Satisfaction (.71). In contrast, Peer Relations was negatively correlated with Difficulty (−.44) and Competitiveness (−.22). Difficulty was also negatively correlated (−.44) with Satisfaction. There were also weak, positive correlations between School Counselor Impact and Peer Relations (.17) and Satisfaction (.18). These findings are consistent with Sink and Spencer (2007).

The “positive climate” scale items largely generated higher mean ratings (Satisfaction, Cohesion, and School Counselor Impact) than for those items situated on the “negative climate” dimensions (Friction, Difficulty, and Competitiveness). Because educators tend to view their classrooms in less nuanced ways (i.e., polarized as either good or bad), skewness and kurtosis indices for some items were less than ideal (>1.00). Alpha reliabilities for four (Satisfaction, Peer Relations, Difficulty, and School Counselor Impact) out of the five scales were adequate for a perceptual survey. Similar to what Sink and Spencer (2007) reported, the Competitiveness scale coefficient in this study was weak (α = .48 current study, α = .66 original study). Unlike the current study, Sink and Spencer reported that the Peer Relations scale comprised five items derived from two factors, with several of the items dropping out of the final model. This minor discrepancy is anticipated when comparing factor structures derived from participant samples obtained from two geographic regions of the United States. In short, whereas the multilevel CFA was less than satisfactory, the single-factor CFA results reported above largely replicated the TMCI-SF’s factor structure and reliabilities presented in Sink and Spencer (2007).

Coupled with the results from Sink and Spencer (2007), the findings here provide further evidence that the TMCI-SF is potentially applicable to a variety of elementary school settings, particularly as a tool to estimate teachers’ perceptions of the classroom climate. Unique to this measure, the School Counselor Impact scale can be used to estimate school counselor contributions to improving various learning environments with high reliability (School Counselor Impact, α = .93). With the data summarized from the other TMCI-SF scales, the information generated provides a relatively comprehensive view of the classroom environment, including peer relations, difficulty of the classroom work, how competitive students appear to be, as well as their satisfaction level.

The demand for school counseling data-driven decision making, use of evidence-based curriculums, and school counselor accountability is well documented (Carey & Dimmitt, 2006; Dimmitt, Carey, & Hatch, 2007; Dimmitt, Carey, McGannon, & Henningson, 2005; Honig & Coburn, 2005; Slavin, 2008; Squier, Nailor, & Carey, 2014; Wiseman, 2010). School counselors do not work in isolation, and they play an integral role in transforming the school culture by providing responsive services to students struggling with mental, emotional, and behavioral issues and by encouraging an environment of acceptance and sensitivity (Grothaus, 2013). Furthermore, Carrell and Hoekstra (2011) found that the school counselor’s role was critical to enhancing teacher quality, by assisting them with managing problem behaviors, helping address pressing student concerns, and freeing them up so that they could focus their efforts on teaching. Despite these claims, there remains a significant dearth in the school counseling literature of sound psychometric instruments available to measure the classroom climate from a teacher perspective as well as the ability to evaluate the impact of school counselors’ efforts toward establishing and sustaining positive learning environments (Sink & Spencer, 2007). The TMCI-SF can yield meaningful quantitative data as school counselors attempt to be more data-driven (Dimmitt et al., 2007) and accountable (Loesch & Ritchie, 2005) for their support of classroom teachers. Counselors can give the survey to classroom teachers prior to and after implementing school counseling interventions as a means of documenting changes in the classroom environment. The instrument can also be used to evaluate the effectiveness of the school counseling programs and can lend support for expanding school counseling services to students and classrooms. In addition, the TMCI-SF instrument is free to educators, is easy to administer and score, and requires no more than 8 minutes to complete.