Abstract
Valid measurement of how students’ experiences in secondary school classrooms lead to gains in learning requires a developmental approach to conceptualizing classroom processes. This article presents a potentially useful theoretical model, the Teaching Through Interactions framework, which posits teacher-student interactions as a central driver for student learning and that teacher-student interactions can be organized into three major domains. Results from 1,482 classrooms provide evidence for distinct emotional, organizational, and instructional domains of teacher-student interaction. It also appears that a three-factor structure is a better fit to observational data than alternative one- and two-domain models of teacher-student classroom interactions, and that the three-domain structure is generalizable from 6th through 12th grade. Implications for practitioners, stakeholders, and researchers are discussed.
Keywords
Recent educational reforms place new emphasis on the important role of teachers in promoting student achievement. States and districts across the country are developing and testing new models for evaluating teachers that provide fair and valid approaches to documenting teachers’ performance. Most of these newer systems include multiple metrics of teacher effectiveness, including value-added scores, student reports, and observation (Kane et al., 2012). Observational methods of assessing teacher performance have particular relevance as they can provide actionable feedback to teachers that student test scores do not. Simply knowing students are (or are not) learning does not tell teachers much about how or what to improve. Observation, if done well, can provide specific and relevant feedback to teachers that can guide and focus their efforts to improve practices that have a direct impact on student achievement (Allen, Pianta, Gregory, Mikami, & Lun, 2011). Unfortunately, there are too few observational measures of teachers’ performance that are grounded in developmental theory, hence relevant to teaching adolescents and applicable to the wide variation in classroom settings across content and ability. This study presents confirmatory evidence of the theoretical structure of an observational assessment of classroom interactions, the Classroom Assessment Scoring System (CLASS), across several samples of secondary school classrooms.
The need for measures of teacher performance with direct application in school settings is high. Unfortunately, most current systems for observing teachers are inadequate. As detailed in the 2009 report “The Widget Effect,” 98% of teachers are given “satisfactory” ratings after being observed by principals. Most of the observational tools used to assess teachers are homegrown, with little to no empirical evidence that they can produce reliable metrics or are accurately measuring components of teaching actually associated with student learning (Pianta & Hamre, 2009). There are exceptions for which empirical support has been obtained, most notably evaluation systems built from the Danielson protocol (Danielson, 2011), such as the Teacher Evaluation System in the Cincinnati Public Schools. When used systematically as a part of teacher evaluation or development systems, standardized and validated approaches to observation can improve teacher performance and student achievement for those teaching adolescents (Allen et al., 2011; Taylor & Tyler, 2011). Thus, harnessing the power of observational protocols to provide meaningful evidence from a teacher’s classroom interactions is critical for the future of the educational system. An important aspect of rigorous evaluation of such protocols is evaluating the underlying factor structure inherent in observational measures, as these factor structures provide both a theoretical and descriptive basis for protocols, such as the CLASS.
Given the context of intense policy pressure on districts to adopt observational measures as a part of teacher evaluation systems and the developmental research on the environments in which adolescents best thrive, it is important that any assessment of teacher practices in secondary school classrooms be linked to an understanding of adolescent development. One observational measure used in several recent studies of middle and high school classrooms is the Classroom Assessment Scoring System–Secondary (CLASS-S; Pianta, Hamre, & Mintz, 2010). The CLASS-S is based on over a decade of research on the elements of teaching that contribute to students’ cognitive and social development throughout school (Hamre & Pianta, 2010). The present article builds from recent work demonstrating associations between CLASS-S scores and students’ achievement (Allen et al., 2011; Allen et al., 2013) and focuses more specifically on examining the factor structure of the CLASS-S across several large studies of middle and high school classrooms. We test the extent to which the previously theorized factor structure fit these diverse data sets and compare that fit with alternative structures. As discussed in greater detail throughout this article, this measurement work has important implications for research, policy, and practice as it provides empirical support to users in the interpretation of CLASS-S scores.
Theoretical Foundations of the Teaching Through Interactions Framework
Developmental theory and research provide strong support for the role of daily interactions of children and adolescents with adults and peers as the driving force behind learning and development (Bronfenbrenner & Morris, 1998). For example, a recent review of effective math programs in middle and high school provides support for an approach to understanding classroom quality that focuses on teacher-student interactions. Slavin, Groff, and Lake (2009) reviewed evidence on 100 math interventions. Findings indicated that interventions focusing on daily interactions between teachers and students had stronger effects than did programs focusing solely on curricula and/or technology. This key role of interactions is somewhat in contrast to the current focus of middle and high schools in which content, standards, and curriculum are viewed as the most important elements of classrooms responsible for student learning (e.g., Confrey, 2006), suggesting a potentially dangerous lack of alignment.
In the present study, we discuss and test the factor structure of the proposed Teaching Through Interactions model (Hamre et al., 2013), using data collected with the CLASS-S observational measure. Consistent with many other descriptions from educational and psychological literature (e.g., Eccles & Roeser, 1999; Pressley et al., 2003; Roehrig et al., 2012), the Teaching Through Interactions model describes three broad domains of teacher-student interactions: Emotional Support, Classroom Organization, and Instructional Support. Each broad domain of teacher-student interactions consists of several specific dimensions of teacher-child interactions (see Table 1); for example, the domain of Emotional Support consists of four dimensions—Positive Climate, Negative Climate, Teacher Sensitivity, and Regard for Adolescent Perspectives. In effect, this model posits that effective teaching behaviors are organized into three broad domains, and these domains are applicable across grades and content area. A prior test of this three-domain conceptual model using data from over 4,000 pre-school to fifth-grade classrooms demonstrated the proposed model fit data well and provided superior fit to alternative one- or two-factor solutions (Hamre et al., 2013). However, this factor structure has yet to be confirmed in middle and high school classrooms where the organizing structure of teacher-student interactions may be different.
Description of Teaching Through Interactions at Secondary Level (Original CLASS-S Factors).
Note. CLASS-S = Classroom Assessment Scoring System–Secondary.
Below, we briefly review the theoretical background that led to the development of the Teaching Through Interactions framework and the observational tool on which it is based, the CLASS-S (Pianta et al., 2010). We then provide evidence supporting the validity of this approach to conceptualizing middle and high school classroom interactions by summarizing recent literature linking CLASS-S observations to student achievement.
Emotional Support
The Teaching Through Interactions framework describes four dimensions of teaching practice that are hypothesized to support students’ social and emotional skills as well as their engagement in academic pursuits: Positive Climate, Negative Climate, Teacher Sensitivity, and Regard for Adolescent Perspectives (see Table 1 for operational definitions of these dimensions). One way the CLASS-S is unique from versions of CLASS for earlier grade levels is that these emotional support dimensions are grounded in adolescent developmental literature. For example, a majority of what is coded in these dimensions are based on decades of research demonstrating that adolescents have particular developmental needs, including relational supports and connections, autonomy and competence, and relevance (Allen, Hauser, Bell, & O’Connor, 1994; Allen & Land, 1999; Ryan & Deci, 2000). Below, we review developmental research confirming the importance of each emotional support dimension.
Positive Climate
A strong student-teacher relationship is one key to positive academic performance, increased school motivation, and positive behavioral outcomes (Rudasill, Gallagher, & White, 2010; Skinner, Furrer, Marchand, & Kindermann, 2008). It is well established that adolescents achieve more in challenging yet supportive environments to which they feel a positive connection (Eccles, 2004; Gentry, Gable, & Rizza, 2002). As a powerful example, one recent national study of almost 8,000 high school students demonstrated that schools in which students had more positive perceptions of the classroom relational climate had lower dropout rates (Barile et al., 2012). In contrast, adolescents who describe relationships with teachers as unsatisfying and unmotivating also report decreased levels of engagement in school (Roeser, Eccles, & Sameroff, 1998). Peer relationships and cooperation are also of vital importance to adolescents and need to be built into classroom interactions in meaningful and productive ways (Crosnoe, Cavanagh, & Elder, 2003).
Negative Climate
The absence of expressions of negative emotion is important for student learning. Attachment theorists posit that when adults provide emotional support in a predictable, consistent, and safe environment, children become more self-reliant and are able to take risks as they explore the world because they know that an adult will be there to help them if they need it (Bowlby, 1969).
Teacher Sensitivity
Sensitive teachers are attuned and responsive to the social, emotional, and academic needs of students in their classrooms, while maintaining a focus on the classroom as a whole. Students in classrooms with sensitive teachers are more engaged and self-reliant in the classroom, have lower levels of mother-reported internalizing problems (National Institute of Child Health and Human Development Early Child Care Research Network, 2003), and display greater gains in academic skills (Connor, Son, Hindman, & Morrison, 2005). Although the construct of teacher sensitivity has not often been used to study middle and high school teachers, it shares some elements of more frequently assessed elements of these environments such as a teachers’ “withitness” (Kounin, 1970), although this construct has tended to apply exclusively to teachers’ attunement to behavioral issues in the classroom.
Regard for Adolescent Perspectives
Middle and high school teachers’ skills in supporting students’ need for autonomy and decision making are critical to creating a classroom environment that engages an adolescent (Hafen et al., 2012). For example, providing students with meaningful choices within the classroom has been shown to increase engagement (Allen et al., 1994). A mismatch between the adolescent’s need for greater autonomy and the teacher’s exercise of control has been shown to result in decreased student learning (Eccles, Wigfield, & Schiefele, 1998). This is particularly true of 6th- to 12th-grade classrooms due to the developmental needs for autonomy in guiding positive development. In other words, inattention to adolescent perspectives diminishes opportunity for student growth, while teacher-student interactions in which the student feels recognized as an individual with his or her own point of view enable an expansion of opportunity and motivation to learn and perform.
Classroom Organization
The Teaching Through Interactions framework describes three dimensions of teaching practice that are hypothesized to support students’ abilities to regulate behavior and attention in the classroom such that they can get the most out of learning opportunities: Behavior Management, Productivity, and Instructional Learning Formats (see Table 1 for operational definitions of these dimensions).
Behavior Management
A multitude of studies indicate that classrooms with positive behavior management tend to have students who make greater academic progress (Tingstrom, Sterling-Turner, & Wilczynski, 2006). Middle and secondary classroom teachers who are effective classroom managers provide predictability allowing adolescents the chance to focus on learning; the establishment of a comfortable and orderly classroom where a teacher is proactively addressing student behavior promotes student engagement and leads to higher academic achievement (Woolfolk Hoy & Weinstein, 2006).
Productivity
Productive classrooms are those in which students are consistently exposed to learning opportunities and downtime is kept to a minimum. Classrooms and lessons tend to run smoothly when students are explicitly taught routines and procedures (Leinhardt, Weidman, & Hammond, 1987). With four to eight different classes daily, middle and secondary teachers must maximize learning time with lessons that get underway quickly, have a clear beginning and end, and have very quick transitions between activities (Arlin, 1979).
Instructional Learning Formats
Student achievement is increased when teachers provide clear learning targets and specific feedback within an organized classroom with few behavioral disruptions (Brophy, 1986). Variety and novelty in modes of presentation and types of activities are also important teacher practices for student learning (Cotton, 2000). Yair (2000), for example, found that students in Grades 6 to 12 demonstrated low engagement during lectures but higher levels of engagement when participating in more active classroom activities (e.g., labs, groups). Thus, offering a more varied instruction that actively facilitates engagement is likely to lead to more learning in secondary school classrooms.
Instructional Support
The Teaching Through Interactions framework describes four instructional dimensions of teaching practice that are hypothesized to enhance students’ cognition and learning: Content Understanding, Analysis and Inquiry, Quality of Feedback, and Instructional Dialogue (see Table 1 for operational definitions of these dimensions). Adolescents learn best in environments in which teachers hold high academic standards and expectations, but these must be accompanied by sufficient support for each individual to ensure that goals are attainable (Weinstein, Tomlinson-Clarke, & Curran, 2004). This is another area of the CLASS-S that is qualitatively different than earlier grades, as the extent to which the classroom environment pushes higher-order thinking and meta-cognition is of paramount importance in secondary school classrooms.
Content Understanding
Many elements of effective instruction promote students’ understanding of material across content areas, and this requires connecting specific material to larger ideas. Teachers, therefore, need to organize instruction within a framework that builds toward the big ideas within an academic discipline but that is fully supported by a strong base of factual knowledge and skills (Bransford, Brown, & Cocking, 2000). Effective teachers teach subject matter in depth, providing many examples in which the same concept is at work and in which similarities and differences are explicitly addressed (Marzano, Pickering, & Pollock, 2001). Effective teachers present new material in small steps, review relevant previous learning and prerequisite skills and knowledge (Bransford et al., 2000), and guide student performance through modeling, numerous examples, and opportunities for extensive practice, both supervised and independent (Swanson, Hoskyn, & Lee, 1999). Supervised practice of procedures and skills builds toward automaticity, thereby freeing up cognitive space for deeper understanding (Bransford et al., 2000). Content learning is strongest when teachers explicitly tie new information to students’ background knowledge and real-world examples (Bransford et al., 2000) and when multiple perspectives are presented (Hooper & Rieber, 1995).
Analysis and Inquiry
Effective teachers tap into the natural problem-solving abilities and curiosity of students by providing them with opportunities to solve ill-defined problems and apply learning to novel contexts (Bransford et al., 2000). Instruction is organized to all students to engage in higher-order thinking (Marzano et al., 2001) and meta-cognitive processes (Bransford et al., 2000) that foster high levels of cognitive engagement and learning. Effective teachers intentionally and explicitly model thinking processes as a way of scaffolding students’ development of meta-cognitive processes (Houtveen & van de Grift, 2007).
Quality of Feedback
High quality feedback also serves to enhance student learning either by bridging the gap between a student’s current level and the target goal (e.g., scaffolding and hints) and/or by “pushing” the student to think or process information in greater depth (e.g., explaining or answering additional questions). Effective feedback is immediate, contingent, corrective and/or specific, and tied to natural settings (Marzano et al., 2001; Swanson et al., 1999). Such feedback serves to control frustration, increase interest and motivation and effort, and promote learning and higher-order thinking (Wood, Bruner, & Ross, 1976).
Instructional Dialogue
A final element of effective instructional support that is relevant across content areas concerns the dialogues that occur between teachers and students and among students. A large body of work demonstrates that students learn more when they are engaged in deep and meaningful conversations about content (Alexander, 2008). This literature highlights the importance of building a shared dialogue, in contrast to the more typically seen classroom conversation patterns in which teachers ask a question, students respond, and teachers ask follow-up questions (Sinclair & Coulthard, 1975). Alexander (2008) also carefully distinguishes more casual instructional conversations from dialogues that are “characterized by purposeful questioning and chaining of ideas into covert lines of thinking and inquiry.” These strategies may be particularly important to employ in the context of collaborative group work, with evidence that these collaborative groups promote better learning in classrooms in which teachers explicitly teach students about how to talk with one another (Mercer, Dawes, & Staarman, 2009).
Empirical Evidence Linking CLASS to Student Performance
There are now dozens of studies demonstrating that pre-school and elementary school-age children in classrooms with higher CLASS scores have better social and academic performance (see Downer, Sabol, & Hamre, 2010 for review). Students in classrooms with higher CLASS scores have been shown to have higher academic skills and achievement (Hamre, Hatfield, Pianta, & Jamil, 2014; La Paro, Pianta, & Stuhlman, 2004; Mashburn et al., 2008), less conflict with their teachers (Hamre & Pianta, 2005), and greater social skills (Burchinal, Vandergrift, Pianta, & Mashburn, 2010; Curby et al., 2009; Mashburn et al., 2008). There is less evidence regarding associations between CLASS-S and student outcomes; however, several recent studies provide important emerging evidence that the CLASS-S assesses components of middle and high school classrooms that are associated with students’ academic and social development (Allen et al., 2011; Allen et al., 2013; Hafen et al., 2012; Kane et al., 2012).
The CLASS-S was one of the several observational measures used in the large Measures of Effective Teaching Study (Kane et al., 2012). Using CLASS-S data from math and English language arts teachers in Grades 4 through 8, the study demonstrated consistent associations between an overall composite of CLASS-S scores and student performance on state and national achievement tests. These links were stronger than those from teachers’ years of experience and level of education. Additionally, students in classrooms with higher CLASS-S scores learned more and reported feeling more connected to school. Associations were stronger in math than in language arts classrooms.
Further evidence for the validity of CLASS-S was obtained in a much smaller sample of middle and high school teachers across multiple academic content areas (Allen et al., 2013). This study used a state-specified achievement score as the outcome and all the dimensions of the CLASS-S observations as the predictor, controlling for each student’s prior achievement. Effects ranged in size from very weak (.01) to moderate (.48) and were strongest for positive climate, teacher sensitivity, regard for adolescent perspectives, instructional learning formats, and analysis and inquiry. These effects were consistent for math, science, and English language arts (ELA) classrooms. The strongest support for the value of the teacher-student interactions, as measured by CLASS-S, comes from a coaching intervention (MyTeachingPartner–Secondary) study that used random assignment to demonstrate not only that teachers who were coached on the CLASS-S dimensions improved the effectiveness of their interactions with students but also that students in their classrooms learned more as a result (Allen et al., 2011; Gregory, Allen, Mikami, Hafen, & Pianta, 2014).
Factor Structure of the CLASS: The Current Study
These early validation studies also point to the need for greater clarity around the factor structure of the CLASS. Published studies by Allen and colleagues (2011) and Allen et al. (2013) and the Measures of Effective Teaching (MET) study (Kane et al., 2012) have each used different approaches to factoring CLASS-S data. In neither case did study authors use only the suggested and hypothesized factor structure discussed above. Allen and colleagues started by confirming the original three-factor structure but then created a single composite from a subset of five CLASS-S dimensions that were most strongly related to student achievement scores, while the MET study suggested that a single-factor solution was the most parsimonious. Although these approaches certainly have their utility, this presents an issue for researchers and practitioners. In an effort to provide researchers and practitioners a more general solution and empirical evidence to support decision making about how to best utilize CLASS-S data in researcher and policy settings, the present study tests a conceptually generated a priori structure across multiple data sets.
Recent studies of the classroom environment also suggest that there may be a global responsive teaching domain that is somewhat independent of the unique effects of emotional or instructional interaction quality. This approach takes advantage of advancements in quantitative methodology and the work around bi-factor models. The bi-factor model specifies both general and domain-specific factors that are uncorrelated and, thus, can be utilized as unique predictors in the same model (Chen, Hayes, Carver, Laurenceau, & Zhang, 2012). In recent work with the CLASS at the pre-K level, there is evidence that a bi-factor model positing a general responsive teaching domain and more specific motivational, management, and cognitive facilitation domains fits the data somewhat better than a typical three-factor structure (Hamre et al., 2014) and is useful in examining associations with student learning gains. Thus, a bi-factor approach may provide an interesting alternative worthy of examination for CLASS-S.
In the current study, we will present results on various approaches for factor analysis and factor structure in an effort to understand the organization of teacher-student interactions in secondary classrooms. The current study systematically assesses the degree to which the hypothesized factor structure of the CLASS-S fits data from several studies of middle and high school classrooms. In a similar article conducted on data from over 4,000 pre-school to fifth-grade classrooms, the factor structure suggested by the Teaching Through Interactions framework fits the data better than alternative one- or two-factor models (Hamre et al., 2013). In this study, the fit of the hypothesized three-factor structure was examined relative to four alternative structures: a one-factor structure that loads all dimensions on a single factor, a two-factor model that combines emotional support and instructional support into one factor while leaving classroom organization as a separate factor, the original three-factor model, and a bi-factor approach that hypothesizes a global factor of responsive teaching with subfactors representing the three-factor approach.
The data set used for these analyses combines observations from four studies (Allen et al., 2011; Bell et al., 2012; Kane et al., 2012) of over 1,000 middle and high school classrooms across the country. These studies cover a broad array of classroom contexts, including schools in rural, suburban, and urban environments, as well as schools with diverse student populations. Thus, the observational data from these studies provide a robust test of applying a specific conceptual model of classroom settings to a broad spectrum of secondary classrooms in the United States. To test the applicability and generalizability of the Teaching Through Interactions’ three-domain organization of teacher-student interactions, we used confirmatory factor analyses (CFA) to examine the extent to which the three-domain latent structure posited in the Teaching Through Interactions framework fits the natural variation in observed classroom processes in comparison with a single-factor, a two-factor, or a bi-factor solution. Consistent with the theoretical and empirical data cited above, we expected that the three-domain model, in which dimensions were organized under Emotional, Organizational, and Instructional interactions, would provide a better fit to the data than the alternative models, and that this data structure would remain similar across the different studies.
Method
Studies and Sample
The present study utilizes data from four large-scale, observational research projects conducted from 2007 to 2011 in 1,482 6th- through 11th-grade classrooms across the United States. Basic information for each study’s sample and the double-scored reliability of the classroom observations are included in Table 2. Although the number of observations of each teacher varied across samples, a single average score for each teacher was used in analyses to avoid issues concerning independence of observations. Readers are referred to individual study citations for more complete information on the data collection procedures and sample.
Descriptive Data for Classroom, Teacher, and Double-Scored Reliability.
Note. MTP = MyTeachingPartner; MET = Measures of Effective Teaching; TUCC = Toward an Understanding of Classroom Context; UTQ = Understanding Teaching Quality; ICC = intraclass correlations.
Measures of Effective Teaching
The MET study (Kane et al., 2012) data were collected between 2010 and 2011. Videotapes of classroom interactions were available from 875 teachers in Grades 6 to 8. The vast majority of teachers were videotaped four times throughout 1 academic year. Observers coded these tapes in 20-minute segments. Since the MET study was more representative demographically and comprised the largest number of teachers, it is used in analyses for both exploratory and confirmatory analyses.
MyTeachingPartner–Secondary (MTP-S)
The MTP-S study (Allen et al., 2011) data were collected between 2007 and 2009. Videotapes of classroom interactions were available from 67 teachers in Grades 6 to 11. Teachers were videotaped for 40 minutes six times throughout 1 academic year. A total of 10 observers coded these tapes in 20-minute segments.
Toward an Understanding of Classroom Context (TUCC)
The TUCC study (Bell et al., 2012) data were collected in 2009 and 2010. Videotapes of classroom interactions were available from 82 algebra teachers in 20 middle schools and 20 high schools. Teachers were videotaped four to five times throughout one academic year. A total of five observers coded these tapes in 15-minute segments.
Understanding Teaching Quality (UTQ)
The UTQ study data were collected in 2009 and 2010. Videotapes of classroom interactions were available from 458 teachers in middle school mathematics and ELA classrooms. Teachers were videotaped four to five times throughout one academic year. A total of five observers coded these tapes in 15-minute segments.
Observational Measure
The CLASS-S (Pianta et al., 2010) was used to code all video observations. The CLASS-S version used in these studies has 10 dimensions. In each study, global ratings during each observation (ranging from 10 to 30 minutes) were made on a 7-point scale, ranging from “Low” to “High,” for the following dimensions of teacher-student interactions: positive climate, teacher sensitivity, regard for adolescent perspectives, negative climate, behavior management, productivity, instructional learning formats, content understanding, analysis and inquiry, and quality of feedback. In the MET study only, there was one additional dimension (instructional dialogue). Observers watched a video of classroom interactions for a prescribed segment of time (anywhere from 10 to 30 minutes) while taking detailed field notes about specific teacher and student behaviors and interaction patterns. Observers then had 10 minutes to compare their notes with a CLASS-S manual and use the behaviorally anchored set of rating scales to record a final code for each dimension. Each session of coding is considered a cycle, which might consist of several segments (e.g., 15-minute time periods). CLASS-S scores used in current analyses were aggregated across cycles, observers, and observation visits to form variables at the classroom level. This aggregation serves to increase the reliability of a teacher’s CLASS-S score, as long as there is no evidence of meaningful variance across cycles.
Training and reliability
Across studies, all observers attended a workshop led by a CLASS-S certified trainer to attain initial reliability on the CLASS-S. The workshops consisted of guided practice with coding videotaped classroom footage. After the training workshops, observers had to pass a videotaped reliability test, involving either five or six cycles of 20 to 44 minutes. Criteria for passing required coders to score at least 80% match (within 1 scale point) with master codes on the global rating scales.
During data collection, all studies completed further reliability checks through independent dual review of videotapes. In the MTP-S study, the TUCC study, and the data used from the MET study, all observations were double-coded. In the UTQ study, 25% of the observations were double-coded. Reliability estimates from this double coding are provided in Table 2. In calculating weighted kappa, the estimates are derived at the segment level within dimensions and then averaged up to the dimension level. In calculating intraclass correlations (ICC), the estimates are averaged to the lesson level within dimensions.
Data Analysis Approach
Due to the relative size of the sample, data from the MET study were used for initial confirmatory and exploratory analyses. Data analysis was modeled after the approach of Hamre et al. (2013) and involved four steps to determine whether the original three-factor structure fit the data best or whether another structure provided a more accurate fit to the data: (a) a CFA on the large MET sample using the original three-factor structure; (b) an EFA on the MET sample to test whether another three-factor structure is more appropriate; (c) CFAs using the MET sample confirming whether a one-, two-, three-, or bi-factor latent structure provided a better fit to the MET data; and finally, (d) CFAs across the remaining data sets (MTP-S, TUCC, and UTQ) to confirm whether the findings from the MET sample replicate. It is of note that the dimension of instructional dialogue is only present in the MET sample.
Our evaluation criteria included a comparative fit index (CFI) above 0.90 and close to 0.95, root mean square error of approximation (RMSEA) under 0.10 and close to 0.06, and a standardized root mean square residual (SRMR) under 0.12 and close to 0.08 (Hu & Bentler, 1999; Schermelleh-Engel, Moosbrugger, & Müller, 2003). Given that data are missing in some studies, we used full information maximum likelihood (FIML) as the method to estimate our structural parameters. Given that the amount of missingness is small for all of the studies (<5%) and that the results were consistent when these data were listwise deleted, it is unlikely that missing data affected the results. The benefits of maximum likelihood estimation with incomplete data have been extensively studied (e.g., Arbuckle, 1996). Multiple fit indices were then compared across the models, because the chi-square overall goodness-of-fit test is unfavorably affected by large sample size, model misspecification, or violation of distribution assumptions (Bollen, 1990).
Results
Comparing Three-Factor Models: The Theoretical Model Versus the Revised Model
The first analysis was intended to establish whether the theoretically derived three-factor model provided a good fit to the data. The largest sample (MET) was used in this analysis. In the theoretical model, there are three factors each with different dimensions: (a) Positive Climate (PC), Negative Climate (NC), Teachers’ Sensitivity (TS), and Regard for Adolescent’s Perspective (RAP) load on the Emotional Support factor; (b) Behavior Management (BM), Productivity (P), and Instructional Learning Format (ILF) load on the Classroom Organization factor; and (c) Content Understanding (CU), Analysis and Inquiry (AI), Quality of Feedback (QoF), and Instructional Dialogue (ID) load on the Instructional Support factor. We tested this structural hypothesis through a CFA framework. As can be seen in Table 3, this original three-factor model provided relatively poor fit to the data, χ2(30) = 1044.34, p < .001, CFI = 0.80, RMSEA = 0.24.
Summary of Model Fit Indices as a Function of Study Samples and Alternative Confirmatory Factor Models.
Note. ES = Positive Climate, Teacher Sensitivity, and Regard for Adolescent Perspectives; CO = Negative Climate, Behavior Management, and Productivity; IS = Instructional Learning Formats, Content Understanding, Analysis and Inquiry, and Quality of Feedback. Residual correlations were included for the associations between Regard for Adolescent Perspectives and Analysis and Inquiry, as well as Regard for Adolescent Perspectives and Instructional Learning Formats. The bi-factor model is presented for the MET data set as a comparison and not for the other data sets as this approach is less optimal in smaller data sets. CFI = comparative fit index; RMSEA = root mean square error of approximation; MET = Measures of Effective Teaching; MTP = MyTeachingPartner; TUCC = Toward an Understanding of Classroom Context; UTQ = Understanding Teaching Quality.
The second analysis was intended to explore whether an alternative factor structure provided a better fit to the data. The largest sample (MET) was again used in this analysis. In order to remove the redundancy of running this exploratory analysis on the same sample a follow-up confirmatory analysis would be conducted, we used a split-sample approach. Thus, we ran the EFA on only half of the sample. This EFA confirmed that a three-factor structure was best but with a slightly different factor structure than the original three-factor structure; thus, this alternative model was included in the confirmatory analyses. The new revised model was exactly the same as the original model except Negative Climate loaded most heavily on the Classroom Organization domain and Instructional Learning Formats loaded on the Instructional Support domain.
In the third analysis, we tested this proposed three-factor structural hypothesis through a CFA framework. We ran the revised three-factor CFA on the half of the MET sample not included in the EFA. As can be seen in Table 3, this revised three-factor model provided a much better fit to the data, χ2(30) = 643.31, p < .001, CFI = 0.94, RMSEA = 0.10. The resulting factor loadings from this analysis are depicted in Table 4. All loadings were significant. Positive Climate, Teacher Sensitivity, and Regard for Student Perspectives loaded positively on Emotional Support. Behavior Management and Productivity loaded positively on Classroom Organization, while Negative Climate loaded negatively. Instructional Learning Formats, Content Understanding, Analysis and Inquiry, Quality of Feedback, and Instructional Dialogue all loaded positively on the Instructional Support factor.
Results of Confirmatory Revised Three-Factor Model.
Note. Est = estimate; SE = standard error.
As expected, each domain of teacher-student interactions was correlated with the other domains. The strongest association was between Emotional Support and Instructional Support (ρco,es = .71 [.02]). Emotional Support and Classroom Organization also were strongly correlated with each other (ρco,is = .52 [.02]), as were Classroom Organization and Instructional Support (ρes,is = .59 [.02]).
Reliability scores were computed for each of the three factors (see Geldhof, Preacher, & Zyphur, 2014). The formula for the reliability of the composite score is defined for the confirmatory factor model as
where m is the number of indicators and λ I and θ I are a factor loading and residual variance for the ith variable respectively. For the hypothesized three-factor model, reliability of Emotional Support was estimated at ρxx = .72. For Classroom Organization, ρxx = .84, and that of Instructional Support was computed as ρxx = .91. These internal consistency measures are all greater than an acceptable cutoff value of ρxx = .70.
Comparing One-, Two-, Three-, and Bi-Factor Models of Classroom Observational Data
Raw data for CLASS-S teacher-student interaction scores were directly fit to alternative confirmatory factor models. For the original and revised three-factor models, we structured the factors as described above. We then tested two alternate structures. In the single-factor model, all indicants loaded on a unitary factor. In the two-factor models, we tested all possible combinations of combining two factors into one latent construct (e.g., Emotional Support and Instructional Support) and leaving the third factor isolated (e.g., Classroom Organization).
Table 3 shows fit indices for the one-factor model, the original three-factor model, the bi-factor model, and the revised three-factor model using the complete MET data. The most noticeable feature in the results is that the revised three-factor solution is the best fitting model among alternative models, as the combined evaluation across fit indices including CFI, RMSEA, and SRMR (not depicted in the table) indicated that the revised three-factor model provided the best fit. The single-factor model (RMSEA = 0.27), the three possible two-factor models (RMSEA = 0.24-0.33), and the original three-factor model (RMSEA = 0.24) all failed to be acceptable-fitting models. The bi-factor model (RMSEA = 0.11) was comparable with the revised three-factor model (RMSEA = 0.10), although it did not provide a better fit.
A final set of CFAs tested whether findings from the MET data would replicate in the other three samples (MTP-S, TUCC, and UTQ). The revised three-factor model was confirmed in each sample as the best fitting (see Table 3).
Follow-Up Analyses
A set of multiple-group CFAs were conducted to determine whether there were differences in the fit of the revised three-factor model based on known demographic characteristics in the MET sample. There were no significant differences in the fit of the revised three-factor model based on course subject (math, ELA), grade, or urbanicity of the school location (urban vs. rural).
Discussion
There is a clear need in educational practice for measures that combine developmental theory with classroom practices, as there is a need for clearer articulations and validation of higher-level theories of effective teaching (Douglas, 2009). This study provides a direct test of the Teaching Through Interactions theoretical model and finds some support for a structure that organizes teacher-student interactions in three factors: Emotional Support, Classroom Organization, and Instructional Support. Ultimately, empirical evidence supporting this theoretical structure has the potential to help researchers and practitioners decide how best to measure classroom interactions in secondary school classrooms in order to capture classroom environments that are most healthy for adolescents’ social and academic development.
The findings from this study suggest that the three-factor structure providing the best fit to the data was only slightly different from the three-factor solution reported for elementary classrooms and yet was best described at the domain level in terms of Emotional Support (Positive Climate, Teacher Sensitivity, and Regard for Adolescent Perspectives), Classroom Organization (Behavior Management, Productivity, and Negative Climate), and Instructional Support (Instructional Learning Formats, Content Understanding, Analysis and Inquiry, Quality of Feedback, and Instructional Dialogue). When tested against other potential models for organizing classroom interactions, such as a one omnibus teacher quality factor or a simpler social and instructional supports model, the three-factor model had the best fit to data from over 1,000 secondary school classrooms. Importantly, this three-factor model fit observational data collected from a range of studies across a broad range of settings, including urban and rural classrooms, and across 6th- to 12th-grade classrooms.
As noted, this structure supported the model posited by the Teaching Through Interactions model with two exceptions: Negative Climate was best integrated in Classroom Organization instead of Emotional Support, and Instructional Learning Formats was organized within Instructional Support instead of Classroom Organization. The former makes logical sense, as when negative climate is high in a classroom, there tends to be disruptions to both the management of the classroom and the productivity of the classroom. The latter also is intuitive as the variety of formats presented in instruction are directly related to the extent to which instruction is varied and emphasizing higher-order thinking (Bransford et al., 2000; Marzano et al., 2001). Although this revised organization is slightly different from the original structure that has been reliably confirmed in pre-K and early elementary samples, it is likely a reflection of the typically more rigidly structured environment in middle and high school classrooms. For instance, scores for Behavior Management and Productivity for CLASS-S tend to be negatively skewed with a mean near 6 on a scale from 1 to 7, while scores for those same dimensions in pre-K classrooms tend to be more uniformly distributed with a mean closer to 5.5.
This study suggests that the proposed factor structure of the CLASS-S is relevant across grade levels, diverse contexts, and across content areas, although specific confirmatory tests for grade level were not conducted. A theory and measure for effective teaching that can be applied so widely, particularly when taken together with validity studies documenting effects of these types of interactions, may be important for scaling and application.
The implications of this study for informing researchers and practitioners are notable. For researchers, as a broad framework based on classroom interactions allows us to better consolidate literature on effective teaching—rather than fragmenting that literature by what may be somewhat artificial distinctions such as grade level or content area (Hamre et al., 2013). However, researchers should not feel confined in their use of the CLASS-S dimensions to the factor structure presented here. Although this study provides evidence that the revised three-domain structure best reflects interactions across the many classrooms studied, there are alternative approaches that may be valuable for varying research contexts and questions. For instance, there may be good reasons to focus efforts on particular dimensions in work that is aimed toward professional development (e.g., Allen et al., 2011). When the goal is improving classroom interactions in order to improve student achievement, a targeted focus on those dimensions most strongly related to achievement is likely beneficial. Also, there is recent evidence that statistical approaches that separate broad frameworks into factor structures that are uncorrelated (e.g., bi-factor models) may allow for greater predictive validity (Hamre et al., 2014). This approach is particularly useful for researchers who not only attempt to isolate the unique effects of the global domain of responsive teaching but are also interested in the underlying effects of more specific interactions (e.g., academic-cognitive facilitation). Additionally, the statistics from this study indicate the bi-factor model fits the data about as well as the revised three-factor structure. However, the use of the bi-factor model for practitioners may be quite difficult given its empirical rather than theoretical basis. Further, the bi-factor model is more susceptible to bias in smaller data sets and often presents convergence issues. Thus, the revised three-factor model tested is likely to remain the predominantly utilized structure for CLASS-S.
For practitioners, the CLASS-S observational protocol has already been translated into an implementation guide describing the dimensions in detail along with suggestions of how to implement those practices more often in their instruction. Emphasizing the importance of interactions to teachers is a powerful tool. Teachers often consider improving the day-to-day interactions they have with their students as a means for improving student learning, but have few tools or guides for describing or improving these behaviors, even though there is ample evidence to suggest their importance (Slavin et al., 2009). Further, the practical use of the CLASS-S framework has already been demonstrated with its inclusion in effective professional development (Allen et al., 2011; Mikami, Gregory, Allen, Pianta, & Lun, 2011). For practitioners, in the context of understanding classroom interactions, it allows for a consistent description of quality interactions that can form the basis of a shared community of practice within schools and across disciplines. It also has the benefit of face validity as teachers can readily understand what it means to implement high quality emotional, organization, and instructional interactions in the classroom.
Another area of research in which the Teaching Through Interactions framework and the three domains of classroom interactions presented in this study may enhance our knowledge is in capturing how teachers develop specific skills over time. In a study that followed teachers from their last year of teacher education into their first 2 years of teaching, evidence showed some distinct patterns of development across teachers (Malmberg, Hagger, Burn, Mutton, & Colls, 2011). Specially, results indicated that, on average, teachers increased their classroom organization and management skills over these early years of teaching, with the strongest improvements seen among teachers who started off relatively lower in this domain. A different pattern was found for emotional support, where initial increases were followed by declines, leading to an inverted U-shape curve over time. This nuanced view of teacher development is only possible when researchers consider multiple domains of teaching quality rather than overall changes over time based on student test scores or teacher report (Hamre et al., 2013).
The goal of any measurement work in the field of education should ultimately lie in promoting positive changes in education practice. Indeed, there is general consensus within the educational community that the professional development of teachers is of paramount importance (Caspary, 2002). However, professional development typically occurs in the absence of a direct link to actual teaching behavior in classrooms, particularly for already-trained and certified teachers (Caspary, 2002). Systematic classroom observation systems provide a standardized approach to measuring and noting teachers’ strengths and weaknesses and evaluating whether professional development activities are actually helping improve the classroom interactions responsible for learning. Along with continuing to develop professional development programs, future research might productively be geared toward exploration of the ways in which these findings could translate into day-to-day practices of teacher assessment and training. For example, further research might consider whether it was possible to conduct CLASS-S ratings via live observation by regular school personnel, as part of ongoing teacher development efforts.
Limitations
The study is not without limitations. First, the inter-rater reliabilities for the rating scales used in this study, while considered acceptable, were not as high as are found in less comprehensive or more discrete measures of teacher behavior. Recent work has suggested that there may be some utility in breaking apart an observational measure into sources of variance using Generalizability Theory (e.g., Casabianca et al., 2013). In this work, error associated with rater, day, and setting can be isolated and incorporated to improve reliability of measurement. This is an area of need across many observational measures although it was beyond the scope of the present study. Second, this study tested a theoretical model of teaching using an observational measure that was developed from a specific theory. It will be important for future work to examine the extent to which data from measures not only based on different theoretical frameworks but also focusing on teacher-student interactions (e.g., Danielson, 2011) may also fit this broad conceptual model for the way teacher-student interactions are organized. Finally, it is quite likely that that measurement of teacher-student interactions are influenced by characteristics of individual students or the group more generally, and so interpretations of these results as indicative of only the teacher are unwarranted. Future research could help determine whether the demographic composition of students in a classroom (e.g., % low income, % ethnic minority) significantly alters the proposed structure of classroom interactions.
Conclusion
The Teaching Through Interactions framework offers only one view into the quality of teacher-student interactions in classrooms, interactions that are among the most important aspects of teachers’ job. There is also initial evidence that this is a robust model for understanding classroom interactions consistent across culture (Malmberg & Hagger, 2009) and grade level (Allen et al., 2011; Mashburn et al., 2008). This study confirms that teacher-student interactions, as observed by the CLASS-S can be reasonably organized into domains of emotional, organizational, and instructional behavior across a range of samples. In doing so, this framework provides a vehicle that may help link basic research to teacher evaluation and professional development.
Footnotes
Declaration of Conflicting Interests
The author(s) declared the potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Bridget K. Hamre and Robert C. Pianta are authors of the CLASS measure and own a company that conducts CLASS trainings.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study and its write-up were supported by grants from the Bill and Melinda Gates Foundation, the William T. Grant Foundation, and the Institute for Education Science (R305A100367).
Author Biographies
). His research interests include understanding the function and development of close relationships in adolescence.