Measuring Text-Focused Reading Instruction

Abstract

Helping students engage with complex texts has been a longstanding challenge, though teachers have received little guidance about practices that help students in engaging with texts. This paper provides a range of empirical evidence about a tool designed to provide formative insight into text-focused teaching, which we used to reliably score more than 500 reading lessons in a large district. We describe the structure of the tool, its relationship to other measures of teaching, teacher and school attributes, and student outcomes. We then provide guidance to practitioners and researchers seeking to employ such a tool for teacher development.

Keywords

teaching literacy educational studies classroom achievement

Helping students engage productively and proficiently with complex texts has been a longstanding challenge in the United States (Gamse et al., 2008; Terry, 2021). Researchers, educators, and policy makers alike have proposed a range of solutions for addressing this challenge including more ambitious standards for student learning, as well as aligned curricular resources for classrooms (Polikoff, 2012, 2015). The texts students are asked to read—whether they are sufficiently “complex,” “leveled,” or align with students’ interests and background knowledge—have been foregrounded in these conversations (Amendum et al., 2018; Goldman & Lee, 2014). Texts alone, however, will not promote positive student outcomes. Teaching is instrumental in these efforts, though there has been less attention paid to supports for teachers that could ultimately help students. Teachers need more clarity about practices that help students engage with texts, along with tools that provide formative information about their use of such practices. Our focus here is the development of such a tool and the provision of validity evidence about its use across hundreds of elementary reading lessons in a large urban district.

Standardized, reliable observational rubrics can provide teachers with a common language for describing and ultimately improving teaching. By naming key instructional practices and delineating different levels of quality instantiation of those practices, teachers can have a clearer and more coherent sense of what to focus on during instructional interactions and what “good” looks like (Allen et al., 2011; Cohen, 2015; Hill & Grossman, 2013). To date, however, there are relatively few observational tools that focus squarely on how teachers support students in making text-based arguments and claims. Existing measures frequently observe generic teaching practices (Danielson, 2007; Pianta & Hamre, 2009). Among content-specific tools, a majority identify disciplinary practices in science and mathematics, rather than reading (Hill et al., 2008; Schoenfeld, 2018; Walkington & Marder, 2018). Rare exceptions include the Protocol for Language Arts Teaching Observation (PLATO; Grossman et al., 2013), used to observe disciplinary practices in reading and writing instruction (Grossman et al., 2014; Kane & Staiger, 2012), the Argument Rating Tool (Reznitskaya & Wilknson, 2021), which focuses on teacher facilitation of student talk, and the Early Language & Literacy Classroom Observation Pre-K Tool (ELLCO Pre-K) which focuses on broad features of the literacy environment in early childhood classrooms, including the characteristics of the books, the organization of the book area, and the approaches to book reading (Smith, et al., 2008). The majority of other research tools developed to analyze reading comprehension instruction have focused on more micro-level classroom interactions, including the proportion of teachers’ and students’ fine-grained discourse moves used during text-based discussion (Connor et al., 2020; Dwyer et al., 2016; Gámez & Lesaux, 2015). While such tools are helpful for characterizing the discourse environment, they do not include rubrics with different levels of quality to help teachers in improvement efforts. Fine-grained, line by line coding of discourse may have value for research but is less practically useful for school or district improvement efforts.

In the absence of readily available, systematically researched tools for supporting text-focused instruction, schools have largely relied on open-source tools created by a range of educational support organizations. Among the most prominent among these are the Instructional Practice Guides (IPGs), coaching rubrics for elementary reading. Developed in 2012 by Student Achievement Partners (SAP), a non-profit organization founded by some of the authors of the Common Core State Standards and dedicated to helping teachers shift instruction in ways that are aligned with ambitious goals for students (achievethecore.org), the IPGs are available for free on SAP’s website and can be modified based on context-specific goals. The IPGs are downloaded approximately 100,000 times each year, and SAP has worked with scores of organizations, schools, and districts to support implementation of IPG coaching for teachers. Despite this widespread use, the schools and districts using the IPGs—or other tools like them—have no way of systematically tracking instructional improvements from IPG coaching because there is limited evidence about the tools’ measurement characteristics.

Rather than having researcher-developed observation instruments with robust psychometric properties exist at a distance from the practitioner-oriented tools used in schools across the country, we would be well-served to work together to develop a suite of tools that could be used in tandem to better support teachers in helping students engage with sophisticated texts. In this paper, we provide empirical validity evidence about a reading-focused observational tool, the Instructional Practice Research Tool for English Language Arts (IPRT-ELA) that we developed in concert with SAP to complement the practices featured in the IPGs. Over the course of several years, we worked together to substantially modify their coaching protocol to create a reliable observational protocol for analyzing reading teaching. After piloting and refining the tool, we used the IPRT-ELA rubrics, coupled with the content-generic CLASS rubrics, to score more than 500 reading lessons in a large, urban school district.

We describe the structure of the tool, the degree to which it can be used to measure text-focused practices consistently, its relationship to content-generic observational measures of teaching quality, as well as the relationship between scores and student achievement and observable teacher and school attributes (Steinberg & Garrett, 2016; Whitehurst et al., 2014). In doing so, we situate this tool in the broader landscape of supports for teachers and provide guidance to practitioners, researchers, and policymakers seeking to employ such a tool in conjunction with coaching protocols like the IPGs. We answer four research questions to help build a validity argument for the IPRT-ELA, which we list here along with our hypotheses for each:

1. What does text-focused instruction look like in upper elementary classrooms of a large school district? How much does instructional practice vary within a teacher’s lessons? How much does practice vary within and across schools?

Hypothesis 1

We hypothesize that most of the variation in text-focused instruction will be among lessons within individual teachers. We also hypothesize that most of the remaining variation would be across teachers, within a school, and the smallest amount of variation would be across schools.

2. What is the relationship between text-focused measures of instructional quality and high and low stakes, content-generic measures of instructional quality?

Hypothesis 2

We hypothesize that text-focused instruction will be modestly correlated with other observational measures of instructional quality, especially those focused on instructional support. We also hypothesize our tool would provide distinct, subject-specific insight into text-focused instructional quality.

3. What is the relationship between scores on text-focused instruction and teacher and student characteristics?

Hypothesis 3

We hypothesize that text-focused instruction will vary with teacher characteristics, especially teacher experience. We hypothesize—or hope—that instructional quality will not systematically vary by student characteristics.

4. To what extent do scores on text-focused instruction predict teacher value-added estimates based on student achievement outcomes?

Hypothesis 4

We hypothesize that text-focused instructional quality positively predicts teacher value-added estimates.

Background and Framework

Features of High-Quality Text-Focused Instruction

The IPRT, like the IPG coaching tools that preceded it, reflect Valencia and colleagues’ (2014) theory of instructional quality, which articulates that various, interconnected elements of comprehension instruction work together to support positive student outcomes. These include the texts teachers use, the tasks students are provided for engaging with those texts, the scaffolding teachers afford, the goals teachers have for students, and the clarity with which those goals are communicated to students (Cervetti et al., 2016; McKeown et al., 2009).

Teachers need to carefully select appropriately challenging texts, and there needs to be sufficient instructional time dedicated to engaging with those texts (Fang, 2016; Shanahan, 2013). Reading scholars emphasize the importance of combining both quantitative and qualitative dimensions to understand a text’s complexity, as each assesses distinct aspects of text quality (Mesmer et al., 2012; Valencia et al., 2014). Quantitative measures focus on text readability and are calculated in different ways. In determining vocabulary difficulty, Lexile levels assume that words with more frequency in the English language are easier for students, whereas Flesch-Kincaid defines difficult words as those with more syllables (Toyama et al., 2017). Standards highlight four qualitative text dimensions: levels of meaning, structure, language conventionality and clarity, and knowledge demands (CCSSO, 2010). Steinbeck’s The Grapes of Wrath has a low Lexile level but is primarily taught in high school because the historical context and themes are more appropriate for older readers (Glaus, 2014).

It is not just the sophistication or complexity of the text that generates disciplinary challenge for students. The tasks teachers ask students to complete working with the same text could be radically different. One teacher could ask students to recall facts made explicit in a text, while another could use the same text to push students to make synthetic arguments about an author’s use of language (Valencia et al., 2014). These two tasks with the same text would demand quite different reading skills, which are vital considerations in assessing and supporting the overall text-focused “instructional quality” (Goldman et al., 2016).

Scores of studies also underscore the vital role teachers play in facilitating meaningful student interactions with texts (Boardman et al., 2018; Dewitz & Graves, 2021; Duke et al., 2011). In particular, the questions teachers ask and how they respond to students’ questions and contributions can support active text engagement, help students revise textual misunderstandings, and make text-based inferences and arguments (Deshler et al., 2007; McKeown et al., 2009; Reznitskaya et al., 2009; Shanahan et al., 2010). In more productive interactions, teachers probe students’ contributions, pushing them to clarify and elaborate their text-based responses, using specific evidence from the text itself (Nystrand & Gamoran., 1991; Taylor et al., 2003). The questions teachers ask—coupled with their responses to student contributions—can support close reading (McKeown et al., 2009; Snow & O'Connor, 2016). Teachers who affirm the use of textual evidence have students who subsequently provide more text-based evidence (Gillies & Khan, 2009; Jadallah et al., 2011). It is thus important to analyze both teacher questions and student responses (Connor et al., 2020; Gámez & Lesaux, 2015).

Text-focusing questions can guide students to develop textual understandings that are more grounded in text evidence (Beck & McKeown, 1981; McElhone, 2012). When teachers support students in justifying their inferences with precise textual evidence, they help develop students’ argumentation skills (Hillocks, 2010; Reznitskaya et al., 2009). Some have argued that students can engage more deeply with texts when teachers begin lessons with a big picture “text-focusing question” (“What are Walt Whitman’s key ideas in this poem, and how does he express them?”)—followed by smaller-grained, aligned questions (“What are the narrator’s feelings in the first stanza?”) (Kaakinen & Hyona, 2005; McCrudden et al., 2010).

Research and standards also suggest the importance of students’ developing vocabulary knowledge during text-focused instruction. This can include instruction on identification of word parts (e.g., prefixes and suffixes and root words) and use of context clues to determine a word’s meaning (CCSSO, 2010; Lubliner & Smetana, 2005). Targeted vocabulary instruction can not only help students understand a particular text, but also build word knowledge and deduction skills for use with other texts (Beck et al., 1982; Lubliner & Smetana, 2005).

There are also teaching moves not exclusively related to literacy instruction that have been well-documented to support student engagement with texts. Illustrating how specific, more immediate academic tasks relate to broader, expansive learning goals can encourage student engagement (Grossman et al., 2014; Borko & Livingston, 1989). Research suggests the benefit of explicitly stating the purpose of the lesson, while also revisiting that purpose throughout a lesson, continually coming back to broader, disciplinary goals (McElhone, 2012; Nystrand & Gamoran, 1991). Finally, interactional scaffolding, or in-the-moment responses to students’ textual misunderstandings, has been shown to help develop students’ comprehension skills (Athanases & de Oliveira, 2014; Clark & Graves, 2005; Reynolds & Goodwin, 2016).

Building a Validity Argument for Classroom Observational Measures

In recent years, scholars have endorsed the use of subject-specific tools that provide teachers with feedback on specific facets of teaching and afford a common vocabulary for describing and supporting improvement on these facets (Hiebert & Stigler, 2017). Empirical evidence suggests teachers carefully attend to the language of a district’s observational rubrics, underscoring that teachers consider a tool’s dimensions not simply as a barometer, but also as a guidepost for quality (Phipps & Wiseman, 2021).While content-generic tools, like FFT (Danielson, 2007) or the CLASS (Pianta & Hamre, 2009), help characterize instruction across a variety of classrooms (Kane & Staiger, 2012), those tools’ macro-level ratings of broad constructs like the “Positive Climate” in the classroom often fall short of the nuanced, subject-specific information critical to helping teachers understand and improve disciplinary practice (Connor et al., 2009; Hill & Grossman, 2013).

Though the technical properties of a tool can seem removed from classroom interactions, such properties have direct implications on information afforded teachers. Does each rubric represent a distinct aspect of teaching, or do some practices “hang together” in empirical factors? If so, rather than focusing on each individual rubric, coaches or principals can provide teachers more parsimonious insight organized around broader factors. Teachers, like all learners, benefit from organizing structures to help them make sense of information gleaned from an observation.

Teachers would also benefit from evidence about whether raters consistently or reliably score instruction so that they know a score is indicative of the quality of their teaching, rather than sources of construct irrelevant variance such the idiosyncratic preferences of a rater (Hill et al., 2012; Zhai et al., 2021). Consistency across raters is needed for teachers to trust the “objectivity” of ratings. Teachers’ observational ratings are also often associated with students’ prior achievement (Whitehurst et al., 2014) or students’ demographics (Campbell & Ronfeldt, 2018). It’s unclear whether these observed differences are driven by “true” differences in instruction or rater bias, but regardless, such patterns can impede teachers’ trust in such ratings.

We also need evidence regarding the stability of instructional constructs from lesson to lesson and the extent to which this measure captures information distinct from other tools (Kane & Staiger, 2012). These kinds of evidence help inform both sampling decisions (e.g., how many lessons do I need to observe to get a reasonable sense of a teacher’s instruction?), as well as the inferences that a tool can support (e.g., does this tool provide different information about a teacher’s practice than other tools we might use?). These day-to-day or week-to-week fluctuations may depend in part on the goals of a lesson or sequence of the unit (Grossman, 1990; Grossman et al., 2014), but such variation can also complicate inferences about individual teachers needed for summative evaluations and personnel decisions (Cohen & Goldhaber, 2016). At the same time, descriptive information documenting temporal variation in instruction may provide invaluable formative information to teachers and school leaders. Similarly, if a content-specific tool is highly correlated with generic observation protocols, it would have little relative advantage.

Finally, a central component of recent classroom observation research has been the predictive relationship between teaching practices and student outcomes, which have tended to be relatively low to modest (Allen et al., 2011; Kane & Staiger, 2012). There are many factors that could mitigate a strong correlation. There may well be omitted, unobservable, or non-school related practices associated with student outcomes but are uncaptured by a particular tool (Condron, 2009). Teaching can also vary greatly across lessons, depending on the instructional practice being measured, which can complicate inferences about relationships with measures from a single time point, such as value-added models, which provide insight into the degree to which the achievement of a teacher’s students is better (a positive value-added coefficient) or worse (a negative value-added coefficient) than one would predict based on students’ prior achievement and demographic characteristics (Garrett & Steinberg, 2015; Grossman et al., 2014; Hill et al., 2012; Pianta & Hamre, 2009; Polikoff, 2012, 2015). Nevertheless, analyzing whether observational tools are correlated with VAMs is useful, as it can provide suggestive evidence of whether the instructional focus of a tool might also contribute to desired student outcomes. Our analysis is organized around these questions about observational measures based on these types of evidence that teachers, schools, and researchers alike might need.

Data and Methods

We leverage data from a larger study focused on the implementation and effects of a content-specific professional learning curriculum including administrative data on schools, teachers, and students, along with teacher-submitted videos of text-focused reading instruction.

Participants

In partnership with the district, the research team invited third-fifth grade ELA teachers clustered in a set of volunteer schools to participate. We selected teachers to maximize variation in demographics, experience, and prior performance (based on the district’s evaluation system). Sixty-six teachers in 24 elementary schools submitted a total of 580 videos of 30-min reading lessons over the course of two years.¹ We excluded the 64 lessons that did not involve a text. The average teacher submitted 8 scoreable lessons, with one teacher submitting 17 lessons.

Our teacher sample is similar to other teachers in the district in terms of their gender, whether they are Black, Hispanic, or Asian, and whether they have earned a graduate degree (see Table 1). They are also comparable in terms of their effectiveness, based on the district’s classroom observational measure and their ELA value-added scores. Teachers in our sample were significantly more likely to be white (44 vs. 32%, p < .05) and have fewer years of experience (6 vs. 10 years, p < .001). The schools at which our sample teachers work compared to other district schools had significantly lower concentrations of Black students (55 vs. 70%, p < .05), higher concentrations of other non-white students (6 vs. 4%, p < .05), and higher concentrations of White students (18 vs. 10%, p < .10). Our sample schools are also less likely to be “high poverty schools” based on the district classification (67 vs. 82%, p < .10) and have higher percent of students testing proficient on the ELA PARCC assessment (36 vs. 22%, p < .01). This suggests that while our findings might not generalize across all district schools, our sample still represents a relatively large and diverse sample of teachers in the district.

Table 1.

Comparison of the Study Sample to the District’s Teacher and School Population.

	Sample	District	Difference
	M (SD)	M (SD)	p-value
Teacher (N)	65	3882
Prior year’s observation score^a	3.34	3.26	.14
Prior year’s observation score^a	(.41)	(.42)
Value-added score^a	.12	−.01	.45
Value-added score^a	(.88)	(1.00)
Proportion with ELA value-added score	.59	.14	.00
Proportion with ELA value-added score	(.50)	(.35)
Proportion Female	.83	.75	.11
Proportion Female	(.38)	(.43)
Proportion Black	.47	.51	.48
Proportion Black	(.50)	(.50)
Proportion Hispanic	.05	.07	.42
Proportion Hispanic	(.21)	(.26)
Proportion Asian	.02	.04	.34
Proportion Asian	(.12)	(.19)
Proportion white	.44	.32	.04
Proportion white	(.50)	(.47)
Proportion missing race	.03	.05	.46
Proportion missing race	(.17)	(.22)
Proportion graduate degree	.65	.72	.25
Proportion graduate degree	(.48)	(.45)
Experience^a	6.14	9.56	.00
Experience^a	(5.45)	(7.52)
School (N)	24	92
Proportion high poverty	.67	.82	.09
Proportion high poverty	(.48)	(.38)
Proportion students Black	.55	.70	.04
Proportion students Black	(.38)	(.30)
Proportion students Hispanic	.20	.16	.35
Proportion students Hispanic	(.26)	(.21)
Proportion students non-White	.06	.04	.02
Proportion students non-White	(.07)	(.04)
Proportion students White	.18	.10	.08
Proportion students White	(.25)	(.19)
Proportion students English learner	.16	.11	.15
Proportion students English learner	(.20)	(.16)
Proportion students special education	.13	.17	.13
Proportion students special education	(.05)	(.14)
Proportion prior Year’s ELA PARCC	.36	.22	.01
Proficient (L4+)	(.26)	(.23)

Note. All data from 2018, the second year of our study with the exception of 10 study teachers whose data comes from 2017 as they did not continue with the study in 2018. One teacher in the study sample could not be linked to the administrative data and thus has missing values for all teacher characteristics. p-values in the last column reflect a simple difference-in-means t-test.

^aDistrict sample N is 3870 for observation scores, is 269 for Value-Added Score, and is 3876 for Experience and the Study Sample N is 39 for Value-Added Score.

Measures

Below, we describe the measures we used in this study. First, we discuss the development and properties of the IPRT-ELA, the tool that we developed to better and more systematically describe text-focused instruction. We designed our tool with an explicit purpose: to provide teachers and school systems with discipline-specific, formative information about text-focused reading comprehension instruction. The tool was not designed for teacher-level, summative evaluation. To assess the convergent and discriminant validity of the IPRT-ELA, we leverage: 1) teacher-level observational scores from the district’s evaluation measure, rubrics derived from Danielson’s Framework for Teaching and 2) lesson-level scores on the Classroom Assessment Scoring System (CLASS), a widely used, content-generic observational tool, shown to have significant associations with students’ reading achievement in some contexts and grade levels (Kane & Staiger, 2012; Pianta et al., 2008).The predictive validity analysis of the IPRT-ELA uses teacher’s value-added scores based on district-provided student achievement data.

Instructional Practice Research Tool for English Language Arts

We designed the IPRT-ELA to support teachers in engaging in text-focused teaching (see Table 2 for tool overview). One of our funders for the broader study of professional learning connected our research team with Student Achievement Partners (SAP), suggesting the Instructional Practice Guides (IPG) coaching tools might be helpful as assessments for understanding the development of standards-aligned reading instruction in the district. The constructs captured in the IPGs were well-aligned with our theory of strong reading instruction, but our initial piloting of the rubrics suggested they were difficult to use consistently or reliably across teams of raters. Thus, we worked closely with SAP to develop the initial version of the IPRT-ELA. This resulted from a careful review of the extant literature, the CCSS-ELA documents, and SAP’s pre-existing coaching protocol (achievethecore.org). Members of our research team and SAP worked together over several months to refine this tool by watching and scoring many language arts classroom videos, discussing the ways in which IPRT rubrics reflected both teams’ perceptions of strong text-focused teaching. Each round of co-observation and scoring surfaced additional areas for rubric refinement, and the final rubrics reflect multiple rounds of discussion. This ongoing and time-intensive collaboration was made possible due to generous funding for both our team and SAP’s team.

Table 2.

IPRT-ELA Items & Descriptions.

Item	Description	Examples
Classroom Materials	Captures the overall complexity of the text by mapping the text’s readability onto a grade band using at least two Lexile-level recommendations (i.e., ATOS, Degree of Reading Power ®, Flesch-Kincaid, the Lexile Framework ®, Reading Maturity, and Text Evaluator)	Esperanza Rising by Pam Munoz Ryan- Lexile Measure: 750LRecommended- Grade Level Quantitative Complexity: 4th – 5th- ATOS Measure:5.3- Recommended Grade Level Quantitative Complexity: 5th
Quantitative complexity
Text quality	Observe if text is of publishable quality (i.e., copyrighted) rather than manufactured or adapted for classroom use	Publishable quality is any text (prose, drama, poetry, lyrics, article, etc.) that is copywritten
Text quality		Non-publishable quality is any text made or adjusted specifically for classroom use (skill packets, teacher-generated stories, etc.)
Text-Focused Teaching	Assesses the proportion of the lesson during which students are engaged with texts (i.e., less than half of the lesson, about half, about three-quarters, or almost all)	Out of a 30-min observation, the teacher spends nearly all of the time facilitating student discussion on character motivation within their novel study. Thus the vast majority of the lesson is focused on text
Text focus
Text-dependent questions	Examines the frequency with which a teacher asks questions or prompt tasks that require students to refer to ideas in the text (e.g., “What is the main idea of the text?“) versus not (e.g. based on student opinion without referencing text; related to management)	Teacher asks the following questions1. What is the structure of this text?2. Why did the author include diagrams?3. Is it more important to read fiction or nonfiction?4. Write a summary of the textQuestions 1, 2, and 4 are text-dependent. Question 3 is not because a student could answer the question without specifically referring to the text
Text-specific questions	A subset of text-dependent questions; assesses the proportion of teacher questions or tasks that prompt students to cite specific elements of text (e.g., “Go to page 25. Why were all eyes on Wyoming?“)	Two text-dependent questions above have been rewritten to be text-specific (which require reference to particular aspects of the text at hand, rather than more general text-dependent questions)- What does the phrase as a result tell us about how the author structured this text?- Why did the author include the diagram on page 4? Compare it with the diagram on page 7
Text-based responses	Measures the frequency with which students use textual evidence to support their answers to teacher questions	Third graders are asked to describe the character- S1: Ada is kind because she is nice- S2: Ada is curious because she wants to find out what thorns are, and what the mystery smell is. She is always asking, “Why?”- S3: Ada is bossy because she is just like my big sisterOnly S2 cites the text to support their response
Vocabulary instruction	Examines the extent to which teachers’ Tier II vocabulary instruction supports students’ comprehension of both: 1) the words’ meaning in the particular text, and 2) overall understanding of the text	High-scoring example- The author wrote, “the crowd erupted in loud cheers.” what does the word erupted mean in this sentence? How do you know?Low-scoring examples- The author wrote, “the crowd erupted in loud cheers.” what does the word loud mean in this sentence? (not Tier II)- What does the word erupted mean? (no context)
Question sequence	Assesses the coherence of the teachers’ questions across a lesson, and the degree to which those questions contribute to a broader comprehension goal	There is an overarching guiding question that focuses students on comprehending the text (e.g., what are the key ideas of the poem and how are they expressed?) with additional questions or tasks that support student understanding (who narrates the poem? What are their feelings in the first stanza? What text evidence supports your thinking?)
Additional Teacher Supports	Focuses squarely on the frequency with which a teacher pushes students to clarify or elaborate on a previous response	S: I think Rufus is braveT: Why do you think that? (Probe 1)S: Because he attends a protestT: How does that show he is brave? (Probe 2)S: He knows it could be dangerous but he does it anyway to stand up for rightsT: Show us in the text (probe 3)
Teacher probing
Scaffolding	Captures the consistency and quality of a teacher’s responses to students’ content misunderstandings in the moment	Scaffolds are instructional supports that move students towards stronger understandingsExamples- Prompting- Re-explanation- Additional questioning- Reference to materials (anchor charts/examples)Non-examples- Repeating the question with no support or adjustment- Stating the correct answer- Telling students to “try again” with no guidance
Purpose	Captures the extent to which a teacher clearly articulates the goals of a lesson, as well as the degree to which the lesson’s activities cohere to the stated goal	- Lesson purpose tied to ELA skills and clearly communicated to students (“Today we will read to understand how authors develop mood through language choice”)- Work of the lesson clearly and consistently supports the purpose (Teacher and students identify the mood of a poem by locating and discussing figurative and descriptive language)- It is evident students understand the purpose of the lesson activities, including referring to the purpose during the lesson

Over several rounds of refinement, we finalized items, modifying the original coaching protocol to emphasize lower-inference and more readily observable behaviors, while not sacrificing key aspects of reading instruction. For instance, we dropped an item from the coaching protocol that asked whether “teacher[s were] pos[ing] challenging questions…[that kept] all students persevering with them,” because we determined that perseverance was a construct that could not be consistently observed across raters (a crosswalk between the IPG and the IPRT-ELA, along with details on scoring are available upon request).

The resulting IPRT-ELA contains eleven items assessing teachers’ text-focused instructional practice. Raters score 30 minutes of consecutive instruction and also collect any supporting materials that teachers provided for that lesson (e.g., student worksheets and lesson plans). Nine of the eleven indicators are scored using 4-point ordinal scales, with 1 representing no opportunities to engage and a 4 indicating consistent engagement. The remaining two indicators are binary scales. For the Text Quality indicator where a 1 represents whether the text is of publishable quality (i.e., copyrighted) rather than adapted for adapted for classroom use. For example, Newsela provides several Lexile level options for a single newspaper article so teachers can select the appropriate level for their students, effectively modifying the news article from its original form. Though these adapted texts may be more accessible for students to read, the adapted version may also be stripped of the original’s unique structure and language, decreasing the text’s overall complexity and quality (Hoffman, 2017). The Quantitative Complexity indicator captures text complexity by mapping readability onto a grade band using at least two Lexile-level recommendations (i.e., ATOS, Degree of Reading Power ®, Flesch-Kincaid, The Lexile Framework ®, Reading Maturity, and Text Evaluator). Texts were coded as being at or above the complexity expected for the class’s grade band or below that expected.

CLASS Observational Tool

The Classroom Assessment Scoring System (CLASS) is undergirded by the idea that positive teacher-student interactions increase student engagement and learning (La Paro et al., 2004). CLASS assesses three domains—Emotional Support, Classroom Organization, and Instructional Support—in which raters score several dimensions on a 1 (low) to 7 (high) scale. The CLASS tool is designed to score 15-min lesson segments, whereas the IPRT-ELA tool requires a 30-min lesson. Therefore, we average scores across the two 15-min segments, so the unit of observation was consistent across tools. Dimensions were then averaged together at the domain level, as is typical with CLASS scoring (Pianta & Hamre, 2009). An overall CLASS score was generated by averaging the three domain scores.

Scoring Lessons

Members of our research team trained a team of five raters to score instruction using the IPRT-ELA and 10 raters to score both reading and mathematics lessons using CLASS (mathematics lessons were used in another component of the larger study from which this study’s data are drawn). For both tools, trainings lasted two days, followed by a certification process. Master raters from the research team scored a set of five videos of reading instruction. Raters were certified to begin scoring videos using the IPRT-ELA tool once they reached at least 80% agreement with the master scores for each IPRT-ELA item or CLASS dimension across the five videos. Throughout the two-year study, raters attended weekly calibration meetings.

Approximately 27% of the lessons were multiple-scored with the IPRT-ELA tool, and 24% of the lesson segments were multiple-scored for CLASS. We examined inter-rater reliability using the exact agreement of raters’ scores, intraclass correlation (ICCs), and Spearman’s rho (see Table 3). Overall, the ICCs of all indicators on the IPRT-ELA were at or above 70%, which is considered good to excellent reliability (Cicchetti, 1994; Hallgren, 2012). All CLASS dimensions across both years of scoring had ICCs greater than 50%, which is typical of CLASS scoring, and which Cicchetti (1994) deems acceptable.

Table 3.

IPRT-ELA and CLASS Reliability Statistics.

	Percent exact agreement, %	Rho, %	ICC (pairs of raters),%
IPRT-ELA (165 lessons)
Text-specific questions	78	72	77
Text-dependent questions	77	78	85
Text-based responses	78	84	91
Text focus	82	85	92
Question sequence	72	60	70
Teacher probing	69	70	78
Purpose	76	60	70
Scaffolding	61	59	70
Vocabulary instruction	80	74	82
Quantitative complexity	90	96	97
Quality text	93	86	95
CLASS (487 lesson segments)
Positive climate	31	46	59
Teacher sensitivity	33	44	59
Regard for student perspective	32	56	73
Behavior management	48	54	68
Productivity	47	40	54
Negative climate	67	48	58
Learning Formats	36	48	61
Content understanding	31	50	70
Analysis and Inquiry	33	43	64
Quality of feedback	34	47	69
Instructional dialogue	30	52	70
Student engagement	33	42	62

Note. CLASS lesson segments include mathematics lessons which were coded by the same group of coders at the same time as the ELA lessons.

District Observational Measures

The district provided us with scores from a content-generic observational measure to evaluate teachers’ instruction, which is based heavily on Danielson’s Framework for Teaching. The measure consists of five rubrics: (1) cultivate a responsive learning community, (2) challenge students with rigorous content, (3) lead a well-planned, purposeful learning experience, (4) maximize student ownership of learning, and (5) respond to evidence of student learning. The five scores, each on a four-point scale, are averaged to create an overall score. Teachers’ yearly observation score is an average of five, 30-min observations. School administrators conduct three, and two are completed by an “expert” who conducts observations at many schools. The final observation score is included as part the annual evaluation process that determines a teacher’s monetary incentive (for descriptive statistics, see Table 1).

Administrators engage in a three-day training about the general principles of the rubrics and receive monthly professional development to practice scoring instructional videos. Expert raters participate in a six-week intensive training, with bi-monthly norming sessions (Curtis, 2011). Though the district has employed multiple methods to increase reliability, there is little work about the reliability of these observational scores (Gitomer et al., 2014).

Individual Teacher Value-Added Student Achievement Data

Teachers’ VAM scores are intended to demonstrate their instructional impact on students’ achievement. The district provided us with the score they estimate as part of their evaluation system and produced by predicting students’ end-of-year test scores for each teacher based on students’ prior test scores and demographic variables. The district then compares the differences between predicted scores and actual test scores to generate a teacher-level VAM score. The tests used for calculating VAM in this study are the PARCC examinations in ELA.

Data Analysis

Multilevel Factor Analysis

The design of the IPRT-ELA tool is based on the theory that the 11 indicators collectively describe the degree to which a teacher’s text-focused practices align with the CCSS-ELA. We conducted multilevel exploratory and confirmatory factor analyses (EFA and CFA) to test this theory. The multilevel factor analyses account for nesting of lesson observations within teachers and, in so doing, allows us to examine the underlying structure of the IPRT-ELA both within and between teachers. Our analyses treat all indicators as categorical to preserve the ordinal nature of the response options. The search for the best fitting factor structure of text-focused instructional practice at each level informed our path through the EFA to the CFA. We estimate all multilevel factor analyses in Mplus 8 (Muthén & Muthén, 2017).

The initial EFA allowed for there to be up to four factors at the lesson observation and/or teacher levels. Our exploration focused on the one and two factor models, given low factor loadings on the third and fourth factors. Several of the indicators performed poorly. We dropped these iteratively and re-estimated the EFA. The first indicators we excluded were Quantitative Complexity and Quality Text as these did not load significantly onto factors at the teacher level (Table 4). From a theoretical perspective, we justify excluding these indicators as they focus on the text itself, whereas the others squarely focus on the teacher’s practices. We next dropped Vocabulary Instruction, which continued to perform poorly, especially at the teacher level. This is likely the result of our observing relatively little vocabulary instruction, with more than half the lessons having none at all. With these three indicators excluded, the model’s fit substantially improved (Table 5, second panel). The final EFA suggested either a 1-factor model of five indicators or a 2-factor model of eight indicators split between the factors. We estimated CFA on each of these (Table 6). The fit of the 1-factor model is both better than the 2-factor model and meets each fit statistic’s threshold for a good fitting model, indicating that the one-factor solution provides technically reliable scores.

Table 4.

Exploratory Factor Analysis (EFA) Results.

Item	Initial EFA			Final EFA
	Single Factor model	2-Factor model		Single Factor model	2-Factor model
	Single Factor model	Factor 1	Factor 2	Single Factor model	Factor 1	Factor 2
Within teachers across lesson observations
Text-specific questions	.872*	.833*	.092	.871*	.823*	.110*
Text-dependent questions	.970*	.982*	−.008	.972*	.987*	−.011
Text-based responses	.881*	.818*	.161*	.880*	.806*	.176*
Text focus	.638*	.656*	−.036	.635*	.659*	−.047
Question sequence	.360*	.150	.480*	.357*	.135	.480*
Teacher probing	.245*	.120	.280*	.245*	.113	.279*
Purpose	.038	−.182	.477*	.040	−.196*	.482*
Scaffolding	.161*	.023	.291*	.163*	.007	.310*
Vocabulary instruction	.180*	.101	.172*
Quantitative complexity	−.034	−.120	.158
Quality text	.314*	.374*	−.127
Between teachers
Text-specific questions	.922*	.905*	.026	.938*	.930*	.017
Text-dependent questions	.976*	.958*	.006	.994*	.987*	.025
Text-based responses	.824*	.834*	−.167	.821*	.925*	−.176
Text focus	.864*	.900*	−.205	.816*	.850*	−.048
Question sequence	.538*	.538*	.321*	.566*	.391	.462*
Teacher probing	.173	.171	.104	.151	.003	.357*
Purpose	.052	.000	.978*	.088	−.242	.796*
Scaffolding	.273	.278	.607*	.296	.033	.794*
Vocabulary instruction	.208	.247	−.389
Quantitative complexity	−.019	−.019	.015
Quality text	.099	.140	−.409
Model fit
Chi-square value	178.210	107.103		144.237	59.277
Chi-square p-value	.000	.002		.000	.000
RMSEA	.039	.029		.062	.043
CFI	.984	.993		.983	.995
TLA	.981	.989		.976	.988
SRMR within	.063	.042		.066	.032
SRMR between	.226	.175		.170	.057

*p < .05.

N = 683 scores within 66 teachers.

Table 5.

Confirmatory Factor Analysis (CFA) Results.

	Single Factor model	2-Factor model
Item	Single Factor model	Factor 1	Factor 2
Within teachers across lesson observations
Text-specific questions	.977***	.972***
Text-dependent questions	.869***	.872***
Text-based responses	.879***	.881***
Text focus	.635***	.635***
Question sequence	.345***		.612***
Teacher probing			.407***
Purpose			.120*
Scaffolding			.275***
Between teachers
Text-specific questions	1.000^a	1.000^a
Text-dependent questions	.942***	.955***
Text-based responses	.873***	.848***
Text focus	.825***	.831***
Question sequence	.498***		1.000^a
Teacher probing			.300
Purpose			.338
Scaffolding			.606*
Model fit
Chi-square value	24.426	91.978
Chi-square p-value	.011	.000
RMSEA	.042	.044
CFI	.998	.992
TLA	.996	.988
SRMR within	.026	.053
SRMR between	.025	.085

***p < .001, **p < .01, *p < .05.

N = 683 scores within 66 teachers.

^aVariances set to 0 for these items to adjust for initial negative residual variances.

Table 6.

Summary of IPRT-ELA Instructional Practices.

	Lesson scores	Lessons	Variance decomposition
	M (SD)	M (SD)	Scores (%)	Teacher (%)	School (%)
Text-specific questions	2.33	2.33	86.8	9.4	3.8
	(.69)	(.67)
Text-dependent questions	2.79	2.79	87.6	9.9	2.5
	(.85)	(.85)
Text-based responses	2.73	2.73	88.8	7.2	4.0
	(.86)	(.86)
Text focus	2.77	2.76	84.8	14.6	0.6
	(1.01)	(.99)
Question sequence	2.70	2.69	86.7	13.3	0.0
	(1.34)	(1.30)
Teacher probing	2.83	2.79	80.4	13.1	6.5
	(.91)	(.90)
Purpose	2.88	2.87	81.0	19.0	0.0
	(.68)	(.67)
Scaffolding	2.92	2.94	88.2	11.8	0.0
	(.90)	(.88)
Vocabulary instruction	1.96	1.97	93.3	5.0	1.7
	(1.23)	(1.21)
Quantitative complexity ^a	.77	.75	89.8	9.8	0.4
	(.42)	(.43)
Quality text ^b	.60	.60	90.5	5.8	3.7
	(.49)	(.49)
Observations	683	516	683	66	24

^aN = 382 scores, 305 lessons, 63 teachers, and 23 schools.

^bN = 510 scores, 404 lessons, 66 teachers, and 23 schools.

Note. Some of the 516 lessons were coded by multiple coders, yielding 683 lesson scores. A cross-classified multi-level model was estimated for the variance decomposition in order to account for the two teachers who switched schools between the first and second year of the study.

Our Text-Focused Instructional Practices (TFIP) measure includes the four indicators about text-focused teacher questions and student responses (Text-specific Questions, Text-Dependent Questions, Question Sequence, and Text-based Responses), as well as the Text Focus indicator that captures how central the text is to the lesson. We theorize that Purpose, Teacher Probing, and Scaffolding indicators do not load on this factor because they are general practices and are not necessarily centered around supporting students to engage with the text.

Multilevel Modeling

With factor structure confirmed, we estimate a series of cross-classified multilevel linear models to explore the contextual variation of text-focused practices, and its association with other measures of instructional practice and value-added scores. Drawing on the factor structure analysis, we average the five text-focused IPRT-ELA indicators across raters of the same lesson and calculate the measure as a simple average (Cronbach alpha = .77).² The data have both a purely hierarchical nesting component with multiple lessons observed per teacher and a cross-classified structure with most teachers nested within a school, but with two teachers who taught in a different school in each of the study’s two years. The cross-classified multilevel linear model specification allows us to analyze associations between theorized predictors and the IPRT-ELA tool’s measure of text-focused practices. Using the classification notation of Browne et al. (2001), we fit the following model to the data

y_{i} = β_{0} + β^{'} x_{i} + u_{t e a c h e r (i)}^{(2)} + u_{s c h o o l (i)}^{(3)} + e_{i}

u_{t e a c h e r (i)}^{(2)} \sim N (0, σ_{u (2)}^{2}), u_{s c h o o l (i)}^{(3)} \sim N (0, σ_{u (3)}^{2}), e_{i} \sim N (0, σ_{e}^{2})

where

y_{i}

is the lesson’s text-focused instructional practice score for the ith lesson, teacher(i) is the teacher delivering the lesson, and school(i) is the school at which the lesson was taught.

x_{i}

is a vector of lesson, teacher, and/or school characteristics associated with the ith lesson.

u_{t e a c h e r (i)}^{(2)}

is the random effect for teacher(i), and

u_{s c h o o l (i)}^{(3)}

is the random effect for school(i).

e_{i}

is a level 1 residual for the ith lesson. We fit these models in Stata 16.0 with the mixed command.

Association with Student Outcomes

In the version of this cross-classified multilevel linear model that explores the association between text-focused practices and teacher value-added scores, the $x_{i}$ vector includes indicators for value-added score quintile (Koedel & Betts, 2007; McCaffrey et al., 2009) because it has historically been difficult to find linear relationships between teaching practices and teacher value-added scores, particularly in smaller samples of teachers. The quintile of our sample teacher’s value-added scores is based on the distribution of value-added scores among all district teachers, not just the distribution among our small sample. In the models, the first quintile (i.e., the lowest value-added scores) serves as the reference group.

Findings

RQ#1: Instruction in Upper Elementary Classrooms

The average text-focused lesson here is somewhat moderately aligned with practices known to support students in making sense of complex texts (see Table 6). Across the IPRT-ELA items using a four-point scale, means fall between a 2 (lower frequency of text-focusing practices) and 3 (moderate frequency of text-focusing practices), with the exception of the Vocabulary Instruction (mean just below 2). The raters were unable to assess text quality in 21.7% of the lessons because they were unable to identify the text’s source. They were also unable to assess the text’s complexity in 40.9% of lessons because the text was not listed in any reference guides. Of the lessons in which text quality and complexity could be assessed, almost half (47.3%) used a quality text, and 44.7% used a complex text. Across all items, most of the score variation is within a teacher (80%–93%). That is, there is little consistency in text-focused instructional quality across an individual teacher’s reading lessons. A meaningful share is across teachers within a school (5%–19%). There is very little variation between schools (see Table 6).

These same lessons exhibit moderately to highly positive teacher-student interactions (see Table 7). The average lesson scores very high on Classroom Organization and in the medium range (i.e., 3-5) on Emotional Support and Instructional Support. The majority of score variation is again within a teacher, but less than for the IPRT-ELA indicators (70%–78%), with some variation at the teacher and school levels (12%–17% and 8%–16%, respectively).

Table 7.

Summary of CLASS Instructional Practices.

	Lesson segment scores	Lessons	Variance decomposition
	M (SD)	M (SD)	Scores (%)	Teacher (%)	School (%)
Instructional support	3.96	3.93	70.0	14.2	15.8
	(1.05)	(.86)
Emotional support	4.39	4.39	78.0	12.1	9.9
	(.99)	(.80)
Classroom organization	6.47	6.47	75.0	17.2	7.9
	(.77)	(.62)
CLASS overall score	4.94	4.93	69.8	17.2	13.0
	(.79)	(.66)
Observations	1,493	580	1,493	66	24

Teachers’ text-focused practices are associated with characteristics of the text used in the lesson (Table 8). Lessons that employ a quality text have a Text-Focused Instructional Practices score a third of a standard deviation higher than in lessons using a low-quality text ( $β$ = .230 (.073), p = .000). There is also evidence that text-focused practices are lower in lessons where we cannot determine the text complexity ( $β$ = -.216 (.106), p = .042).

RQ#2: Convergent and Discriminant Validity

The text-focused practices captured by our measure are modestly but statistically significantly correlated with the teacher practices assessed by the district and some domains of the CLASS (Table 9). TFIP is positively correlated with the district’s overall observational rating for the high-stakes evaluation system (

β

= .217 (.103), p = .036) and the CLASS’s overall score (

β

= .110 (.053), p = .038). The positive correlation with the CLASS overall score is driven by the Instructional Support domain, as we would hypothesize, given its focuses on practices more similar to those captured by the IPRT-ELA, including the teacher’s feedback on student contributions and facilitation of classroom dialogue (

β

= .176 (.055), p = .001). The Emotional Support domain is negatively correlated with the IRPT measure (

β

= -.133 (.062), p = .028). There is a small, statistically insignificant positive correlation between the IPRT-ELA and the Classroom Organization domain (

β

= .065 (.063), p = .304) (see Table 9).

Table 8.

Contextual Variation in Text-Focused Instructional Practices.

	$β$	S.E.
Quality text	.230**	(.073)
{Missing}	.166	(.094)
Quantitative complexity	−.054	(.089)
{Missing}	−.216*	(.106)
Female	.043	(.096)
Black	.097	(.092)
Hispanic	.061	(.176)
Asian	−.312	(.256)
Missing race	.346	(.298)
Graduate degree	.101	(.076)
Experience (years)	.018*	(.007)
Has ELA value-added score	−.009	(.075)
High poverty school	.017	(.370)
Prop. Students Black	−.018	(.566)
Prop. Students Hispanic	−1.295	(1.234)
Prop. Students other non-white	2.618	(2.205)
Prop. Students English learner	1.993	(1.542)
Prop. Students special education	−3.942*	(1.796)
Prop. Students ELA proficient	−.810	(.567)
School year 2018	−.064	(.093)
Intercept	2.928***	(.670)
Log likelihood	−519.124
N lessons	511
N teachers	65^a
N schools	24
Variance: school	.030
Variance: Teacher	.005
Variance: Residual	.407

^aWe exclude one teacher with five lessons who we were unable to link to DCPS’s teacher-level administrative data.

***p < .001, **p < .01, *p < .05.

Table 9.

Convergent and Divergent Validity of Text-Focused Instructional Practices.

	Model 1		Model 2		Model 3
	$β$	S.E.	$β$	S.E.	$β$	S.E.
District observation score	.217*	(.103)
CLASS overall			.110*	(.053)
Instructional support					.176**	(.055)
Emotional support					−.133*	(.062)
Classroom organization					.065	(.063)
Intercept	1.916***	(.351)	2.102***	(.258)	2.108***	(.351)
Log likelihood	−528.462		−528.418		−524.153
N lessons	511		511		511
N teachers	65		65		65
N schools	24		24		24
Variance: school	.014		.013		.013
Variance: Teacher	.026		.035		.029
Variance: Residual	.434		.430		.425

***p < .001, **p < .01, *p < .05, + p < .1.

RQ#3: Contextual Variation

The use of text-focused practices is also associated with teacher experience and a school’s concentration of special education students (see Table 8). Teachers with more experience have higher Text-Focused Instructional Practices scores than teachers with fewer years of experience, although the association is small (.02 additional points on the rubric, or 3% of a standard deviation, with each additional year of teaching experience; $β$ = .018 (.007), p = .017). Additionally, lessons taught in schools with a higher concentration of special education students have lower quality text-focused practices than those in schools with lower concentrations of special education students. The average lesson’s Text-Focused Instructional Practices score in a school with 10-percentage-points higher concentration than an observationally similar school is .39 points lower (57% of a standard deviation; $β$ = -3.942 (1.796), p = .028). There is no evidence that text-focused teaching varies with teacher gender, race, or whether they had earned a graduate degree. We also find no evidence that text-focused practices vary systematically with characteristics of a school’s student body, other than the percentage of students who qualify for special education. That said, our sample of 65 teachers and 24 schools is underpowered to detect small differences.

RQ#4: Predictive Validity

Our analysis of teachers’ value-added scores found no evidence of a statistically meaningful relationship between teacher value-added and observed scores on the IPRT measure (Table 10). This is not necessarily surprising, given the difficulty others have had in similar analyses and that only 58% of the sample teachers had a value-added score.

Table 10.

Predictive Validity of Text-Focused Instructional Practices.

	$β$	S.E.
Quintile 2	−.041	(.149)
Quintile 3	.208	(.173)
Quintile 4	−.015	(.137)
Quintile 5	−.184	(.155)
Intercept	2.687***	(.152)
Log likelihood	−316.794
N lessons	308
N teachers	38
N schools	20
Variance: school	.000
Variance: Teacher	.048
Variance: Residual	.424

Limitations

We recognize that the IPRT-ELA has its limitations. It is designed to only assess 30 consecutive minutes of text-based instruction and does not analyze how teachers might build on their instruction over several lessons, a pitfall of many observational measures (Cohen et al., 2020). Teachers selected the time and day for the research team to videotape, so it is possible that these lessons are not representative of a teacher’s overall instructional practice. In addition, the tool focuses only on whole group interactions around text. There may well be important small-group and/or individual-level instructional interactions that support students’ engagement with texts. These would not be captured in IPRT scores, but there are also other tools with a strong evidence base that focus on the reading classroom environment, from the perspective of an individual student, including the Individualizing Student Instruction (ISI; Connor et al., 2009).

The IPRT-ELA tool was designed to focus only on reading instruction. Other aspects of language arts teaching, including writing, speaking, and listening would not be well-aligned with the IPRT-ELA. It is worth noting that this tool is not designed to assess instruction on foundational reading skills (i.e., instruction on phonics or phonemic awareness), and thus would be less helpful for either formative or summative assessment of primary grades reading instruction.

We also acknowledge that our measures of text complexity and text quality are blunt and limited by what we could discern in video. In future research, we hope to collect texts used and develop additional methods for analyzing their quality and complexity. A central part of Valencia and colleagues’ (2014) framework is the nature of the instructional task, which we found impossible to meaningfully quantify. However, we asked raters to qualitatively note the nature of the task, and in future research, we plan to analyze these qualitative data. In addition, scholars have emphasized the importance of attending to the sociocultural contexts in which text-focused instruction is occurring (Goldman & Lee, 2014). While we, too, agree that context is crucial, we were also unable to quantify this in our observational measure. Future mixed-methods research should explore the degree to which and ways in which our findings may look different in distinct sociocultural contexts. No single observational tool is comprehensive, though we developed the IPRT-ELA to capture a wider range of reading focused practices than other existing tools. Our hope is that this tool may provide formative feedback for teachers about their text-focused teaching practices.

Discussion and Implications for Practice

Part of the challenge involved in improving students’ ability to engage with complex texts has been a lack of reliable observational tools that can be used to identify and support improvement on specific features of instruction (Grossman et al., 2014). In contrast to mathematics, where there are many observational tools (Hill et al., 2008; Schoenfeld et al., 2014; Walkington & Marder, 2018), there are very few tools to reliably assess reading instruction in elementary classrooms. Of those, none focus squarely on text-focused instruction that promotes students’ comprehension skills. Teachers would likely benefit from tools that capture the quality and quantity of practices known to support students (McDuffie et al., 2018; Porter et al., 2015). In particular, the Common Core and other ambitious standards foreground the need for students to engage deeply with authentic, challenging texts, but few observation tools feature text-focused teaching. Some tools like the PLATO include a broad scale for “Text-Based Instruction” but stop short of parsing different facets of such instruction (Grossman et al., 2014).

In this paper, we provide evidence about such a text-based observation tool, the IPRT-ELA. Over the course of several years, we used this tool to reliably assess hundreds of reading lessons in a large school district’s upper-elementary classrooms. We find encouraging evidence that many reading lessons in the district were indeed text-focused and frequently incorporated text-dependent questions. We also found considerable variability in other text-centric practices, such as whether teachers use scaffolded sequences of questions to support students’ engagement with texts. Districts will have to support teachers in continuing to develop their text-based instruction. The practices highlighted here and their corresponding rubrics, coupled with examples of high-level enactment, could be supportive tools for formative assessment and professional development, though more research is needed to explore this potential.

Variation Across Lessons and Formative Insight About Text-Focused Teaching

Like other subject specific tools, the IPRT-ELA scores fluctuated considerably more at the lesson level than at the school level. This finding resonates with Hill and Grossman’s (2013) argument that subject-specific tools measure more fine-grained practices that likely vary across lessons in ways that generic and broad measures of classroom climate and behavior management do not. Not every ELA lesson focuses on the same text-focused reading skills, and nor should it. However, even though this temporal variation may reflect natural—and true—variation in practices in reading, it does make a tool like the IPRT more challenging to use for consequential, summative evaluations of teachers. Our data suggest it would take many more observations than is typical to make appropriate inferences about a teacher’s overall practice. That said, we see the major benefit of the IPRT-ELA as the provision of more targeted, formative data for teachers, schools, and educational systems. In other words, this tool would be useful to offer teachers, instructional coaches, and districts information on where to improve teachers’ text-focused instruction. In particular, our factor analyses helped us identify a group of five text-focusing practices. This could provide teachers and schools a parsimonious set of practices for professional development and coaching, though more research is needed on the usability of the tool by school-based professionals in live observations, along with additional work on how teachers make sense of and respond to the information provided by the tool. We are in the midst of collecting those data.

Operationalizing Text Quality

We also found that Quantitative Complexity and Text Quality, measures of the classroom materials used, were associated with scores on text-focused practices. Indeed, it may well be difficult to engage in text-focused scaffolding absent rigorous texts. While prior literature theoretically highlights the importance of using complex texts in the classroom (Fang, 2016; Shanahan, 2013), our data provide early empirical evidence that quality reading instruction may indeed be contingent upon teachers using grade-level, appropriately challenging texts. In this way, there may be an important relationship between the quality of materials and supports for engaging with those materials. Though understanding the nuances of this relationship is beyond the scope of this study, we see this as an important direction for research (Cohen et al., 2020).

These findings also point to the need for more robust and validated measures of text complexity. We acknowledge that quantitative reading measures are only one way of examining the difficulty of a text. The extent to which a text is considered complex also involves readers’ prior background knowledge, whether the text has multiple levels of meanings, and the degree to which the author’s language is conventional (Fisher & Frey, 2014; Mesmer et al., 2012). We also need better insight into the ways in which text selection occurs based on the sociocultural context in which reading instruction takes place (Goldman & Lee, 2014). We need more robust evidence about such decision-making processes, and the implications of them, to better support literacy coaches and teachers in their selection of complex texts that will engage their students and cultivate their comprehension skills.

Is Text-Focused Instructional Quality Just “Good Teaching”?

We also set out to understand the degree to which the IPRT-ELA measured distinct facets of instruction from those captured by the CLASS and the district’s high-stakes observation measure. While we find statistically significant correlations between our measures and the district’s observational measure and the CLASS domain of Instructional Support, those correlations were low to moderate (i.e., r < .25). This suggests that the IPRT-ELA is assessing different aspects of reading instruction than those prioritized by content-generic tools. A tool like the IPRT-ELA could provide teachers with helpful data about their disciplinary teaching practices (Hill & Grossman, 2013), as well as illuminate for districts the relative strengths and weaknesses of reading instruction across classrooms. Such data could then inform the focus of district-wide professional development efforts, along with tailored coaching supports for individual teachers. In other work, we have argued for the value of using discipline-specific tools in concert with more general tools like the CLASS (Berlin & Cohen, 2020). Some teachers may well need support with generic practices—including the cultivation of warm and productive classroom environments—along with support in the text-focused practices captured here. Understanding the interplay between the two, as well as how schools use different tools to support teacher development, is also beyond the scope of this study but is a vital area for ongoing research.

We also found some unexpected correlations between IPRT scores and other CLASS domains. There was a significant, negative relationship between text-focused practices and the CLASS domain of Emotional Support, which focuses on teacher sensitivity, and the classroom climate. This is counter to what we found in mathematics lessons in the same district (Berlin & Cohen, 2020). Though our data preclude inferences about mechanisms underlying this relationship, the Emotional Support dimensions of CLASS do privilege student autonomy and peer-to-peer interactions. The teacher-focused instructional scaffolds highlighted in the IPRT-ELA may well occur less often in student-led lessons. Future research should investigate these relationships in greater depth, complemented by qualitative insight into mechanisms.

Variation by Context and Predicting Student Outcomes

Before using a given observational tool, it’s also important to understand the degree to which it captures the quality of instruction, distinct from the characteristics of students. Scholars have found that some observational tools may be biased, particularly in that teachers of predominantly students of color may systematically receive lower scores (Campbell & Ronfeldt, 2018; Chaplin et al., 2014). Fortunately, most of the student characteristics examined—including percentage of Black students, ELL students, and students receiving subsidized lunch—were not significantly associated with IPRT scores. However, we do see significant, negative associations between the percentage of students receiving special education services and scores IPRT-ELA text-focused scales. It is unclear whether this finding suggests an actual difference in teachers’ instruction in classrooms with these students or rater bias, but it indicates more research is needed to ensure students with disabilities in the district are receiving the same levels of text-focused instructional support as other students. Future studies should be conducted in a way that helps the field disentangle whether and how rater bias may influence scores. This, along with other threats to measurement reliability, should be examined with data collected on text-focused teaching in a range of contexts, to assess the generalizability of the measure.

Finally, we explored the relationship between text-focused practices and teacher value-added measures. Similar to previous studies analyzing the relationship between observational tools and student outcomes, we find small, not statistically significant correlations between the two (Bell et al., 2012; Grossman et al., 2013; Hill et al., 2011; Kane & Staiger, 2012; Strong et al., 2011). Many teachers in our sample did not have value-added data, so we could only test a smaller subset of participants, and hence our estimates are quite noisy. Examining these relationships with larger samples is an important next step in our research.

Conclusion

Despite the plethora of tools for assessing high-quality, standards-aligned teaching in mathematics, we have comparably few tools that measure reading instruction. Here we provide an array of evidence about such a tool, the IPRT-ELA, which we developed in partnership with Student Achievement Partners (SAP) to help districts, schools, and researchers both understand and support teachers in helping students engage with complex texts. Scores of schools and districts are already using SAP’s Instructional Practice Guides in informal walkthroughs and for coaching, though there is limited research about such efforts. A central goal of this work was to build an empirical base about these particular practices. Our efforts also provide a proof of concept that research teams can work productively with educational non-profits who provide much of the instructional support to districts around the country. Rather than treating practitioner-facing tools as distinct from research tools, we must find ways to bridge the two so that we develop rigorous empirical evidence about the tools being used in American schools.

Decades of research suggest that measuring core teaching practices and providing teachers with such information can help improve both instruction and student outcomes (Dee & Wyckoff, 2015; Dixit, 2002; Taylor & Tyler, 2012). However, in this district, as well as in most districts across the country, teachers receive only content-generic information about their teaching (Cohen & Goldhaber, 2016). These data suggest that such information would obscure content-focused aspects of text-based instruction. Though more research is needed in larger and more representative samples of teachers across districts, we see these findings as an exciting first step towards helping teachers support students in engaging with rigorous, grade-level texts.

Footnotes

Author’s Note

We could not have completed this work without our colleagues at Student Achievement Partners. In particular, Jessica Eadie, Lisa Goldschmidt, and Amanda Vitello worked alongside us throughout our work. Rachel Etienne spent countless hours discussing videos with us, as we worked together to develop the measures described here. We are also grateful to our district partners for supplying the data employed in this research and for answering our many questions.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Finally, the authors wish to thank the Charles and Lynn Schusterman Family Philanthropies and the Overdeck Family Foundation for their financial support of this work. The research reported here was also supported by the Institute of Education Sciences, U.S. Department of Education, through Grant #R305B140026 to the Rectors and Visitors of the University of Virginia. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. Errors are attributable to the authors.

ORCID iD

Julie Cohen

Notes

References

Allen

J. P.

Pianta

R. C.

Gregory

Mikami

A. Y.

Lun

(2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333(6045), 1034–1037. https://doi.org/10.1126/science.1207998

Amendum

S. J.

Conradi

Hiebert

(2018). Does text complexity matter in the elementary grades? A research synthesis of text difficulty and elementary students’ reading fluency and comprehension. Educational Psychology Review, 30(1), 121–151. https://doi.org/10.1007/s10648-017-9398-2

Athanases

S. Z.

de Oliveira

L. C.

(2014). Scaffolding versus routine support for Latina/o youth in an urban school: Tensions in building toward disciplinary literacy. Journal of Literacy Research, 46(2), 263–299. https://doi.org/10.1177/1086296X14535328

Beck

I. L.

McKeown

M. G.

(1981). Developing questions that promote comprehension: The story map. Language Arts, 58(8), 913–918. https://doi.org/10.2307/41961422

Beck

I. L.

Perfetti

C. A.

McKeown

M. G.

(1982). Effects of long-term vocabulary instruction on lexical access and reading comprehension. Journal of Educational Psychology, 74(4), 506–521. https://doi.org/10.1037/0022-0663.74.4.506

Bell

C. A.

Gitomer

D. H.

McCaffrey

Hamre

Pianta

(2012). An argument approach to observation protocol validity. Educational Assessment, 17(3), 62–87. https://doi.org/10.1080/10627197.2012.715014

Berlin

Cohen

(2020). The convergence of emotionally supportive learning environments and college and career ready mathematical engagement in upper elementary classrooms. AERA Open, 6(3), 1-20. https://doi.org/10.1177/2332858420957612

Boardman

A. G.

Boelé

A. L.

Klingner

J. K.

(2018). Strategy instruction shifts teacher and student interactions during text‐based discussions. Reading Research Quarterly, 53(2), 175–195. https://doi.org/10.1002/rrq.191

Borko

Livingston

(1989). Cognition and improvisation: Differences in mathematics instruction by expert and novice teachers. American Educational Research Journal, 26(4), 473–498. https://doi.org/10.3102/00028312026004473

10.

Browne

W. J.

Goldstein

Rasbash

(2001). Multiple membership multiple classification (MMMC) models. Statistical Modelling, 1(2), 103–124. https://doi.org/10.1177/1471082x0100100202

11.

Campbell

S. L.

Ronfeldt

(2018). Observational evaluation of teachers: Measuring more than we bargained for? American Educational Research Journal, 55(6), 1233–1267. https://doi.org/10.3102/0002831218776216

12.

Cervetti

G. N.

Wright

T. S.

Hwang

(2016). Conceptual coherence, comprehension, and vocabulary acquisition: A knowledge effect? Reading and Writing, 29(4), 761–779. https://doi.org/10.1007/s11145-016-9628-x

13.

Chaplin

Gill

Thompkins

Miller

(2014). Professional practice, student surveys, and value- added: Multiple measures of teacher effectiveness in the Pittsburgh Public Schools (REL 2014–024). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic.

14.

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284

15.

Clark

K. F.

Graves

M. F.

(2005). Scaffolding students' comprehension of text. The Reading Teacher, 58(6), 570–580. https://doi.org/10.1598/rt.58.6.6

16.

Cohen

(2015). Challenges in identifying high-leverage practices. Teachers College Record: The Voice of Scholarship in Education, 117(7), 1–41. https://doi.org/10.1177/016146811511700702

17.

Cohen

Goldhaber

(2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387. https://doi.org/10.3102/0013189x16659442

18.

Cohen

Hutt

Berlin

Wiseman

(2020). The change we cannot see: Instructional quality and classroom observation in the era of common core. Educational Policy, 36(6), 1261–1287. https://doi.org/10.1177/0895904820951114

19.

Condron

D. J.

(2009). Social class, school and non-school environments, and black/white inequalities in children's learning. American Sociological Review, 74(5), 685–708. https://doi.org/10.1177/000312240907400501

20.

Connor

C. M.

Morrison

F. J.

Fishman

B. J.

Ponitz

C. C.

Glasney

Underwood

P. S.

Piasta

S. B.

Crowe

E. C.

Schatschneider

(2009). The ISI classroom observation system: Examining the literacy instruction provided to individual students. Educational Researcher, 38(2), 85–99. https://doi.org/10.3102/0013189X09332373

21.

Connor

C. M.

Kelcey

Sparapani

Petscher

Siegal

S. W.

Adams

Hwang

J. K.

Carlisle

J. F.

Carlisle

J. F.

(2020). Predicting second and third graders’ reading comprehension gains: Observing students’ and classmates talk during literacy instruction using COLT. Scientific Studies of Reading: The Official Journal of the Society for the Scientific Study of Reading, 24(5), 411–433. https://doi.org/10.1080/10888438.2019.1698583

22.

Council of Chief State School Officers . (2010). Common core state standards ELA. National Governors Association Center for Best Practices.

23.

Curtis

(2011). District of Columbia Public Schools: Defining instructional expectations and aligning accountability and support. The Aspen Institute, Education & Society Program.

24.

Danielson

(2007). Enhancing professional practice: A framework for teaching. Association for Supervision and Curriculum Development.

25.

Dee

T. S.

Wyckoff

(2015). Incentives, selection, and teacher performance: Evidence from IMPACT. Journal of Policy Analysis and Management, 34(2), 267–297. https://doi.org/10.1002/pam.21818

26.

Deshler

D. D.

Palincsar

A. S.

Biancarosa

Nair

(2007). Informed choices for struggling adolescent readers: A research-based guide to instructional programs and practices. International Reading Association (NJ3).

27.

Dewitz

Graves

M. F.

(2021). The science of reading: Four forces that modified, distorted, or ignored the research finding on reading comprehension. Reading Research Quarterly, 56(S1), S131–S144. https://doi.org/10.1002/rrq.389

28.

Dixit

(2002). Incentives and organizations in the public sector: An interpretative review. Journal of Human Resources, 37(4), 696–727. https://doi.org/10.2307/3069614

29.

Duke

N. K.

Pearson

P. D.

Strachan

S. L.

Billman

A. K.

(2011). Essential elements of fostering and teaching reading comprehension. What research has to say about reading instruction, 4(1), 286–314.

30.

Dwyer

Kelcey

Berebitsky

Carlisle

J. F.

(2016). A study of teachers’ discourse moves that support text-based discussions. The Elementary School Journal, 117(2), 285–309. https://doi.org/10.1086/688896

31.

Fang

(2016). Teaching close reading with complex texts across content areas. Research in the Teaching of English, 51(1), 106–116.

32.

Fisher

Frey

(2014). Addressing CCSS anchor standard 10: Text complexity. Language Arts, 91(4), 236–250.

33.

Gámez

P. B.

Lesaux

N. K.

(2015). Early-adolescents’ reading comprehension and the stability of the middle school classroom-language environment. Developmental Psychology, 51(4), 447–458. https://doi.org/10.1037/a0038868

34.

Gamse

B. C.

Bloom

H. S.

Kemple

J. J.

Jacob

R. T.

(2008). Reading first impact study: Interim Report. NCEE 2008-4016. National Center for Education Evaluation and Regional Assistance.

35.

Garrett

Steinberg

M. P.

(2015). Examining teacher effectiveness using classroom observation scores: Evidence from the randomization of teachers to students. Educational Evaluation and Policy Analysis, 37(2), 224–242. https://doi.org/10.3102/0162373714537551

36.

Gillies

R. M.

Khan

(2009). Promoting reasoned argumentation, problem‐solving and learning during small‐group work. Cambridge Journal of Education, 39(1), 7–27. https://doi.org/10.1080/03057640802701945

37.

Gitomer

D. H.

Crouse

Joyce

(2014). A review of the DC IMPACT teacher evaluation system Paper prepared for the Committee for the Five-Year (2009–2013). Summative evaluation of the District of Columbia public schools. The National Academy of Sciences.

38.

Glaus

(2014). Text complexity and young adult literature: Establishing its place. Journal of Adolescent & Adult Literacy, 57(5), 407–416. https://doi.org/10.1002/jaal.255

39.

Goldman

S. R.

Lee

C. D.

(2014). Text complexity: State of the art and the conundrums it raises. The Elementary School Journal, 115(2), 290–300. https://doi.org/10.1086/678298

40.

Grossman

Cohen

Ronfeldt

Brown

(2014). The test matters: The relationship between classroom observation scores and teacher value-added on multiple types of assessment. Educational Researcher, 43(6), 293–303. https://doi.org/10.3102/0013189x14544542

41.

Grossman

Loeb

Cohen

Wyckoff

(2013). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores. American Journal of Education, 119(3), 445–470. https://doi.org/10.1086/669901

42.

Grossman

P. L.

(1990). The making of a teacher: Teacher knowledge and teacher education. Teachers College Press.

43.

Hallgren

K. A.

(2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in quantitative methods for psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023

44.

Hiebert

Stigler

J. W.

(2017). Teaching versus teachers as a lever for change: Comparing a Japanese and a U.S. perspective on improving instruction. Educational Researcher, 46(4), 169–176. https://doi.org/10.3102/0013189x17711899

45.

Hill

H. C.

Blunk

M. L.

Charalambous

C. Y.

Lewis

J. M.

Phelps

G. C.

Sleep

Ball

D. L.

(2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26(4), 430–511. https://doi.org/10.1080/07370000802177235

46.

Hill

H. C.

Charalambous

C. Y.

Kraft

M. A.

(2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64. https://doi.org/10.3102/0013189X12437203

47.

Hill

H. C.

Grossman

(2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384. https://doi.org/10.17763/haer.83.2.d11511403715u376

48.

Hill

H. C.

Kapitula

Umland

(2011). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48(3), 794–831. https://doi.org/10.3102/0002831210387916

49.

Hillocks

(2010). Teaching argument for critical thinking and writing: An introduction. English Journal, 99(6), 24–32. Retrieved from: https://www.ncte.org/library/nctefiles/ej0996focus.pdf

50.

Hoffman

J. V.

(2017). What if “just right” is just wrong? The unintended consequences of leveling readers. The Reading Teacher, 71(3), 265–273. https://doi.org/10.1002/trtr.1611

51.

Jadallah

Anderson

R. C.

Nguyen-Jahiel

Miller

B. W.

Kim

I. H.

Kuo

L. J.

Dong

(2011). Influence of a teacher’s scaffolding moves during child-led small-group discussions. American Educational Research Journal, 48(1), 194–230. https://doi.org/10.3102/0002831210371498

52.

Kaakinen

J. K.

Hyona

(2005). Perspective effects on expository text comprehension: Evidence from think-aloud protocols, eyetracking, and recall. Discourse Processes, 40(3), 239–257. https://doi.org/10.1207/s15326950dp4003_4

53.

Kane

T. J.

Staiger

D. O.

(2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Gates Foundation.

54.

Koedel

Betts

J. R.

(2007). Re-examining the role of teacher quality in the educational production function (p. 49). National Center on Performance Incentives.

55.

La Paro

K. M.

Pianta

R. C.

Stuhlman

(2004). The classroom assessment scoring system: Findings from the Prekindergarten year. The Elementary School Journal, 104(5), 409–426. https://doi.org/10.1086/499760

56.

Lubliner

Smetana

(2005). The effects of comprehensive vocabulary instruction on Title I students’ metacognitive word-learning skills and reading comprehension. Journal of Literacy Research, 37(2), 163–200. https://doi.org/10.1207/s15548430jlr3702_3

57.

McCaffrey

Sass

Lockwood

Mihaly

(2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4(4), 572–606. https://doi.org/10.1162/edfp.2009.4.4.572

58.

McCrudden

M. T.

Magliano

J. P.

Schraw

(2010). Exploring how relevance instructions affect personal reading intentions, reading goals and text processing: A mixed methods study. Contemporary Educational Psychology, 35(4), 229–241. https://doi.org/10.1016/j.cedpsych.2009.12.001

59.

McElhone

(2012). Tell us more: Reading comprehension, engagement, and conceptual press discourse. Reading Psychology, 33(6), 525–561. https://doi.org/10.1080/02702711.2011.561655

60.

McKeown

M. G.

Beck

I. L.

Blake

R. G. K.

(2009). Rethinking reading comprehension instruction: A comparison of instruction for strategies and content approaches. Reading Research Quarterly, 44(3), 218–253. https://doi.org/10.1598/rrq.44.3.1

61.

Mesmer

H. A.

Cunningham

J. W.

Hiebert

E. H.

(2012). Toward a theoretical model of text complexity for the early grades: Learning from the past, anticipating the future. Reading Research Quarterly, 47(3), 235–258. https://doi.org/10.1002/RRQ.019

62.

Muthén

(2017). Mplus. In Handbook of item response theory (pp. 507–518). Chapman and Hall/CRC.

63.

Nystrand

Gamoran

(1991). Instructional discourse, student engagement, and literature achievement. Research in the Teaching of English, 25(3), 261–290.

64.

Phipps

A. R.

Wiseman

E. A.

(2021). Enacting the rubric: Teacher improvements in windows of high-stakes observation. Education Finance and Policy, 16(2), 283–312. https://doi.org/10.1162/edfp_a_00295

65.

Pianta

R. C.

Belsky

Vandergrift

Houts

Morrison

F. J.

(2008). Classroom effects on children’s achievement trajectories in elementary school. American Educational Research Journal, 45(2), 365–397. https://doi.org/10.3102/0002831207308230

66.

Pianta

R. C.

Hamre

B. K.

(2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38(2), 109–119. https://doi.org/10.3102/0013189X09332374

67.

Polikoff

M. S.

(2012). ”The association of state policy attributes with teachers’ instructional alignment”. Educational Evaluation and Policy Analysis, 34(3), 278–294. https://doi.org/10.3102/0162373711431302.

68.

Polikoff

M. S.

(2015). How well aligned are textbooks to the common core standards in mathematics? American Educational Research Journal, 52(6), 1185–1211. https://doi.org/10.3102/0002831215584435.

69.

Porter

R. E.

Fusarelli

L. D.

Fusarelli

B. C.

(2015). Implementing the common core: How educators interpret curriculum reform. Educational Policy, 29(1), 111–139. https://doi.org/10.1177/0895904814559248

70.

Reynolds

Goodwin

(2016). Supporting students reading complex texts: Evidence for motivational scaffolding. AERA Open, 2(4), 5–16. https://doi.org/10.1177/2332858416680353

71.

Reznitskaya

Kuo

L.-J.

Clark

A.-M.

Miller

Jadallah

Anderson

R. C.

Nguyen-Jahiel

(2009). Collaborative reasoning: A dialogic approach to group discussions. Cambridge Journal of Education, 39(1), 29–48. https://doi.org/10.1080/03057640802701952

72.

Reznitskaya

Wilkinson

I. A.

(2021). The Argumentation rating tool: Assessing and supporting teacher facilitation and student argumentation during text-based discussions. Teaching and Teacher Education, 106(1), 103464. https://doi.org/10.1016/j.tate.2021.103464

73.

Roth McDuffie

Choppin

Drake

Davis

J. D.

Brown

(2018). Middle school teachers’ differing perceptions and use of curriculum materials and the common core. Journal of Mathematics Teacher Education, 21(6), 545–577. https://doi.org/10.1007/s10857-017-9368-0

74.

Schoenfeld

A. H.

(2018). Video analyses for research and professional development: The teaching for robust understanding (TRU) framework. ZDM, 50(3), 491–506. https://doi.org/10.1007/s11858-017-0908-y

75.

Schoenfeld

A. H.

Floden

R. E.

(2014). The TRU Math scoring rubric. Graduate School of Education, & College of Education, Michigan State University.

76.

Shanahan

(2013). Letting the text take center stage. American Educator, 37(3), 4–11.

77.

Shanahan

Callison

Carriere

Duke

N. K.

Pearson

P. D.

Schatschneider

Torgesen

(2010). Improving reading comprehension in kindergarten through 3rd grade: IES practice guide. Ncee 2010-4038. What Works Clearinghouse.

78.

Smith

M. W.

Brady

J. P.

Anastasopoulos

(2008). User's guide to the Early Language & literacy classroom observation pre-K tool (ELLCO pre-K). Education Review.

79.

Snow

O'Connor

(2016). Close reading and far-reaching classroom discussion: Fostering a vital connection. Journal of Education, 196(1), 1–8. https://doi.org/10.1177/002205741619600102

80.

Steinberg

M. P.

Garrett

(2016). Classroom composition and measured teacher performance: What do teacher observation scores really measure? Educational Evaluation and Policy Analysis, 38(2), 293–317. https://doi.org/10.3102/0162373715616249

81.

Strong

Gargani

Hacifazlioğlu

Ö.

(2011). Do we know a successful teacher when we see one? Experiments in the identification of effective teachers. Journal of Teacher Education, 62(4), 367–382. https://doi.org/10.1177/0022487110390221

82.

Taylor

B. M.

Pearson

P. D.

Peterson

D. S.

Rodriguez

M. C.

(2003). Reading growth in high-poverty classrooms: The influence of teacher practices that encourage cognitive engagement in literacy learning. The Elementary School Journal, 104(1), 3–28. https://doi.org/10.1086/499740

83.

Taylor

E. S.

Tyler

J. H.

(2012). The effect of evaluation on teacher performance. The American Economic Review, 102(7), 3628–3651. https://doi.org/10.1257/aer.102.7.3628

84.

Terry

N. P.

(2021). Delivering on the promise of the science of reading for all children. The Reading Teacher, 75(1), 83–90. https://doi.org/10.1002/trtr.2031

85.

Toyama

Hiebert

E. H.

Pearson

P. D.

(2017). An analysis of the text complexity of leveled passages in four popular classroom reading assessments. Educational Assessment, 22(3), 139–170. https://doi.org/10.1080/10627197.2017.1344091

86.

Valencia

S. W.

Wixson

K. K.

Pearson

P. D.

(2014). Putting text complexity in context: Refocusing on comprehension of complex text. The Elementary School Journal, 115(2), 270–289. https://doi.org/10.1086/678296

87.

Walkington

Marder

(2018). Using the UTeach observation protocol (UTOP) to understand the quality of mathematics instruction. ZDM, 50(3), 507–519. https://doi.org/10.1007/s11858-018-0923-7

88.

Whitehurst

G. J.

Chingos

M. M.

Lindquist

K. M.

(2014). Evaluating teachers with classroom observations: Lessons learned in four districts. Brookings.

89.

Zhai

Haudek

K. C.

Stuhlsatz

M. A.

Wilson

(2020). Evaluation of construct-irrelevant variance yielded by machine and human scoring of a science teacher PCK constructed response assessment. Studies in Educational Evaluation, 67(4), 100916. https://doi.org/10.1016/j.stueduc.2020.100916