Abstract
This article measures and discusses six teacher groups’ (a) time spent working with data and (b) time devoted to particular kinds of inquiry activities, and explores various contextual factors that influence these results. The authors make use of a framework useful in describing and analyzing the stance taken by teachers when they engage with student-learning data. Their findings suggest that most teacher groups spend the vast majority of their time collecting and analyzing data, with little time devoted to exploring potential data sources and reflecting on implications of their data analysis. Furthermore, “time on task” is less important than stance in determining the nature of the inquiry activity. Implications of these results are explored.
“Looking at student work” has emerged as an important process in teacher professional development (PD). This activity usually occurs during some form of collaborative inquiry, such as a professional learning community or lesson study group. About one decade ago, Little (2003) stated that much of this work occurs in a black box, with little empirical data to inform and support teachers engaged in this collaborative process. There continue to be few frameworks for coding, categorizing, and quantifying teachers’ collaborative inquiry processes. This article provides a descriptive framework for identifying phases of a collaborative teacher inquiry process and a conceptual framework for examining teachers’ dialogic interactions around student work. Both can be used as a basis for understanding a group’s potential for affecting teacher learning and instructional change.
Background
The culture of a school is a significant influence on teachers’ dialogic interactions in collaborative inquiry groups. McLaughlin and Talbert (2006) delineate the differences among traditional school communities and learning communities. In the latter, teachers’ professional norms are collegial as they construct shared values and beliefs about the active involvement and potential of all students in attaining learning goals. In schools characterized by a learning community, teachers are oriented toward surfacing problems of practice and view these as opportunities for generative conversation about instruction, student thinking, and interpretations of content goals (Gallimore, Ermeling, Saunders, & Goldenberg, 2009; Nickerson, 2008).
Lieberman and Miller (2008) describe this orientation as a learning stance. Others, such as Jaworski (2006), discuss the notion of inquiry stance as a “way of being” that involves, as Cochran-Smith and Lytle (2009) state, “an open and questioning viewpoint about practice” (p. 121). We claim that a teacher group’s stance toward student-learning data can determine the nature of their collaborative work. For example, through multiple studies of teachers’ collective examination of student work, Little (2007) found that when teachers share classroom events as “war stories,” they shift responsibility for learning to students, parents, and other external factors. There is little potential for these stories to serve as artifacts that “advance both individual understanding and the collective capacity of a school” (p. 221). However, artifacts of student thinking, such as written work and classroom conversations, can expose problems of practice and help focus teachers’ conversations. Kazemi and Franke (2004) identify shifts in how teachers interacted dialogically as they examined evidence of student thinking, which included changes in their instructional trajectory. Their study highlights the importance of group facilitation in a teacher group’s ability to establish an environment conducive to wondering or asking “tough questions.”
Horn and Little (2010) argue that the burgeoning of collaborative teacher inquiry throughout the educational landscape calls for large-scale investigations into the practice of teacher work groups. While we are beginning to better understand the specific processes and outcomes of this work, there is still a need for empirical evidence on the ways in which teachers conceive of, collect, and use student-learning data. Horn and Little (2010) state that “focusing on selected group-level conversational routines provides an important and strategic means for conceptualizing and investigating opportunity to learn within workplace settings” (p. 184). We agree, and have found that teacher interactions reveal key beliefs and knowledge that shape, and sometimes determine, teachers’ work.
We have come to realize that the most important conversations in which teachers engage revolve around the use of student-learning data. Specifically, we believe that an analysis of conversational routines should attend to the teacher groups’ collective stance toward student-learning data, which led us to develop a framework to empirically analyze its nature and consequence. This framework is a result of analyzing 5 years of data on 11 teacher groups and targets two key dimensions of the stance teachers take toward student-learning data. Our framework also incorporates our identification of four phases of inquiry that use student-learning data. The framework provides a way of quantifying, analyzing, and describing the reasons for and nature of teacher interactions in this context. After discussing the framework, we provide evidence on the ways in which teacher groups conceive of, collect, and use student-learning data.
Framework
Figure 1 provides an overview of the two-dimensional framework we have developed to analyze and describe teachers’ engagement with student-learning data in collaborative inquiry contexts. Although we acknowledge the tension between individual teachers’ stances and that of the group (Shulman & Shulman, 2004), we have chosen to focus our framework and subsequent analysis on the collective stance exhibited by a teacher group as evidenced in their dialogic interactions, particularly in the context of analyzing student-learning data during collaborative inquiry meetings. While we provide an overview here, a more detailed characterization of our framework can be found elsewhere (Nelson, Slavit, & Deuel, in press).

Two dimensions of a teacher group’s collective stance toward student-learning data
Dimension 1: Epistemological Stance Toward Student-Learning Data
We describe an epistemological stance as a way of thinking and being in relation to a particular phenomenon. In this study, we are analyzing teacher interactions in the context of collaborative inquiry processes that incorporate student-learning data. Our framework for considering and discussing teachers’ epistemological stance toward data lies on a continuum from proving to improving and is based on the teacher group’s collective stance toward four essential features of collaborative inquiry: nature of learning goals, nature of student-learning data, student-learning focus of data analysis (DA), and implications on practice.
Part of Dimension 1 involves distinctions between student understanding and achievement. The literature is replete with definitions of understanding and achievement in both general (e.g., Gardner, 1985; Thorndike, 1932) and content-specific (e.g., Hiebert & Carpenter, 1992) contexts. We use the term understanding in consideration of student sense-making activity. It is both the process and result of internally representing and making sense of ideas, statements, and procedures. Achievement is the result of displaying knowledge and skills consistent with predetermined learning goals and benchmarks.
Improving stance
Teachers with an improving epistemological stance toward student data seek to surface limitations in classroom practice through an examination of the data. Using Wells’s (1999) notion of wondering, a teacher group with an improving stance displays a willingness to take risks and explore possibilities. An improving stance involves seeking out questions and evidence, not fixes or narrow solutions. By surfacing limitations, teachers with an improving stance generalize problems of practice to larger principles of teaching (Horn & Little, 2010). Such a perspective is similar to Cochran-Smith and Lytle’s (2001) description of knowledge-of-practice:
Through inquiry, teachers . . . make problematic their own knowledge and practice as well as the knowledge and practice of others and thus stand in a different relationship to knowledge.
For teachers with an improving stance, the production of knowledge is not the end. Data are used to problematize practice, and knowledge becomes dynamic, an ongoing negotiation of learning goals, student understandings, and implications on practice.
We distinguish two levels of an improving stance. First, a nuanced stance seeks to critically reflect on and reconsider practice through detailed analyses of student understandings of specific content. A nuanced stance opens widely the door to change and is consistent with Thompson and Zeuli’s (1999) notion of transformation, or “changes in deeply held beliefs, knowledge, and habits of practice” (p. 342). A learning-focused stance seeks to identify general levels of and trends in student understanding to make student-learning needs more explicit. The rethinking of beliefs and practice is not deeply considered, but targeted changes to instructional approaches and techniques are actively explored.
Proving stance
Teachers with a proving epistemological stance toward student data seek to identify and verify strengths and weaknesses in their practice by measuring changes in student achievement. When discussing student outcomes, teachers emphasize instructional delivery and are more concerned with measuring impacts of past practice than considering changes to future practice. When working with data, generalizations of student understanding are commonplace to measure student performance. This is often due to the kinds of data analysis that are enacted; teachers with a proving stance tend to express student understanding as “got it/don’t got it” or high/medium/low. Phrases that use broad descriptions and unidentified pronouns, such as “I think he’s finally getting it” and “my class has been struggling with that,” dominate the discussion. Such descriptions often lack an empirical base or other rationale. For teachers with a proving stance, data lead to results, not questions. Whenever evidence surfaces and becomes the object of teacher interactions, it is generally used to systematically move the inquiry work forward rather than as a stimulus for exploring, or even contemplating, underlying questions about teaching and learning.
We also distinguish two levels of a proving stance. A teaching-focused stance seeks results from student-learning data that uncover trends in student achievement. The purpose of working with data is to generate broad measures of student learning to determine the need for modifications of instructional approaches targeting specific content. When deemed necessary, an approach to change is restricted to existing teaching paradigms and often involves some degree of reteaching. A categorical stance seeks to use measures of student learning to verify the impact of practice as it relates to achievement-oriented, targeted learning goals. Only minor consideration, if any, is given to the potential utility of these findings to inform future practice, as the purpose is to show the degree to which past instruction was effective.
Dimension 2: Dialogic Stance Toward Student-Learning Data
The second dimension of our framework involves the ways in which teachers position and interact with each other in the context of collaborative uses of student-learning data. When collaboratively examining data, numerous opportunities arise to make joint meaning, individually accept or reject new ideas, or articulate one’s own ideas. Throughout this process, teachers continuously choose whether to activate, or fail to activate, their own beliefs and understandings. Teachers engaged collaboratively can also choose to develop a tone of wondering for the purpose of questioning, or a tone of certainty for the purpose of resolution. Dimension 2 involves a teacher group’s collective stance toward four essential features of dialogic interactions: connectedness of conversational turns, purpose of questions, nature of statements, and level of certainty in the talk.
We label a stance toward interactions in which cognitive conflicts are actively confronted and collaboratively addressed as negotiation. Our analysis suggests that there exist three levels of negotiation. First, inquiry-based talk involves a stance defined by uncertainty and meaning making. Conversational turns build continuously in a purposeful, reflective, and critical manner. Beliefs and knowledge are questioned in authentic ways, and the goal is to analyze and reflect on evidence to co-construct new understandings and questions that can further inform links between curriculum, teaching, and student learning. Second, exploratory talk maintains a level of uncertainty but to a lesser degree than inquiry-based talk. Conversational turns are connected and authentic questions still emerge, but these lead to more superficial examinations of beliefs and knowledge. Finally, connected talk is the weakest form of negotiation and involves a task-oriented stance toward data. Conversational turns may be related to each other but remain at the descriptive or procedural level and fail to challenge beliefs and knowledge.
At the other extreme, conversations may not exhibit any characteristics of negotiation. When participants do not build substantively on or from each other’s statements, we categorize the dialogue as disconnected. These conversational turns may be superficially connected through sharing related stories about teaching or students but do not substantively relate to or build on each other. We claim that if negotiation is not present in teacher interactions, particularly in the form of inquiry-based or exploratory talk, then attempts at collaborative inquiry are reduced to shared work. Furthermore, in collaborative PD contexts, we maintain that inquiry-based interactions and a nuanced stance toward student-learning data offer the most promise for transforming teacher beliefs, knowledge, and practice (Nelson, 2009; Slavit & Nelson, 2010; Slavit, Nelson, & Kennedy, 2010).
Method
This article targets the final year of a broader 5-year research project funded by the National Science Foundation (NSF) and using the above framework. The previous 4 years of data collection (DC) informed both the DC and analysis of the target year. In Year 1, our research focused generally on teachers’ professional learning and then narrowed over subsequent years to fine-grained analyses of teachers’ conceptions of, actions with, and uses of student-learning data. Numerous case and cross-case empirical studies have been previously disseminated from these analyses (cf. Nelson, 2009; Slavit, Kennedy, Lean, Nelson, & Deuel, 2011; Slavit & Nelson, 2010; Slavit, Nelson, & Kennedy, 2009).
Mixed methods are used in this study, with a modified explanatory design (Creswell & Clark, 2007). Qualitative data were initially collected to develop cases that used thematic analysis and thick description. Cross-case analysis led to the above framework using a grounded theoretical approach (Strauss & Corbin, 1990). Four particular types of inquiry activity (referred to as data phases and described below) were also developed from this analysis. The inquiry stance and data phase descriptions were then used as analytic coding tools to quantitatively characterize the teachers’ dialogic interactions and activity relative to student-learning data. Descriptors related to the two dimensions and four data phases were used to produce specific codes, and open coding of transcripts produced quantitative results related to the dimensions and phases. Qualitative data were then used to provide context for, explanation of, and elaboration on these findings. Therefore, quantitative findings yielded descriptive results to address the first three questions, and qualitative data provided contextual and explanatory information to further address each question.
Research Questions
Hypothesis 1: What percentage of time do teacher groups spend working with student-learning data?
Hypothesis 2: What percentage of time is devoted to each particular data phase when teachers interact with student-learning data?
Hypothesis 3: What, if any, is the relationship between the two dimensions of the inquiry stance framework? That is, does a teacher group’s stance toward the collection and use of student-learning data reciprocate with the ways in which they position and interact relative to these data?
Hypothesis 4: What factors influence teacher groups’ inquiry activity (data phases) and epistemological and dialogic stances toward student-learning data?
Participants and Setting
PD context
Five of the six groups had one or more teachers participate in the Partnership for Reform in Secondary Science and Mathematics (PRiSSM) PD program. PRiSSM was a 3-year project that targeted high-leverage instructional practices, leadership, and collaborative inquiry skills (see Slavit et al., 2010, for further details). Collaborations with building and district administrators worked to ensure that PRiSSM would be sustainable; 2 years after the project (the target year of this study), all schools continued to enact collaborative inquiry, and most used the models developed and supported by PRiSSM. The 1st year of PRiSSM was devoted to developing a cadre of teachers with leadership and collaborative inquiry skills. The next 2 years supported these teachers in organizing and leading their own teacher groups through collaborative inquiry cycles at their schools. Teachers received school-embedded meeting time to conduct collaborative inquiry meetings as well as annual summer institutes.
PRiSSM was designed to develop improving and negotiation stances toward student-learning data. PRiSSM provided a facilitator with experience and knowledge of the content and collaborative inquiry to support the lead teachers’ ability to facilitate the group, eventually transitioning the lead teacher into the formal facilitator role. Groups were also provided time and support for developing their own inquiry foci, which was a factor in increasing teacher buy-in for the project. The teachers’ selection of inquiry focus was not only informed by the collection of classroom-based data but also emerged from the teachers’ own felt need (Dewey, 1933).
Although variation exists, the cases in this study are representative of collaborative teacher groups who receive targeted PD, define their own inquiry foci, make significant use of student-learning data, and are led by a recognized teacher leader. Because we focus our study on the 2nd year post-PRiSSM, our teacher groups also represent cases of sustainability, decline, or generativity with respect to the PD focus on collaboration and formative use of student-learning data.
Teacher groups
Table 1 provides an overview of the six groups from schools located in the northwest United States. Two of the groups, Silver Valley Math and Science (SVMath and SVSci), were composed of middle school teachers in a small, rural district with one mathematics and one science teacher for each of Grades 5 to 8. During the target year, Silver Valley had 25% more students receiving free/reduced lunch than the state average. Scores on achievement tests were on par with state averages in mathematics but below par in science. Ongoing school improvement efforts over the past 15 years, as well as strong administrative support, resulted in the adoption of reform-based mathematics and science instructional materials, a variety of PD initiatives delivered by providers from inside and outside the district, and a dedication to collaborative inquiry. These opportunities arose despite the fact that the nearest major population center is more than 2 hours away. Both groups remained consistent with PRiSSM inquiry processes, particularly with regard to teacher choice of inquiry focus and emphasis on student-learning data. Because of the varied nature of grade levels taught, the groups selected inquiry foci related to broad student-learning goals, rather than specific content objectives.
Teacher and Group Characteristics
Note: CG6 = Cedar Grove 6; CG7 = Cedar Grove 7; GB = Grays Bay; SVMath = Silver Valley Math; SVSci = Silver Valley Science.
Cedar Grove 6 and 7 (CG6 and CG7) were both composed of four science teachers in a middle school located in an affluent area of a suburban school district. CG’s free/reduced lunch student population was approximately half the state average, and scores on standardized tests in mathematics and science were well above the state average. CG7 was seen as a model inquiry group by the district, and their work was highlighted by the school principal and presented to the school board. Both groups met weekly for approximately 50 minutes throughout the school year.
The Madrid group consisted of four high school mathematics teachers in a suburban school district with achievement scores and student demographics near state averages. The adopted curriculum integrated several mathematical topics into two courses. The group met weekly for 30 minutes and would often break into smaller groups related to the individual courses. Madrid’s inquiry focus and work was shaped by a district-level initiative that focused on identifying measurable, relatively short-term goals related to percentage changes in student achievement. When two recognized leaders of the mathematics staff (including the only PRiSSM participant) left for district positions, the principal imposed more control over the group’s inquiry process. Furthermore, because of rising failure rates that surfaced during the final half of the school year, the principal directed Madrid to abandon their chosen inquiry focus in favor of an all-school PD initiative targeting failing students.
Grays Bay (GB) consisted of 11 science teachers from two different middle schools in the same district as Madrid. GB met weekly as a large group, but also broke into smaller groups by grade level. Despite district accountability structures for collaborative teacher inquiry, GB remained consistent with PRiSSM inquiry norms, largely due to the teachers’ voiced desires and one principal’s belief in and recognition of the past success of her teachers. In effect, this principal was able to deflect the district’s inquiry requirements away from this group in a way that Madrid’s principal did not. We will discuss the implications of these administrative actions later.
Data
The primary data source was 88 audio recordings and transcripts of meetings from the six teacher groups (this constitutes more than 95% of all meetings). The number of meetings across the six groups ranged from 10 to 19, with an average of 743 lines of text per transcript. The median number of teachers per group was 4.5. Meetings typically lasted 40 to 45 min. Researchers were present at more than 50% of group meetings and collected a variety of qualitative data including field notes, taken as participant observers, and artifacts (e.g., student work, meeting agendas).
During the target year, the research team created case reports for each group. These reports focused on the teachers’ use of student-learning data and the nature of their dialogue. Yearly interviews with lead teachers, interviews with administrators and teachers groups, and emails and other informal conversations with participants occurred frequently and were used in the qualitative analysis. These data are used in this study to provide context to the quantitative findings. Complete methodological details of the larger qualitative analysis can be found in Nelson et al. (in press).
Data Analysis
Data phases
To fully characterize a teacher group’s collective stance toward student-learning data, we realized the importance of identifying the specific purposes that teachers have for data and the various ways in which data become important over time. We also realized an inquiry cycle is quite fluid and does not occur in discrete steps. Rather, it is a continuously adaptive process that cycles between particular kinds of activities.
Our analyses revealed four distinct, but related, phases in collaborative inquiry grounded in student-learning data. Descriptions of these phases were iteratively described to produce specific codes. Collaborative inquiry meetings involved none, some, or all of these phases, and teachers engaged in the phases in various ways. However, these phases collectively represent the totality of teacher activity related to the use of student-learning data in the collaborative inquiry settings that we analyzed. When we use the phrase working with student-learning data, we are referring to work consistent with one or more of these data phases.
First, data exploration (DE) involved activity and discussion related to potential topics and assessments for student learning. Talk inside the DE phase usually consisted of brainstorming or exploring the perceived future benefits of various data collection methods and tools. Second, data collection (DC) involved activity and discussion related to the creation, revision, and implementation of the data collection tools. This involved methodological discussion, but conversation that involved topics such as clarifying learning goals was also considered DC if it had direct bearing on the data collection methods or tools. Furthermore, discussion that clarified the logistics and criteria for analyzing student work was also classified as DC. Third, data analysis (DA) occurred when student data had previously been or was being analyzed, and teacher interpretations of the data were the focus of the interactions. DA typically involved teachers’ individual or collective attempts at making meaning of student-learning data by sharing scores, exploring specific ways that students think about content, or seeking trends. Fourth, data implications (DI) involved links from the teachers’ analyses of student data to learning goals and future approaches to classroom practice. This involved discussing potential applications of the data analysis in a content area consistent with the group’s inquiry focus.
Coding
Qualitative techniques led to a grounded framework for characterizing a teacher group’s stance toward student-learning data in the context of collaborative inquiry. This framework (Figure 1) and the above descriptions of data phases served as the primary tools for quantitatively coding transcripts and transcript segments.
Coding of transcripts from the 88 meetings was performed by a four-person team. Descriptive statistics were calculated on percentage of time working with data (i.e., data talk), percentage of time in a given phase per meeting, and percentage of time in a given phase in relation to overall time working with data. Consistent with iterative coding, one transcript was purposefully selected and coded across all four raters. Selection was based on the presence of multiple data phases as determined by one of the raters. Discrepancies were discussed, and code descriptors were refined. Four additional transcripts were then coded in a similar manner, and further refinements made.
Two levels of coding occurred for each transcript. First, entire meetings were given two codes that corresponded to a specific category of each dimension of the inquiry stance framework (Figure 1). For example, a given meeting might be coded as “learning-focused, connected talk.” Second, a more fine-grained coding was used to identify the specific data phases present throughout a meeting. To do this, individual transcript segments (of at least five lines) were initially coded as “data talk”; these segments were then coded as one of the four data phases, using codes consistent with the above descriptions.
Interrater reliability for each of the two dimensions (epistemological stance and dialogic stance) was measured across all four raters. Agreement across at least three of the raters constituted positive agreement. For example, in the case of epistemological stance toward data, a transcript scored as nuanced/learning-focused/nuanced/nuanced by the four raters constituted positive agreement, while nuanced/learning-focused/nuanced/learning-focused constituted disagreement. Given this standard, interrater reliability was 100% for epistemological stance and dialogic stance across all five transcripts.
Interrater reliability on data talk was measured by calculating line numbers of agreement on data talk and dividing it by the total number of lines in the transcript. For reliability on data phases, only the portions of transcripts commonly coded as data talk were used; hence, this measure was calculated by dividing the number of lines of agreement on data phase within portions agreed to be data talk by the total number of lines agreed to be data talk. Two measures of reliability for both data talk and data phase were then calculated using these procedures. First, agreement among all four raters was measured, with universal agreement serving as the standard. Second, agreement among all six rating pairs was calculated, and the overall pairwise agreement average was found. For data talk, pairwise agreement was at 82.4%, with 59.0% agreement across all four raters. For data phase, pairwise agreement was at 71.9%, with 78.2% agreement across all four raters.
To further ensure validity in the analysis, efforts were made to select a researcher with firsthand observational experience of group activity to perform the subsequent coding. When possible, field notes and meeting artifacts were provided to a coder to add contextual grounding to the transcript at hand. In addition, as the analysis ensued, eight meeting transcripts were purposely sampled for further reliability measures. Pairwise reliabilities on data talk and data phase were calculated at 80% and 81%, respectively, for these eight meetings. Finally, meeting transcripts were occasionally “spot checked” when an individual coder sought consultation with one or more other coders after encountering portions of a meeting particularly difficult to code.
Results
Question 1
Collectively, the teacher groups spent approximately half of their time engaged in data talk (Table 2). Two of the groups with the highest percentage of data talk, CG7 and Madrid, worked with data approximately two thirds of the time. However, as will be discussed later, CG7 and Madrid had quite different inquiry processes and outcomes, largely due to their different stances toward student-learning data. While the evidence suggests that teacher groups have different amounts of data talk for a variety of reasons, we will show that stance is much more important to the inquiry process than “time on task.”
Percentage of Data Talk and Data Phase Talk of Each Group During the Target Year
Note: CG6 = Cedar Grove 6; CG7 = Cedar Grove 7; GB = Grays Bay; SVMath = Silver Valley Math; SVSci = Silver Valley Science.
Teachers have few opportunities to talk with each other during a school day, and one motivation behind the proliferation of collaborative inquiry structures is to provide time to discuss practice and associated student outcomes (McLaughlin & Talbert, 2006). It is not surprising that talk within this time shifts across a variety of teaching- and school-related topics. Teachers addressed logistical aspects of group functioning, or had situational or personal needs to address. At times, groups followed tangents initiated by teacher comments on specific students, or told classroom stories unrelated to their inquiry focus.
Question 2
Overall, the DC and DA phases comprised 35% and 40% of the total data talk, respectively, and DE and DI just over 10% each (Table 3). Parsing the work of the teachers into these data phases reveals the nature of their engagement with student-learning data. The evidence shows that teacher groups can get drawn to a collection-analysis cycle without first exploring what data might be most informative to their understanding of student thinking, or to how their analysis might inform subsequent instructional actions.
Data Phase Talk as a Percentage of Data Talk Only
Note: CG6 = Cedar Grove 6; CG7 = Cedar Grove 7; GB = Grays Bay; SVMath = Silver Valley Math; SVSci = Silver Valley Science.
A closer analysis of the DE phase reveals the importance of the epistemological stance taken by a teacher group. For example, CG6’s relatively high amount of DE involved identifying test questions that they could also use to frame instruction as a means of raising test scores. In contrast, after involvement in a PD activity that explored multiple data sources, SVSci teachers desired to “look at the aquarium from different angles” and placed greater emphasis on the use of multiple measures of student learning, increasing their level of DE.
In some cases, the inquiry design dictated that certain data phases would need greater attention. GB devoted 3 years to the same inquiry focus. As such, they did not need to explore data sources or develop new instruments, as their work focused on more deeply understanding the way they scored student work. Contextual factors also affected a group’s data phase engagement. At Madrid, district administrators provided teachers with a specific framework for collecting and analyzing data that were based on the identification of specific, measurable goals and the use of pre/posttests. Their relative balance across the four data phases is likely due to the fact that they were told to teach at least 10 chapters during the academic year and report the results of each pre/posttest cycle to the principal. This contextual force also helps explain why Madrid had such a relatively high percentage of data talk.
Question 3
We hypothesized that the two dimensions of a teacher group’s stance toward student-learning data would be related. Table 4 suggests that, for the six groups in the target year of this study, a relationship did exist. More improving-oriented stances toward student-learning data reciprocated with more negotiation-based teacher interactions, whereas more proving-oriented stances reciprocated with lesser degrees of negotiation.
Overall Stances of Each Group During the Target Year
Note: GB = Grays Bay; SVSci = Silver Valley Science; CG7 = Cedar Grove 7; SVMath = Silver Valley Math; CG6 = Cedar Grove 6.
Yearly averages for both dimensions were based on the application of a 1 to 4 Likert-type scale to the codes of both dimensions, with 4 corresponding to the left-most code (i.e., 4 = nuanced and inquiry based; 1 = categorical and disconnected). After translating the dimensional code to the 1 to 4 scale for each meeting transcript, the means of both dimensions were subsequently calculated.
Averages were translated into categories using the following scale: 1-1.75 = categorical/disconnected, 1.75-2.5 = teaching-focused/connected, 2.5-3.25 = learning-focused/exploratory, 3.25-4 = nuanced/inquiry based.
These data empirically support the notion that “looking at student work” as a PD tool is most effective when teachers are actually looking at student thinking and are open to changes in practice. However, teachers who seek verification of the success of their past instructional practices through a general analysis of student achievement tend to have procedural, task-oriented discussions.
As we will show, CG7 and Madrid had two very different approaches to analyzing student-learning data and reflecting on potential changes to instruction. Table 4 shows that these groups also engaged in different kinds of data talk, which corresponded to our hypothesis. An abbreviated example from CG6, which typically undertook a categorical, connected stance, further illustrates the synergy between the two dimensions:
These were my low—4 or more wrong. I had 37 [on the pretest]. This time I had 18.
In all your classes?
Yeah. But I think that by looking at each one of these [quantitative results of teacher sets of assessments] individually, I do believe it shows us the same thing, it’s the same thing from pre to post.
But they have to learn to be able to analyze it and figure it out.
I think it’s way too much of a leap for them.
They just haven’t had their hands on it.
And the question wasn’t worded the same [from pre- to posttest]. I don’t think those 2 questions are bad questions. I think we just didn’t teach it well . . . I thought that was the whole point. That we all teach the same thing, and we retest.
This example is illustrative of teacher groups who take a proving stance and seek trends in student achievement data. The data talk consisted of authoritative and generalized statements about student learning that provided limited opportunities to question beliefs, or invite others to do so. The group’s collective stance toward student-learning data produced numeric results that afforded little opportunity to uncover and discuss specific student understandings, and the corresponding stance toward dialogue produced interactions that minimized opportunities to reflect on and potentially change instruction.
We also hypothesized that improving-oriented and negotiation-based stances would yield higher amounts of data talk and implication phases. Neither of these hypotheses is supported by these data. The case of Madrid, which was provided with a specific framework and tools for collecting and analyzing data, illustrates how a teaching-focused, connected stance can still generate a good deal of data talk; alternately, GB demonstrated a nuanced, exploratory stance but had the same amount of data talk as Madrid. We continue to analyze why more improving-oriented, negotiation-based stances do not lead to more dialogue about DI.
Question 4
We now more specifically address the contextual factors that led teacher groups to use specific kinds of student-learning data, undertake a particular data phase, and enact stances toward student-learning data.
Administrative context
Principals were invited to participate in selected PRiSSM PD activities, but most delegated involvement to an assistant. They remained aware of project goals and activity through site-based research showcases and regular project communiqués. Most principals were also frequent attendants at teacher group meetings.
PRiSSM espoused inquiry- and negotiation-based approaches to collaborative teacher inquiry. However, initiatives grounded in skills-based, prescribed inquiry methods with short-term, measurable goals emerged in some of the districts of this study. The ways in which administrators handled these contrasting approaches was pivotal for both the processes and outcomes of the teacher work. Specifically, the principal was the main influence on mediating the impact of district accountability measures related to collaborative inquiry, which were sometimes in opposition to PRiSSM philosophically and practically. We contrast the cases of Madrid and CG7 to show how principals shaped a group’s inquiry process and stance toward student-learning data.
Madrid’s principal reinforced the group’s proving stance. In line with district initiatives, he provided the group with a meeting form that predetermined the inquiry process by requiring the identification of a target standard and employing short cycles of pre/posttest data focused on measuring discrete mathematical skills. Despite sections that asked teachers to analyze students’ strengths and weaknesses, and then link instruction to achievement, the short cycles and pressures to prove student gains shaped Madrid’s task-oriented approach, which maintained focus and led to a high percentage of data talk. However, when the teachers explored reasons for or implications of these results, they focused on generalizations of student understanding and remediation. The following illustrates typical DA discussion by Madrid:
So far we have 81% mastery. 81% on . . .
Oh, on this one, okay.
Yeah, which was the only one we’re really focusing on.
Correct.
81% on skill one.
Yeah.
Okay. So that’s, by and large, that should be history then? Overall, right?
Yeah.
At least as far as collecting data.
That was our goal.
Madrid’s goal, across multiple cycles, was to have 80% of their students master the skills-based learning target, and their conversation stayed focused on analysis at this level. While individual teachers sometimes expressed wonder about why students were not achieving mastery of a skill, they almost never pursued this line of inquiry.
In contrast, CG7 viewed their principal as a support for their current inquiry approach, and did not feel the need to adapt to district initiatives similar to those in Madrid. As a result, rather than quickly developing and implementing data collection tools, CG7 spent considerable time clarifying student-learning goals and negotiating details of their inquiry focus. As the year progressed, these discussions were often grounded in student-learning data collected previously. This allowed the group to subsequently refine tools, or develop new tools, that could target the specific facets of student understanding they wished to consider. For example, during a segment in which CG7 exhibited nuanced and inquiry-based stances, Logan proposed to measure student understanding with a single question, “Why do plants and animals need each other to survive? Explain.” Discussion (DC phase) ensued regarding the potential implications of this approach on the group’s ability to uncover specific aspects of student understanding:
They could get this right by saying plants give us O2, and we give them CO2.
That’s what the lower kids would say, yeah.
That could be right. The higher kid might remember the formulas and put that into their answer. But I think it’s okay for us to still have that, because that’s really the only photosynthesis related question.
So that would be a synthesizing question, accessing prior knowledge from the 6th grade combined with 5th, then you come up with the conclusion.
The really low kid is going to be like, “Because animals eat plants and that’s where we get our food from.”
That is partly true.
It is, all food starts with sun.
So, there’s a few answers, and I think those are sometimes the best questions, there’s more than one answer.
Prior fine-grained analyses of student-learning data positioned CG7 to develop an approach to assessment that targeted important scientific concepts that could lead to rethinking instructional practice, and their high level of negotiation supported this approach. Rhett later remarked that, through this assessment, “you actually find out where the problem (of student understanding) was.” CG7 teachers then analyzed their own students’ assessments to determine specific aspects of understanding and, 1 week later, met to discuss potential changes in their instructional approach to meet the needs of the collective variety of learners in their classrooms. CG7 then established “intervention groups” based on this analysis that targeted instruction on four different aspects of the main scientific concept. Logan stated the following in a follow-up email to this meeting:
Today we spent most of our time dividing students up into intervention groups. Monday we will be reteaching key digestion concepts based upon generally poor exam results for all of us. We’ll probably be looking at the exit slips from our interventions next week.
This episode illustrates CG7’s perspective that inquiry is an ongoing, iterative process, exemplified by their improving stance toward data as a tool for surfacing student understanding and rethinking instructional approaches. Had the principal at CG7 forced the district inquiry perspective and processes on to CG7, it is unlikely this would have occurred. However, the stances of the CG7 teachers supported their own ability to help create this collaborative inquiry environment.
Brandon, the Silver Valley principal, participated in his own collaborative inquiry with fellow district principals, an experience that led him to appreciate the advantages of allowing teacher groups to develop and follow their own lines of inquiry (Slavit et al., 2011):
Our wrestling with the tough questions and issues gave us new insights into the difficulties of the process as it faces teachers.
When SV administrators provided all teacher groups with a specific, district-determined inquiry focus, the SV teachers quickly pushed back. Brandon reflected on his own experiences and decided to allow the teachers to continue in their chosen direction and with methods of their own choosing.
Although tangible actions of the principal influenced the scope of teacher group activity, and therefore the stance from which they could think about their work, further analysis revealed that the teacher group’s perception of the principal’s intentions was nearly as important as the actual actions of the principal and that the lead teacher was pivotal in the construction of this perception. For example, CG’s principal discussed a district initiative, similar to that in Madrid, which promoted the use of quantitative measures on precise, skills-based learning targets. Kevin, the lead teacher at CG6, perceived this to be a directive and led his group toward this method of inquiry. However, Logan in CG7 emphasized the district initiative as only a minor focus of the group’s work, which led to the results above. At a meeting in the middle of the school year, Logan reflected on this perspective:
We should probably spend some time next time doing our SMART goals [the district-recommended inquiry model]. We’re kind of drifting from our original focus a little bit from science journals. I think we have a SMART goal related to these intervention days and a SMART goal related to science journals that represent most of the stuff we’re doing.
We see that CG7 gave SMART goals secondary attention, although the group did not wish to completely reject this approach. Rather, Logan and the group designed their SMART goals to be consistent with those they had developed previously, and then used their own methods to address them. This allowed the group to abandon the district model, but still respond to the district’s accountability measures, which was amenable to their principal. In fact, she regularly asked questions of the group that encouraged further analysis of student thinking, such as
What about this one where you ask “Explain why plants and animals, humans, need each other.” She [a student whose response is being read] goes into “Animals need plants because plants release oxygen, and butterflies etc., pollinate plants.” Nothing else. If they don’t go into why they need each other, does that concern you?
While the district inquiry model focused teachers’ collaborative work on quantitative measures, this principal not only supported but also led CG7 in more nuanced analyses of student thinking. CG6 either did not benefit from such questions, or failed to recognize the principal’s flexible view of adherence to district inquiry policies.
Content knowledge and inquiry skills
The evidence suggests a relationship between the depth of teachers’ content knowledge and the tendency to adopt inquiry-based and negotiation-oriented stances. In CG6 and SVSci, half the teachers did not have a content degree in the subject they were teaching. Transcript analysis revealed few instances in which CG6 elicited particular aspects of student understanding in their collective examination of student responses, a finding confirmed by their overall teaching-focused stance throughout the year (Table 4). SVSci made attempts to question and rethink their instructional practice and did display an overall learning-focused stance. However, these attempts were limited due to an inability to fully explore and discuss their content goals or deeply examine their students’ work. In contrast, all the teachers in CG7 and all but one of the 11 GB teachers had science degrees; these two groups enacted some of the strongest improving and negotiating stances (Table 4).
PRiSSM did not focus on the development of teacher content knowledge but instead focused on facilitating collaborative norms and meaningful interactions, with secondary attention to specific inquiry processes. Collaboratively looking at student work for facets of understanding was the dominant inquiry method presented. This encouraged groups to choose student work as their main data source, although some groups did make use of surveys, interviews, and large-scale data analysis. However, some groups failed to engage in what Ikemoto and Marsh (2007) call “complex” uses of student-learning data, which involve the use of extended time frames, multiple sources and types of data, and disaggregation. CG7 and GB were the only groups to regularly collect and analyze disaggregated or qualitative data that elicited student understandings. Other groups focused the majority of their data collection on aggregated, pre/post student-learning measures. Our results agree with the conclusion that teachers “often lack adequate capacity to formulate questions, select indicators, interpret results, and develop solutions” (Ikemoto & Marsh, 2007, p. 121), and it appears PRiSSM did not support teacher growth in this area in several cases.
Our evidence suggests that a teacher group’s overall stance toward student-learning data affects, and is affected by, the group’s inquiry capacity. Specifically, stances consistent with improving and negotiation, such as in GB and CG7, tended to correspond with more fine-grained uses of student-learning data, more expansive teacher content knowledge, and an increased capacity to develop effective inquiry methods.
Group composition and leadership
Shulman and Shulman (2004) discuss the rather problematic choice of taking a dichotomous view of individual and group stances toward inquiry. While our framework uses the teacher group as the unit of analysis, we also agree that an analysis of the stance of individual teachers can support an understanding of the overall group dynamic. Our knowledge of the teacher leaders in each group allows us to incorporate this level of analysis into our discussion.
With the exception of Frank in Madrid, each recognized teacher leader had some involvement in the PRiSSM project and additional inquiry support at the school and/or district level. As discussed, Frank’s somewhat forced capitulation to district mandates blended with his limited experience with complex uses of data to significantly affect the overall stance of Madrid. Kevin, the leader of CG6, had an ability and strong desire to partake in complex inquiry, but three members of his group were neither ready nor willing to do so, which significantly curtailed the group’s inquiry stance. Serena and Amanda, the co-leaders of SVSci, were in a similar situation but were further hampered by the limited content understanding among group members. In CG7, Logan and Kyle had a strong desire to understand learners and an ability to develop assessment items that surfaced student ideas; as importantly, the other two members in the group saw potential benefit and were willing to adopt this approach to inquiry. Leon, GB’s group leader, perhaps possessed the most inquiry-based perspective of any of the teachers we studied. Table 4 suggests that Leon’s stance translated to the group as a whole, even though case study analysis indicates that several GB teachers were resistant to this approach (Nelson, 2009).
Group composition and leadership also influenced a group’s need to spend time in the DE phase. The limited content knowledge and multigrade composition inherent in SVSci led them to focus their inquiry on the general notion of student engagement. Exploratory conversations that sought a working definition of engagement produced a great deal of DE. Toward the end of the year, SVSci realized limitations in this focus and began to explore alternatives, which led to additional time spent in the DE phase.
The interplay between the stances of individual teachers and the overall group is quite complex, but the evidence suggests that the group leader is an important piece in this tapestry. Our evidence suggests that the leader’s stance, the stances of other “highly participating” teachers, the group’s perception of the principal’s particular desires or mandates, the group’s collective vision of student-learning data, and what the group decides as the nature of student understanding all interplay to affect the overall group stance in significant ways. Specifically, a critical mass of teachers who are ready and able to engage in critical conversations, challenge beliefs, and analyze facets of student understanding is necessary for teacher groups to enact nuanced and inquiry-based stances.
Limitations
Our study synthesizes a variety of data across multiple sites over a long period of time. Our modified exploratory research design sought to minimize the inevitable limitations in a study of this kind. Our reliance on the “counting” of discourse units risked oversimplifying the ways in which humans connect or do not connect through the resources of language. The dependence on conversations during teacher group meetings as nearly the sole source of discourse data narrows the contextual scope of the investigation, ignoring things such as “hallway conversations” and other teacher interactions potentially significant to the development and enactment of a group stance toward student-learning data.
To simplify our analytic methods, our discussion of “percentage of time” actually related to lines of transcript, not actual time. Group pauses and various transcribing factors would likely lead to differences in percentages calculated using time as the unit of analysis, but these were deemed acceptable, particularly given our assumption that transcript lines might actually be a more representative measure of the amount of teacher talk. We also chose to at times foreground quantitative measures of teacher talk and shift descriptive data to a supplementary, explanatory role. While this hides an enormous amount of contextualized information, our mixed methods design allowed us to adequately address the given research questions.
Implications
Unless professional dialogue is tightly focused on the collective examination of student work from an inquiry stance, opportunities for teacher growth can arise in a haphazard manner. This study reveals the importance of a teacher group’s collective stance toward student-learning data on the processes and outcomes of collaborative teacher inquiry. Our quantitative results provide specific information on the amount and nature of teacher talk in these settings. Through more fine-grained analyses, we also attempted to uncover specific reasons for these results and broaden the potential implications of our work.
As a growing number of schools are systematically enacting collaborative approaches to PD grounded in student-learning data, the need for frameworks to support and understand this work is increasing. Our two-dimensional framework is useful to researchers seeking to code and analyze collaborative activity related to student-learning data. More importantly, the framework allows for understanding how teachers conceive of these data, why teachers select particular kinds of data, and the overall stance that defines the underlying reasons why they make particular uses of data. The framework can also be used by PD providers wishing to monitor and support this kind of teacher work. Such frameworks are needed to support large-scale studies that identify precisely why, and not just how, teachers think about and work with student-learning data, providing the means to support particular stances and subsequent approaches to working with these data.
The teacher groups in our study spent the vast majority of their time in the DC and DA phases; they tended to dive into these two phases of the data process and stay there. Preparing for the data collection process by exploring possible data sources is minimized, and there is not much reflection on what the data mean with regard to classroom practice. Teachers need to be encouraged to appreciate the need for DE to more carefully reflect on the potential paths and implications of their work (Kazemi & Franke, 2004; McLaughlin & Talbert, 2006).
Stance is a much greater influence on collaborative teacher inquiry than time on task. The six groups collectively spent approximately half their time engaged in data talk, but the results show that the overall stance taken toward student-learning data framed the processes and outcomes of this work. Furthermore, the empirical relationship established between the dimensions suggests that, if we are to provide opportunities for teachers to problematize and rethink practice through examinations of student-learning data, then “looking at student work” must be reframed as “looking at student thinking.” Because the vast majority of data talk involved the DC and DA phases, teacher groups had plenty of opportunity to explore student thinking. However, as Little, Gearhart, Curry, and Kafka (2003) state, “Putting student work on the table (does) not ensure whether or how it (will) be taken up in conversation” (p. 190). The stance taken during these crucial interactions determines the degree to which student work and student thinking are highlighted, and the opportunities for teachers to engage in genuine negotiation of existing or novel approaches to instruction.
Specifically, there is a need to support the development of more nuanced stances toward student-learning data and more inquiry-based stances toward dialogic interactions among teacher groups; such stances are not common, nor are they inherent. All of the groups in this study received specific support over many years for enacting collaborative inquiry, but some of the groups still had difficulty staying focused on the data process and engaging in data talk. More importantly, some groups could not use student-learning data to generate or maintain conversations that challenged prevailing beliefs and instructional norms. Specific supports that target both dimensions may be necessary for PD of this kind to be transformational and not just additive or verifying (Thompson & Zeuli, 1999). Supports could target broader and more focused administrator awareness of the importance of group stance; teacher observations of others’ classroom practice; questioning norms of a wondering, clarifying, or challenging nature; active listening; and the analysis of student-learning data for detailed levels of content understanding. Our evidence suggests that these are important focal points for PD designers to consider when emphasizing collaborative investigations of student-learning data.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based on work supported by the National Science Foundation under Grant No. 0554579.
