Debugging inequities: Data use,“gumshoe work,” and problem identification in district-wide computer science education initiatives

Capacity: surfacing teacher capacity factors in introductory CS courses

Chicago Public Schools is one of the largest school districts in the United States, and has been engaged in a comprehensive, district-wide CSed initiative since 2013. Notable among the work the district has done to advance CSed was the establishment of a CS graduation requirement for high school students in 2016 (Dettori et al., 2018). The policy change was a substantial step toward reducing inequities in CSed. This policy created a mandate for not only offering, but promoting full participation in, CSed experiences. However, the policy also contained risks around potentially reproducing broader inequities: if CSed implementation was not well-supported, it could result in students not graduating from high school.

In the second year of the graduation policy’s implementation, the team involved in both implementation and research began to broadly hear district stakeholder concerns related to these risks. Specifically, there was a sense that the introductory course (Exploring Computer Science: Goode et al., 2014) that was in widespread implementation in CPS high schools was too difficult and that there was a high course failure rate. If this concern was well-founded, it would present a substantial risk for both students experiencing course failure as well as for the sustainability of the broader CPS CSed initiative itself.

In response, the CPS Office of Computer Science (OCS) collaborated with CAFÉCS, a longstanding research-practice partnership supporting the CPS CSed initiative (Henrick et al., 2019) to understand failure rates within their ECS courses. To support the analysis, leaders in OCS and CAFÉCS took advantage of existing data that had been collected since the introduction of the ECS course, as well as collaborative structures they had in place to engage in discussions of data in relation to the initiative’s strategic goals. Steven McGee, a leader of the CAFÉCS partnership, described the process this way:

“These meetings are two to three dozen people that get together. [...] The idea there is we want to create an atmosphere where we are doing this shared visioning, shared interpretation of data, getting the whole community input so that we’re guided by everyone’s voice. So it was at one of these monthly meetings that we then said, hey, there’s this issue and we brainstormed. Everyone had an opportunity to say ‘well, I think these are the factors that I think affect the failure rate.’ GPA, attendance, the quality of the teaching, there was some hypotheses about, well, maybe it’s higher in the second semester because that’s where there’s more collaborative work, that’s where attendance really would become an issue for people, well there are school level factors, teacher level factors on content knowledge.”

This described process of hypothesis generation around possible factors affecting failure rates was significant—in response to the larger concern that was raised about failure rates, the group simply could have gone to their historical data to answer the question of whether or not there was, in fact, a problem with failure rates. Doing so would mean using available data to simply monitor the degree to which an equity issue around course failure was present. Instead, they engaged in a process of both answering the initial question of the degree of course failure but also used additional data they had available to them diagnostically—asking what might be “underneath” cases where course failure did occur in terms of potential root causes (e.g. “GPA,” “attendance,” “quality of teaching,” noted in the excerpt above). Essentially, engaging in a process of diagnostic problem framing vis-a-vis course failure rates.

The analysis proved to have important impacts when it came to how the initiative approached questions of systems change, in particular when it came to issues of teacher capacity (e.g. teacher content knowledge). McGee shared that the results, first and foremost, showed that the initial concern around failure rates did not exist to the degree that those who raised them may have imagined.

“The failure rate was about 11%, which as near as we could tell, was not any different than any other course [in the district]… And in fact the consortium [University of Chicago Consortium on School Research], who actually has access to all subject areas, found that the grades in computer science were actually higher on average than grades on other courses.”

But just because the degree of course failure in reality did not match the stakeholder concerns did not mean there was no room for improvement. The deeper exploration of factors that correlated with course failure offered important insights around potential root causes of the problem. Outlined in a technical report by McGee et al. (2018), the results pointed to two critical factors that could be focused on to reduce failure rates: (1) Whether teachers had participated in ECS professional development and (2) the number of times a teacher had taught the course. Each was found to be statistically significant factors correlated with students passing the course.

These results, in turn, were then able to be mobilized to encourage administrators to make evidence-based decisions on both supporting teacher participation in professional development as well as in assigning teachers to the course. As Andy Rasmussen, an administrator within Chicago’s OCS, shared:

“We were able to see from the research that the failure rate for students in classes taught by somebody who had actually been through the professional development was nearly half that of teachers who had not been through that. So that’s been validating in one sense, but it also gives us a really good talking point when speaking to principals and others about the importance of starting and completing the professional development sequence and taking advantage of what we have to offer.”

McGee shared a similar view on how the team was able to mobilize findings around teacher experience in teaching ECS:

“That was then data that the [OCS] implementation specialists or integration specialists could take back to the principals to say, ‘Hey, if you’re worried about failure rate, please make sure you assign the same teacher, don’t keep changing teachers every year.’”

If the group had simply monitored course failure data, rather than taking a stance of diagnostic problem framing around what factors might be behind instances of course failure, they may have missed an important opportunity to reduce inequities. The group did report that in the year after the report’s results were shared with administrators, they saw increased attendance in ECS professional development, a result suggesting that the findings prompted evidence-based decision-making on the part of principals with regard to engaging with teacher professional development opportunities.

Access: reaching into district systems to understand access issues for English Language Learners in secondary CS

A second case shared by the Chicago CS team concerned lack of access to secondary CS courses by students who were English Language Learners (ELLs). As Andy Rasmussen of the CPS Office of Computer Science (OCS) shared:

“I’d been looking at intersectional slices of who has not had access to a Computer Science course by the time that they’re in their senior year. So there were a small, small percentage of students that hadn’t been offered a class, and we’re trying to figure out why. And looking at every slice, so like gender and race, or diverse learner status and race, looking at all these sorts of demographics. One thing that really stuck out is that if you’re an English Language Learner you had a much higher chance of not having been offered the course. And this isn’t something that had popped out to us before.”

The group had surfaced an equity issue around access for this particular subgroup, but did not have a clear sense of what was causing the problem. As in the previous case, the graduation requirement that CPS had put in place around secondary CS meant that all students should have, in theory, been offered the opportunity to take a CS course in order to fulfill the requirement.

In contrast to the case involving course failure rates, the group did not have a clear set of hypotheses around what might explain this discrepancy. To better understand the problem, the group reached out to a representative from CPS’s Office of Language and Cultural Education (OLCE), which focused on English Language Learners within the district. This representative was invited to one of OCS’s aforementioned “consultancy” meetings where stakeholders regularly explored data and how it might be used to improve goals. As another OCS administrator shared:

“Her presence was invaluable, because she was able to articulate why schools have problems scheduling English language learner students into computer science. I didn’t realize that there were these tiers of English Language Learners, and depending on what tier they fell in, that determined their English proficiency. And some students are never elevated to the level of proficiency where they’re able to [participate]... They have to continue... I think it’s if you’re level one or level two, then you have to continue to take all of these English classes until you get to a level of proficiency where you can partake in other things.”

In consultation with the representative from OLCE, the computer science office came to understand a set of district policies they had, up to that point, been unaware of: ELL students had particular proficiency requirements that resulted in certain courses not being available to them. They determined that this policy likely explained the under-enrollment trend in ELL students that they had found in their secondary CS course data. Andy Rasmussen, an OCS administrator, described the importance of gathering information and diagnostic problem framing around this issue:

“She was just this fount of information about all that schools and students have to think about when they are a foreign transfer student, or any other thing where their English level is low, and so they’re treated very differently by the school. So we just hadn’t had that insight very much. We kind of knew from conversation, but hadn’t had the insight there.”

At the time of writing, OCS was considering how to respond, exploring the possibility of putting in place a waiver for certain students when it comes to the CS graduation requirement.

In reflecting on this case and others like it, Rasmussen noted that “there’s a fair amount of data work that’s sort of gumshoe work.”, pointing to the reality of needing to track down both forms of data or people with knowledge of the inner workings of particular aspects of the instructional system that might be impacting data:

“There’s so many things we didn’t know to ask the question. [...] It’s often one person in the district that manages that thing. And you have to be able to know that the person exists, know the right questions to ask and how things relate. So, there’s a fair amount of the data work that’s sort of gumshoe work, and figuring out what are the things we need to be thinking about, who are the students that are hidden? What are the experiences that they have that are hidden? And learning to track some of that down.”

In contrast to the first case shared around CPS’s data work, this one did not involve either direct generation of hypotheses to explain trends they were seeing in their broader monitoring data, or formal processes of “asking questions of the data.” Instead, the diagnostic problem-framing process took the form of consultations with other district actors who understood elements of the system that they were unaware of. As stated by Rasmussen, this “gumshoe work” allowed them to figure out how to address inequities they were observing in the data.

Participation: Explaining trends in “undermatching” within secondary CS courses

In Yorkville school district, two stakeholders monitored and analyzed the enrollment data of AP Computer Science A over multiple years across student subgroups: a high school CS teacher, Rosalyn, and the math department supervisor who managed the district’s secondary CS work, Nicole. When examining the percentage of female enrollment in AP CS-A, they noticed the highest it ever reached was a one-to-two ratio: for every female enrolled in AP CS-A there were two males enrolled. To make salient how problematic these enrollment numbers were for CS specifically, they compared them to the district’s female enrollment in AP Calculus which held a one-to-one ratio. Rosalyn explained her use of AP Calculus rather than AP Physics as a comparison point:

“Calc we’re fifty-fifty, that’s not the problem, but that’s why it’s a really good standard. It doesn’t make sense to compare it with physics. In other words, the people missing in AP Physics are pretty much the same people missing in AP CS-A. It’s females and minorities.”

Rosalyn’s point here echoes Barbara Ericson’s (2020) research that shows AP CS-A has the second lowest female enrollment rates among all AP courses, second only to AP Physics. Instead, AP Calculus has a relatively balanced percentage at approximately 50% female.

Keeping track of this data was not straightforward, however, especially as more teachers were added and involved. As Rosalyn describes,

“I like to keep track of the percent of minority and then the percent of female, and then the percent of free and reduced lunch across all courses. And again, it was easier when I was the only teacher. Now that there’s so many other teachers involved, it needs to be done through a Google Form, all teachers need to administer that form. It’s grown.”

Procuring a comprehensive picture of their enrollment numbers across all classes helped them identify the occurrence, magnitude, and trends in disparities over time. However, this process did not, in of itself, help Rosalyn or Nicole to sort out the underlying variables that might be acting as root causes for the disparity in the first place. This needed to be accomplished through diagnostic problem framing.

Rosalyn began to identify a possible root cause, in her opinion, as she witnessed another phenomenon happening in her AP Computer Science Principles class. There were more female students who she felt were “undermatched,” and should have been enrolled in AP CS-A based on Rosalyn’s perception of their proficiency. Rosalyn had a hunch that guidance counselor recommendations might explain the undermatching participation trends she was seeing, and thus the low enrollment numbers for AP CS-A. She described that this hunch may have emerged from a combination of factors. First, Rosalyn had numerous experiences with faculty in her district who voiced stereotypical views about the kind of students computer science was well-suited for (i.e. male technology enthusiasts). Second, she attended national training sessions that articulated the worldwide pervasiveness of these stereotypes and how they worked to keep women out of Computer Science. During one such training she reflected on her own high school experiences as a student talking with guidance counselors, and realized that “I am one of those statistics.”

Acting on this hunch, Rosalyn spoke with the “misplaced” female students who were not in AP CS-A to figure out who their guidance counselors were, which led her to notice a pattern as to which counselors were undermatching female students. She then spoke with those specific guidance counselors to both advocate for female students as well as to figure out if there were other female students who might be undermatched. As Rosalyn described this informal data practice:

“Some of that data is word of mouth. So I’ll have a student that’s in my class, Abby, she was in CS Principles, and she should have been in [AP CS] A based on her abilities, and I’ll ask her who was her counselor, and she’ll tell me. And then two periods later, it’s September, we’re still trying to work out the kinks. And again, I’ve got Sally, who’s in the class, and she should have been in CS-A as well and it’s the same counselor, so now that’s data. And now I’m gonna go talk to that counselor and explain, like I said, adult to adult, ‘This is what I’m seeing. Are you aware of this and what happened here?’ So that it doesn’t happen in the future. ‘And do you think it might have happened to anybody else? Anybody that wasn’t in my class?’ So that’s an example of data that’s sort of informal.”

Rosalyn’s diagnostic problem framing was informed by the combination of various sources of data: annual formal comparisons of enrollment numbers between AP Calculus and AP CS-A, Rosalyn’s own formative understandings of students’ abilities and whether they are misplaced or not, and beginning-of-the-year informal word-of-mouth data from students and their counselors. Rosalyn attributed the issue of low female enrollment in AP CS-A not to the students themselves (as if females were reluctant to push themselves), but to the recommendations of guidance counselors. She suspected that the problem may be related to counselors being unfamiliar with or even afraid of computer science as well as unconscious stereotypical views they may have held about who CS is for. This diagnostic problem framing set her up to enact solutions that focused on inviting counselors to see themselves as part of the solution to increasing female enrollment in computer science. To do this, Rosalyn had a series of conversations with counselors to familiarize them with Computer Science, debunk the myths of who Computer Science is for, and to back up her claims about the severity of the problem by directly sharing the low enrollment data for females in Computer Science.

Experience: unpacking student disengagement in CS across grades 3–6

Beverly served as Rosewood Public Schools’ data, assessment and technology facilitator since 2018. In this role, she collected student data across a variety of content areas and presented these back to the administration to inform improvement efforts. In other subject areas, she had well-established methods of tracking growth over time. However, in CS, she lacked any formalized benchmarks such as state-based, standards aligned tests. In lieu of these, Beverly decided to focus on student engagement in CS. To measure engagement, she went directly into CS classrooms with a stopwatch and a notebook and collected what she described as “[Beverly’s] mucky data.” While she considered these data to be “informal” and “anecdotal,” she was in fact fairly systematic in her approach, both iterating on how she considered accounting for disengagement and collecting data longitudinally and across grade levels.

Her approach to “mucky data” evolved over time. First, she recorded “tallies of how many times did I see kids not engage.” Then, she used her phone’s stopwatch feature to document, “here’s the longest that everybody was engaged. And then I restart the timer. Okay, here’s the next longest, here’s the next longest.” As her classroom observations progressed over the school year, Beverly noticed, for example at the sixth grade level, that when the school year started students were “scared to death of the teacher. So they were quiet…but then as time went on, and they built rapport and they got to know her…they loosened up a bit.” As engagement seemed to increase at the aggregate level, she modified her approach to her mucky data again, this time attending to individual students.

When this method revealed to her that some individual students were not engaging for extended periods of time (e.g. 2–3 min) she would talk with the students’ teacher about how they might more effectively engage them. However, these conversations about individual students were not always fruitful as teachers tended to attribute the problem of disengagement back to the students themselves. Beverly would try to gather additional data about students’ engagement:

“How do you know that students are grasping this? And then I’d name a specific student. So then saying, depending on whatever the answer was, which was usually, ‘I don't know.’ ... Or ‘That student just isn’t getting it.’ To then being able to say, ‘Okay. What are some ways that this instruction can change so that they do get it or they are able to engage more and they’re not sitting back and just staring at their computer?'”

Our data suggest that Beverly had a hunch about one factor affecting student engagement: the lack of immediate assistance when a student became stuck during a CS activity. Beverly guessed that this was specifically true during whole group instruction, which has tendencies to lose student engagement when struggling students were “stuck” and needed to wait for teacher assistance. Due to her work as data, assessment, and technology facilitator across multiple content areas, she had a unique vantage point that made her familiar with what she saw as the shortcomings of whole group instruction:

“So in the past, we did everything whole group. We’re getting away from that more because we have kids that are at all different levels and if we do solely and just whole-group instruction and then set them loose, we know that our strugglers are not learning. And so that’s true whether it’s computer science, or whether it’s math, or whether it’s reading.”

She also appreciated that placing students in smaller groups was not enough. For example, if “the strugglers” were all placed together, students did not have the opportunity for “more differentiated” groups that could influence their growth. This assumption was reflected in the solutions to student disengagement that she devised with the computer science teachers. At the third-fifth grade level she worked to create differentiated small groups. At the sixth grade level, the teacher wanted to continue with whole group instruction, but was willing to bring in a teacher aide student from her upper-level programming class. These moves suggest that Beverly suspected that student disengagement was caused not by student disinterest in computer science, but by a specific feature of whole group instruction—students becoming stuck and having to wait at their computer until a teacher was free to come over and help them move forward again.

Based on her measurements around time on task as a proxy for engagement, Beverly’s actions suggest that she saw prolonged periods of disengagement as resulting from lack of individualized support within whole group instruction, which then drove her actions to bring in more individualized supports for students by reconfiguring the third-to-fifth grade classrooms into small differentiated groups where peers could support each other, and by bringing in a student teacher aide in the sixth grade classroom. Beverly described the resultant change in engagement for the sixth grade classroom this way:

“She has been able to strategically pick a teacher aide student who is also in her upper level programming class to be able to be a teacher aide for her, say, an eighth grader who can work with a sixth grader and be able to... The two of them, after she gives a short 10-minute lesson on whatever the skill is for the day, and they can go around the room and work individually with students, or work with pairs and the students. And that’s been interesting to watch, because the level of engagement has amazed me in that room. Because kids, they don’t sit wondering what they should be doing, they have a pretty clear picture on what they should be doing and if they have a question, it gets answered pretty quickly. So they can move on.”

Yet, midway through the year Beverly discovered that whole group instruction was not the only problem interfering with student engagement. As part of her work, Beverly also offered computer science professional development to science teachers in her district. After 5 days of training, these science teachers were able to offer computer science in their science rooms. At the fifth grade level, students had computer science opportunities both in their science classroom and in a 30-min specialist class with a CS instructional specialist. Unbeknownst to Beverly, however, was that the science teachers felt like they had to teach the same lessons that the CS instructional specialist was teaching, and unbeknownst to both Beverly and the science teachers was that a handful of students were attending the CS specialist course right before attending the science classroom for a CS lesson—the rest of the students were going to other specialist courses such as P.E. or the library. The result of this was that some but not all fifth graders were immediately repeating the same CS lessons. As Beverly describes it, “so the kids that already got it felt like they were being punished.” In response to this situation, Beverly started gathering a new form of data—fifth graders’ schedules—and shared these with the science teachers. Together, they devised solutions to increase the engagement of students receiving double doses of CS by providing them with additional pathways for engagement such as helping other students or creating their own CS projects, instead of simply repeating a CS lesson.

Conceptualization of equity as a multifaceted phenomenon

Within each of our cases, we see how ideas about what counts as “equitable” informed and shaped district leaders’ practices of diagnostic problem framing. In the first case concerning capacity, after learning that ECS failure rates were not, in fact, higher than average compared to other subject areas, district staff could have set the issue aside. However, they did not. They maintained a more expansive equity goal of reducing failure rates, a goal which then drove further inquiry into root causes. In the second case regarding access, an underlying conceptualization of equity through the lens of reducing subgroup inequities in access led district actors to seek out explanations for the disparities they identified in their data. In Yorkville’s participation case, a conceptualization of not simply enrollment, but enrollment in courses that were appropriately matched to female student competencies, drove Rosalyn to seek out causes for undermatching. Finally, in Rosewood, Beverly conceptualized equity within the context of student experiences within courses, leading her to explore why disengagement might be occurring. In each, a normative stance concerning what might be considered equitable, or not, undergirded the process of then diagnosing underlying causes. Indeed, it’s possible to imagine that different conceptualizations of equity, especially ones that are perhaps more narrow, would constrain which activities, if any, district leaders might have engaged in around problem diagnosis.

Problem attribution to systems, not students

Rather than explaining disparities in system outcomes or opportunities as the fault of students themselves, district leaders used diagnostic problem framing to show that the system itself consisted of shortcomings, contradictions, and redundancies, and/or was steeped in processes that reproduced historical inequities. In Chicago, district leaders countered the claim that the course failure rate was evidence that computer science was simply too difficult for students, by showing, instead, that when teachers had adequate professional development and previous experience teaching the course, students were successfully learning computer science.

In Yorkville, district leaders encountered a stereotype-laden view amongst counselors that female students were not well-suited for participating in CS courses. Instead of blaming female students themselves for not choosing more advanced courses, district leaders attributed the problem to the presence of historically inaccurate stereotypes that were being reproduced in everyday interactional dynamics between students and their teachers as well as in counselors’ course selection advice.

Lastly, in Rosewood, Beverly confronted a deficit problem framing from certain teachers when they explained students’ disengagement as “that student just isn’t getting it.” She worked to counter this framing, by redirecting the nature of the problem of engagement to the system itself: “Okay. What are some ways that this instruction can change so that they do get it or they are able to engage more and they’re not sitting back and just staring at their computer.”

Ability to analyze data for indicators of inequities

District leaders made one or more intentional data analysis moves to search for potential inequities. These included disaggregating data sets into student groups, comparing data sets across instructional systems, and closely analyzing cases of specific students. District leaders in Chicago and in Yorkville worked to disaggregate student data into various student groups to glean the presence of disparities in student access or student participation in computer science courses, respectively. District leaders in both Chicago and Yorkville also compared computer science data sets such as course failure rates and enrollment rates (of males and females) to other subject areas that served as appropriate points of reference (such as AP Calculus rather than AP Physics in Yorkville), to grasp the severity of a given inequity. Last, district leaders in both Yorkville and Rosewood worked to build relationships with specific students and gather individual data points on them to notice irregularities that suggested an underlying equity issue such as students being undermatched in Yorkville and students not receiving timely support in Rosewood. This problem-framing work entailed more than monitoring and collecting data, but rather strategically triangulating data across multiple sources to infer and surface underlying inequities.

Knowledge of local instructional systems

As part of their problem-framing work, district leaders drew on their pre-existing knowledge of local instructional systems to locate the root causes of the inequities they surfaced. In Yorkville, Rosalyn connected ideas between her knowledge of under-enrollment of female students in AP CS-A courses, her formative knowledge of capabilities around CS displayed by female students in the AP CS-P course she taught, and her familiarity with counselors’ stereotype-laden views about who CS is for. This led to her diagnostic problem framing to form a “hunch” about the cause of under-enrollment being related to guidance counselor brokerage and recommendation practices.

In Rosewood, Beverly was familiar with how issues of whole group instruction were inadequately serving struggling students across her district’s different subject areas. This familiarity allowed her to extrapolate the issue into the context of computer science courses as a potential explanation for student disengagement. She eventually located an additional root cause as she became familiar with additional aspects of the local instructional system. This root cause was a teaching practice in which science teachers provided the same computer science lessons as the CS specialist, and a scheduling quirk that allowed certain students to get a double dose of these lessons back-to-back.

This points to the importance of district leaders working not just to connect the dots between their pre-existing knowledge of local instructional systems, but to expand their purview of other potentially relevant aspects of instructional systems. In Chicago, for example, district leaders worked to learn about the division of English Language Learners into tiers and the scheduling and resultant access issues that emerged from that. Likewise, in the experience case, Beverly’s intuition about whole group instruction being a root cause of disengagement was only a part of the explanation.

Embeddedness in instructional networks

The cases also highlight the reality that diagnostic problem framing does not occur in a social vacuum. District leaders were able to advance their problem framing through actively embedding themselves in larger organizational systems and building personal relationships. In Chicago, district leaders were able to engage in processes of hypothesis generation through a standing consultancy meeting that involved a large number of district administrators and research partners, which led to formal analysis of large datasets on the part of the CAFÉCS research partners in order to identify factors correlated with course failure. Additionally, in their problem-framing work around access, district leaders in Chicago leveraged their social networks with the district to broker a meeting with a representative from the district office responsible for policies surrounding ELL students. Their notion encompassed not about what an explanation might be, but who might possess it. By gathering additional background knowledge from the ELL administrator, the team was able to receive knowledge of ELL instructional policy that was highly plausible as the root cause of the access issues at play.

In Yorkville, Rosalyn utilized her positionality as a teacher of AP-CSP who had direct relationships with students, her role collaborating with Nicole on monitoring enrollment data, and her experiences at national equity-focused trainings in order to develop her hunch around “undermatching” and its relationship to guidance counselor practices. In Rosewood, Beverly’s positionality as both the actor responsible for data monitoring as well as professional development around CSed positioned her to be exposed to the underlying “double-dosing” issue that explained why some students were disengaged in CS lessons.

In all of these examples of diagnostic problem framing, district leaders unpacked and made sense of the data they collected and monitored by partnering with researchers, leveraging brokers, attending equity-focused trainings, and building relationships with students impacted by a specific equity issue.