A Multiyear Investigation of Student Computational Thinking Concepts,Practices,and Perspectives in an After-School Computing Program

Abstract

In this work, we examine whether repeated participation in an after-school computing program influenced student learning of computational thinking concepts, practices, and perspectives. We also examine gender differences in learning outcomes. The program was developed through a school–university partnership. Data were collected from 138 students over a 2.5-year period. Data sources included pre–post content assessments of computational concepts related to programming in addition to computational artifacts and interviews with a purposeful sample of 12 participants. Quantitative data were analyzed using statistical methods to identify gains in pre- and post-learning of computational thinking concepts and examine potential gender differences. Interview data were analyzed qualitatively. Results indicated that students made significant gains in their learning of computational thinking concepts and that gains persisted over time. Results also revealed differences in learning of computational thinking concepts among boys and girls both at the beginning and end of the program. Finally, results from student interviews provided insights into the development of computational thinking practices and perspectives over time. Results have implications for the design of after-school computing programs that help broaden participation in computing.

Keywords

computational thinking object-oriented programming informal learning environment equity mixed methods

Introduction

The Computer Science (CS) for All initiative in the United States emphasizes the need to help all students acquire a deeper understanding of computing in the world around us by drawing on CS principles (President’s Council of Advisors on Science and Technology, 2010; The White House, 2016). These objectives have been described in the literature under the term computational thinking (CT). CT involves the use of CS concepts and cognitive processes to solve problems efficiently and creatively (Anderson, 2016). Programming is one way of promoting CT and has strong ties to CS (Shute, Sun, & Asbell-Clarke, 2017). Programming enables students to create and innovate, develop a sense of competency that may lead to further interest in computing, and ultimately take advantage of the growing career opportunities in computing-related fields (Google, 2014; Google & Gallup, 2015). The U.S. Bureau of Labor Statistics estimates that by 2022 there would be more than 1.3 million job openings in computer and mathematical occupations. Traditionally, however, opportunities to develop knowledge and skills related to CT, including programming, have been limited in K-12 schools. While efforts to improve access to such opportunities are currently underway in secondary education, participants continue to represent a homogeneous group of students, while females remain severely underrepresented (Cuny, 2012; Master, Cheryan, & Meltzoff, 2016).

Prior research suggests that one approach to broadening participation in computing is to provide engaging experiences at an early age (e.g., Ching, Hsu, & Baldwin, 2018; Dasgupta & Stout, 2014; Franklin et al., 2013). Younger students have limited knowledge of career options, and thus middle school presents an important juncture for providing experiences that introduce students to a variety of future opportunities (Friend, 2015). Similarly, middle school presents a prime time for breaking down stereotypes related to computing. Research indicates that by the time students start middle school, both gender differences as well as disparities in prior knowledge and experiences with computing become more apparent (Ardito, Czerkawski, & Scollins, 2020; Master et al., 2016; Witherspoon, Higashi, Schunn, Baehr, & Shoop, 2017). Providing middle school students with opportunities to engage in computing, however, can reduce gender imparities (Master et al., 2016).

While K-12 systems play a crucial role in providing students with early computing experiences, enrichment programs offered by universities or other private entities can also promote learning and support interest in computing (Hsu, Irie, & Ching, 2019). Higher education faculty, in particular, should play an active role in enriching school efforts. Research, for example, recommends investing in local partnerships, implementing service-learning programs where undergraduates assist local teachers, offering after-school programs, using age-appropriate programming tools, and implementing sound pedagogical practices that build upon children’s innate curiosity (Maloney, Peppler, Kafai, Resnick, & Rusk, 2008; President’s Council of Advisors on Science and Technology, 2010).

In this work, we report on efforts to foster the development of CT at the upper elementary and middle school level through the design of an after-school enrichment program. This program was developed through a partnership between university faculty, CS undergraduates, teachers, and students. The overall intention was to foster the development of CT knowledge and skills through opportunities to create computational artifacts using Scratch, a block-based programming language (Maloney et al., 2008). We adopt a multiyear longitudinal design where we examine students’ learning over time as they repeatedly participate in the after-school program. Specifically, we use quantitative data to examine participants’ learning of programming concepts over a 2.5-year period. Further, we use qualitative data to examine participants’ motivation as well as CT practices and perspectives related to programming. Understanding how the program influenced student learning of CT concepts, practices, and perspectives is important for the design of effective learning experiences that help broaden participation in computing. Specifically, we explore two questions:

How does repeated participation in an after-school computing program influence student learning of CT concepts related to programming? Are there any gender differences among student outcomes?

What is the motivation, CT practices, and perspectives of students who choose to participate in the after-school computing program?

Theoretical Perspectives

Definition of Computational Thinking

CT is often situated in the theoretical framework of constructionism, which highlights the pedagogical importance of constructing a “public entity,” such as a computer program, that can be shared with others (Papert, 1980; Papert & Harel, 1991). Wing (2006) expanded on Papert’s ideas and popularized the term CT describing it as a “fundamental skill for everyone, not just for computer scientists” (p. 33). The renewed popularity and interest in the development of CT knowledge and skills prompted a number of curricular changes in K-12 contexts around the world (Rich, Browning, Perkins, Shoop, Yoshikawa, & Belikov, 2019). In the United States, for instance, CT has been identified as a scientific practice in both the Common Core State Standards and the Next Generation Science Standards (Common Core State Standards Initiative, 2010; NGSS Lead States, 2013). In turn, educational policies and organizations have been leading efforts to provide a definition of CT that is relevant for K-12 contexts. In particular, the Computer Science Teachers Association in collaboration with the International Society for Technology in Education convened a group of leaders from education and industry to define concepts, dispositions, and artifacts associated with CT in K-12. This group identified core CT concepts and provided examples of how they might be embedded across different subject areas. Specifically, identified CT concepts and competencies include problem decomposition, algorithmic thinking, abstraction, data, automation, parallelization, and simulation (Computer Science Teachers Association & International Society for Technology in Education, 2011).

Programming is considered a fundamental component of CT, as it provides a context for students to develop CT-related skill sets (Brennan & Resnick, 2012; Grover, Jackiw, & Lundh, 2019; Grover & Pea, 2013; Guzdial, 2004). Traditionally, learning to program focused primarily on coding skills, an effort that has been criticized because it fails to demonstrate what students learn and how such learning can be transferred in other subjects (Kafai & Burke, 2013). Recently, efforts to support CT through programming focused on the design of software applications such as interactive games and apps with applicability in a variety of subject areas. Further, efforts to support novice programmers have resulted in a range of visual programming construction tools such as Scratch, which lower the floor of entering into programming and facilitate the creation of interactive media that can be shared with others (Brady et al., 2017; Kafai & Burke, 2013). Programming, however, is not valuable in and of itself. Rather, teachers as well as instruction are critical in both what and how students learn (Grover, Pea, & Cooper, 2016).

Focusing specifically on CT-related skills associated with programming in Scratch, Brennan and Resnick (2012) proposed a hierarchical CT framework that involves three dimensions including (a) computational concepts: concepts that programmers use including sequences, loops, events, parallelism, conditionals, operators, and data; (b) computational practices: problem-solving practices involved in programming including being incremental and iterative, testing and debugging, reusing and remixing, and abstracting and modularizing; and (c) computational perspectives: students’ understanding of themselves, their relationship to others, and the technological world around them. Given the focus of this work on the use of programming in Scratch as a means of supporting the development of CT, we draw heavily on this framework.

Approaches for Developing Computational Thinking in K-8

To date, a number of studies have been implemented in middle school settings to examine the role of curricula and instruction in fostering student learning of CT concepts. Grover et al. (2016) designed a 7-week curriculum to introduce middle school students to CT-related concepts with a focus on algorithmic thinking and programming. Using pre- and post-measures of computational concepts and other survey data, Grover et al. found that students achieved substantial learning goals in algorithmic thinking skills and demonstrated growth toward an understanding of computing as a discipline. They also found that certain CT ideas were more difficult than others, while prior experience as well as math and English ability were predictors of learning outcomes.

Other work introduced project and curriculum that aimed to educate students across gender in a range of educational settings. These studies concluded that both boys and girls could achieve equivalent levels of skills and attitudes toward CT, though training time and pedagogy were important variables. Atmatzidou and Demetriadis (2016), for instance, investigated student’s CT development in the context of educational robotics learning activities. Their study indicated that female and male students eventually reached the same CT skill level; however, female students claimed more training time to reach the same CT skill level compared with male students. Examining the issue of pedagogy in advancing students’ CT development, Webb, Repenning, and Koh (2012) also found some gender differences. Specifically, they found that female students were more likely to be motivated through pedagogical approaches that utilized “guided discovery scaffolding,” meaning just enough assistance to reach their goals, while they were given options and choices to design and program their own games. In contrast, prescriptive teacher-directed approaches exacerbated gender differences, particularly in relation to student interest in taking another game design class. Finally, focusing specifically on developing CT among girls, Brady et al. (2017) found that a focus on physical and social computing can have a positive impact on attitudes toward CS.

While these studies are valuable, they are primarily short-term, focusing on pre- and post outcomes alone. Multiyear studies examining student learning outcomes when they repeatedly participate in such programs are scarce. Although prior experience has been associated with improved learning outcomes, research that documents learning of CT concepts, practices, and perspectives over a long period of time is not widely available. The few studies that exist to date focus primarily on examining the computational products of Scratch users in the online community (e.g., Matias, Dasgupta, & Hill, 2016); they do not address longer term outcomes as a result of participation in specific after-school or other enrichment programs. In this work, we attempt to bridge this gap by examining student outcomes related to Scratch programing as a result of repeated participation in a 9-week after-school computing program over a 2.5-year period of time. Through a set of purposeful case studies, we also examine student perspectives of their experience.

Pedagogical Practices to Support Equitable CT Development

With no doubt, there is need to increase female participation in computing (Master et al., 2016). The underrepresentation of females in computing is problematic for both economic and scientific reasons. First, it limits female participation in potentially lucrative career fields (Friend, 2015). According to Barker and Aspray (2006), computing is essential across many fields, including science and economics. Thus, to have successful careers, females must be computationally literate. Second, the U.S. economy depends on highly skilled workers in computing fields. The underrepresentation of females in computing has resulted in shortages of qualified workers (National Research Council, 2010). Finally, innovations in computing products depend highly on diverse teams (Ashcraft & Breitzman, 2007; Google, 2014). As a result, lack of female participation in computing could result in products that are not responsive to the needs of female consumers (Cukier, Shortt, & Devine, 2002).

To address the issue, researchers have explored factors associated with lack of female interest in computing. Such factors typically include negative stereotypes of computer scientists as isolated, lack of understanding related to computing jobs, and lack of female role models in computing fields (Master et al., 2016; Papastergiou, 2008; Varma & Hahn, 2008; Vekiri, 2010). Middle school represents an important time for helping females gain exposure to computing and broaden their career perspectives (Google, 2014; Witherspoon et al., 2017). A growing body of research, however, indicates that learning experiences aimed at broadening participation in computing should be carefully orchestrated. Shah et al. (2013), for instance, proposed a framework for equitable instruction in computing consisting of four dimensions: rich content, quality instruction, productive peer relationships, and identities as computer scientists. In the framework, rich content and quality instruction are viewed as access to material resources. They include both a focus on curricular materials and teaching practices that make the curriculum more accessible to students. Peer relationships and identities as computer scientists are viewed as access to non-material resources. These dimensions focus on building students’ self-efficacy, as well as awareness of the role of their peers as learning resources.

A number of studies have been conducted to date focusing explicitly on promoting female participation in computing. Denner, Werner, Bean, and Campe (2005), for example, designed an after-school and summer program for middle school girls built on four strategies about the personal, social, and structural barriers to female participation in computing. These strategies included a focus on game design and production as a means for problem-solving and self-expression, collaboration with peers, and activities that challenged stereotypes and supported exploration of social identities.

Similarly, Lang, Fisher, Craig, and Forgasz (2015) created an outreach program just for females called the Digital Divas Club delivered in Australian schools. The program was also built on strategies that were found to break barriers to female participation in computing including single-sex classes, a curriculum focused on female interests (e.g., health, body image), role models, and a supportive learning environment focused on breaking down stereotypes. Findings indicated that the program significantly influenced student confidence in computing but did not necessarily influence their desire to pursue a career path in computing.

While not focusing explicitly on program design, other researchers have also studied the experiences of females both during regular school and after-school programs (e.g., Denner et al., 2005; Friend, 2015). These studies also demonstrate that females typically have limited experiences as well as limited understanding of career paths in computing. Females expressing more interest in computing careers typically had higher confidence, more social support for computing, and stronger social identities (Friend, 2015). In this work, we do not focus exclusively on females. Rather, we built on this body of work to ensure that the design of the after-school program encourages positive experiences that support the learning of all students—males and females.

Context of This Work

This work is situated in a larger effort to improve the teaching of computing in the United States through a three-pronged approach: teacher professional development, a college field-experience course, and sustainable school partnerships (Pollock, Mouza, Atlas, & Harvey, 2015). Here, we focus on the latter two strategies. The field-experience course is open to undergraduates with at least one prior course in CS. It combines college classroom meetings with field-experience in schools. During the college meetings, undergraduates and faculty in CS and education work together to (a) identify CT lessons and activities relevant to students’ age-group, interests, and prior experiences; (b) model pedagogical strategies for teaching CT that have shown promise in broadening participation in computing; (c) prepare and analyze computing lesson plans; and (d) reflect on successes and challenges during the field experience (Mouza, Marzocchi, Pan, & Pollock, 2016). Although participating undergraduates do not intend to pursue a teaching career, they enroll in the course to fulfill service-learning requirements and their desire to share their CS expertise and interest with teachers and younger audiences. In the field, undergraduates meet with teachers to discuss lesson plans, solicit input, and work out logistics. They also co-facilitate classroom activities or after-school programs with their partner teacher (see Mouza et al., 2016; Pollock et al., 2015). Here, we examine one such partnership in an after-school program at a K-8 school, which had been a partner for 2.5 years.

Design of After-School Computing Program

As part of the partnership, the school’s technology teacher (Teacher A) established her first 9-week after-school computing program for students in Grades 4 to 6, at Town Charter School (TCS) in Spring I semester. TCS is a suburban K-8 school enrolling approximately 1,350 students. School demographics included 67.5% Whites, 11.6% Asians, 11.1% African Americans, 4.2% Hispanic, and 5.5% multiracial. Approximately 16% qualified as low income. Here, we report on data collected from the TCS program over a period of five semesters (Table 1).

Table 1.

TCS Computing Programs From Spring I to Spring V.

	Spring I	Fall II	Spring III	Fall IV	Spring V
Lead teacher	Teacher A	Teacher A	Teacher A	Teacher A	Teacher A
Number of students	41	24	27	23	23
Number of undergraduates	4	2	2	2	2

Each semester, two to four CS undergraduates partnered with Teacher A. The undergraduates worked with Teacher A to discuss their CT lessons and solicit input regarding the relevance of the activities for the target audience. The undergraduates had primary instructional responsibilities, while Teacher A facilitated logistics, classroom management, provided assistance to students, and offered pedagogical feedback to undergraduates after each session. The 70-minute-long computing program met once a week after-school for 9 weeks. The design of the program was grounded on existing curricula designed for implementation in formal educational settings. Given the shorter time frame of the TCS computing program and prior literature, the design team (CS and education faculty, undergraduates, and Teacher A) selected topics considered foundational and developmentally appropriate for middle school students, similar to those addressed in other short-term informal curricula (e.g., Grover et al., 2016; Mouza et al., 2016). Those topics were introduced through short segments of instruction followed by open-project construction that incorporated the targeted concept. Although iterations occurred from semester to semester over the 2.5-year period, the key elements of the program (e.g., CT concepts and pedagogy) remained stable. Table 2 describes the TCS computing program.

Table 2.

Overall Description of Content Covered at the TCS Computing Program.

CT concepts	Description/example of programming exercises
Intro to programming & sprites	Introduction to Scratch programming: customizing sprites and backgrounds
Algorithms: self-contained sequence of steps	Understanding that computers are programmed to follow instructions; using Scratch to start creating a program of choice (motion, sprites, keyboard input)
Loops: repeated execution of specific code	Scratch activities that include programming blocks such as “repeat#times,” “forever,” and “repeat until.” Example: Students make Scratch sprites dance
Conditionals: if–then	Reviewing and developing Scratch projects that include conditionals (e.g., guessing games)
Broadcasting: sending and receiving messages	Creating a skit in Scratch to understand how one action prompts another within a project of choice. Example: Have two sprites interact with one another
Variables: manipulation and modification of data	Creating a guessing game in Scratch that includes variables. Example: Create a pong game that records the player’s scores
Boolean logic: variables reduced to true or false	Scratch activities that include Boolean logic such as true, false, and, and or. Example: Students play a game of 20 yes/no questions until guessing the answer
Tracing: reading and interpreting code	Examining Scratch code and explaining the actions associated with each block. Example: In pairs students review Scratch projects, varying in complexity, to predict the execution of the script; students discuss/share new insights
Project demo	Project development, peer review, and demonstrations

Note. CT = computational thinking.

Importantly, prior literature has established that pedagogy is equally, if not more important, than content in efforts to stimulate student interest and broaden participation in computing (Shah et al. 2013). The TCS computing program drew on learning sciences and CS education literature (Grover et al., 2016; Shah et al., 2013) to introduce and address issues related to how children learn foundational CT concepts and how females become more confident and develop social identities in computing. Table 3 illustrates the pedagogical practices that guided the design and delivery of the TCS after-school program.

Table 3.

Pedagogical Strategies for Teaching CT in Ways That Broaden Participation.

Dimension	CT pedagogies	Description/examples
Rich content	Use of metaphors	CS unplugged are activities done away from the computer to make CS concrete
	Diverse problem-solving	Students received small chunks of formal instruction followed by project-based work individually or in pairs, where they followed their own paths, explored different design options, and controlled the overall direction of their work
	Multiple solutions	During project-based instruction, CS undergraduates encouraged diverse problem-solving approaches and multiple solutions
Quality instruction	Differentiated plans based on prior knowledge	Students with prior experience were provided with differentiated exercises to advance their learning, including advanced challenges, integration of multiple CT concepts, and opportunities for project updates and remixing
	Formative assessment	Formative assessment through exit tickets (e.g., interpret the output of Scratch code) to indicate understanding
	Reflection	Students reflected on their designs using a design journal. Experienced students were required to respond to additional prompts, such as Which extensions did you add to your project?
Peer relationships	Peer interactions	Peer interactions were supported through pair programming (two students work together at one computer, serving as a driver or navigator), design of open-ended projects in pairs, and CS undergraduate modeling during their instruction
Social identities	Role models	Each pair of CS undergraduates included one male and one female who worked together to serve as role models to students and encourage female participation in computing
Social identities	Public sharing of work	Students engaged in public sharing of their work to build confidence

Note. CT = computational thinking; CS = Computer Science.

Research Design

In this work, we adopt a mixed-methods research design drawing on both quantitative and qualitative data. Quantitative data were collected from all students (N = 138), while qualitative interview data were collected from a purposeful sample of 12 students (N = 12). Mixed-methods approaches are valuable because they allow researchers to generate more detailed and contextualized findings that can inform future program development (Creswell & Plano-Clark, 2018).

Participants

Participants included 138 students in Grades 4 to 6 who participated in the TCS computing program over the 2.5-year period (Spring I, Fall II, Spring III, Fall IV, and Spring V). Students volunteered to attend. Of those students, 72 attended the TCS program for the first time, 39 students attended twice, while 27 attended multiple times (more than twice). Students who were first-time participants typically had limited background knowledge, while returning and multiple-time participants had more experience. Of the 138 participants, 74 were boys and 64 were girls.

Participants selected for interviewing included 12 students in Grades 4 to 6 who participated in the program during that period. Participants were selected to represent different levels of experience (i.e., first-time, returning, and multiple-time participants), grades, and gender. Based on these criteria, we chose 12 students: 6 boys and 6 girls. All interviews were conducted either at the end of students’ participation in Spring III or Spring V (see Table 4).

Table 4.

Interview Participants.

Name	No. of times participated	Gender	Grade	Interview time: Spring III	Interview time: Spring IV
Sarah	1	F	4	x
Jaime	1	F	4	x
Caleb	1	M	4	x
Stella	1	F	5	x
Sal	1	M	5	x
Rick	1	M	5	x
Gia	2	F	4	x
Maggie	2	F	4	x
Jim	2	M	4	x
Cynthia	3	F	6		x
Jack	3	M	6		x
George	3	M	6		x

Data Sources

Data for the study were collected from multiple sources: (a) observations of the TCS program meetings (N = 32); (b) pre/post-assessment of student learning of CT concepts; (c) interviews with a subset of 12 students (N = 12); and (d) collection of students’ Scratch projects (N = 59).

Program observations

Most program meetings were observed and documented using detailed field notes to provide an accurate description of the activities. Each after-school program meeting was 90 minutes long. Each observation included information on the lesson, including materials and programming concepts addressed. Further, all teaching materials created by undergraduates throughout the duration of the after-school program were collected to gain a better understanding of the CS activities.

Pre/post-assessment

To document changes in students’ learning, we administered a pre- and post-multiple-choice content assessment every semester in which a student attended the after-school program. The assessment includes 10 questions examining students’ knowledge of CT concepts associated with Scratch programming, which served as the foundation of this work (Ericson & McKlin, 2012). The first question asks students to select the category for a block. Questions 2 to 9 ask students to match a Scratch script to a computing concept such as loops, conditionals, variables, or parallelism and determine the result of executing a script. Question 10 asks students to read a code segment and predict the outcome.

In this work, we focused on examining differences among student participants based on the number of times they attended the program (first-time, returning, and multiple-time participants). Accordingly, we selected to compare pre- and post-scores among the three student groups utilizing the following principles: For the first-time participants (N = 72), we used the initial pre- and post-assessment scores; for the returning (N = 39) and multiple-time (N = 27) participants, we used their pre- and post-assessment scores at their latest participation in the TCS program. For instance, the pre- and post-scores of three-time participants referred to their scores during their participation in the TCS program at their third 9-week program. It is worth noting that we did not use all students’ initial pre-test scores, especially for the returning and multiple-time groups. However, there was no statistically significant difference in students’ baseline scores (initial pre-test scores) among these three groups—first-time group: M = 5.02, SD = 2.13; returning group: M = 5.53, SD = 1.80; multiple-time group: M = 5.67, SD = 1.49; F = 1.504, df (2, 135), p = .226.

Interviews

We interviewed a subset of 12 students to elicit information on their experiences with the program (e.g., motivation, CT practices, and perspectives). The interviews asked students to discuss (a) motivation for attending the TCS program; (b) ways in which they framed their projects (e.g., idea, process, challenges, and enjoyment); (c) aspects of their work they were most proud of; and (d) best part of working with Scratch programming. Each semistructured interview was about 15 to 20 minutes and was tape-recorded and transcribed.

Scratch projects

A number of Scratch projects (N= 59) developed during student participation in the program were collected. Scratch projects helped situate and triangulate both observation and student interview data.

Data Analysis

Program observations

Observation data and teaching artifacts collected from undergraduates were used to triangulate findings and confirm or dismiss emergent themes (Merriam, 2009).

Pre/post-assessments

Content assessments were scored for correctness. Each correct answer received one point, while each incorrect answer received no points. Further, a total score was calculated for the instrument as a whole for both the pre- and post-administration. To test for the significance of the gain score for each group (first-time, returning, and multiple-time groups), a paired sample t test was conducted respectively. Subsequently, to understand whether there were differences in CT knowledge based on gender and the number of times that students participated in the TCS computing program, a two-way factorial analysis of variance (ANOVA) was conducted to examine (a) the main effects of gender and number of participation times, and (b) the interaction between gender and number of participation times. This analysis was conducted for both the pre- and post-administration of the content assessment.

Interviews

Interview data were analyzed qualitatively to identify emerging themes (Hatch, 2002). We first read all interview transcripts and summarized key ideas related to student motivation, practices, and perspectives. Subsequently, all entries were entered into the NVivo qualitative software (QSR International, 2010). The first coding sweep involved creating nodes that captured student views. This resulted in a list of codes for all the practices and perspectives discussed by the students that were subsequently categorized and organized into three primary codes: (a) motivation for attending the TCS after-school program (i.e., enjoys programming, heard about it from a friend); (b) Scratch practices and perspectives (i.e., generating ideas about Scratch projects, challenges faced while programming, ways of overcoming challenges, and aspects of work a student was most proud of); and (c) working with Scratch (i.e., learning how computers work and sharing with others).

Results

Student Learning of CT Programming Concepts

As shown in Table 5, there was a significant improvement (p < .05) for participants in each group from pre assessment to post-assessment. Further, a two-way ANOVA was conducted on the influence of two independent variables (gender and number of participations) on the pre- and post-assessments. The result indicated that the interaction of gender and the number of participations had no statistically significant effect on either the pre assessment, F(2, 132)= .099, p = .905, or post-assessment, F(2, 132)= .013, p = .987. The number of participations, however, yielded an F ratio of F(2, 132)=9.581, p < .01 in pre administration and F(2, 132)=4.996, p < .05 in post-administration, indicating a statistically significant difference between groups.

Table 5.

Dependent (Paired) t Test of Pre- and Post-content Assessment Among the Three Groups.

Groups	Pre-knowledgeM	Pre-knowledgeSD	Post-knowledgeM	Post-knowledgeSD	Mean differences (Post–Pre)	p Value	Effect size(d)
First-time participants (N=72)	5.02	2.13	6.17	2.09	1.15	.000**	0.54
Returning participants (N=39)	6.12	1.82	6.96	2.23	0.84	.008**	0.41
Multiple-time participants (N=27)	6.76	1.93	7.43	1.70	0.67	.000**	0.36

**p < .01. *p < .05.

As shown in Table 6, both the returning and the multiple-time participants scored better on both the pre- and post-administration of the assessment; for multiple-time participants, scores were significantly higher than first-time participants in both the pre administration of the assessment (p = .001) with large effect size of 0.84 and post-administration (p = .023) with medium effect size of 0.63. For returning participants, scores were also significantly higher than first-time participants but only in the pre administration of the assessment (p = .021) with medium effect size of 0.54. There was no statistically significant difference in the post-administration (p = .154).

Table 6.

Multiple Group Comparisons With p Value and Effect Size.

	Comparison	Pre-knowledge		Post-knowledge
	Comparison	Significance (p value)	Effect size (d)	Significance (p value)	Effect size (d)
1	First-time vs. returning	.021*	0.54	.154	0.37
2	Returning vs. multiple	.492	0.34	.748	0.23
3	First-time vs. multiple	.001**	0.84	.023*	0.63

**p < .01. *p < .05.

Finally, two-way ANOVA also indicated gender had a significant effect on student knowledge of CT concepts both at the beginning, F(1, 132) = 3.989, p = .048, and end of the program, F(1, 132) = 4.588, p = .034. As shown in Table 7, boys (N = 74) scored significantly higher than girls (N = 64) in both pre- and post-assessments. Despite differences in pre-assessments, the gain scores between boys and girls did not have statistically significant difference by the number of participations, F(1, 132) = . 177, p = .838.

Table 7.

CT Content Assessment Results by Gender.

Groups	Gender	Pre-assessment		Post-assessment
Groups	Gender	M	SD	M	SD
First-time participants	Boys (41)	5.43	2.18	6.49	2.05
First-time participants	Girls (31)	4.48	1.98	5.74	2.09
Returning participants	Boys (22)	6.43	1.18	7.32	1.98
Returning participants	Girls (17)	5.71	2.40	6.50	2.51
Multiple-time participants	Boys (11)	7.09	2.01	7.95	1.27
Multiple-time participants	Girls (16)	6.53	1.90	7.06	1.90

Student CT Practices and Experiences

Results from student interviews revealed insights related to their motivation for attending the TCS program as well as their CT practices and perspectives during their participation.

Motivation for attending the TCS program

Findings indicated that the majority of the students (83%) initially attended the TCS program because of some initial experience they had with programming in fourth grade through their mathematics class. As Sal, a first-time participant, noted: “I first learned about Scratch when we used it last year in fourth grade. The teacher worked with us in math class and I enjoyed it. I couldn’t do the program earlier because I had band practice so when I saw the chance this semester I signed up.” In addition, a number of girls noted a prior programming experience offered by Teacher A in the past just for girls. Cynthia, a multiple-time participant noted,

Last year I did not attend the TCS because I did not know what Scratch was. But I did Girls Code in the month of January and I liked it a lot. Once we started it I was like, oh this is fun, so I came back.

Similarly, all students noted that the reason they returned to the program for another semester was because they really enjoyed working with programming. George, also a multiple-time participant, noted: “I really liked it before. I liked it so much that I wanted to come back and do it again so I could do more advanced things.” Interestingly, two first-time participants (16%) noted that they enrolled because a friend told them about the program. Caleb explained:

Well, a friend told me about it so I decided to just try it out. I realized why everyone was telling me how fun it was. So I tried it and that’s pretty much it. My friend told me about it and then I liked it.

Similarly, Stella noted: “My friend told me that you get to make cartoons, sort of, and I thought that was really cool to make something like that come to life and do whatever you tell it. That’s why I joined.”

Computational practices and perspectives

Findings indicated a number of computational practices and perspectives employed by students. Specifically, students discussed how they typically generated ideas for new programming projects in Scratch, identified challenges encountered during the process, and noted ways of addressing those challenges. During their participation in the program, students received small chunks of formal instruction around programming, followed by project-based work that allowed them to apply new learning into practice (see Table 2). For instance, students learned first about conditional statements and subsequently were asked to create a guessing game that allowed them to incorporate conditional statements. Nonetheless, students were free to decide the storyline of their projects, the types of questions to ask, characters, backgrounds, and so forth. Further, students had the freedom to design their own products in the last 2 weeks of each semester.

Discussing how they generated ideas for projects, the majority of the students (70%) identified the Scratch online community. Specifically, students noted that they like to review games other users created in the online community, which typically served as an inspiration for new ideas or modifications (remix) to existing projects. The CS undergraduates routinely allowed students the first 10 minutes to review other projects of interest in the Scratch online community. For example, after introducing variables and Boolean logic, CS undergraduates asked students to create a game of 20 questions using yes/no answers. Discussing this project, Jack, a returning participant, explained how he created his project following an idea he saw online:

Jack: I created this project that is called Ask squirrel yes or no questions. The squirrel tells you to ask him yes or no questions and then answers with yes, no, maybe or never.

Researcher: How did you come up with this idea?

Jack: I got this idea from another project online, it’s called Ask Elsa yes or no questions. So you press the green flag and then you ask the squirrel a yes or no question. Like, do you like ice cream? No.

Researcher: Oh, I see. Does the squirrel randomly pick yes or no?

Jack: Yes, no, maybe, never. The squirrel also changes costumes based on the answer. It picks a random costume between 1-4 based on the answer.

Similarly, Sarah (first-time participant), explained how the online community provided ideas for her work: “Sometimes when I see certain projects online I think it would be cool to do a part of them, but then add other things to them as well.” Just like Jack, George also discussed how he remixed an existing Scratch project called Elmo goes nuts by adding new features, his own characters, and music. He explained, “I had to program it so that it glides and then changes color. I had to do a lot to make it work and go forever. The whole thing goes on forever until you get annoyed.” A smaller percentage of students indicated that they generated ideas while talking or brainstorming with friends. Maggie (returning participant) indicated, “I just look at some of the projects and I also talk about it sometimes with my friends during recess or lunch, just to get other ideas and brainstorm.”

Discussing challenges faced when working with Scratch, most students noted difficulties in identifying specific Scratch blocks for each desired action or instances where the program output did not behave as expected. As Jack explained, “ . . . the most challenging part is sometimes you don’t know what each block is for and sometimes stuff don’t work out as you planned.” Typically, students addressed these challenges by looking at similar Scratch projects in the online community, asking for help from their peers or the CS undergraduates, or simply problem-solving and debugging on their own. Describing some of the challenges she faced while developing a guessing game that used variables to keep track of score (see Figure 1), Maggie explained: “I looked into it and I watched Sarah’s project. I saw how she had the variables in the lists and I tried it out, and it worked.” Just like Maggie, Jack looked online at the Ask Elsa game to troubleshoot challenges with having the squirrel change costume with each response in his project. He explained, “I didn’t know how to pick a random costume so I went back to the Ask Elsa game. I looked at the script and found the part that made the program pick random costumes from 1-4.”

Figure 1.

Guessing game using variables.

In her interview, Cynthia also illustrated a set of computational practices and perspectives utilized by students while describing a Scratch project called The Unicorn Game (see Figure 2). In this game, users navigate the unicorn using the arrow keys while trying to avoid touching the rainbow-colored dots coming down. Instead of looking at other projects as a way of addressing challenges in her project, however, Cynthia problem-solved and eventually addressed the issue on her own. It was clear that multiple-time participants such as Cynthia, Jack, and George gradually became more confident trouble-shooting and debugging on their own. She explained:

Researcher: What gave you the idea for this project?

Cynthia: I always liked games like these. They were fun for me and kind of frustrating, too. I wanted to make a game like that the last time I attended the TCS but I could not figure it out. So after the undergraduate assistants showed us a similar project this semester I wanted to try again.

Researcher: How did you go about making this game?

Cynthia: I made it so some of the colored dots go really fast and some go really slow . . . if you get hit, it says you lose. It uses cloning. This is the first time that I did stuff with cloning and I think it is really cool!

Researcher: What kinds of challenges did you face while making this unicorn game?

Cynthia: Well, with this one I had to figure out how to make it so the clones didn’t appear halfway through the page and instead appeared from the top and then went down.

Researcher: And how did you figure that out?

Cynthia: I worked with the coordinates, it was like, I set the x and y. I set the y where it spawns the same and I changed the x. The x is “pick random.”

Researcher: Did you receive any support from the undergraduate assistants?

Cynthia: No, not this time. I feel like I’m really confident that I can figure something out. I look at the code because that helps. If there’s a problem that I can’t figure out at the time I can always go back to it. Sometimes I like to go back to projects that didn’t work and try to figure out the problem and fix it.

Discussing which aspects of their work they felt more proud about, most students (58%) noted satisfaction with implementing something new or challenging in Scratch. Maggie (returning participant), who worked on a guessing game that every time picked a random number from 1 to 10 while also kept track of the player’s score (see also Figure 1) explained:

I think I’m most proud of the variable part . . . because I struggled with picking the secret number and everything. I wanted to be able to make it harder and more challenging if they got it wrong. So it kind of stumped me and so that’s why I’m proud of figuring out how to set a random secret number.

Figure 2.

Unicorn game.

Working with Scratch programming

As students continued to work with Scratch, they were able to create interactive products that helped them enjoy and feel pride in their work. Interview data indicated that for most students (75%) the best part of working with Scratch was learning how to program and share their work with others. As Rick (first-time participant) explained, he really enjoyed working with Scratch because you can “program stuff and control everything . . . there aren't rules, it’s like anything can happen, you can do anything, if you know how to code.” Caleb, a first-time participant, explained that programming in Scratch helped him learn “how computers actually work” and how other interactive products are developed. Cynthia similarly noted that the best part of working with Scratch is the ability to “think of something and watch yourself make it happen.” Further, the ability to share that work with others was highlighted by students. Jack explained,

Showing your work so everyone can see how good your work is. You can program games and it’s really fun and I like that other people can see your skills with Scratch. You can share your work and let your friends know how good you are with Scratch.

Adding to this perspective, Cynthia noted that she was also proud showing her creativity and expertise to others. She explained: “I like letting people see how creative I can be and how good I am at working on this stuff . . . and I like showing boys that girls can do cool computer stuff, too.”

Discussion

In this work, we described the design and implementation of an after-school computing program for upper-elementary and middle-school students and its impact on learning of CT concepts, practices, and perspectives. Based on the quantitative results and analysis, it appears that students made significant gains in their learning of CT concepts associated with Scratch programming. Specifically, results indicated statistically significant improvements from the pre to the post-administration of the content assessment. Nevertheless, it is important to note that students continued to find certain programming concepts challenging. Looking at the results from the content assessment, only a small percentage of students responded correctly to items related to event handling (36%), conditionals using if statements (39%), and script execution (32%). The items of event handling and conditionals used vocabulary that was not explicitly addressed in the TCS computing program, which may explain the results. Further, the script execution question asked students to interpret the output of a computer script, which included drawing a triangle. For students to respond to this item correctly, they must understand Cartesian coordinates, a mathematical concept that may still be unfamiliar to some elementary school students. Students also experienced challenges with the question on variables. In the post-administration of the assessment, only 61% of students responded correctly to the relevant item. This finding is not surprising. Prior research also found that concepts related to conditionals and manipulation of variables are difficult for novice programmers (Grover et al., 2016; Maloney et al., 2008; Meerbaum-Salant, Armoni, & Ben-Ari, 2013). For example, only about 20% of projects in the online Scratch community use variables (Meerbaum-Salant et al., 2013). Results indicated that students performed well on items related to the identification of Scratch blocks, loops, broadcasting, and conditional test.

In addition, results indicated benefits for students who attended the program more than once. While in prior work we have examined student learning of CT concepts over a one-semester period (Mouza et al., 2016), in this work we have adopted a longitudinal design that provided a repeated measure of CT knowledge for students who participated more than once (returning and multiple-time participants). Results demonstrated that gains in student learning of CT concepts persisted over time; both returning and multiple-time participants scored higher than first-time participants on the pre-assessment. Consistent with findings from other studies (e.g., Grover et al., 2016), prior experience was also associated with higher performance on the post-test as shown by the scores of multiple-time participating students. Further, students who attended the program more than two times benefited the most with significantly higher pre- and post-scores on CT concepts compared with students who only attended the program once. The differentiated plans implemented during the TCS program, which allowed students with prior experience to engage with advanced challenges and integration of multiple CT contexts, may have contributed to student continuous learning over time.

Findings, however, revealed persistent differences in CT outcomes among boys and girls both at the beginning and end of the program. This finding needs further attention. It is possible that participating boys had further prior experiences with computing that better prepared them to engage with learning of programming concepts. Research, for instance, indicates that parents encourage boys to engage with computing more than girls (Anderson, Welch, & Harris, 1984). Other studies also found that even girls who voluntarily joined computing camps had very limited knowledge of and experience with computing (Hur, Andrzejewski, & Marghitu, 2017). Both Margolis and Fisher (2003) and Barron (2004) refer to this issue as an “experience gap,” which is directly related to female participation in computing. Nonetheless, it is important to note that there were no statistically significant differences in gain scores on the CT assessment, indicating that the TCS program provided a positive learning experience with promise to close the gender gap related to the understanding of CT concepts associated with Scratch programming.

Findings from student interviews provided insights into the development of CT concepts, practices, and perspectives over time as well as student motivation and perceptions of Scratch programming. Specifically, findings indicated that students were initially motivated to attend the TCS program primarily because of an earlier school experience but later enjoyed programming, which served as a motivator for returning. The nature of the program, which allowed students the opportunity to engage with activities that did not necessarily resemble school or products that can be developed with other means (e.g., paper and pencil) may have also continued to encourage student participation. Cynthia, for example, who first utilized Scratch in her math class to design geometric shapes noted that initially she did not enjoy it as much because “it was all just drawing stuff, which I can do by hand.”

Findings also indicated that students not only acquired new CT concepts but they were also able to apply those concepts in the design of computational artifacts using Scratch. The design of the TCS program which only provided small chunks of formal instruction followed by project-based work was instrumental in allowing students opportunities to apply new learning into practice. Further, findings indicated that participants improved in their understanding of CT practices involved in thinking and learning about programming; they adopted a range of strategies for developing computational artifacts that involved problem-solving, learning by examining the code of similar products or remixing, and debugging their products to achieve the desirable action. These strategies were encouraged in the TCS program. As previously shown in Table 3, CS undergraduates routinely encouraged diverse problem-solving and multiple solutions to problems by encouraging students to work with peers, ask questions, and reflect on their work. Finally, findings indicated that students enjoyed working with Scratch and were proud to share their work with others. Public sharing of students' work was frequently encouraged during the TCS program as a way of building student confidence.

Importantly, participation in the program instilled an interest in computing given the number of students who returned back. Ericson and McKlin (2012) argue that beyond assessments of learning, one way to measure the impact of computing programs is by the number of students who return back. In this work, we had 66 students who returned back to the program, with 27 students returning more than once (i.e., multiple-time participants). Of those, 33 (50%) were females, indicating that the program appealed to both boys and girls. This finding is important given that interest in computing declines as students progress in their studies (Carter, 2006).

Limitations

We recognize that results are limited in generalizability; the assessment focused exclusively on CT concepts associated with Scratch programming and students volunteered to attend (e.g., self-selection), which may have influenced their motivation to learn. Further, although our sample included gender diversity, it was limited in terms of socioeconomic and ethnic diversity with most students coming from middle-class families who enjoyed wide access to technology at home. Finally, the number of students interviewed was fairly small compared with the large sample of students. Future studies should pay attention to issues of ethnic diversity and self-selection. Future studies should also examine student career aspirations over time to further understand ways in which participation in computing programs during middle school can help broaden career interests and female participation in computing.

Conclusion

This work presented an after-school computing program for upper elementary and middle school students focusing on building CT knowledge and skills through programming. The program was developed through a partnership between university faculty, CS undergraduates, teachers, and students. Using a mixed-methods design, the study examined students’ learning of CT concepts, practices, and perspectives over time as they repeatedly participated in the program over a 2.5-year period of time. Findings from this work are important for two reasons. First, they provide insight into the design, implementation, and outcomes of after-school programs that help both boys and girls learn foundational CT concepts and develop interest in computing, which could lead to future pursuits of STEM. Second, although our work was conducted in an after-school program, it is situated in a K-12 setting and engages middle school teachers through a partnership with university faculty and CS undergraduates. Therefore, this work provides a model that holds promise for engaging teachers with CS, who in turn can help their students reach the goals of CS for All.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported in this article was supported by National Science Foundation under award numbers: 1240905 and 1639649. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Science Foundation.

Author Biographies

Chrystalla Mouza is distinguished professor and director of the School of Education at the University of Delaware. Her research interests focus on the development of computational thinking in K–16 settings, teacher technological pedagogical content knowledge, and emerging technologies for the classroom. Dr. Mouza serves as editor of Contemporary Issues in Technology and Teacher Education (CITE) journal.

Yi-Cheng Pan is an educational consultant with the Pennsylvania Training and Technical Assistance Network. His research interests focus on the development of computational thinking among teachers and students and the application of new technologies in K–16 settings.

Hui Yang is a postdoctoral associate in the Department of Information Science at Cornell University. Her research examines how technology-rich learning environments support learners’ cognitive and affective outcomes in STEM education.

Lori Pollock is alumni distinguished professor of computer science at the University of Delaware. Her research focuses on computer science education and on the ways in which software engineers maintain and test large software systems. Dr. Pollock has been actively working to improve the participation of women and other underrepresented groups in computer science for many years.

ORCID iD

Chrystalla Mouza

References

Anderson

N. D.

(2016). A call for computational thinking in undergraduate psychology. Psychology Learning & Teaching, 15, 226–234.

Anderson

R. E.

Welch

Harris

L. J.

(1984, April). Inequities in opportunities for computer literacy. The Computing Teacher, 11(7), 10–12.

Ardito

Czerkawski

Scollins

(2020). Learning computational thinking together: Effects of gender differences in collaborative middle school robotics program. TechTrends. Advance online publication. doi:10.1007/s11528-019-00461-8

Ashcraft

Breitzman

(2007). Who invents IT? An analysis of women’s participation in information technology patenting. Boulder, CO: National Center for Women in Information Technology.

Atmatzidou

Demetriadis

(2016). Advancing students’ computational thinking skills through educational robotics: A study on age and gender relevant differences. Robotics and Autonomous Systems, 75, 661–670.

Barker, L., & Aspray, W. (2006). The state of research on girls and IT. In J. M. Cohoon & W. Aspray (Eds.), Women and information technology: Research on underrepresentation (pp. 3–54). Cambridge: MIT Press.

Barron

(2004). Learning ecologies for technological fluency: Gender and experience differences. Journal of Educational Computing Research, 31(1), 1–36.

Brady

Orton

Weintrop

Anton

Rodriguez

Wilenski

(2017). All roads lead to computing: Making, participatory simulations, and social computing as pathways to computer science. IEEE Transactions on Education, 60(1), 59–66.

Brennan

Resnick

(2012, April 13–17). New frameworks for studying and assessing the development of computational thinking. Paper presented at the Annual meeting of the American Educational Research Association, Vancouver, BC, Canada.

10.

Carter

(2006). Why students with an apparent aptitude for computer science don’t choose to major in computer science. SIGCSE Bulletin, 38, 27–31.

11.

Ching, Y., Hsu, Y., & Baldwin, S. (2018). Developing computational thinking with educational technologies for young learners. TechTrends, 62(6), 563–573.

12.

Common Core State Standards Initiative. (2010). Common core state standards for mathematics. Retrieved from http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf

13.

Computer Science Teachers Association & International Society for Technology in Education. (2011). Computational thinking. Teacher resources. Retrieved from http://csta.acm.org/Curriculum/sub/CurrFiles/472.11CTTeacherResources_2ed-SP-vF.pdf

14.

Creswell

J. W.

Plano-Clark

V. L.

(2018). Using and conducting mixed methods research. Los Angeles, CA: Sage.

15.

Cukier

Shortt

Devine

(2002). Gender and information technology: Implications of definitions. ACM SIGCSE Bulletin, 34, 142–148.

16.

Cuny

(2012). Transforming high school computing: A call to action. ACM Inroads, 3(2), 32–36.

17.

Dasgupta

Stout

J. G.

(2014). Girls and women in science, technology, engineering, and mathematics. Policy Insights from the Behavioral and Brain Sciences, 1(1), 21–29.

18.

Denner

Werner

Bean

Campe

(2005). The girls creating games programs: Strategies for engaging middle-school girls in information technology. Frontiers: A Journal of Women Studies, 26(1), 90–98.

19.

Ericson

McKlin

(2012, February 29–March 3). Effective and sustainable computing summer camp. Paper presented at the Proceedings of the 43th SIGCSE Technical Symposium on Computer Science Education (pp. 290–294), Raleigh, NC, USA.

20.

Franklin

Conrad

Boe

Nilsen

Hill

Len

, . . . Waite

(2013, March 6–9). Assessment of computer science learning in a scratch-based outreach program. Paper presented at the Proceeding of the 44th ACM Technical Symposium on Computer Science Education (pp. 371–376), Denver, CO, USA.

21.

Friend

(2015). Middle school girls’ envisioned future in computing. Computer Science Education, 25, 152–173.

22.

Google. (2014). Women who choose computer science – What really matters: The critical role of encouragement and exposure. Retrieved from https://docs.google.com/file/d/0B-E2rcvhnlQ_a1Q4VUxWQ2dtTHM/edit

23.

Google & Gallup. (2015). Searching for computer science: Access and barriers in U.S. K-12 Education. Retrieved from https://services.google.com/fh/files/misc/searching-for-computer-science_report.pdf

24.

Grover

Jackiw

Lundh

(2019). Concepts before coding: Non-programming interactives to advance learning of introductory programming concepts in middle school. Computer Science Education, 29(2–3), 106–135.

25.

Grover

Pea

(2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43.

26.

Grover

Pea

Cooper

(2016, March 2–5). Factors influencing computer science learning in middle school. Paper presented at the Proceedings of the 47th ACM Technical Symposium on Computer Science Education (pp. 552–557), Memphis, TN, USA.

27.

Guzdial

(2004). Programming environments for novices. Computer Science Education Research, 2004, 127–154.

28.

Hatch

J. A.

(2002). Doing qualitative research in education settings. New York, NY: SUNY University Press.

29.

Hsu

Irie

N. R.

Ching

(2019). Computational thinking educational policy initiatives (CTEPI) across the globe. TechTrends, 63, 260–270.

30.

Hur

J. W.

Andrzejewski

C. E.

Marghitu

(2017). Girls and computer science: Experiences, perceptions, and career aspirations, Computer Science Education, 27(2), 100–120, doi:10.1080/08993408.2017.1376385

31.

Kafai

Y. B.

Burke

(2013, March 6–9). The social turn in K-12 programming: Moving from computational thinking to computational participation. Paper presented at the Proceedings of the 44th ACM Technical Symposium on Computer Science Education (pp. 603–608), Denver, CO, USA.

32.

Lang

Fisher

Craig

Forgasz

(2015) Outreach programmes to attract girls into computing: How the best laid plans can sometimes fail. Computer Science Education, 25(3), 257–275. doi:10.1080/08993408.2015.1067008

33.

Maloney

Peppler

Kafai

Resnick

Rusk

(2008, March 12–15). Programming by choice: Urban youth learning programming with Scratch. Paper presented at the Proceedings of the 39th ACM Technical Symposium on Computer Science Education (pp. 367–371), Portland, OR, USA.

34.

Margolis

Fisher

, (2003). Unlocking the clubhouse—Women in computing. Cambridge, MA: MIT Press.

35.

Master

Cheryan

Meltzoff

A. N.

(2016). Computing whether she belongs: Stereotypes undermine girls’ interest and sense of belonging in computer science. Journal of Educational Psychology, 108(3), 424–437.

36.

Matias

J. N.

Dasgupta

Hill

B. M.

(2016, May 7–12). Skill progression in Scratch revisited. Paper presented at the CHI, San Jose, CA, USA.

37.

Meerbaum-Salant

Armoni

Ben-Ari

(2013). Learning computer science concepts with Scratch. Computer Science Education, 23(3), 239–264. doi:10.1080/08993408.2013.832022

38.

Merriam

(2009). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey-Bass.

39.

Mouza

Marzocchi

Pan

Pollock

(2016). Development, implementation and outcomes of an equitable computer science after-school program: Findings from middle school students. Journal of Research on Technology in Education, 48(2), 84–104.

40.

National Research Council. (2010). Committee for the workshops on computational thinking: Report of a workshop on the scope and nature of computational thinking. Washington, DC: National Academies Press.

41.

NGSS Lead States. (2013). Next generation science standards. Retrieved from http://www.nextgenscience.org/

42.

Papastergiou

(2008). Are computer science and information technology still masculine fields? High school students’ perceptions and career choices. Computers & Education, 51, 594–608.

43.

Papert

(1980). Mindstorms: Children, computers, and powerful Ideas. New York, NY: Basic Books.

44.

Papert

Harel

(1991). Situating constructionism. Constructionism, 36(2), 1–11.

45.

Pollock

Mouza

Atlas

Harvey

(2015, March 4–7). Field experience in teaching computer science: Course organization and reflections. Paper presented at the Proceedings of Special Interest Group in Computer Science Education, Kansas City, MO.

46.

President’s Council of Advisors on Science and Technology. (2010). Prepare and inspire: K–12 education in science, technology, engineering, and mathematics (STEM) for America’s future. Washington, DC: Author. Retrieved from http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-ste medreport.pdf

47.

QSR International. (2010). NVivo qualitative data analysis software. Doncaster, Australia: Author.

48.

Rich

P. J.

Browning

S. F.

Perkins

Shoop

Yoshikawa

Belikov

O. M.

(2019). Coding in K-8: International trends in teaching elementary/primary computing. TechTrends, 63, 311–329.

49.

Shah

Lewis

Caires

Khan

Qureshi

Ehsanipour

Gupta

(2013, March 6–9). Building equitable computer science classrooms: elements of a teaching approach. Paper presented at the Proceedings of the 44th ACM Technical Symposium on Computer Science Education (pp. 263–268), Denver, CO, USA.[10.1145/2445196.2445276][Mismatch]

50.

Shute

Sun

Asbell-Clarke

(2017). Demystifying computational thinking. Educational Research Review, 22, 142–158.

51.

The White House (2016). Fact Sheet: President Obama announces Computer Science for All initiative. January 30, 2016. Washington, DC: Office of the Press Secretary, The White House.

52.

Varma

Hahn

(2008). Gender and the pipeline metaphor in computing. European Journal of Engineering Education, 33, 3–11.

53.

Vekiri

(2010). Boys’ and girls’ ICT beliefs: Do teachers matter? Computers & Education, 55, 16–23.

54.

Webb

Repenning

Koh

K. H.

(2012, February 29–March 3). Toward an emergent theory of broadening participation in computer science education. Paper presented at the Proceedings of the 43th SIGCSE Technical Symposium on Computer Science Education (pp. 265–269), Raleigh, NC, USA.

55.

Witherspoon

E. B.

Higashi

R. M.

Schunn

C. D.

Baehr

E. C.

Shoop

(2017). Developing computational thinking through a virtual robotics programming curriculum. ACM Transactions on Computing Education (TOCE), 18(1), Article 4.

56.

Wing

J. M.

(2006). Computational thinking. Communications of the ACM, 49(3), 33–35.