Exploring the Effectiveness and Moderators of Block-Based Visual Programming on Student Learning: A Meta-Analysis

Abstract

Block-based visual programming tools, such as Scratch, Alice, and MIT App Inventor, provide an intuitive and easy-to-use editing interface through which to promote programming learning for novice students of various ages. However, very little attention has been paid to investigating these tools’ overall effects on students’ academic achievement and the study features that may moderate the effects of block-based visual programming from a comprehensive perspective. Thus, the present study carried out a meta-analysis to systemically examine 29 empirical studies (extracting 34 effect sizes) using experimental or quasi-experiments involving the programming learning effects of employing block-based visual programming tools to date (until the end of 2019). The results showed a small to medium significant positive overall mean effect size (fixed-effect model g = 0.37; random-effects model g = 0.47) of the use of these block-based visual programming tools with respect to students’ academic achievement. Furthermore, the overall mean effect size was significantly affected by the educational stage, programming tool used, experimental treatment, and school location. Discussions and implications based on the findings are provided.

Keywords

block-based visual programming meta-analysis academic achievement study features

“Learning to program” has become a popular trend in educational institutions around the world. Students of all ages are encouraged to engage in various programming activities (Scherer et al., 2019; Yildiz Durak, 2018). Previous empirical studies have pointed out that the act of learning to program can facilitate learners’ creativity and problem-solving skills, as well as promote their abilities to engage in logical and computational thinking (Bustillo & Garaizar, 2016; Deng et al., 2019; Korkmaz, 2016a; H. Y. Wang et al., 2014). Durak (2016) emphasized that students should possess basic programming skills and be able to cope with the challenges of the 21st century. However, programming is a challenging subject for students—especially novices—to perceive and learn (Baser, 2013; Durak, 2016; Kelleher et al., 2007). When beginners learn to program in the initial operating stage by utilizing text-based programming tools (e.g., Python, Java, C), overly complicated program syntax might make them feel frustrated, and, thus, reduce their enthusiasm for learning how to program (Altun & Mazman, 2012; Chang et al., 2017; Gomes & Mendes, 2007; Guzdial, 2004; Kinnunen & Malmi, 2006; Lahtinen et al., 2005; Salleh et al., 2013; Topalli & Cagiltay, 2018; Weintrop & Wilensky, 2018).

Given this fact, some block-based visual programming tools, such as Scratch, Alice, and MIT App Inventor, have been developed by dragging the visualization blocks and setting the configuration parameters within the blocks to provide an easy-to-operate editing interface for beginners to learn programming (Chang et al., 2017; Chao, 2016; Kelleher & Pausch, 2005; Lye & Koh, 2014; Xinogalos et al., 2017). When using block structure, puzzle pieces can be moved via a drag and drop function to produce a programming construction that is consistent with the constructivist concept (Buckleitner, 2007). Therefore, for example, teachers in Finland, China, Singapore, Taiwan, and South Korea, have approved that using block-based visual programming tools in their programming instruction as one of the best ways for students to learn programming (Wu et al., 2020). At present, more educational institutions are adopting some of the different block-based visual programming tools to help students learn programming, and empirical studies have reported on the different effects on academic achievement (success), which represents actual performance outcomes (Steinmayr et al., 2014) and refers to the declarative and procedural knowledge that a student acquires through teaching and learning. Several researchers have concluded that block-based visual programming tools have the potential to promote better academic achievement among students than the use of conventional text-based programming tools (Al-Linjawi & Al-Nuaim, 2010; Cetin, 2016; Cooper et al., 2003; Sykes, 2007; Weintrop & Wilensky, 2017). For instance, Sykes (2007) investigated university students’ learning outcomes after they adopted either Alice or a text-based programming tool in computer science courses; the researcher found that students using Alice had higher academic achievement than did those who used text-based programming tools. In addition, Cetin (2016) explored undergraduate students’ programming concepts and revealed that students who used Scratch performed better in programming learning than did students who used text-based programming tools. Weintrop and Wilensky (2017) conducted a quasi-experimental study in an introductory high school programming class and found that students using block-based visual programming tools demonstrated better academic achievement than did students using an isomorphic text-based programming environment. Nevertheless, some studies found that, in terms of the ability to learn programming, there was not much difference between students who adopted block-based visual programming tools and those who adopted text-based programming tools (de Kereki, 2008; Garlick & Cankaya, 2010; Kyfonidis et al., 2017; Mihci & Ozdener, 2014, 2017).

These inconsistent effects on student academic achievement between block-based visual programming tools and text-based programming tools were moderated, which may stem from differences in participants’ educational stages (Xu et al., 2019), the different block-based visual programming tools used, the different conditions of the experimental treatments (Costa & Miranda, 2017), or the different school locations. To analyze these different results, Costa and Miranda (2017) synthesized six quasi-experimental studies on learning effects for students by comparing the use of Alice and a text-based programming tool from 2000 to 2004. They found a significant medium effect size (k = 6, random-effects model d = 0.54, 95% confidence interval: 0.34 to 0.74) on student academic achievement in terms of the use of Alice. Similarly, Xu et al. (2019) examined learning outcomes between students using block-based visual programming tools and text-based programming tools based on 13 studies collected from 2000 to 2018; they discovered a small but non-significant overall effect size (k = 10, random-effects model g = 0.25, 95% confidence interval: −0.08 to 0.57) on students’ academic achievement with regard to adopting block-based visual programming tools as compared to the use of text-based programming tools. Nevertheless, these two meta-analyses may have either shown a comparison of one block-based visual programming tool (Costa & Miranda, 2017) or presented the issues of publication bias (Xu et al., 2019). Therefore, in this study, we attempted to (1) systematically examine existing research results involving the impacts of the use of block-based visual programming tools on students’ academic achievement; and (2) compare the size of the effects in different variables by using a meta-analysis approach. In particular, this study explores the moderator variables, such as the participants’ educational stage (e.g., elementary schools, middle schools, high schools, colleges/universities), the various kinds of block-based visual programming tools (e.g., Alice, Scratch, MIT App Inventor), the condition of the experimental treatment (e.g., the alternative condition or the complementary condition), and the participating school location (e.g., Asia, Europe, North America, or others). Two research questions are as follows:

RQ1: What are the overall effects of the use of block-based visual programming tools on the academic achievement of students?

RQ2: Are these effects significantly affected by educational stage, programming tool used, experimental treatment, and school location?

Literature Review

Block-Based Visual Programming Tools

Providing a simple and intuitive programming environment to alleviate the difficulty of syntactic operation in the programming learning process is very important for beginners (Mladenović et al., 2018). To this end, a variety of block-based visual programming tools, such as Alice, Scratch, and MIT App Inventor, have been developed to help beginners learn to program. These tools typically possess a visual, modular editing interface that enables learners to construct programs by dragging-and-dropping the blocks and setting parameters, rather than reciting too much specific syntax (Chao, 2016; Kelleher & Pausch, 2005; Lye & Koh, 2014). Laine and Haaranen (2018) indicated that the greatest benefit of block-based visual programming tools is that, when they are compiled, they can reduce students’ memorization of complex syntactical statements, which encourages students to focus more on basic and general programming concepts. Furthermore, block-based visual programming tools provide multiple function settings and further interface with external devices (such as Arduino and micro:bit) that enable students to create interactive games and animations, engage in storytelling, etc. (Mladenović et al., 2018). The three most representative programming tools—Alice, Scratch, and MIT App Inventor—are described below.

Alice

Alice (http://www.alice.org) is a 3D block-based visual programming tool that can help students learn fundamental programming concepts and may promote their logical and computational thinking skills. This tool was developed by Carnegie Mellon University and has been widely applied for programming instruction (Johnsgard & McDonald, 2008; Moskal et al., 2004; Sykes, 2007). Through Alice, students can build dynamic animations, narratives, and interactive games by assembling the blocks into the code window instead of code writing in plain text (Cooper et al., 2003; T. C. Wang et al., 2009).

Scratch

Scratch (https://scratch.mit.edu) is a 2D block-based visual programming tool developed by the Media Laboratory of the Massachusetts Institute of Technology (MIT). Scratch features a “low floor” to get started, a “high ceiling” to create complex projects, and “wide walls” to enable learners of different school ages to perform different programming tasks (Maloney et al., 2008; Resnick et al., 2009). It provides an intuitive and simple drag-and-drop platform with which to construct 2D animated stories, immersive narratives, digital art projects, and games.

MIT App Inventor

MIT App Inventor (https://appinventor.mit.edu/), proposed by Hal Abelson and his colleagues, is an intuitive block-based visual programming platform that can help students design and develop mobile applications (apps) such as games, social applications, and quizzes for smartphones and tablets (Zaranis et al., 2016). The user interface of MIT App Inventor has three distinctive sections: designer window, blocks editor, emulator/mobile device. In addition, MIT App Inventor offers, in the edges of the blocks, highlighted feedback indicating whether or not the blocks can be combined.

These three block-based visual programming tools have some similarities and differences in their interface operation and functions. A basic comparison is presented in Table 1 (João et al., 2019; Papadakis et al., 2014).

Table 1.

Comparison of Alice, Scratch, and MIT App Inventor.

Description	Alice	Scratch	MIT App Inventor
Usability	A little difficult	Simple	Requires familiarity
2D/3D	3D	2D	2D
User interface	One screen	One screen	Three sub-screens
Emulator	No	No	Yes
Operating environment	Computer software for Windows, macOS, or Linux	Computer software or web browser on Windows, macOS, or Linux	Computer software or web browser on Windows, macOS, or Linux
Types of projects	Stories, games, and virtual worlds	Stories, games, and animations	Functional applications for smartphones and tablets
Price	Free	Free	Free
Target group	Students from middle school (and sometimes even younger) to university	Students aged 8 to 16 (and even all ages)	All ages, especially young people

Moderator Variables in Block-Based Visual Programming

Several previous studies have used block-based visual programming tools to help students learn how to program and have achieved greater learning effects. We carried out a literature review under the four moderator variables, which are (1) educational stage; (2) programming tool used; (3) experimental treatment; and (4) school location.

Educational Stage

Block-based visual programming tools have been used in different educational stages, including elementary schools (e.g., Mladenović et al., 2018), middle schools (e.g., Price & Barnes, 2015; Tabet et al., 2016), high schools (e.g., T. C. Wang et al., 2009; Weintrop & Wilensky, 2017; Zaranis et al., 2016), and colleges/universities (e.g., Garlick & Cankaya, 2010; Mihci & Ozdener, 2014; Sykes, 2007; Topalli & Cagiltay, 2018).

Programming Tool Used

Block-based visual programming tools are currently used to help students learn how to program. These include Alice (e.g., Cooper et al., 2003; Johnsgard & McDonald, 2008; Sykes, 2007), Scratch (e.g., Cetin, 2016; Erol & Kurt, 2017; Korkmaz, 2016a, 2016b; Mladenović et al., 2018), MIT App Inventor (e.g., Mihci & Ozdener, 2014, 2017; Zaranis et al., 2016), and others.

Experimental Treatment

In several studies, block-based visual programming tools have been used in two kinds of experimental treatments to facilitate students’ learning of programming, including the alternative condition and the complementary condition. The alternative condition refers to when block-based visual programming tools are used in place of traditional text-based programming tools in programming education, while the complementary condition is when block-based visual programming tools are used together with text-based programming tools in a programming curriculum. In the alternative condition, the experimental group uses a block-based visual programming tool, while the control group uses a traditional text-based programming tool. This condition aims to explore the effects of using block-based visual programming tools as an alternative to a text-based programming tool in programming education. For example, Mihci and Ozdener (2017) applied MIT App Inventor as an alternative to text-based programming tools to teach programming concepts to undergraduate university students. They found that the block-based visual programming tool that is used has no significant effect on academic success. The experimental group in the complementary condition takes the same text-based programming classes as the control group. In addition, the experimental group is exposed to block-based visual programming, which is aimed at investigating the effects of the use of a block-based visual programming tool as a complement to a conventional text-based programming curriculum. For instance, Topalli and Cagiltay (2018) tried to improve students’ understanding of programming concepts by incorporating a block-based programming tool into a text-based introductory programming course. They found that students in the enriched introductory programming course performed better than did those in the traditional introductory programming course. While block-based visual programming tools have often been applied to two conditions, little research has compared their effectiveness.

School Location

Many schools in different regions of the world use block-based visual programming tools to teach programming. These regions include Europe (e.g., Kyfonidis et al., 2017; Mladenović et al., 2018; Ruf et al., 2014; Zaranis et al., 2016); North America (e.g., Garlick & Cankaya, 2010; Johnsgard & McDonald, 2008; Moskal et al., 2004; Price & Barnes, 2015; Sykes, 2007; Weintrop & Wilensky, 2017); Asia (e.g., Deng et al., 2019; Korkmaz, 2016a, 2016b; Mihci & Ozdener, 2014; Tabet et al., 2016), and so forth.

Method

This study used a meta-analysis approach to synthesize and evaluate the findings of several empirical studies about the effects of block-based visual programming to explore its influence on students’ academic achievement and investigate the impacts of these moderating variables. Accordingly, four procedures of the meta-analysis, including study searching, study selection, study coding, and data analysis, are performed as follows.

Study Searching

In mid-December 2019, at the corresponding author’s university, we conducted a keyword search combination (“visual programming” OR “block-based programming” OR “block-based visual programming” OR “graphic programming” OR “GUI programming”) AND (“course” OR “education” OR “instruction” OR “teaching” OR “learning” OR “pedagogy”) in five pre-selected academic databases, including ACM Digital Library, Education Resource Information Center (ERIC), IEEE Xplore, ScienceDirect, and Web of Science Core Collection. After the preliminary searching, we examined all relevant literature that may have been related to these topics in the selected databases. As a result, we obtained 1471 articles and another 14 articles from the reference lists of existing reviews and meta-analyses obtained from the above searches. In total, 1485 potential published articles were obtained as a result of the literature search.

Study Selecting

We selected the articles following the two screening criteria in Table 2. First, to determine initial eligibility for the meta-analysis, two educational researchers used the inclusion criteria to screen the titles and abstracts of 1485 articles resulting from the literature search. The rate of agreement between the two researchers was about 97.44%. The researchers dealt with their disagreements by reading and debating the content of each article on which they had not initially reached a consensus (cf. Chen & Yang, 2019). The first screening excluded 1337 articles. After the removal of 21 duplicates, 127 articles remained. Second, to judge whether the articles were suitable for inclusion in the meta-analysis, the same two researchers carefully screened the full texts of 127 articles according to the exclusion criteria. The rate of agreement between them was roughly 93.70%. The researchers resolved their disagreements by discussing the content of each article on which they had not agreed. The second screening eliminated 98 articles, leaving 29 eligible studies published from 2003 to 2019, which were used for the meta-analysis. The article selection process is portrayed in a flow chart presented in Figure 1.

Table 2.

Inclusion and Exclusion Criteria.

Inclusion criteria

• Must be experimental studies or quasi-experimental studies.
• Must be published in English.

Exclusion criteria

• The application of block-based visual programming tools was not the core variable of the study. The control group did not use traditional text-based programming tools.
• The study results were inadequate for calculating an effect size, for example, not providing the sample sizes of each group, lacking any inferential statistical results.
• The results were not presented with academic achievement (success) as a major dependent variable measured using students’ grades (by exams) and success rates in the programming curriculum.

Figure 1.

Flow Chart of the Article Selection Process.

Study Coding

Four variables, including educational stage (i.e., elementary school, middle school, high school, and college/university), programming tool used (i.e., Alice, Scratch, MIT App Inventor, and others), experimental treatment (i.e., the alternative condition and the complementary condition), and school location (i.e., Asia, Europe, North America, and others) were evaluated to determine whether they moderate the effects of block-based visual programming.

Data Analysis

Calculating Effect Sizes for Each Study

The effect sizes of each study were calculated. If a study assessed academic achievement using more than one outcome measure, we weighted its effect sizes (e.g., Cetin, 2016; Deng et al., 2019). For example, Deng et al. (2019) used computational concepts and computational practice tests as measures of students’ academic achievement. We adopted Hedges’s g as the standardized indicator of the mean weighted effect sizes for all studies because it has the greatest properties for limited samples (Borenstein et al., 2011). Additionally, to explain our study’s effect size, we applied Cohen’s criterion, in which 0.2 is small, 0.5 is medium, and 0.8 is large (Cohen, 1988). This study calculated a 95% confidence interval (CI) for Hedges’s g to test for significant differences. The Comprehensive Meta-Analysis Software (Version 2.2) was employed to calculate all statistical analyses.

Calculating the Overall Effect Sizes and Testing for Heterogeneity

The overall effect size was estimated using the fixed-effect and random-effects models. The fixed-effect model permits inferences about the studies included in the meta-analysis, while the random-effects model allows for generalizations, beyond the limited collection of studies included in the meta-analysis, to comparable studies that have been or may be carried out (Schmidt et al., 2009).

Heterogeneity was calculated using Cochran’s Q statistic and I² statistic. The Q statistic (Q_T) represents the observed dispersion in effect sizes. The I² statistic is used to evaluate how much of the variance between studies can be attributed to actual variance instead of sampling bias (Borenstein et al., 2011). If significant variation existed, further moderator analyses were needed to obtain an understanding of which moderator variables could have accounted for such heterogeneity (Higgins et al., 2003; Lipsey & Wilson, 2001).

Moderator Analysis

Moderator analysis was performed under the fixed-effect model to optimize the ability to identify correlations between moderator variables and effect sizes, as the fixed-effect model is more statistically robust than the random-effects model for testing moderator variables (Roorda et al., 2017). In addition, if the between-class variance variable Q_B showed significant variation between moderator subgroups with more than two subgroups, then post-hoc subgroup analysis was performed using the fixed-effect model.

Investigating Publication Bias

Publication bias was examined by using two methods. First, the study used Rosenthal’s fail-safe N test to calculate the number of unpublished findings needed to cause the mean effect size to become insignificant. According to Rosenthal (1991), publication bias will not affect the results of the meta-analysis if the fail-safe number N is larger than the tolerance level of 5k + 10 (where k refers to the total number of reported effect sizes in the meta-analysis). Second, this study investigated publication bias by applying the trim-and-fill method (Duval & Tweedie, 2000), which calculates the number of missing studies and generates an adjusted mean effect size by inserting the missing studies on the biased side.

Results

Effect Sizes of Each Selected Study

Table 3 presents the characteristics and calculated effect size of the 29 eligible published studies with 34 effect sizes, representing 3655 participants. Thirty effect sizes are used to analyze the academic achievement from students’ grades on exams, and the remaining four effect sizes analyze it from students’ success rates in the programming curriculum. It is noteworthy that success rates are a measure of students’ academic achievement (scores or success rate) in programming learning (Costa & Miranda, 2017). The selected studies using success rates to measure academic achievement in this meta-analysis applied experimental control group design and can be calculated to obtain the Hedges’s g effect size with the same criteria as the remaining studies measuring academic achievement using students’ grades. Among these, 22 of the 34 effect sizes (64.71%) showed statistically significant positive effects, indicating that block-based visual programming learning significantly increased students’ academic achievement as compared to traditional text-based programming learning; 2 of the 34 effect sizes (5.88%) demonstrated statistically significant negative results, indicating that students’ academic achievement in text-based programming learning significantly outperformed that of block-based visual programming learning; and 10 of these (29.41%) failed to reveal significant effects, indicating that the use of block-based visual programming tools did not have a significant effect on students’ academic achievement as compared to the use of text-based programming tools. The effect sizes of selected studies ranged from −0.50 to 1.46, which lay within three standard deviations of the overall effect size; thus, no studies were removed (Lipsey & Wilson, 2001).

Table 3.

Characteristics and Effect Sizes of Selected Studies.

Authors (Year)	No. of comparisons	No. of students	No. of the experimental group	No. of the control group	Educational stage	Programming tool used	Experimental treatment	School location	Measurement of academic achievement	Effect size (Hedges’s g)
Cooper et al. (2003)	1	21	11	10	College/University	Alice	Complementary	North America	Performance scorcs in CS1	1.46*
Moskal et al. (2004)	1	55	25	30	College/University	Alice	Complementary	North America	Grades in CS1	0.98*
Sykes (2007)	2	105	71	23	College/University	Alice	Alternative	North America	Performance scores	0.66*
				11	College/University	Alice	Alternative	North America	Performance scores	1.31*
de Kereki (2008)	2	30	15	15	College/University	Scratch	Alternative	Other (Uruguay)	Task results	0.01
		30	15	15	High school	Scratch	Alternative	Other (Uruguay)	Task results	−0.34
Johnsgard & McDonald (2008)	1	106	37	69	College/University	Alice	Complementary	North America	Success rates in programming I course	0.55*
Mullins et al. (2009)	1	1149	414	735	College/University	Alice	Alternative	North America	Pass rates	0.10
T. C. Wang et al. (2009)	1	166	81	85	High school	Alice	Alternative	Asia	Test scores	0.36*
Al-Linjawi & Al-Nuaim (2010)	1	45	24	21	College/University	Alice	Alternative	Asia	Final quiz scores	0.81*
Garlick & Cankaya (2010)	1	156	82	74	College/University	Alice	Alternative	North America	Exam grades	−0.35*
Rizvi & Humphries (2012)	1	97	44	53	College/University	Scratch	Complementary	North America	CS1 pass rates	0.58*
Rizvi et al. (2012)	1	65	23	42	College/University	Scratch	Complementary	North America	Grades in CS1	0.55*
Mihci & Ozdener (2014)	1	76	42	34	College/University	MIT App Inventor	Alternative	Asia	Exam grades	−0.50*
Ruf et al. (2014)	1	56	29	27	Middle school	Scratch	Alternative	Europe	Exam grades	−0.04
Zhang et al. (2014)	1	52	26	26	College/University	Alice	Alternative	Asia	Final scores	0.65*
Dwarika & Villiers (2015)	1	103	55	48	College/University	Alice	Complementary	Other (South Africa)	Final marks	0.23
Price & Barnes (2015)	1	31	17	14	Middle school	Other	Alternative	North America	Test scores	−0.16
Cetin (2016)	1	56	28	28	College/University	Scratch	Alternative	Asia	Achievement test scores	0.82*
Korkmaz (2016a)	1	44	20	24	College/University	Scratch	Alternative	Asia	Achievement test scores	0.78*
Korkmaz (2016b)	1	49	28	21	College/University	Scratch	Complementary	Asia	Achievement test scores	0.59*
Tabet et al. (2016)	1	141	71	70	Middle school	Alice	Complementary	Asia	Grades on final exam	0.97*
Zaranis et al. (2016)	2	87	35	18	High school	MIT App Inventor	Alternative	Europe	Test scores	1.10*
			34		High school	Scratch	Alternative	Europe	Test scores	0.33
Erol & Kurt (2017)	1	52	26	26	College/University	Scratch	Alternative	Asia	Achievement test scores	1.00*
Kyfonidis et al. (2017)	1	32	16	16	College/University	Other	Alternative	Europe	Success rates on final exam	−0.14
Mihci & Ozdener (2017)	1	101	54	47	College/University	MIT App Inventor	Alternative	Asia	Exam scores	−0.14
Weintrop & Wilensky (2017)	1	52	25	27	High school	Other	Alternative	North America	Scores in commutative assessment	0.57*
Mladenović et al. (2018)	2	207	46	98	Elementary school	Scratch	Alternative	Europe	Scores in programming skill assessment	0.81*
				63	Elementary school	Scratch	Alternative	Europe	Scores in programming skill assessment	0.40
Topalli & Cagiltay (2018)	1	322	48	274	College/University	Scratch	Complementary	Asia	Grades on final exam	0.34*
Vasilopoulos & Van Schaik (2018)	2	70	23	25	Middle school	Other	Alternative	Europe	Test scores	0.70*
				22	Middle school	Other	Alternative	Europe	Test scores	0.75*
Deng et al. (2019)	1	99	50	49	High school	Other	Alternative	Asia	Test scores	0.44*
Sum	34	3655	1515	2115

*p < .05.

Overall Effect on Academic Achievement

The results of the analysis of the overall effect size and the heterogeneity test are reported in Table 4. For the 34 effect sizes of academic achievement, the fixed-effect model meta-analysis revealed a mean effect size of 0.37 (95% confidence interval 0.30–0.44); the random-effects model analysis resulted in a mean effect size of 0.47 (95% confidence interval 0.32–0.62). The result indicated that the effect of block-based visual programming tools on students’ academic achievement is significantly better than that of traditional text-based programming tools and achieved a small to medium level in Cohen’s (1988) criterion.

Table 4.

Overall Effect Size and the Homogeneity Test.

Model	N	k	Hedges’s g	95% CI		Test of homogeneity
Model	N	k	Hedges’s g	Lower	Upper	Q_T	df	I²
Fixed-effect model	3655	34	0.37	0.30	0.44	135.55*	33	75.65%
Random-effects model			0.47	0.32	0.62

Note. k = total number of effect sizes; CI = confidence interval.

The heterogeneity test generated a Q statistic of 135.55 versus the expected value of 33. The null hypothesis test is statistically significant, with a p-value of less than 0.001. This suggests that there is a need to explain the significant variability of the effect size. The I² statistic shows that 75.65% of the observed total variance is not due to sampling errors within the same population; thus, the effects of moderator variables should be investigated. Therefore, this study conducted moderator analyses to examine the potential effect of moderator variables on students’ academic achievement in block-based visual programming learning.

Moderator Analyses

To examine possible significant differences among the studies, four moderator variables were included in moderator analyses: educational stage, programming tool used, experimental treatment, and school location. For the educational stage, elementary schools and middle schools were combined into the category of “elementary and middle schools” (i.e., grades 5 to 9) due to the presence of few effect sizes in each subgroup. Table 5 shows the distribution of moderator variables and their effect sizes under the fixed-effect model. The largest group of studies focused on university or college students (61.76%). Scratch (38.23%) was the most studied of the block-based visual programming tools, followed by Alice (35.29%). Most studies in this meta-analysis adopted block-based visual programming tools as an alternative to text-based programming tools (73.53%). Studies from Asia composed the largest proportion of studies in this meta-analysis (35.29%), followed by North America (32.35%). Four moderator variables (educational stage, programming tool used, experimental treatment, and school location) significantly influenced students’ academic achievement under the fixed-effect model, as shown in Table 5. The details are displayed in the following section.

Table 5.

Effect Sizes by Moderator Variables on Students’ Academic Achievement.

Moderator variables	k	Hedges’s g	z	95% CI	Q_B	Q_W	Contrast
Educational stage	34				14.48*	121.07*	1 > 3
1. Elementary and middle schools	7	0.64	7.56*	0.48–0.81		16.03*
2. High schools	6	0.43	4.91*	0.26–0.59		9.98
3. Colleges or universities	21	0.29	6.55*	0.20–0.37		95.06*
Programming tool used	34				15.15*	120.39*	2 > 1 > 3
1. Alice	12	0.32	6.40*	0.22–0.41		72.22*
2. Scratch	13	0.51	7.44*	0.37–0.64		21.39*
3. MIT App Inventor	3	−0.02	−0.16	−0.29–0.24		18.09*
4. Others	6	0.49	5.34*	0.31–0.66		8.69
Experimental treatment	34				11.04*	124.50*	2 > 1
1. Alternative	25	0.31	7.67*	0.23–0.39		108.70*
2. Complementary	9	0.59	7.87*	0.44–0.74		15.81*
School location	34				16.22*	119.33*	1 & 2 > 3
1. Asia	12	0.46	7.84*	0.35–0.57		45.48*
2. Europe	8	0.58	6.55*	0.41–0.75		14.33*
3. North America	11	0.25	4.62*	0.14–0.36		57.51*
4. Others	3	0.08	0.52	−0.22–0.39		2.01

Note. k = number of effect sizes; CI = conﬁdence interval. The “others” subgroups were eliminated from the subgroup comparisons.

*p < .05.

Educational Stage

The results show a medium to large effect size for elementary and middle school students (g = 0.64, z = 7.56), a small to medium effect size for high school students (g = 0.43, z = 4.91), and a small effect size for college or university students (g = 0.29, z = 6.55). The results of the heterogeneity test (Q_B = 14.48) reveal that education stage was a significant moderator variable influencing academic achievement in block-based visual programming learning. In addition, the results of post-hoc subgroup comparisons indicate that the effects of block-based visual programming learning in elementary and middle schools were significantly greater than those in colleges or universities.

Programming Tool Used

While ignoring the “others” category (including 1 using Block-C, 2 using Koios, 2 using Pencil Code, and 1 using Tiled Grace), Scratch (g = 0.51, z = 7.44) was associated with a medium significant effect size, while Alice (g = 0.32, z = 6.40) was associated with a small to medium significant effect size. However, MIT App Inventor (g = −0.02, z = −0.16) did not show a significant effect size. The results of the heterogeneity test (Q_B = 15.15) demonstrate that the programming tool used was a significant moderator variable influencing academic achievement in block-based visual programming learning. Furthermore, the results of post-hoc subgroup comparisons demonstrate that the effect size of Scratch was larger than that of Alice, whose effect, in turn, was greater than that of MIT App Inventor.

Experimental Treatment

When a block-based visual programming tool was used as an alternative to the traditional text-based programming tool (the alternative condition), the effect size was small to medium (g = 0.31, z = 7.67). By contrast, when a block-based visual programming tool was used as a complement to the programming curriculum (the complementary condition), the effect size was medium (g = 0.59, z = 7.87). The results of the heterogeneity test (Q_B = 11.04) demonstrate that the condition of experimental treatment was a significant moderating variable influencing academic achievement in block-based visual programming learning. This indicates that block-based visual programming tools had significantly outperformed alternative activities in terms of students’ academic achievement.

School Location

The results suggest a nearly medium effect size in Asia (g = 0.46, z = 7.84), a medium to large effect size in Europe (g = 0.58, z = 6.55), and a small effect size in North America (g = 0.25, z = 4.62). The results of the heterogeneity test (Q_B = 16.22) indicate that school location was a significant moderator variable influencing academic achievement in block-based visual programming learning. Furthermore, while ignoring the “others” category (including two effect sizes in Uruguay and one in the Republic of South Africa), the results of post-hoc subgroup comparisons indicate that the effect sizes of studies in the contexts of Asia (including seven effect sizes in Turkey, two effect sizes in Mainland China, one in Qatar, one in Saudi Arabia, and one in Taiwan) and Europe (including five in Greece, two in Croatia, and one in Germany) were significantly greater than those in North America (including nine effect sizes in the United States of America and two in Canada).

Evaluation of Publication Bias

The calculated fail-safe N was 1108, which was greater than the tolerance level of 180 (5 × 34 + 10), indicating that the publication bias would not impact the results of this meta-analysis, according to Rosenthal (1991). Furthermore, a fixed-effect model approach to trim-and-fill found nine missing studies on the left and adjusted the overall effect in the fixed-effect model from 0.37 to 0.24 (p < .05) and in the random-effects model from 0.47 to 0.27 (p < .05); a random-effects model approach to trim-and-fill found three missing studies on the left and adjusted the overall effect in the fixed-effect model from 0.37 to 0.34 (p < .05) and in the random-effects model from 0.47 to 0.41 (p < .05). Although the overall effect size was reduced, the results can still be statistically significant.

Discussion

The result indicated that students who learned programming with block-based visual programming tools had significantly better academic achievement than those who learned programming only with traditional text-based programming tools, with a small to medium effect size. A possible explanation is that students may perceive block-based visual programming tools as more intuitive and easier to use because these tools have simpler syntax operations as compared to those of traditional text-based programming tools (Sykes, 2007; Weintrop & Wilensky, 2015). This could save students’ time and effort in memorizing complex programming syntax and debugging syntax errors (T. C. Wang et al., 2009), as well as decrease their cognitive load (Cetin, 2016; Vasilopoulos & Van Schaik, 2018), increase their confidence (Deng et al., 2019), keep them focused on programming (Mladenović et al., 2018; Moors et al., 2018; Sykes, 2007), and enable them to perform better academically. Another possible explanation is that these block-based visual programming tools provide students with more concrete and interesting experiences (Mladenović et al., 2018), enabling them to use the program to solve different learning problems, such as math or science, or to construct games and animations, as well as to tell stories, which boosts students’ interest in programming (Moors et al., 2018; Topalli & Cagiltay, 2018). The tools also help them comprehend the deeper concepts of programming (Mihci & Ozdener, 2017; Mladenović et al., 2018) and, eventually, attain better academic achievement. These findings lead us to believe that block-based visual programming tools should be applied to enhance students’ academic achievement in programming learning.

Educational Stage

This meta-analysis shows that the majority of the studies had been conducted at the college or university stage, although block-based visual programming tools are developed mainly for young students. This result is consistent with Costa and Miranda’s (2017) finding, which showed that the majority of the empirical studies had used Alice in a university setting. A possible explanation for this is that programming was only recently integrated into K–12 compulsory education in some countries and regions (Wu et al., 2020); thus, few studies have been conducted and published in K–12 settings. Another possible reason may be that most researchers are at university and, thus, have better access to data from the college or university stage than data from other stages. The results also indicate that the effects of block-based visual programming are significantly different in different educational stages. One of the possible reasons for this might be that, as compared to college or university students, elementary and middle school students have more motivation and intention to learn programming by using block-based visual programming tools. Cheung et al. (2009) noted that learning programming with block-based visual programming tools might be interesting and enjoyable for elementary school students, but a bit too simple for middle school students and too straightforward for high school and college or university students. More specifically, the block-based visual programming tools provide limited flexibility and limited creative space and do not provide enough challenges for high-level students. Thus, these students do not encounter any problems in learning to program; as a consequence, they get bored, which may decrease their enthusiasm and desire to learn. Additionally, Mihci and Ozdener (2017) noted that some university students are not interested in block-based visual programming tools because they know that experience with such tools is not a requirement for a coding job; instead, experience with text-based programming tools is a requirement stated in employment ads. Because the effects of block-based visual programming on academic achievement were significantly better at elementary and middle schools as compared to colleges and universities, future research could focus more on the use of block-based visual programming tools in the elementary and middle school stages. Nevertheless, according to the results, students’ academic achievement in block-based visual programming is significantly better than that in traditional text-based programming, regardless of students’ stage of education.

Programming Tool Used

The use of block-based visual programming tools has a significant influence on the effectiveness of programming learning with respect to student academic achievement. According to the moderator analysis results, Scratch is more effective in boosting academic achievement in programming learning; this echoes the finding that university students viewed Scratch as the most powerful and appropriate tool for an introduction to programming for all educational stages (Xinogalos et al., 2017). The study also found that the effect size of applying Alice in programming learning (g = 0.32) was smaller than that found in Costa and Miranda’s (2017) meta-analysis, which found an effect size for Alice of 0.54 (Cohen’s d) , with the number of effect sizes being k = 6. The result may be partially explained by the fact that, in this meta-analysis, we included more studies and effect sizes. Surprisingly, this study did not show a significant positive effect of the use of MIT App Inventor on block-based visual programming learning in comparison to the use of text-based programming tools; this result must be interpreted with caution because our meta-analysis included only three MIT App Inventor articles. For example, one study, conducted by Zaranis et al. (2016), demonstrated that the use of MIT App Inventor achieved a large effect size (1.10) in comparison to the use of the text-based programming tool. We suggest that more empirical studies be used to investigate the effect of MIT App Inventor on programming learning achievement in comparison to text-based programming tools. One of the reasons for the different effect sizes of these block-based visual programming tools may be their different features. More specifically, Scratch is a game-oriented environment, Alice is a story-telling 3D animation environment, and MIT App Inventor focuses on developing mobile applications for Android devices (Xinogalos et al., 2017). Scratch and Alice might be more appealing to students. Another possible reason for the result may be differences in the target groups of these block-based visual programming tools. do Nascimento et al. (2019) reported that Scratch seems to be more suitable for all levels, especially for young students, while Alice seems to be more appropriate for the high school stage or above. Robins et al. (2003) stated that MIT App Inventor may be more suitable for a more formal introduction to programming, in which the ultimate objective is to promote programming skills and transition to traditional text-based programming. Zaranis et al. (2016) suggested beginning with Scratch in elementary-school or early-middle-school courses and then moving on with MIT App Inventor in the next classes and school stages in programming learning. Overall, MIT App Inventor and Alice are considered to be more difficult for students to learn as compared to Scratch.

Experimental Treatment

The results show that there was a small but significant effect size (k = 25, g = 0.31) and a medium effect size (k = 9, g = 0.59) when block-based visual programming tools were used as an alternative or complement, respectively, to traditional text-based programming tools. The result for the effects of block-based visual programming in the alternative condition is not consistent with the result of Xu et al. (2019), who found no significant effect size (k = 10, g = 0.25) of block-based visual programming versus text-based programming. This inconsistency may be because publication bias was detected in Xu et al.’s (2019) meta-analysis, which could have influenced the validity of their results. A future study could continue evaluating the impact of the use of a block-based visual programming tool as an alternative to traditional text-based programming. The use of block-based visual programming tools as complementary tools to traditional programming learning resulted in better academic achievement than it did among those who used it as an alternative. The result is consistent with Costa and Miranda’s (2017) finding that the use of block-based visual programming tools as complementary tools was more effective. One underlying reason for this result could be the exposure time for learning to program. Some students in the complementary courses were exposed to programming for a longer period of time (Costa & Miranda, 2017). For instance, in the study conducted by Cooper et al. (2003), students took traditional text-based programming classes. Additionally, the experimental group had concurrently or previously taken classes complementary to block-based visual programming during the first part of the semester. In the study by Sykes (2007), the experimental group had the same traditional classes as the control group in the C language but was exposed to Alice for an extra half-hour in each class. However, the exposure time, which may have had an impact on the effects of block-based visual programming on students’ academic achievement, was not analyzed in this meta-analysis because the exposure times in the complementary condition for the experimental and control groups are different in most of the included studies (e.g., Cooper et al., 2003; Johnsgard & McDonald, 2008; Moskal et al., 2004; Rizvi & Humphries, 2012; Rizvi et al., 2012).

School Location

The results show that school locations affected the effectiveness of block-based visual programming with respect to student academic achievement. It is noteworthy that the use of block-based visual programming tools was shown to be more effective for programming learning in Asia. A partial explanation for this finding is that students in Asia may emphasize their achievement scores more than other students do. However, the result must be interpreted with caution. In addition, the results indicate that the effect sizes of studies in the contexts of Asia and Europe were greater than those in North America. Again, the result should be interpreted with caution because other variables were not controlled by the subgroup comparisons. Future researchers should conduct empirical studies and further investigate the effects of school location on block-based visual programming.

Conclusion

This meta-analysis has revealed that, in general, block-based visual programming learning is valid and can lead to an improvement in students’ academic achievement as compared to text-based programming learning, with a small to medium significant overall mean effect size (fixed-effect model g = 0.37; random-effects model g = 0.47). In addition, the effect of block-based visual programming on academic achievement was significantly affected by school location, educational stage, programming tool used, and experimental treatment. More specifically, the effect of such usage was more pronounced for elementary and middle schools than for colleges or universities. The use of Scratch was more effective than the use of Alice, and the use of Alice was more effective than the use of MIT App Inventor. It helps to select the most appropriate block-based visual programming tool to support students’ learning according to students’ educational stage. The complementary condition tends to be more successful than the alternative condition in increasing students’ academic achievement. This result can guide instructors in the design of programming instruction. Future practices may pay more attention to instructional treatment. The effect of block-based visual programming (as compared to text-based programming) on academic achievement was higher for Asia and Europe than it was for North America. Researchers and educators could pay more attention to students’ educational stage, the choice of block-based visual programming tool and experimental treatment, and the participating school location.

Although these findings provide a better understanding of block-based visual programming learning and present a more concise view of the potential factors influencing its effects on academic achievement, there are several limitations. First, due to the low number of eligible studies, this meta-analysis focused on the effects of block-based visual programming only on students’ academic achievement rather than on their affective outcomes or computational thinking skills, which are often a focus of block-based visual programming in educational practices (Grover et al., 2017). Future researchers could conduct a new meta-analysis, with a sufficient number of papers, to explore the effectiveness of block-based visual programming on students’ affective outcomes or computational thinking skills. Second, we analyzed the educational stage, programming tool used, experimental treatment, and school location as moderator variables. However, the number of effect sizes in a few subgroups is a bit small, which may limit the power and generalizability of the results; some related empirical studies were excluded due to a lack of adequate statistical information with which to calculate the effect size. Third, some possible moderator variables that might influence the effectiveness of block-based visual programming instruction, such as students’ prior programming learning experience, existing programming knowledge and the type of instructional strategies (Mihci & Ozdener, 2014, 2017), were not analyzed in this study due to a lack of information about the variables in many studies. For instance, instructional strategies are considered to be essential for effective learning with block-based visual programming tools (Mihci & Ozdener, 2017), but many studies analyzed in this research were not clear about the type of instructional strategies used in the classroom. Therefore, we could not compare the effects of different kinds of instructional strategies. Further investigation of variables that potentially influence effect size can be performed only when more data are available about the possible moderator variables of effect size.

Footnotes

Acknowledgment

The authors thank Ms. Hao-Yue Jin for her assistance in analyzing the data.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was funded by the Philosophy and Social Sciences Planning Project of Zhejiang Province, China, under grant no. 18NDJC026Z, and was also supported by the teaching and scientific research project for liberal arts teachers in Zhejiang University.

Author Biographies

Yue Hu is a lecturer in the Department of Educational Technology at Hangzhou Normal University, China. Her research interests include block-based visual programming, acceptance of new technologies, and instructional design.

Cheng-Huan Chen is an assistant professor in the Department of M-Commerce and Multimedia Applications at Asia University, Taiwan. His research interests include technology-enhanced learning, computer-supported collaborative learning, and educational technology.

Chien-Yuan Su is an associate professor in the Department of Curriculum and Learning Sciences at College of Education, Zhejiang University, China. His research interests include e-learning and educational technology.

ORCID iDs

Yue Hu

Cheng-Huan Chen

Chien-Yuan Su

References

*Al-Linjawi

Al-Nuaim

(2010). Using Alice to teach novice programmers OOP concepts. Journal of King Abdulaziz University-Science, 22(1), 59–68.

Altun

Mazman

S. G.

(2012). Developing computer programming self-efficacy scale. Journal of Measurement and Evaluation in Education and Psychology, 3(2), 297–308.

Baser

(2013). Attitude, gender and achievement in computer programming. Middle-East Journal of Scientific Research, 14(2), 248–255.

Borenstein

Hedges

L. V.

Higgins

J. P.

Rothstein

H. R.

(2011). Introduction to meta-analysis. John Wiley & Sons.

Buckleitner

(2007). Why scratch is significant. Children’s Technology Review, 15(6), 17.

Bustillo

Garaizar

(2016). Using scratch to foster creativity behind bars: Two positive experiences in jail. Thinking Skills and Creativity, 19, 60–72. https://doi.org/10.1016/j.tsc.2015.08.003

*Cetin

(2016). Preservice teachers’ introduction to computing: Exploring utilization of scratch. Journal of Educational Computing Research, 54(7), 997–1021. https://doi.org/10.1177/0735633116642774

Chang

C.-K.

Yang

Y.-F.

Tsai

Y.-T.

(2017). Exploring the engagement effects of visual programming language for data structure courses. Education for Information, 33(3), 187–200. https://doi.org/10.3233/EFI-170108

Chao

P.-Y.

(2016). Exploring students’ computational practice, design and performance of problem-solving through a visual programming environment. Computers & Education, 95, 202–215. https://doi.org/10.1016/j.compedu.2016.01.010

10.

Chen

C.-H.

Yang

Y.-C.

(2019). Revisiting the effects of project-based learning on students’ academic achievement: A meta-analysis investigating moderators. Educational Research Review, 26, 71–81. https://doi.org/10.1016/j.edurev.2018.11.001

11.

Cheung

J. C.

Ngai

Chan

S. C.

Lau

W. W.

(2009). Filling the gap in programming instruction: A text-enhanced graphical programming environment for junior high students. ACM SIGCSE Bulletin, 41(1), 276–280. https://doi.org/10.1145/1539024.1508968

12.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.

13.

*Cooper

Dann

Pausch

(2003). Teaching objects-first in introductory computer science. In Proceedings of the 34th SIGCSE technical symposium on computer science education (pp. 191–195). ACM. https://doi.org/10.1145/792548.611966

14.

Costa

J. M.

Miranda

G. L.

(2017). Relation between Alice software and programming learning: A systematic review of the literature and meta-analysis. British Journal of Educational Technology, 48(6), 1464–1474. https://doi.org/10.1111/bjet.12496

15.

*de Kereki

I. F.

(2008). Scratch: Applications in computer science 1 [Conference presentation]. Thirty Eighth ASEE/IEEE Frontiers in Education Conference (pp. T3B-7–T3B-11). IEEE. https://doi.org/10.1109/FIE.2008.4720267

16.

*Deng

Lei

Zhou

Zhang

(2020). Pencil code improves learners’ computational thinking and computer learning attitude. Computer Applications in Engineering Education, 28(1), 90–104. https://doi.org/10.1002/cae.22177

17.

do Nascimento

M. D.

Félix

I. M.

Ferreira

B. M.

de Souza

L. M.

Dantas

D. L.

de Oliveira Brandão

(2019). Which visual programming language best suits each school level? A look at Alice, iVProg, and Scratch [Conference presentation]. 2019 IEEE World Conference on Engineering Education (pp. 1–6). IEEE. https://doi.org/10.1109/EDUNINE.2019.8875788

18.

Durak

(2016). Design and development of an instructional program for teaching programming process to gifted students [Unpublished doctoral dissertation]. Gazi University, Ankara.

19.

Duval

Tweedie

(2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463.

20.

*Dwarika

Villiers

M. R. R. d.

(2015). Use of the Alice visual environment in teaching and learning object-oriented programming [Conference presentation]. 2015 Annual Research Conference on South African Institute of Computer Scientists and Information Technologists (pp. 1–10). ACM. https://doi.org/10.1145/2815782.2815815

21.

*Erol

Kurt

A. A.

(2017). The effects of teaching programming with scratch on pre-service information technology teachers’ motivation and achievement. Computers in Human Behavior, 77, 11–18. https://doi.org/10.1016/j.chb.2017.08.017

22.

*Garlick

Cankaya

E. C.

(2010). Using Alice in CS1—A quantitative experiment [Conference presentation]. Fifteenth Annual Conference on Innovation and Technology in Computer Science Education (pp. 165–168). ACM. https://doi.org/10.1145/1822090.1822138

23.

Gomes

Mendes

A. J.

(2007, September 3–7). Learning to program-difficulties and solutions [Conference presentation]. International Conference on Engineering Education 2007, Coimbra, Portugal. http://icee2007.dei.uc.pt/proceedings/papers/411.pdf

24.

Grover

Basu

Bienkowski

Eagle

Diana

Stamper

(2017). A framework for using hypothesis-driven approaches to support data-driven learning analytics in measuring computational thinking in block-based programming environments. ACM Transactions on Computing Education, 17(3), 1–25. https://doi.org/10.1145/3105910

25.

Guzdial

(2004). Programming environments for novices. Computer Science Education Research, 2004, 127–154.

26.

Higgins

J. P. T.

Thompson

S. G.

Deeks

J. J.

Altman

D. G.

(2003). Measuring inconsistency in meta-analyses. BMJ (Clinical Research ed.), 327(7414), 557–560.

27.

João

Nuno

Fábio

S. F.

Ana

(2019). A cross-analysis of block-based and visual programming apps with computer science student-teachers. Education Sciences, 9(3), 181. https://doi.org/10.3390/educsci9030181

28.

*Johnsgard

McDonald

(2008). Using Alice in overview courses to improve success rates in programming I [Conference presentation]. Twenty first Conference on Software Engineering Education and Training (pp. 129–136). IEEE. https://doi.org/10.1109/CSEET.2008.35

29.

Kelleher

Pausch

(2005). Lowering the barriers to programming: A survey of programming environments and languages for novice programmers. ACM Computing Surveys, 37(2), 83–137. https://doi.org/10.1145/1089733.1089734

30.

Kelleher

Pausch

Kiesler

(2007). Storytelling Alice motivates middle school girls to learn computer programming [Conference presentation]. SIGCHI conference on Human factors in computing systems (pp. 1455–1464). ACM. https://doi.org/10.1145/1240624.1240844

31.

Kinnunen

Malmi

(2006). Why students drop out CS1 course? In Proceedings of the second International workshop on Computing Education Research (pp. 97–108). ACM. https://doi.org/10.1145/1151588.1151604

32.

*Korkmaz

Ö.

(2016a). The effect of scratch and lego mindstorms Ev3 based programming activities on academic achievement, problem solving skills and logical mathematical thinking skills of students. Malaysian Online Journal of Educational Sciences, 4(3), 73–88.

33.

*Korkmaz

, Amasya University, Faculty of Technology, Department of Computer Engineering, Amasya, Turkey (2016b). The effects of scratch-based game activities on students’ attitudes, self-efficacy and academic achievement. International Journal of Modern Education and Computer Science, 8(1), 16–23. https://doi.org/10.5815/ijmecs.2016.01.03

34.

*Kyfonidis

Moumoutzis

Christodoulakis

(2017). Block-C: A block-based programming teaching tool to facilitate introductory C programming courses. In proceedings of 2017 IEEE global engineering education conference. (pp. 570–579. IEEE.

35.

Lahtinen

Ala-Mutka

Järvinen

H.-M.

(2005). A study of the difficulties of novice programmers. ACM SIGCSE Bulletin, 37(3), 14–18.

36.

Lindberg

R. S. N.

Laine

T. H.

Haaranen

(2019). Gamifying programming education in K-12: A review of programming curricula in seven countries and programming games. British Journal of Educational Technology, 50(4), 1979–1995. https://doi.org/10.1111/bjet.12685

37.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta-analysis. Sage.

38.

Lye

S. Y.

Koh

J. H. L.

(2014). Review on teaching and learning of computational thinking through programming: What is next for K-12?. Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012

39.

Maloney

Peppler

Kafai

Resnick

Rusk

(2008). Programming by choice: Urban youth learning programming with scratch. ACM SIGCSE Bulletin, 40(1), 367–371.

40.

*Mihci

Ozdener

(2014). Programming education with a blocks-based visual language for mobile application development. In Proceedings of the 10th International Conference on Mobile Learning (pp. 149–156. ACM.

41.

*Mihci

Ozdener

(2017). Teaching GUI-programming concepts to prospective K12 teachers: MIT app inventor as an alternative to text-based languages. International Journal of Research in Education and Science, 3(2), 543–559.

42.

*Mladenović

Boljat

Žanko

Ž.

(2018). Comparing loops misconceptions in block-based and text-based programming languages at the K-12 level. Education and Information Technologies, 23(4), 1483–1500. https://doi.org/10.1007/s10639-017-9673-3

43.

Moors

Luxton-Reilly

Denny

(2018). Transitioning from block-based to text-based programming languages. In Proceedings of 2018 6th International Conference on Learning and Teaching in Computing and Engineering, LaTiCE 2018. IEEE. https://doi.org/10.1109/LaTICE.2018.000-5

44.

*Moskal

Lurie

Cooper

(2004). Evaluating the effectiveness of a new instructional approach. In Proceedings of the 35th SIGCSE technical symposium on Computer Science Education (pp. 75–79). ACM. https://doi.org/10.1145/971300.971328

45.

*Mullins

Whitfield

Conlon

(2009). Using Alice 2.0 as a first language. Journal of Computing Sciences in Colleges, 24(3), 136–143.

46.

Papadakis

Kalogiannakis

Orfanakis

Zaranis

(2014). Novice programming environments. Scratch & App Inventor: A first comparison [Conference presentation]. 2014 workshop on interaction design in educational environments (pp. 1–7). ACM. https://doi.org/10.1145/2643604.2643613

47.

*Price

T. W.

Barnes

(2015). Comparing textual and block interfaces in a novice programming environment [Conference presentation]. Eleventh annual International Conference on International Computing Education Research (pp. 91–99). ACM. https://doi.org/10.1145/2787622.2787712

48.

Resnick

Maloney

Monroy-Hernández

Rusk

Eastmond

Brennan

Millner

Rosenbaum

Silver

Silverman

Kafai

(2009). Scratch: Programming for all. Communications of the ACM, 52(11), 60–67. https://doi.org/10.1145/1592761.1592779

49.

*Rizvi

Humphries

(2012). A scratch-based CS0 course for at-risk computer science majors [Conference presentation]. 2012 Frontiers in Education Conference (pp. 1–5). IEEE. https://doi.org/10.1109/FIE.2012.6462491

50.

*Rizvi

Humphries

Major

Jones

Lauzun

(2012). A CS0 course using scratch. The Journal of Computing Sciences in Colleges, 26(3), 19–27.

51.

Robins

Rountree

(2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137–172. https://doi.org/10.1076/csed.13.2.137.14200

52.

Roorda

D. L.

Jak

Zee

Oort

F. J.

Koomen

H. M. Y.

(2017). Affective teacher–student relationships and students’ engagement and achievement: A meta-analytic update and test of the mediating role of engagement. School Psychology Review, 46(3), 239–261. https://doi.org/10.17105/SPR-2017-0035.V46-3

53.

Rosenthal

(1991). Meta-analytic procedures for social research. SAGE.

54.

*Ruf

Mühling

Hubwieser

(2014). Scratch vs. Karel: Impact on learning outcomes and motivation [Conference presentation]. Ninth Workshop in Primary and Secondary Computing Education (pp. 50–59). ACM. https://doi.org/10.1145/2670757.2670772

55.

Salleh

S. M.

Shukur

Judi

H. M.

(2013). Analysis of research in programming teaching tools: An initial review. Procedia—Social and Behavioral Sciences, 103, 127–135. https://doi.org/10.1016/j.sbspro.2013.10.317

56.

Scherer

Siddiq

Sánchez Viveros

(2019). The cognitive benefits of learning computer programming: A meta-analysis of transfer effects. Journal of Educational Psychology, 111(5), 764–792. https://doi.org/10.1037/edu0000314

57.

Schmidt

F. L.

I.-S.

Hayes

T. L.

(2009). Fixed- versus random-effects models in meta-analysis: Model properties and an empirical comparison of differences in results. The British Journal of Mathematical and Statistical Psychology, 62(1), 97–128. https://doi.org/10.1348/000711007X255327

58.

Steinmayr

Meißner

Weidinger

Wirthwein

(2014). Academic achievement. In L. H. Meyer (Ed.), Oxford bibliographies. Oxford University Press. https://doi.org/10.1093/OBO/9780199756810-0108

59.

*Sykes

E. R.

(2007). Determining the effectiveness of the 3D Alice programming environment at the computer science I level. Journal of Educational Computing Research, 36(2), 223–244. https://doi.org/10.2190/J175-Q735-1345-270M

60.

*Tabet

Gedawy

Alshikhabobakr

Razak

(2016). From Alice to Python. Introducing text-based programming in middle schools [Conference presentation]. 2016 ACM Conference on Innovation and Technology in Computer Science Education (pp. 124–129). ACM. https://doi.org/10.1145/2899415.2899462

61.

*Topalli

Cagiltay

N. E.

(2018). Improving programming skills in engineering education through problem-based game projects with scratch. Computers & Education, 120, 64–74. https://doi.org/10.1016/j.compedu.2018.01.011

62.

*Vasilopoulos

I. V.

Van Schaik

(2019). Koios: Design, development, and evaluation of an educational visual tool for Greek novice programmers. Journal of Educational Computing Research, 57(5), 1227–1259. https://doi.org/10.1177/0735633118781776

63.

Wang

H. Y.

Huang

Hwang

G. J.

(2014). Effects of an integrated Scratch and project-based learning approach on the learning achievements of gifted students in computer courses [Conference presentation]. 2014 IIAI Third International Conference on Advanced Applied Informatics (pp. 382–387). IEEE. https://doi.org/10.1109/IIAI-AAI.2014.85

64.

*Wang

T. C.

Mei

W. H.

Lin

S. L.

Chiu

S. K.

Lin

J. M. C.

(2009). Teaching programming concepts to high school students with Alice [Conference presentation]. Ninth IEEE Frontiers in Education Conference (pp. 1–6). IEEE. https://doi.org/10.1109/FIE.2009.5350486

65.

Weintrop

Wilensky

(2015). To block or not to block, that is the question: Students’ perceptions of blocks-based programming [Conference presentation]. Fourteenth International Conference on Interaction Design and Children (Vol. 85, pp. 199–208). ACM. https://doi.org/10.1145/2771839.2771860

66.

*Weintrop

Wilensky

(2017). Comparing block-based and text-based programming in high school computer science classrooms. ACM Transactions on Computing Education, 18(1), 1–25. https://doi.org/https://doi.org/10.1145/3089799

67.

Weintrop

Wilensky

(2018). How block-based, text-based, and hybrid block/text modalities shape novice programming practices. International Journal of Child-Computer Interaction, 17, 83–92. https://doi.org/10.1016/j.ijcci.2018.04.005

68.

Looi

C. K.

Multisilta

How

M. L.

Choi

Hsu

T. C.

Tuomi

(2020). Teacher’s perceptions and readiness to teach coding skills: A comparative study between Finland, mainland China, Singapore, Taiwan, and South Korea. The Asia-Pacific Education Researcher, 29(1), 21–34.

69.

Xinogalos

Satratzemi

Malliarakis

(2017). Microworlds, games, animations, mobile apps, puzzle editors and more: What is important for an introductory programming environment? Education and Information Technologies, 22(1), 145–176. https://doi.org/10.1007/s10639-015-9433-1

70.

Ritzhaupt

A. D.

Tian

Umapathy

(2019). Block-based versus text-based programming environments on novice student learning outcomes: A meta-analysis study. Computer Science Education, 29(2–3), 177–204. https://doi.org/10.1080/08993408.2019.1565233

71.

Yildiz Durak

(2018). Digital story design activities used for teaching programming effect on learning of programming concepts, programming self‐efficacy, and participation and analysis of student experiences. Journal of Computer Assisted Learning, 34(6), 740–752.

72.

*Zaranis

Orfanakis

Papadakis

S. S. J.

Kalogiannakis

(2016). Using scratch and app inventor for teaching introductory programming in secondary education. A case study. International Journal of Technology Enhanced Learning, 1(1), 1–233. https://doi.org/10.1504/IJTEL.2016.10001505

73.

*Zhang

J. X.

Liu

de Pablos

P. O.

She

(2014). The auxiliary role of information technology in teaching: Enhancing programming course using Alice. International Journal of Engineering Education, 30(3), 1–6.