Developing Educational 3D Games With StarLogo: The Role of Backwards Fading in the Transfer of Programming Experience

Abstract

The emergence of block-based environments aims to facilitate the problems caused by the abstractness of text-based languages. Recent studies generally focus on the effect of having block-based experience on programming education. This study is an attempt to observe the transfer of previous programming experiences (block-based vs. text-based) into a three-dimensional game-making environment through the use of backwards fading. In addition to observation of transfer, students' perceptions about the difficulty of practices were also investigated. Twenty-one senior university students participated in the study. They practiced through worked example, completion example, and full practice. Moreover, the comparison of the contribution of three examples, their perceived difficulty, and cognitive load has also been observed. There are four main findings that add value to the current literature. First, students having text-based programming experience had higher scores, which may be a sign of far transfer; second, completion example format was more efficient for students having block-based programming experience; third, full practice format was perceived as more difficult than either worked example or completion example; and fourth, based on the efficiency of example formats, completion example represented high efficiency for all students. However, average efficiency of all example formats has represented high efficiency for students, who had text-based programming experience.

Keywords

programming and programming languages teaching/learning strategies backwards fading completion example worked example efficient instruction game development game-based learning block-based programming text-based programming

Introduction

Playing games constitute an important part of society. Traditional games have been changing their forms as computer technologies spread through people's daily routines. Today, for the young generation, playing computer games has become a significant part of their lives and culture (Wilson, Connolly, Hainey, & Moffat, 2011). There are different game genres, and educational games are one of the most crucial ones. They can serve as appropriate tools for both formal and informal learning situations (Kirriemuir & Mcfarlane, 2004). Besides the growing popularity of playing computer games, it is believed to result in diverse benefits, such as increasing interest and motivation of students as well as reduction of teaching time and work load for instructors (Petri & Gresse von Wangenheim, 2017). The literature shows that educational games are one of the major environments that aim to help learners to learn computer programming and to adopt logical thinking skills in a more interesting manner (Backlund & Hendrix, 2013; Battistella & Wangenheim, 2016; Malliarakis, Satratzemi, & Xinogalos, 2013; Maloney, Peppler, Kafai, Resnick, & Rusk, 2008; Petri & Gresse von Wangenheim, 2017). There is a considerable amount of research investigating both educational games and programming languages (Law, 2017; Leutenegger & Edgington, 2007; Malliarakis et al., 2013; Vasilateanu, Wyrazic, & Pavaloiu, 2016).

For the past few years, in programming language education, using games has become popular. Games are used for various purposes based on game programming approaches (Fernández Leiva & Civila-Salas, 2013). One of the purposes of the integration of games into learning environments is to motivate students. Moreover, games can be used as tools for design and development of students' own games. Students complete a project to demonstrate knowledge and skill acquired during the course. From that perspective, the concept of students as designers emerges. Interfaces such as block-based programming environments combine the issues of programming, designing, and even playing games. Thanks to those environments, children have the opportunity to get to know coding concepts in their early lives. Unlike their predecessors in history, modern programming languages appeal to a wide range of people. Beyond doubt, the programming course has become one of the main courses of this century. The value of text-based programming languages has been increasing in education. However, for teaching complex subjects such as programming, there have always been many difficulties. Although learning how to code is a popular issue, developing a complete program is still a challenge, especially for novices.

The majority of youth have already become consumers of video games, but with the approach of making games for the purpose of learning, game players have started to program their own games and have begun learning to work on related software and interface design (Hayes & Games, 2008; Peppler & Kafai, 2007). Through developing games in block-based environments, students have become the designers or makers of their own products. Based on the promise of being more concrete, block-based programming uses drag-and-drop colored code blocks and creates flow charts instead of writing codes to tell a computer what to do. Students combine code blocks in a meaningful way using a mouse and receive visual and sometimes audio feedback. Code blocks having similar shapes and colors can be put together to run in a functional way. Once a meaningful set of blocks has been set, the learner can easily run and see the output instantly. Lego, Pet Park Blocks, Alice, Tinkertoy, Scratch, StarLogo series, and many more are well-known coding environments.

Recently, according to some educators, it is assumed that making games for learning is an efficient instructional strategy. There are examples in education of enabling students to design and develop their own games (Hayes & Games, 2008). The combination of programming language and game design has drawn the attention of many educators. There have been even some attempts to design a programming language-specific game engine (Serrano-Laguna, Torrente, Manero, & Fernandez-Manjon, 2014). In particular, there are many studies mostly using Scratch (Malan & Leitner, 2007; Maloney et al., 2008; Wilson et al., 2011; Wilson, Hainey, & Connolly, 2013) and other tools such as Alice 2.0 (Kelleher & Pausch, 2006); EToys (Lee, 2011); ToonTalk (Kahn, 1996); and StarLogo The Next Generation (TNG; Begel & Klopfer, 2004; Klopfer, Scheintaub, Huang, Wendel, & Roque, 2009). Game design method has offered many opportunities to promote learning in an effective way (Garris, Ahlers, & Driskell, 2002). Students are encouraged to develop their own games with the integration of their knowledge and skills.

Although the approach of students as designers seems effective to provide learners with more concrete learning experiences, how and when to use this approach is still an evolving issue in programming education. There is a general tendency to start with block-based approach for introductory concepts (Franklin et al., 2016; Grover, Pea, & Cooper, 2015). In other words, many studies have investigated learners' ability to transfer their (previous) block-based programming knowledge and skills to a text-based environment. Tabet, Gedawy, Alshikhabobakr, and Razak (2016) examined the transfer from programming with Alice (block-based) to programming with Python (text-based) and found that starting programming education with Alice helped with the transfer of knowledge, thus facilitated the adaptation to Python. Armoni, Meerbaum-Salant, and Ben-Ari (2015) investigated the use of the Scratch environment for teaching a professional text-based programming language (Java or C#) at secondary school. They observed that such an environment contributed to students' motivation in addition to learning how to write code. In a recent study, Weintrop and Holbert (2017) investigated how learners use both text-based and block-based environments using the Pencil Code programming environment. In their study, there were two groups of participants: high school and graduate. Both groups started using the block-based environment of Pencil Code. During assignments, while learners were shifting from text to blocks, they preferred to use drag-and-drop over typing, thereby preventing syntax errors. Findings also showed that learners generally shift from text to block-based modality to add new commands or edit existing commands. It is shown that novice learners tend to work in the block-based environment, while experienced learners are eager to work in the text-based environment. In short, the depth of previous experiences (novice vs. expert) might shape the transfer of learning (Franklin et al., 2016).

As in many other fields, previous knowledge and skills can shape how students develop expertise within a design–develop–code cycle. From the cognitive load theory perspective, it is expected that experts may skip many steps due to automaticity; however, novice learners' steps may tend to be more precise and slow, which results in engaged working memory (Clark, Nguyen, & Sweller, 2006). Educational transfer of learning can enable students to apply their prior knowledge and skills anytime and anywhere they need. There are two types of transfer of learning: near and far transfer. Near transfer is about procedures of a task that are done more or less the same way each time they are performed. For instance, near transfer occurs when students solve problems in math exam that is similar to the problems they have solved earlier in their homework. Far transfer is about tasks that require adapting one's skill to a new situation each time, which is more like transfer across contexts. Learning a new language or playing a new musical instrument can be shown as examples. Although the transfer is a desired state of learning, it may become complex for such sophisticated tasks as coding. Studies focusing on the transfer of learning can be found in math and other positive sciences that require problem-solving skills, and programming is one of those fields, due to the underlying algorithms. Because the ultimate aim of any instructional design is to facilitate learning, the harmony among content, learner, and the context is very important. In this way, the efficient learning conditions can be ensured. In cognitive load theory, “instructional environments that result in higher learning outcomes with less mental effort are more efficient than environments that lead to lower outcomes with greater mental effort” (Clark et al., 2006, p. 19). Therefore, performance and the mental effort spent during the performance have unique roles in efficient learning environments. In the transfer of previous experiences into new ones, the efficiency may be affected by the familiarity of the content, which in turn affects the performance.

There are techniques to facilitate transfer process: worked examples, completion examples, and full practice examples. It is known that worked examples contribute both far and near transfer (Moreno, Reisslein, & Delgoda, 2006), and the inclusion of fading can sometimes be beneficial with regard to expertise levels of learners. In a backwards fading situation, learners move from worked-out problems to full problems, and thus extraneous cognitive load reduces, especially when the cases are integrated in an adaptive way (Najar, Mitrovic, & McLaren, 2016). The role of fading in worked example is to help learners to remember what they have learned in a previous stage and apply it to their current task. One of the advantages of fading is the expectation of fewer errors. There are two kind of fading steps according to their order: backward fading and forward fading. In backward fading, learners accomplish the last one step of the first problem, the last two steps of the second problem, and so on until they succeed all the steps. In forward fading, learners accomplish the problem-solving as the first step rather than the last step, which is opposite to the backward fading process.

There are a few studies examining how students design and develop their own (educational) games and its impact on students coding skills from a cognitive load theory perspective (e.g., Tabet et al., 2016). In Song's (2015) study, there is an attempt for a special environment designed for interactive worked examples in computer science education. Vieira, Yan, and Magana (2015) used worked examples for the introductory concepts of programming and reported a considerable benefit of novice learners, which is in line with cognitive load theory. Another study based on cognitive load theory for programming education used worked examples to facilitate learning as a result of scaffolding (Salleh, Shukur, & Judi, 2018). However, none of these studies had an emphasis on the transfer of programming skills based on different previous experiences. The current study aims to observe how students create educational games both using a block-based programming language and designing their game in a three-dimensional (3D) environment. In addition, it attempts to observe how students can transfer their previous programming knowledge and skills into new situations when the backwards fading method was included. It is assumed that developing code blocks in a 3D game design environment is a new experience for students with text-based background and therefore can be a far transfer condition. On the other hand, it is not a new experience for those with block-based background because they are used to block-based interfaces, which is assumed a near transfer condition in this study. Backwards fading has been integrated into many subject areas, but the examples are very rare in programming education; therefore, this study is important in terms of shedding light on further studies on the similar subjects. Moreover, majority of the literature includes a design to observe block-based to text-based, and this study is unique in terms of comparing the transition from text-based to block-based.

In light of the reviewed literature, the current study aims to investigate the following research questions:

Does the use of the backwards fading facilitate the transfer of block-based programming experience into 3D game making in comparison with that of text-based programming experience?

Does the format of practice examples affect the 3D game-making performance of participants?

2.1. Does the format of practice examples affect the perceived difficulty?

2.2. Does the format of practice examples affect cognitive load?

Method

Using the case study approach, this study explores how students having different programming backgrounds create their 3D games in a block-based programming environment with the help of backwards fading method considering the cognitive load perspective. To observe the students' game-making process and find out the outcome, this case study includes a series of embedded units of analysis (Yin, 2009). Although the main case is the students making 3D games, their different backgrounds and the details in the backwards fading approach constitute the smaller units. The embedded design type of the current study is demonstrated in Figure 1.

Figure 1.

The embedded design of the study (adapted from Yin, 2009).

Participants

The participants of the study are Computer Education and Instructional Technology students (N = 21) enrolled to an elective course about games. The majority of them are male students (N = 19), which is quite typical for that department, who have had all completed a programming language course for two semesters. Half of them had already experienced Scratch (N = 11), which is a block-based programming environment. They knew how to design 2-D scenes and define codes for simple games with Scratch. The rest of them (N = 10) had never used Scratch or any similar block-based environment but wrote codes for simple games in Pygames. None of the participants have had ever heard about StarLogo TNG, which is a block-based environment specific to 3D game making. The participants were not selected deliberately, that is, the groups were intact. Their differences in programming backgrounds encouraged researchers to observe how they differed when learning something similar in comparison with learning something different from previously experienced ones.

Context: StarLogo TNG

StarLogo TNG, which is the next version of StarLogo, provides graphical programming language/blocks and a 3D world (Klopfer et al., 2009). It constitutes a client-based modeling and simulation software and builds on the tradition of Logo-based languages developed by the MIT Scheller Teacher Education Program and is free for all operating systems used for instructional purposes. Its main aim is to reduce barriers by making programming easy for novice learners, persuading younger learners into programming by making games, and providing with an effective environment for creating 3D games and simulations. At first, StarLogo TNG had been designed for secondary students who are primarily new and willing to learn programming. StarLogo TNG is useful not only for programming but also for the science content area. Therefore, with its easy programming environment, StarLogo TNG is also appropriate for teachers and enables both teachers and students to understand and develop their own 3D world (Begel & Klopfer, 2004). The major developments of StarLogo are StarLogoBlocks, which represent code in the shape of puzzle pieces, and Spaceland that provides a rich 3D interface for building games (Begel & Klopfer, 2004; Klopfer et al., 2009).

StarLogoBlocks is a visual programming language interface where codes are presented as in the shape of puzzles. Moreover, StarLogoBlocks is considered “an instruction-flow language, where each step in the control flow of the program is represented by a block” (Begel & Klopfer, 2004, p. 7). Blocks are placed as a categorized palette of blocks on the left side of the interface and can be replaced by dragging onto workspace (see Figure 2). Code blocks are puzzle-piece shaped and colored differently according to the programming function. One of the conveniences provided while using puzzle-piece blocks is the software itself making a “click” sound when syntactically appropriate commands are placed together. Such feedback can help beginners to overcome hesitations and shape further development.

Figure 2.

StarLogoBlocks, which contains all commands available to the programmer in a categorized palette.

Another convenience of StarLogo TNG is the output visualized within a 3D environment (Klopfer et al., 2009). The tab named Edit Terrain allows users to choose certain region and create a hill or crater in a round or square shape. Once a student clicks the edit terrain, the terrain will show a grid to assist in selecting a region for editing. When students choose the region to edit, it will be highlighted in purple. Code blocks that need to run the game are located in a tab labeled runtime. The level tab is about the stages of the game. Another tab labeled drawing consists of five drawing tools, namely, rectangle, circle, polygon, pencil, and image. When a student clicks the rectangle tool, it draws a rectangle and fills it in with the selected RGB colors. The circle tool draws a circle or oval and fills it in with the selected RGB colors. The polygon tool draws a series of connected lines. The pencil tool draws free forms. With the image tool, the student can put a 2-D image (.jpg or.png) on the terrain.

Procedures

This study was conducted in Samsun. The course title was “E-Game Based Learning.” The class met every week for 180 minutes for one semester. The students in the class created computer games using StarLogo TNG, which is client-based modeling and simulation software that teaches students and teachers how to design and create their own 3D games and simulations for understanding complex systems. StarLogo TNG is a game-making tool to motivate novice learners and allows them to begin programming immediately without requiring any level of text-based programming knowledge (Smith & Dunchan, 2011). In this study, StarLogo TNG was appropriate for all participants in terms of their prior programming knowledge. The students were provided with theoretical information about electronic/educational game for 6 weeks. During the next 2 weeks, students were given basic information about creating 3D game using StarLogo TNG via PowerPoint slides including tips about how to work with code blocks and the interface. Tasks were completed at a computer laboratory, where each student had the opportunity to work individually. All tasks were the same for all participants and provided within the same sequence by the same instructor. For the next 2 weeks, students were shown how to use some basic code blocks including procedure blocks and if-else blocks. The worked example method was applied during those 2 weeks. During the following 2 weeks, using the completion example method, the researcher showed students some other code blocks including collision block, and so forth. During the past 2 weeks, the researcher gave two game scenarios to the students and asked them to create games according to the scenario, this time using the full practice example method. One of the game scenarios was easy, and other one was more difficult. All the games were collected during these 6 weeks to evaluate them according to the rubrics created specific to each week. At the end of the semester, participants were asked to create their own games in 2 weeks, and they were collected as a final project. The procedures described earlier are summarized in Table 1.

Table 1.

Weekly Plan of the Study.

	Content	Method	Game
Week 1–6	Educational game	Presentation	–
Week 7	Introduction to StarLogo TNG	Presentation + StarLogo TNG program interface	Movement of main character
Week 8	Logic blocks: if-else	Worked example	Mario
Week 9	Logic blocks: if-else	Worked example	Treasure
Week 10	Collision block	Completion example	Running away from the bears
Week 11	Collision block	Completion example	Catching crashed cars
Week 12	Summary	Full practice	Red Riding Hood: Collecting carrots
Week 13	Summary	Full practice	Crossing bridge (collecting coins)
Week 14–16		Students' final project

Note. TNG = The Next Generation.

Instruments

The data collection instruments of this study included students' weekly activity difficulty ratings and the rubric for each game design as a sign of performance. Each student had performance scores for each condition (worked example [WE], completion example [CE], and full practice [FP]). To calculate the efficiency metric, students' self-ratings for the difficulty of the activity were gathered. At the end of each lesson, students were asked to rate the difficulty on the scale ranging from 1 (very low) to 9 (very high) as suggested by Paas (1992). To calculate efficiency metric of each example, the following equation was used (Clark et al., 2006, p. 334):

Average Performance in Z Scores - Average Difficulty Rating in Z Scores / \sqrt{2}

The assessment of the outputs was guided by rubrics, which were prepared separately for each task. All criteria were in line with the instruction and the examples. Rubrics also helped researchers to reach some information on each task such as completion status, difficulties, transfer of knowledge, and examining if these methods are appropriate for programming education. Sample of our rubric consists of six fundamental statements. Each rubric has some supplementary statements depending on the session. Each statement has three grading: not tried, incomplete, and complete. They were all graded and converted to a total point per week. An example rubric is summarized in Table 2.

Table 2.

Rubric Example.

Code blocks	Not tried	Incomplete	Completed
1. Edit breeds
2. Creating Procedure block
3. Using Logic-If block (movement)
4. Math block
5. Creating Setup block (creating characters and other objects)
5.1. Set xy and size
6. Creating Run Block
7. Terrain screen

Data Analysis

The present study includes mainly quantitative data. For the first research question, the scores of final projects were used. The mean scores were compared by running an independent t test. To see the contributions of examples' formats, we ran Friedman's analysis of variance (ANOVA) due to violated assumptions of parametric tests. For that analysis, the scores of weekly activities were used: first-second worked examples, first-second completion examples, first-second full practice examples, and one final full practice. The same analyses were done for perceived difficulty ratings across different formats. Students' difficulty ratings were transformed into z scores, and then efficiency metrics were calculated for each activity. In this way, the efficiency metrics were calculated to see how efficient the inclusion of examples was. Two researchers separately evaluated each product of the students to avoid biased scores. When there was no consensus on the scores, then two researchers discussed and evaluated again, but this rarely happened.

Findings

The Role of Backwards Fading in the Transfer of Block-Based Programming Experience Into 3D Game Making

The following hypotheses were tested to explore the first research question:

H₀: There are no significant differences between the final scores of students having text-based programming experience and those of block-based programming experience.

H_a: The final scores of students having block-based programming experience are higher than the final scores of others.

For the analyses, students' final projects, except for one student, were evaluated according to a rubric. Those scores are assumed as dependent variables, and the grouping variable was the condition of having previous experience on block-based programming. First, to test the null hypothesis, we ran an independent t test. Before the main analysis, we tested the assumption of normality and found no departures from normality (see Table 3).

Table 3.

Tests of Normality for Groups.

	Kolmogorov–Smirnov			Shapiro–Wilk
	Statistic	df	Sig.	Statistic	df	Sig.
Block-based experience	.270	10	.047	.841	10	.056
Text-based experience	.244	10	.093	.888	10	.160

Although the descriptives showed that students with a previous block-based programming experience (M = 83.5, SD = 10.55) had lower scores than the ones with text-based programming experience (M = 85.5, SD = 6.85), such a difference was not found significant with respect to t-test results, t(18) = −.503, p > .05. As a result, we fail to reject the null hypothesis. Table 4 summarizes the results of the test.

Table 4.

Independent t-Test Results.

	Block-based		Text-based
	M	SD	M	SD	t(18)	p
Final scores	83.5	10.55	85.5	6.85	−.503	.622

The slight differences in the final projects of participants can be observed in the coding blocks. The projects were evaluated according to a rubric, and the final products were not very much different from one another regarding coding accuracy, functioning, and the design quality. Table 5 includes the screenshots of coding blocks from final projects that demonstrated similar performance. Although the details of each game depend on the overall design approach, the depth of coding blocks seems very similar.

Table 5.

Comparison of Final Projects.

Sample screenshot	Score	Group
	85	Block-based
	85	Text-based
	80	Block-based
	80	Text-based

The Role of Practice Examples' Formats on 3D Game Making Performance

The following hypotheses were tested to understand the role of example formats:

H₀: There are no significant differences among the worked example, completion example, and full practice scores of participants.

H_a: There are significant differences among the worked example, completion example, and full practice scores of participants.

The grouping variable for this analysis was the conditions of worked example, completion example, and full practice, whereas the scores of students' weekly games were the dependent variables. To see the significant differences among data sets, ANOVA was an appropriate test, but before conducting the test, a group of assumptions were checked starting with the normality assumption.

The histograms and skewness–kurtosis values were examined. Some of the distributions, such as worked example, were skewed left as observed both on the histogram (see Figure 3) and skewness values. Moreover, both Kolmogorov–Smirnov and Shapiro–Wilk tests reported that all of the scores were significantly nonnormal, D_WE(21) = 0.283, D_CE(21) = 0.335, p < .001 (see Table 6). Despite the full practice scores' tendency to be normal, D_FP(21) = 0.174, p > .05, the normality assumption was violated. That is why we continued with a nonparametric alternative of ANOVA.

Figure 3.

Histogram of worked example scores.

Table 6.

Normality Tests for Example Formats.

	Kolmogorov–Smirnov			Shapiro–Wilk
	Statistic	df	Sig.	Statistic	df	Sig.
Worked example	.283	21	.000	.660	21	.000
Completion example	.335	21	.000	.761	21	.000
Full practice	.174	21	.095	.935	21	.171

The Friedman's ANOVA test is based on ranks. The results of the analysis indicated that the 3D game-making scores of students significantly changed across different example types, χ²(2) = 35.11, p < .001, and therefore we reject the null hypothesis. Table 7 summarizes the distribution ranks, and there are slight differences among mean ranks that are between 1 and 2.69. To understand the details of these differences, we conducted Wilcoxon signed-rank tests as post hoc with Bonferroni corrections (α/3 = .0167 level of significance).

Table 7.

Distribution Ranks.

	Mean rank
Worked example	2.31
Completion example	2.69
Full practice	1.00

Follow-up tests all resulted in significant values. Based on negative ranks, the scores changed from worked example to completion example conditions. The scores of game design were significantly higher in completion example (Mdn = 100) than those of worked example (Mdn = 96.5), z = −2.80, p < .017, r = −.35. There are performance samples of two participants in Figures 4 and 5. The transition seems similar across example formats, but there is a slight difference in terms of functioning accuracy in completion example. In other words, the text-based background may have facilitated the transfer; see Figure 4(b) and Figure 5(b) for comparison.

Figure 4.

(a) The worked example performance of a participant with block-based background. It shows an acceptable performance. Code blocks are not accurate, and one collision block is missing. The final product is not properly working. (b) The completion example performance of the same student. It shows the output of excellent performance. It works fine.

Figure 5.

(a) The worked example performance of a participant with text-based background. It shows an acceptable performance. Code blocks are not accurate, and one collision block is missing. The final product is not properly working. (b) The completion example performance of the same student. It shows the output of excellent performance. These code blocks deliver an accurately functioning game.

Based on positive ranks, the scores changed through full practice (Mdn = 82.5). In other words, the scores of game design were significantly lower in full practice than either those of worked example, z = −4.02, p < .001, r = −.51, or completion example, z = −4.02, p < .001, r = −.51. The results of the comparisons are summarized in Table 8.

Table 8.

Wilcoxon Signed-Ranks Results for Performance Scores.

	CE – WE	FP – WE	FP – CE
z	−2.80^a	−4.02^b	−4.02^b
Asymp. Sig. (two-tailed)	.005	.000	.000

Note. CE = completion example; WE = worked example; FP = full practice.

^aBased on negative ranks.

^bBased on positive ranks.

Unlike other conditions, the variety of performances was more obvious in the full practice condition. Figure 6 shows three different performances: a complete and well-functioning code, a complete but not functioning code, and an incomplete and nonfunctioning code. The performance gap clearly exists between Figure 6(a) and (c).

Figure 6.

(a) The code blocks are complete and accurate. The game runs correctly. (b) The code blocks need to be improved to run correctly. (c) The code blocks are incomplete, and the game does not run.

The role of practice examples' formats on perceived difficulty while making 3D games

In this section, the following hypotheses were examined:

H₀: There are no significant differences among the worked example, completion example, and full practice scores of participants' perceived difficulty.

H_a: There are significant differences among the worked example, completion example, and full practice scores of participants' perceived difficulty.

Each week, as soon as students completed and submitted their work, they were asked to rate the difficulty of weekly activities. The value ranged from 1 to 9. We calculated the average of activities for similar activities. In other words, we had two worked example activities and two ratings for each, and then calculated the average to have one value.

Some of the distributions, such as the scores for the completion example, were skewed to left on both histograms and skewness values. Moreover, either Kolmogorov–Smirnov or Shapiro–Wilk tests reported that the scores were significantly nonnormal, D_CE(21) = 0.20, D_FP(21) = 0.23, p < .05, except for worked example difficulty ratings that showed no significant departure from normality, D_WE(21) = 0.16 (see Table 9). However, we did not risk the overall results and continued with the nonparametric test Friedman's ANOVA.

Table 9.

Normality Tests for Difficulty Ratings.

	Kolmogorov–Smirnov			Shapiro–Wilk
	Statistic	df	Sig.	Statistic	df	Sig.
WE_Difficulty	.157	21	.192	.942	21	.236
CE_Difficulty	.199	21	.029	.891	21	.023
FP_Difficulty	.230	21	.005	.912	21	.059

Note. CE = completion example; WE = worked example; FP = full practice.

The results of the Friedman's ANOVA generated meaningful differences across formats. The mean ranks already showed differences in ranks, which are between 1.31 and 2.95 (see Table 10), but to decide the significance, just observing different rankings is not enough. The test statistics for this case showed that the perceived difficulty scores of students significantly changed across different example types, χ²(2) = 31.63, p < .001, and therefore we reject the null hypothesis.

Table 10.

Distribution Ranks.

	Mean rank
Worked example difficulty	1.74
Completion example difficulty	1.31
Full practice difficulty	2.95

Wilcoxon signed-rank tests were used as post hoc with Bonferroni corrections (α/3 = .0167 level of significance). It was found that students perceived the full practice (Mdn = 5.50) as significantly more difficult than either worked example (Mdn = 3.00, z = −4.03, p < .001, r = −.88) or completion example (Mdn = 2.50, z = −3.991, p < .001, r = −.87) based on negative ranks. On the other hand, based on positive ranks, it can be inferred that students did not find either worked examples or completion examples more difficult in comparison with each other, z = −2.25, p > .017, r = −.45. Table 11 summarized the results of post hoc results.

Table 11.

Wilcoxon Signed-Ranks Results for Difficulty Ratings.

	CE – WE	FP – WE	FP – CE
z	−2.254^a	−4.025^b	−3.991^b
Asymp. Sig. (two-tailed)	.024	.000	.000

Note. CE = completion example; WE = worked example; FP = full practice.

Based on positive ranks.

Based on negative ranks.

The role of practice examples' formats on cognitive load

In this research question, we tried to understand the cognitive load students experienced during different formats of examples of game making. For this purpose, we used both performance scores (the average scores for each example type) and difficulty ratings (the average ratings for each example type). To calculate cognitive load, an efficiency metric was used. We transformed all values into z scores and then calculated efficiency values. The efficiency metric for WE represents low efficiency with average performance value of 5 × 10⁻⁷ and an average difficulty value 1 × 10⁻⁶. In contrast, the metric for CE represents high efficiency with average performance value of 3 × 10⁻⁶ and an average difficulty value 2 × 10⁻⁶. However, the metric for FP represents efficiency on line E = 0.00 with equal average performance value and an average difficulty value of 5 × 10⁻⁷. In other words, full practice condition has neither high nor low efficiency. For an intervention to be highly efficient, the performance should be high, whereas the mental load should be low. Considering the placement of efficiency values within the efficiency graph, the WE condition can be interpreted as low, CE as high, and FP as neutral in terms of efficiency. The summary of the efficiency scores can be found in Table 12 and also in Figure 7. Moreover, the averages of all practice problems of two groups were taken and are compared. Cognitive loads of all practice problems were taken into consideration for two different groups of students: Group 1 students who had experience with block-based programming language (Scratch), and Group 2 students who only had experience with text-based programming language (Python). The efficiency value of all practice problems of Group 2 was higher than the efficiency value of other group.

Table 12.

Efficiency Metric.

Example format	Performance	Mental effort	Efficiency
Worked example	5 × 10⁻⁷	1 × 10⁻⁶	−14 × 10⁻⁴
Completion example	3 × 10⁻⁶	2 × 10⁻⁶	−1 × 10⁻³
Full practice	5 × 10⁻⁷	5 × 10⁻⁷	0.00

Figure 7.

Efficiency metrics for (a) worked example, (b) completion example, (c) full practice, (d) Group 1: block-based, and (e) Group 2: text-based.

Discussion

To achieve acquisition of a new skill, the role of practice is the key. There are many ways to increase the efficiency of practice, and fading is one of the effective ones. According to Renkl, Atkinson, Maier, and Staley (2002), fading from worked example to problem-solving produced reliable effects on near transfer but not on far transfer items within computer-based environments. Moreover, the fading is more beneficial when the worked-out solution steps are removed in a backwards manner (Atkinson, Renkl, & Merrill, 2003). The same approach was also found effective in different disciplines such as physics education (Renkl, Atkinson, & Maier, 2000). In this study, we tried to understand the role of backwards fading in the transfer of previous programming experience into block-based 3D game-making environment. We observed the transferring process of programming experience in two different groups of students with different prior knowledge of programming. From the cognitive load perspective, we expected that students with block-based experience could easily adapt the StarLogo's block-based environment due to being familiar with a similar environment. Therefore, we expected better performance of those students. The results, however, did not support this hypothesis. Instead, we found that the students having text-based programming experience had higher scores than the final scores of others, which we assumed as a far transfer case. Renkl et al.'s (2002) study suggests that the backwards fading condition produced more accurate solutions on far transfer problems, and this is parallel to our findings.

Students with different programming backgrounds were exposed to three example formats in the same sequence: worked, completion, and full practice. Students' scores on each example format were compared. The results showed that students' performances significantly changed across different example types. The scores in completion example were significantly higher than those of worked example. Effectiveness of example format in educational settings depends upon the learners' level of prior knowledge (Franklin et al., 2016; Reisslein, Atkinson, Seeling, & Reisslein, 2006). For novice learners, studying with worked examples helps them to experience or form relevant problem-solving schemas, which help them to gain expertise as a result (Renkl & Atkinson, 2003; Vieira et al., 2015). However, for learners who already have some schema in their long-term memory, studying with worked examples can produce no benefit (Kalyuga, Ayres, Chandler, & Sweller, 2003). However, as knowledge increases, completion example solving becomes the more effective learning activity. For more knowledgeable learners, worked example becomes the source of extraneous cognitive load. This phenomenon is called expertise reversal effect. However, full practice scores of game design were significantly lower than either worked example or completion example. It is clear that full practice format requires more effort.

In this study, students rated difficulty on a 1 to 9 scale each week. The rating scale technique can be regarded as a valuable research tool for estimating cognitive burden in instructional research (Paas, 1992). The hypothesis was to investigate whether students' perceived difficulties were affected by any of the example formats. Depending on the difficulty of example formats, we expected the full practice format would be perceived as the most difficult one. According to participants of this study, the full practice was more difficult than either worked example or completion example. Students reported very high levels of mental effort required during full practice and got low scores on their full practice activity. This result might be because of high demands of higher order cognitive processes such as computational thinking, problem-solving, and design thinking simultaneously. However, there was no significant difference on students' perceived difficulties between worked example and completion example. Worked example provides learners with full guidance needed for problem-solving. Completion example also provides learner with solution steps, yet one or more solution steps are omitted. Therefore, it requires no/very little mental effort (Skudder & Luxton-Reilly, 2014). As students gained the acquisition of a new skill with the help of worked example, solving problem with completion example format would be easier than solving the whole problem with no guidance. Full practice requires learners to complete the full solution on their own. One of the reasons that students' perceived difficulties differ between worked example and completion example may result from their prior knowledge about block or text-based programming language. Moreover, whether students are novice or not, providing them with full practice format diminished their motivation because relatively more cognitive effort was required for that format. They might felt overwhelmed to finish the codes in a limited amount of time.

The results regarding the measures of cognitive loads revealed that efficiency value of completion example was the highest among all other example formats. However, the efficiency value of worked example was computed as low and that of full practice was computed as neutral. This finding is not surprising because the completion examples guide both novice and experts, that is, it is a kind of balance point throughout the gain of expertise. Worked examples may be redundant for experts, and full practice can be too challenging for the novices. According to flow theory (Csíkszentmihályi, 2008), the challenge and skills should be balanced; otherwise, one can easily give up either for feeling anxious or boredom. The same point is valid for efficiency of learning in different example formats. The worked examples may bring about boredom, whereas the full practice may result in high anxiety. In this case, the completion examples can be attributed as the flow experience. In short, to keep learners engaged and let them learn efficiently in a block-based programming environment, providing completion examples appealing to the level of learners should be fulfilled. On the other hand, students with text-based programming background benefitted from all example formats more than the other students. This highlights the importance of the quality of prior experiences in programming education (Armoni et al., 2015). Although the hypothesis was built upon the ease of block-based to block-based transfer, the results were in favor of text-based to block-based transfer. The results need more research on this area because the observations in the literature generally focus on the transfer of block-based to text-based.

Implications

It is known that the design of any instructional message is important. In our study, the examples designed in a full practice format were perceived as difficult, whereas the completion examples served in a highly efficient manner. Based on these findings, instructors can be advised to include more completion examples, especially to enable far transfer. Moreover, worked examples can be skipped for students having any type of background programming knowledge. The design of the full practice examples should be more appealing to prevent prejudice. Once the automaticity of steps was ensured, building the whole blocks may not be scary for learners; therefore, the transition from worked examples through full practice should be designed more smooth in terms of time and the level of steps. To decrease students' perceived mental effort level, variety of worked and completion example can be enriched.

Limitations

It is important to indicate that the findings of this study are limited to similar students in similar learning environments. One should be cautious to generalize the results due to relatively small size of sampling. The participants of this study are the senior university students of Computer Education and Instructional Technology who enrolled in an E-Game Based Learning course. Although this study is built upon students' prior programming experiences (block-based vs. text-based), their levels of programming skills were not evaluated by assuming that they had already completed introductory programming courses. Another important limitation of this study is time. The fact that students' full practice scores were low might result from an insufficient amount of time spent during worked example and completion example sessions because the schedule of full practice (constructing the whole solution from the first step through the last one) is more demanding in nature. The variety of examples provided during this study was limited, especially the completion examples. Finally, the students self-rated the difficulty levels of the tasks, which may result in biased results.

Recommendations for Future Studies

Participants of this study practiced different types of example formats throughout the study. Students' scores of completion example displayed results higher than their scores of worked example. However, their scores of full practice showed results that were significantly lower than either worked example or completion example. These findings suggest that for students with prior (block-based or text-based) programming skill, worked example became extraneous. However, to achieve sufficient knowledge transfer to accomplish full practice examples, duration of completion example sessions can be increased. Duration can be beneficial for achieving sufficient schemas to be able to accomplish full practice. In findings regarding students' perceived difficulty, full practice was also more difficult than either worked example or completion example. Full practice format was assumed as the most difficult format in this study because it requires high/different level of mental effort simultaneously. However, there was no significant difference on students' perceived difficulties between worked example and completion example. These findings suggest that to decrease students' perceived mental effort level, the variety of worked and completion examples can be enriched. Moreover, the duration of completion example sessions can be increased to make sure of whether students have sufficient skill of solving problems with no guidance. The results regarding the measures of cognitive loads that are based on efficiency metrics revealed that the efficiency value of all practice problems of Group 2 was higher than the efficiency value of the other group. Findings suggested that future studies could be conducted without the worked example format for nonnovice learners. An overall suggestion for further studies is to provide sufficient completion examples to students, who have prior knowledge and skill. This practice could help determine whether students achieved enough knowledge and skill to complete the full practice with less guidance.

Conclusion

The value of games in education is well known. In this study, students became designers and developers of their own educational games instead of being consumers of the ones already developed. Because the participants had prior programming knowledge, students' ability to transfer their prior knowledge was observed. During the game-making process, different example formats were used: worked example, completion example, and full practice. The evidence suggests certain worked example techniques with backwards fading effect are an improvement over conventional problem-solving techniques, in terms of learning time and performance on far transfer for novices in block-based programming environments. In situations, where the student is not a novice, the worked example format appears more to be redundant. However, providing students with completion examples appears to be a more effective example format. For full practice problems, it is assumed that providing students with sufficient time and variety of examples can lead to better learning outcomes.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Nomin Boldbaatar recently graduated with master’s degree from computer education and instructional technologies department at Ondokuz Mayıs University, Turkey. She works as an ICT teacher at Ulaanbaatar Elite International School in Ulaanbaatar, Mongolia. Cognitive load theory and computer science education are her main research topics.

Emine Şendurur completed her PhD at Middle East Technical University, Turkey. She recently works as a instructor at computer education and instructional technologies department at Ondokuz Mayıs University, Turkey. Human–computer interaction, informal learning, social networking sites, and instructional message design are among her main research interests.

References

Atkinson

R. K.

Renkl

Merrill

M. M.

(2003) Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-outs steps. Journal of Educational Psychology 95(4): 774–783.

Armoni

Meerbaum-Salant

Ben-Ari

(2015) From Scratch to “real” programming. ACM Transactions on Computing Education (TOCE) 14(4): 25.

Backlund, P., & Hendrix, M. (2013, September). Educational games -Are they worth the effort?: A literature survey of the effectiveness of serious games. Paper presented at the Games and Virtual Worlds for Serious Applications Conference, Poole, England.

Battistella

Wangenheim

C. V.

(2016) Games for teaching computing in higher education – A systematic review. IEEE Technology and Engineering Education 9(1): 8–30.

Begel

Klopfer

(2007) Starlogo TNG: An introduction to game development. Journal of E-Learning 53: 146.

Clark

R. C.

Nguyen

Sweller

(2006) Efficiency in learning: Evidence-based guidelines to manage cognitive load, San Francisco, CA: Pfeiffer.

Csíkszentmihályi

(2008) Flow: The psychology of optimal experience, New York, NY: Harper Perennial.

Fernández Leiva

A. J.

Civila Salas

A. C.

(2013) Practices of advanced programming: Tradition versus innovation. Computer Applications in Engineering Education 21(2): 237–244.

Franklin, D., Hill, C., Dwyer, H. A., Hansen, A. K., Iveland, A., & Harlow, D. B. (2016, February). Initialization in Scratch: Seeking knowledge transfer. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (pp. 217–222). ACM.

10.

Garris

Ahlers

Driskell

J. E.

(2002) Games, motivation, and learning: A research and practice model. Simulation & Gaming 33(4): 441–467.

11.

Grover

Pea

Cooper

(2015) Designing for deeper learning in a blended computer science course for middle school students. Computer Science Education 25(2): 199–237.

12.

Hayes

E. R.

Games

I. A.

(2008) Making computer games and design thinking: A review of current software and strategies. Games and Culture 3(3–4): 309–332.

13.

Kahn

(1996) ToonTalk: An animated programming environment for children. Journal of Visual Languages and Computing 7: 197–217.

14.

Kalyuga

Ayres

Chandler

Sweller

(2003) The expertise reversal effect. Educational Psychologist 38(1): 23–31.

15.

Kelleher, C., & Pausch, R. (2006). Lessons learned from designing a programming system to support middle school girls creating animated stories. In IEEE Symposium on Visual Languages and Human-Centric Computing, 2006. (pp. 165–172). IEEE.

16.

Kirriemuir

McFarlane

A. E.

(2004) Literature review in games and learning, Bristol, England: Nesta Futurelab.

17.

Klopfer

Scheintaub

Huang

Wendel

Roque

(2009) The simulation cycle: Combining games, simulations, engineering and science using StarlogoTNG. E-Learning 6(1): 71–96.

18.

Law, R. (2017, October). Teaching programming using computer games: A program language agnostic approach. In M. Pivec & J. Gründler (Eds.) European Conference on Games Based Learning (pp. 368–376). Academic Conferences International Limited: UK.

19.

Lee

Y. J.

(2011) Empowering teachers to create educational software: A constructivist approach utilizing Etoys, pair programming and cognitive apprenticeship. Computers & Education 56(2): 527–538.

20.

Leutenegger

S. T.

Edgington

(2007) A games first approach to teaching introductory programming. ACM SIGCSE Bulletin 39(1): 115–118.

21.

Malan, D. J., & Leitner, H. H. (2007). Scratch for budding computer scientists. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education (pp. 223–227). ACM: Kentucky, USA.

22.

Malliarakis

Satratzemi

Xinogalos

(2013) A holistic framework for the development of an educational game aiming to teach computer programming. Academic Conferences International Limited 1: 359–368.

23.

Maloney, J., Peppler, K., Kafai, Y. B., Resnick, M., & Rusk, N. (2008). Programming by choice: Urban youth programming learning with Scratch. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (pp. 367–371). New York, NY: ACM. doi:10.1145/1352135.1352260.

24.

Moreno, R., Reisslein, M., & Delgoda, G. M. (2006, October). Toward a fundamental understanding of worked example instruction: Impact of means-ends practice, backward/forward fading, and adaptivity. In Frontiers in Education Conference, 36th Annual (pp. 5–10). IEEE.

25.

Najar

A. S.

Mitrovic

McLaren

B. M.

(2016) Learning with intelligent tutors and worked examples: Selecting learning activities adaptively leads to better learning outcomes than a fixed curriculum. User Modeling & User-Adapted Interaction 26(5): 459–491.

26.

Paas

F. G.

(1992) Trainig strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology 84(4): 429–434.

27.

Peppler

Kafai

Y. B.

(2007) What videogame making can teach us about literacy and learning: Alternative pathways into participatory culture. In: A. Baba (ed.) Situated play: Proceedings of the Third International Conference of the Digital Games Research Association (DiGRA), Tokyo, Japan: The University of Tokyo, pp. 369–376.

28.

Petri

Gresse von Wangenheim

(2017) How games for computing education are evaluated? A systematic literature review. Computers & Education 107: 68–90.

29.

Reisslein

Atkinson

R. K.

Seeling

Reisslein

(2006) Encountering the expertise reversal effect with a computer-based environment on electrical circuit analysis. Learning & Instruction 16(2): 92–103.

30.

Renkl, A., Atkinson, R. K., & Maier, U. H. (2000). From Studying Examples to Solving Problem: Fading Worked-Out Solution Steps Helps Learning. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 22, No. 22).

31.

Renkl

Atkinson

R. K.

Maier

U. H.

Staley

(2002) From example to problem solving: Smooth transitions helps learning. Journal of Experimental Education 70(4): 293–315.

32.

Renkl, A., & Atkinson, R. K. (2003). Structuring the transition from example study to problem solving in cognitive skill acquisition: A cognitive load perspective. Educational psychologist, 38(1), 15–22.

33.

Salleh

S. M.

Shukur

Judi

H. M.

(2018) Scaffolding model for efficient programming learning based on cognitive load theory. International Journal of Pure and Applied Mathematics 118(7): 77–83.

34.

Serrano-Laguna, Á., Torrente, J., Manero, B., & Fernandez-Manjon, B. (2014). A game engine to learn computer science languages. In Frontiers in Education Conference (FIE), 2014 IEEE (pp. 1–8). IEEE: Madrid, Spain.

35.

Skudder, B., & Luxton-Reilly, A. J. (2014). Worked examples in computer science. In J. Whalley, & D. D'Souza (Eds.), Proceedings of the Sixteenth Australasian Computing Education Conference – Volume 148, Vol. 148 (pp. 59–64). Auckland, New Zealand.

36.

Smith

V. A.

Duncan

(2011) Biology students building computer simulations using StarLogo TNG. Bioscience Education, Special Edition 18(1): 1–9.

37.

Song Y. (2015). An authoring and presentation environment for interactive worked examples (Doctoral dissertation). University of Glasgow, Scotland.

38.

Tabet, N., Gedawy, H., Alshikhabobakr, H., & Razak, S. (2016). From Alice to Python. Introducing text-based programming in middle schools. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (pp. 124–129). New York, NY, USA: ACM.

39.

Vasilateanu, A., Wyrazic, S., Pavaloiu, B. (2016). A science fiction serious game for learning programming languages. Proceedings of ELSE 2016 The 12th International Scientific Conference eLearning and Software for Education (pp. 561–564), Bucharest: Editura Universităţii Naţionale de Apărare ,,Carol I”.

40.

Vieira

Yan

Magana

A. J.

(2015) Exploring design characteristics of worked examples to support programming and algorithm design. Journal of Computer Science Education 6(1): 2–15.

41.

Weintrop, D., & Holbert, N. (2017). From blocks to text and back: Programming patterns in a dual-modality environment. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (pp. 633–638). ACM: Washington, USA.

42.

Wilson, A., Connolly, T., Hainey, T., & Moffat, D. (2011). Evaluation of introducing programming to younger school children using a computer game making tool. In M. Meimaris & D. Gouscos (Eds.) Proceedings of the Fifth European Conference on Games Based Learning (pp. 639–649). Dechema e.V: Greece.

43.

Wilson

Hainey

Connolly

T. M.

(2013) Using Scratch with primary school children: An evaluation of games constructed to gauge understanding of pragramming concepts. International Journal of Game-Based Learning 3(1): 93–109.

44.

Yin, R. K. (2009). Case study research: Design and methods (applied social research methods). London and Singapore: Sage.