Abstract
This article discusses the design and implementation of a new programming tool for Greek novices as a means to improve introductory programing instruction in Greece. We implemented Koios, a new highly interactive and visual programming tool for Greek novices, based on the body of research in the field of psychology of programming. The main contribution of this article is the empirical demonstration of the benefit of this tool in novice programming, compared with two other popular programming tools for Greek novices. The results show that users of Koios performed significantly better than users of the other two programming tools.
Introduction
Learning to program is an important area of computer literacy included in the majority of curricula worldwide. All levels of programming instruction involve devising programming solutions to problems and implementing them into programs using a programming language (PL) and, ideally, an integrated development environment (IDE; Winslow, 1996). However, learning and teaching programming is notoriously reported in literature as a very difficult task, especially for novice programmers (Robins, Rountree, & Rountree, 2003). To alleviate some of novice programmers’ difficulties, the design of educational PLs and IDEs aims to better support novices in program creation than professional PLs and IDEs. Taking into consideration, principles that are identified and informed by research in the psychology of programming and other relevant fields could offer guidelines and rules of thumb for producing educational programming software which could provide a supportive learning environment and better facilitate novices in learning how to program.
However, little research has been conducted in this field in Greece, with few exceptions (Efopoulos, Dagdilelis, Evangelidis, & Satratzemi, 2005; Kordaki, 2010; Vrachnos & Jimoyiannis, 2008). To help Greek novices, complete their first programming course, we designed and implemented a new Greek programming tool called Koios for teaching and learning introductory programming. The first part of this article discusses the design and implementation of Koios, which was substantially inspired and heavily influenced by the theories of constructivism and cognitive load theory (CLT) as well as by reviewing novice programmers’ difficulties and previous work in the field. Koios’ design and implementation also followed a user-centered approach because developing a software tool that helps and supports its users in efficiently accomplishing tasks is particularly important, especially for complex and challenging tasks, like programming. More specifically, Koios’ distinctive combination of characteristics (e.g., textual code automatic generation, error prevention, informative feedback, and constant guidance) supports its users in overcoming common difficulties novices face and enabling them to focus on the task of programming. In this context, Koios’ design was inspired by similar software such as BACCII (Calloni, Bagert, & Haiduk, 1997), jGRASP (Hendrix, Cross, & Larry, 2004), ALVIS Live! (Hundhausen & Brown, 2007; Hundhausen, Farley, & Brown, 2009), Scratch (Resnick et al., 2009), Alice (Bishop-Clark, Courte, Evans, & Howard, 2007), JEliot (Ben-Ari et al., 2011), and Ville (Rajala, Laakso, Kaila, & Salakoski, 2008). The second part of the article presents the empirical evaluation of Koios and demonstrates the benefits of this tool on learning outcomes in comparison with two popular programming tools for Greek novice programmers. The study involved assessing the programming performance (end-test scores) of three groups of novice programmers, each using one of the three tools in a practical test at the end of their introductory course.
The rest of the article is organized as follows: The next section presents the design and implementation of Koios, which is followed by the section that reports the empirical study used for the evaluation of Koios and its results, then a further section that discusses the study findings, and the final section that concludes this article.
Koios—A New Greek Visual Programming Tool for Greek Novices
Koios was named after a titan from Greek mythology, who was the titan god of intelligence and an eager learner. This section briefly describes the conceptual background, design, and development of Koios, but more details regarding Koios can be found elsewhere (Vasilopoulos, 2014).
Theoretical Background of Koios
Koios’ design and development was theoretically supported by two learning theories, namely constructivism and CLT. The two theories together with the most-often cited difficulties that novices face in introductory programming and the existing body of work in the field formed the conceptual framework that inspired and guided Koios’ design and development.
Constructivism is a theory introduced by Piaget in 1967, and its claim point is that each individual constructs new knowledge differently (Ackermann, 2001; Crook & Sutherland, 2017; Kay & Kibble, 2016). According to constructivism, due to the lack of an objective truth to which all individuals can refer for acquiring new knowledge, new knowledge is constructed based on already-acquired knowledge and previous experiences that are actively and repeatedly used in this construction (Sjøberg, 2007). Constructivist views emphasize the active role of the learner in the educational environment and the importance of exploratory learning. Schema, the basic building block of knowledge used in this process, is an internal hierarchical cognitive structure that represents a set of attributes of a concept from the real world, such as information, the vocabulary, the actions, and the experiences related to that specific concept.
New experiences and information combined with previous knowledge dynamically add to (process of assimilation) or modify (process of accommodation) existing schemata. Therefore, the two processes construct and modify learners’ knowledge according to their experiences and perception of the world.
CLT was introduced by John Sweller (1988) and postulates that the learning process is based on three types of memory (sensory, working, and long term) and three types of cognitive load (intrinsic, extraneous, and germane), which are imposed on working memory by learning processes according to the learner’s level of expertise. However, the latest revision of the theory distinguishes only between intrinsic and extraneous cognitive load; depending on the type of load working-memory resources are allocated to, they are characterized as germane resources (intrinsic load) and extraneous resources (extraneous load; Sweller, Ayres, & Kalyuga, 2011).
Sensory memory receives all sensory stimuli from the outside world and are retained for only a small fragment of time and, if not transferred to working memory, are considered lost. Working—or short term—memory is hypothesized to be the place where all conscious processing of information occurs and to have a limited capacity. However, Paas and Sweller (2012) argued that these limitations are crucial only when novel biologically secondary information is processed. Biologically secondary information is cultural knowledge that humans were not specifically evolved into acquiring and originates from social or cultural environments, for example, reading or solving problems. 1 Long-term memory is believed to process information unconsciously, has a virtually limitless capacity, and stores processed information in the form of schemata.
Intrinsic cognitive load is the load imposed on working memory by the nature and structure of the subject taught and is considered to be manageable but not amenable to change. The main source of intrinsic cognitive load is the element interactivity of the subject, which is determined by the learners’ expertise and the number of elements of a task or subject that can be successfully learned without needing to refer to their relation with other elements (Sweller, van Merriënboer, & Paas, 1998). The cognitive load produced by the instructional material is called extraneous cognitive load. Sweller (2010) attributed extraneous cognitive load to element interactivity of instructional material and methods. This formulation seems to propose element interactivity as a common basis for explaining both intrinsic and extraneous cognitive load.
CLT can be used to support learning by freeing up memory resources devoted to processing extraneous cognitive load in order to reallocate them, as germane resources, to processing intrinsic cognitive load; this can be achieved through instructional design (Caspersen & Bennedsen, 2007; Sweller et al., 2011). A number of techniques that have been demonstrated to aid learning through the application of CLT in various disciplines include (a) the goal-free effect, (b) the worked-example effect, (c) the completion problem effect, (d) the split-attention effect, (e) the modality effect, (f) the redundancy effect, and (g) the expertise reversal effect (Morrison, 2015, 2017; Morrison, Decker, & Margulieux, 2016; Paas & van Merriënboer, 1994; Sweller et al., 1998; van Merriënboer & Sweller, 2005; van Mierlo, Jarodzka, Kirschner, & Kirschner, 2011).
Design of Koios
Motivation for the creation of Koios
Many modern educational programming environments, like Scratch (Resnick et al., 2009), Alice (Bishop-Clark et al., 2007), and Greenfoot (Kölling, 2010), have been influenced (indirectly) by principles of constructivism, although constructivism is often not explicitly referenced as part of their design rationale. Based on CLT, other environments for novice programmers like CORT (Garner, 2009), ReadJava simulator (Williams, 2014), XLogoOnline (Hromkovič, Serafini, & Staub, 2017) were designed explicitly to reduce cognitive load on its users. Moons and De Backer (2013) combined principles of these two learning theories in the design of their interactive learning environment for introductory programming, and its evaluation showed that it facilitated its users in understanding programming constructs most students find difficult.
There is a generation of modern educational programming tools, like Scratch or Alice, which are used for teaching and learning programming worldwide, targeting primarily an English-speaking audience with available versions of these tools in other languages, like Greek. However, we were motivated to design and implement our own programming tool especially tailored for Greek novice programmers by the following four reasons. First, because of the positive findings of Moons and De Backer (2013) and the fact that, at least to our knowledge, an educational programming tool designed based on principles of both these two theories has not been used for introductory programming in Greece. Second, based on CLT, we argue that extraneous cognitive load could be reduced by providing a less complex IDE (Mason, Cooper, Simon, & Wilks, 2015) and a smaller set of commands can be beneficial for users (Koulouri, Lauria, & Macredie, 2014). Third, we intended users to focus on learning the mechanics of programming (Williams, 2014) by providing a more transparent and detailed view of program execution, which is not supported by existing IDEs. Fourth, at the time of the study, educational programming environments that were popular elsewhere were not in use in Greece. This was also the reason for deciding to include two programming tools that were popular for programming instruction in Greece at the time of the study.
In addition to the two theories, the design process was also influenced by previous studies and educational programming software relevant to novice programming. Thus, a number of design choices was inspired by or based on recommendations that were proposed in the work of Pane and Myers (1996), Green and Petre (1996), McIver (2000), Kelleher and Pausch (2005), Mannila and de Raadt (2006), and Sorva, Karavirta, and Malmi (2013). In addition, we were inspired by educational programming tools such as BACCII (Calloni et al., 1997), FLINT (Ziegler & Crews, 1999), jGRASP (Hendrix et al., 2004), VIP (Virtanen, Lahtinen, & Järvinen, 2005), RAPTOR (Carlisle, Wilson, Humphries, & Hadfield, 2005), ALVIS Live! (Hundhausen & Brown, 2007; Hundhausen et al., 2009), Scratch (Resnick et al., 2009), Alice (Bishop-Clark et al., 2007), JEliot (Ben-Ari et al., 2011), and Ville (Rajala et al., 2008). Next, we present the most important choices and heuristics used for designing Koios.
Linguistic attributes of Koios match users’ natural language
Koios was designed for novice programmers who study at Greek educational institutes. Therefore, Koios’ programming commands as well as IDE were developed to match the target users’ natural language, namely Greek, 2 with the aim of reducing extraneous cognitive load (Bouvier et al., 2016). In addition, Roussel, Joulia, Tricot, and Sweller (2017) found that presenting material in a foreign language decreased language and content learning.
Providing a minimum set of commands for novices
The concepts supported by Koios were determined by (a) the target users of Koios, (b) computer science curricula of educational institutes, and (c) the aim to reduce extraneous cognitive load. The target audience of Koios is novice programmers and thus all the basic programming concepts should be supported, namely constant, variable, array, arithmetic operators, relational operators, logical operators, input-statement, output-statement, assignment-statement, conditional statement, iteration-statement, function, and procedure.
Creating programs visually
Another design decision was that the cognitive load imposed by the syntax of commands should be minimized in order to allow novices to focus on programming concepts during program creation (Gaspar, Langevin, & Boyer, 2008; Grandell, Peltomäki, & Salakoski, 2005; Lye & Koh, 2014; McIver, 2000; Myers, 1990; Scarr, Cockburn, Gutwin, & Quinn, 2011; Scott, 2010). Thus, Koios users can create their programs only in a graphical or visual way (Petre, Blackwell, & Green, 1998). Typically, programs created in an introductory course are small and therefore, creating small programs visually can be a powerful method, especially in programming tools for novices. Moreover, visual tools have been found to promote transfer-of-learning when users move to textual PLs (Stefik & Hanenberg, 2014). A list of icons represents the available programming commands because images seem to facilitate the construction of schemata (Tudoreanu & Kraemer, 2008), which according to constructivism, takes place during knowledge acquisition. A programming command can be added into a program by dragging-and-dropping or copying-and-pasting its icon into the program (Lee & Ko, 2011).
Dropping (or pasting) a command’s icon in programs initiates a dialog session between users and Koios via dialog windows (Hsu, 2003), during which, they are guided through the creation of this command and provide all the necessary parameters regarding it. Thus, the detection of syntax errors during the creation of each command can be facilitated, and immediate feedback can be given to users (Carroll & Carrithers, 1984; Grandell et al., 2005) via error messages. Hence, the functionality of an interpreter is simulated, which is considered beneficial for novice programmers (Gaspar et al., 2008). We acknowledge that the use of dialogs could become frustrating for experienced users, but the benefit of having an always syntax-error-free program is more important than a potential inconvenience. From a constructivist viewpoint, users are able to construct their new programming knowledge, given that they are actively engaged in the creation of commands through a potentially repetitive process and supported by immediate feedback that could expose any misconceptions. Most important, a command is added in the program only when its parameters are specified correctly, which guarantees that programs are always without syntax errors and can be executed at any time. This process could also reduce extraneous cognitive load, because users do not need to recall the syntactic rules and required parameters for correctly creating a command (Gaspar et al., 2008; Scott, 2010). Because Koios guides novices through the creation of each command, it may enable them to focus on programming concepts and their functionality.
Supporting graphical and textual notations
A basic consideration throughout the design process was that Koios should be a programming tool for the initial stages of programming instruction and, therefore, should provide features that would render the transition to professional PLs as smooth as possible (Hewett, 2005; Stefik & Hanenberg, 2014). This is the reason that programs can be viewed in a graphical or a textual way (Chilana et al., 2015; Pane, Myers, & Ratanamahatana, 2001; Petre et al., 1998). Users can switch at any time between views and familiarize themselves with both notations as well as the association between them. This frequent-switch functionality facilitates users not only make clearer associations between graphical and textual forms of commands and increase intrinsic cognitive load but also avoid increasing harmful cognitive load, as would be the case of a single switch between views before program execution. Programming icons do represent parameterized pieces of code in Koios’ PL. However, this code is hidden from users when they are interacting with the programming icons. This approach underlies the distinction between designing a program and writing code (van Merriënboer, 1990).The textual view provides a traditional textual code that follows a specific set of syntactic and semantic rules (Vasilopoulos, 2014) and is created automatically by the graphical representation of the program.
Providing natural syntax and unambiguous semantics of commands
As the syntax and semantics of PLs are a source of mistakes for novices (Brown & Altadmri, 2017), the design aim was to produce a syntax with Greek keywords that feels natural in the Greek language while conforming to a certain level of formality. A trade-off here was that in order to facilitate users’ transition to and familiarize them with professional PLs, Koios’ syntax (Vasilopoulos, 2014) deliberately includes a small number of widely used programming words or symbols that we acknowledge that could be considered as less natural or intuitive for novices (Stefik & Siebert, 2013). Because the textual form of programs is created automatically and syntactic rules do not need to be remembered, the phrasing of commands in textual view can be verbose and more descriptive than typical programming commands. This, in turn, can make the textual form of programs more readable and informative and reduce extraneous cognitive load. Koios’ semantics of commands were designed to convey their programming meaning and to be as unambiguous as possible, while reducing extraneous cognitive load. Thus, the auto-generated code follows conventional rules of indentation, while different combination of colors and font styles are used to distinguish different programming constructs.
Providing information during program execution
Programs are executed in a new window and shown in textual form. Koios users create their programs in a graphical way and, thus, they are required to pay much attention to this form of programming. Involvement with the textual form of programs is important for any novice programmer as well, because at present, the dominant way of creating programs is by using text, and a visual representation of large programs is still inefficient. The textual form of programs supported by Koios can be more readable and informative—due to the verbose and more descriptive phrasing of commands in textual view—than the textual form of programs supported by other programming tools. Therefore, presenting programs in textual form during execution draws users’ attention to this form as well. This feature, however, would be useful if users have already familiarized themselves with the textual code, during program creation. Otherwise, such a switch right before program execution could increase extraneous load.
To make the execution of programs more comprehensible and reduce working-memory load, each command is highlighted while it is being executed, and explanatory and commentary messages are shown regarding the function of the command (du Boulay, O’Shea, & Monk, 1999; Petre et al., 1998). Two features that further increase users’ active involvement, which in turn, according to constructivism, can facilitate knowledge construction are (a) buttons that can start, pause, and terminate as well as control the speed of programs’ execution are available and (b) data input during execution is performed via dialog windows (Hsu, 2003).
Two important additional design choices are (a) the programming paradigm supported by Koios and (b) the provision for comments in programs. The two dominant paradigms are the procedural and the object-oriented (OO) paradigm. Despite the ongoing debate on the appropriateness of each paradigm (Milne & Rowe, 2002; Wiedenbeck, Ramalingam, Sarasamma, & Corritore, 1999), the first design choice was to select the procedural paradigm as more appropriate, based on the set of programming constructs that are supported by Koios. This set can be supported by both paradigms, but it does not contain or require any OO concepts, as this would introduce unnecessary programming concepts, which, in turn, would increase extraneous cognitive load. Besides the increased complexity and possible unnatural programming logic for novices introduced by the OO paradigm (Koulouri et al., 2014; Rist, 1996), the procedural paradigm was considered more appropriate for introductory programming for the following three reasons. First, it seems that the OO paradigm is not more natural, easier, or powerful than the procedural paradigm as hypothesized (Green, 1990; Wiedenbeck et al., 1999); second, OO concepts are difficult for novices to understand (Milne & Rowe, 2002); and, third, novices need less time to familiarize themselves with the procedural paradigm (McCracken et al., 2001).
The second design choice was that the prototype of Koios does not need to support comments. Even though the inclusion of comments in programs is recognized as a helpful source in reading and understanding code, and a good programming habit (Raskin, 2005), the findings presented by Barr, Holden, Phillips, and Greening (1999) and Grandell et al. (2005) suggest that the inclusion of comments by novices may yield undesired results. A number of students in the study of Barr et al. (1999) inserted comments that were similar to the code they had written, and in other cases, comments did not explain or were not relevant to the code. Grandell et al. (2005) reported that more than 65% of the participants in their study did not include comments in their code and that the quality of the comments did not always match the quality of the program. They also reported that reasons for omitting comments could be that students consider comments unnecessary for small programs, that the inclusion of comments delays the programming process and that properly conveying their programming ideas via comments is hard. This design decision perhaps seems controversial, but it does not suggest that comments in programs should altogether be removed from introductory courses, only that the programming tool does not necessarily need to support them. Further choices and decisions regarding the development and implementation of Koios are discussed next.
Implementation of Koios
The primary aim of Koios’ design and development was producing a programming environment that could serve as an efficient tool for improving the teaching and learning of introductory programming in Greece. However, a secondary aim was that this tool could also serve as a stepping-stone for the transition of Greek novices to more advanced and powerful professional PLs. We invite our readers to try the Greek or English version of Koios, available on line at https://github.com/koiosVPT/koiosvpt_en.
The Koios programming environment was developed in Java, using the NetBeans IDE—Versions 6.5 and 6.7—and Java Development Kit 6, 7, and 8. Koios was developed in Java because it is hardware independent and platform independent (Gosling & McGilton, 1996). This way, hardware dependence is avoided, a deadly sin of PLs for novices (McIver & Conway, 1996).
Koios’ user-interface
Koios provides a simple and easy user-interface for novice programmers, avoiding unnecessary exposure to more complex features. Furthermore, Koios provides two view modes for initial and advanced states of introductory programming, respectively.
As shown in Figure 1, the user-interface of Koios in program creation mode displays a menu bar, a toolbar, a status bar, and three main windows: (a) the program window (left), (b) the programming commands window (top right), and (c) the properties window (bottom right). The programming commands window contains all the available programming commands, grouped in six categories. The implemented programming commands-items in Advanced mode are constants, variables, arrays, write, read, assign, call, if-statement, for-statement, while-statement, do-while-statement, return procedures, and functions. The Basic mode provides only the following programming commands-items: variables, write, read, assign, call, if-statement, for-statement, and procedures. Each programming command is represented by an icon; the available programming commands or items in Advanced mode and their grouping can be shown in Figure 2.
The main window of Koios programming environment. Icons of programming items.

Programming items are represented as icons-nodes and are associated with brief text, which reads programming items’ most important properties. The aim of this is not only to help users to identify different programming items that have the same icon but also to prevent them from splitting their attention between different parts of the user-interface by providing critical on-time information regarding items’ properties (Shaffer, Doube, & Tuovinen, 2003). The nodes of programs are organized in a tree structure, which resembles the structure used for the representation of files and folders by the majority of operating systems. This choice was based on the following four reasons. First, it is possible that this view can convey the concept that there are two categories of nodes or programming items: (a) compound statements, which are represented by nodes or programming items that can include one or more nodes, like the if-statement and for-statements and (b) simple statements, which are represented by nodes or programming items that cannot, like an output statement. Second, according to constructivism, previous experiences can be used to construct new knowledge and thus, users who may be already familiar with the tree view from their interaction with computers’ operating system could better grasp the parallel. Even if this is not the case, this representation could be a good metaphor for distinguishing the functionality of these two categories. Third, this view can provide a good mechanism for showing or hiding the nodes or programming items within a node or item, such as the commands within a for-statement and thus, the size of the visible program can be adjusted. Fourth similar showing or hiding techniques are used by modern professional textual PLs and IDEs.
New commands or items are added to a program by selecting an icon from the programming commands window and dragging-and-dropping or copying-and-pasting it to a program. To prevent errors, only valid commands are allowed to be entered in programs, for example, the icon of a return statement can be added in functions but not in procedures. When a new item is dropped or pasted in the program, a dialog between Koios and a user is triggered to guide them through the creation of the command and determine its properties. Before the user moves away from a dialog window, the properties entered so far are checked. If a property is not entered correctly, then an error message appears in a pop-up dialog window. When all the properties have been entered correctly, a new icon is created and a message appears informing the user that this programming item has been successfully added in the program. Hence, only items without errors are entered in the program; therefore, the program is always in a syntax-error-free state.
Every time an action is performed, a message appears on the left side status bar that describes this action, while on the right side of the status bar, a balloon appears that provides information about the status of programs. Available actions are adding, deleting, cutting and pasting, and moving up or down programming items.
Code generation and error check
The graphical and textual view of a program can be shown in Figure 3. Figure 3(b) shows the text code that is automatically generated from the graphical view of the program (Figure 3(a)). Different combination of colors and font styles are used for formatting the text in order to highlight the different programming constructs.
The (a) graphical and (b) textual view of a program.
Koios provides a check command, which performs three actions on the current program. First, it reorders the nodes of constants, variables, and arrays that are declared and moves them to the beginning of the procedure or function. This action guarantees that every item is declared before it is used. Second, it checks whether any constants, variables, or arrays are declared but not used and any variables or array elements are used without initialization. In the first case, the name of the item is reported, while in the second case, the name of the item as well as the statement(s) that include(s) it are reported. Third, it detects any nonsyntax errors, for example, functions without return statements, except for dynamic semantic (run-time) errors, for example, accessing an array element beyond the size of the array, which can only be detected during run-time. From a pedagogical viewpoint, users are advised to run the check command during program creation and before compiling the program. However, if this action is omitted the compile command automatically first runs the check command before actual compilation of the program. When the check or compile commands are executed, the balloon in the status bar informs users whether warnings or errors exist or not.
Koios’ complier
Deciding which compiler would be appropriate is a matter that comes with a number of technical considerations. Since Koios uses the Java Virtual Machine, the solution was to transcompile the Koios programs to equivalent Java programs. These programs can, in turn, be executed within the Koios programming environment. It should be noted that the equivalent Java programs are carefully transcompiled to produce as few Java exceptions (errors) as possible. However, if such an exception occurs, it is handled transparently to Koios users. An important feature of Koios is that it provides the Export Java Program command, which can produce the equivalent Java version of the Koios programs. This feature can be useful for a smooth transition from an introductory PL to a professional one or if the equivalent Java code is needed.
Program execution mode
Once a program contains no errors, it can be executed using the run command. This action changes Koios’ mode from program creation to program execution, used for tracing program execution. At the top of the window, there are control buttons for starting, pausing, and stopping the execution and a slide bar for adjusting its speed. When a statement is being executed, it is highlighted in yellow, while in the case of a compound statement, for example a for-statement, the end statement is highlighted in orange (see Figure 4). The output of running programs is shown in the output window of the program execution mode.
A snapshot of the execution of a program.
An important and unique feature of Koios is the comments or description window. This window provides messages with overall information about the statement that is being executed at the time. This information includes a brief description of the statement’s properties, how this statements works, what the result of its execution is, and how it affects the control flow of the program (if applicable). Thus, through the play or pause buttons and these messages users can be supported in better understanding while tracing code execution. These features were developed because, according to constructivist principles, users—based on their personal learning needs—can individually and actively construct their knowledge structures of program execution at their own pace and with minimum help from instructors and promote explanatory learning under proper guidance with always-available on-time information.
Empirical Study and Evaluation of Koios
Method
The method followed for evaluating Koios programming tool (Vasilopoulos, 2014) is by comparing their impact with that of other programming tools on users (Stefik & Hanenberg, 2014; Vogts, Calitz, & Greyling, 2008). The comparison of a new programming tool with a widely used one can provide evidence for its effectiveness and contribution.
Differences Across Features Between the Three Programming Tools Used in the Quasi-Experimental Design.
Table 1 shows that Koios provides more natural syntax and unambiguous semantics than Glossomatheia and MicroworldsPro that provide Pascal-like and Logo-like syntax with Greek keywords, respectively. Moreover, Koios, compared with the medium level supported by Glossomatheia and MicroworldsPro, provides a high level of visual techniques. All programing tools provide a small set of instructions, but Koios’ error messages are designed to be more comprehensible and informative than the error messages provided by Glossomatheia and MicroworldsPro. Finally, Koios’ IDE maintains a higher level of interaction with its users during program creations and execution than the other two programming tools.
Research hypothesis
Koios’ design was inspired by principles of both constructivism and CLT, but the design of Glossomatheia is not explicitly based on either of these. However, MicroworldsPro, as a programming tool based on LOGO (Papert, 1980), could be considered to have been influenced by constructivism. Therefore, the following research hypothesis was tested: Novice students who use Koios develop a higher level of programming competence than novice students who use Glossomatheia and MicroworldsPro.
Design
A quasi-experimental design was used, because this study took place with actual school students as participants and, due to administrative reasons, a random allocation of participants was not possible (Lye & Koh, 2014; Stefik & Hanenberg, 2014). The independent variable of this design was the programming tool, and the dependent variable was the level of programming performance of novice programmers.
Participants
Statistical power analysis revealed that a total of 66 Greek secondary-school pupils were required in order to detect a large effect size (η2 = .138) with a significance level of .05 and a statistical power of .80. The 70 participants—37 male students and 33 female students (mean age = 14)—that took part in this study were third-grade secondary-school students with no previous formal instruction in programming. Three whole school-classes were allocated by the school administration as the three quasi-experimental groups, and each one consisted of 23 (Koios group), 25 (Glossomatheia group), and 22 (MicroworldsPro group) participants-students. Participants were informed about the study, and consent forms were signed by their parent(s) or guardian(s) before the start of the study. Participants had no previous programming experience nor had they used a programming tool before.
Procedure
Each group used the corresponding programming tool for its introductory programming course for nine lessons (one per week) and had the same teacher (first author). The programming concepts, teaching method, classroom examples, homework, and teaching material were the same for all experimental groups. The only necessary, though slight, variation was the teaching material for each of the three groups in order to appropriately present the corresponding programming tool. Programming performance of participants was assessed with a practical test at the end of the course, as this type of task has been found to be provide an accurate assessment of programming performance (Koulouri et al., 2014, p. 14).
Material
Participants were asked to create three programs in computer lab with the programming tool they had been using during their programming course. The three programs are presented in Appendix A.
Data analysis
For the analysis of the practical-test scores, correlation analysis, simple regression, and analysis of covariance ran as hierarchical multiple regression (HMR; Field, 2017; Pedhazur & Pedhazur Schmelkin, 1991) were employed. These techniques were used for testing the effect of programming tool on the level of programming skills. The programs that students produced were scored by the first author using an assessment scheme (see Appendix B), and the minimum and maximum score of the practical test was 0 and 30, respectively. The same scheme was used by an external marker for marking the programs of the practical test in order to analyze reliability of marking.
Gender, assessments of homework, grades from other relevant subjects (e.g., mathematics and physics), and grade point average (GPA) were used as candidate covariates. However, only participant’s GPA correlated significantly with the end-test scores, r(66) = 0.51, p < .001 and therefore was the only covariate used in the analysis.
Because Koios programs are always syntax-error-free, the data collected regarding syntax test-items were analyzed separately from the nonsyntax items. This separation removed the potential bias that the syntax items could introduce if all items would be analyzed together.
Results
Descriptive Statistics for Syntax Scores of the Practical Test.
Descriptive Statistics for Nonsyntax Scores of the Practical Test.
Results of Regression Analysis for the Syntax Scores of the Practical Test.
Note. GPA = grade point average; R2 = .21 for Model 1; ΔR2 = .21 for Model 2 (p < .001).
**p < .01. ***p < .001.
Results of Regression Analysis for the Nonsyntax Scores of the Practical Test.
Note. GPA = grade point average; R2 = .23 for Model 1; ΔR2 = .19 for Model 2 (p < .001).
**p < .01. ***p < .001.
Again, in the first regression model, GPA was a significant predictor and explained 23% of variance in the scores, R2 = .23, F(1, 68) = 20.35, p < .001. In the second model, programming tool explained a significant additional 19% of variance in the scores, ΔR2 = .19, Fchange(2, 66) = 10.89, p < .001, while a total of 42% of variance was explained by the two predictors, R2 = .42, F(3, 66) = 16.00, p < .001. GPA was also a significant predictor, t(66) = 5.92. Finally, with GPA held constant, Koios users performed significantly better, with on average 4.64 points (Table 3) higher on the end test than Glossomatheia users, t(66) = −3.83, p < .001; they also performed significantly better, with on average 5.60 points (Table 3) higher on the end test than MicroworldsPro users, t(66) = −4.26, p < .001.
The bias of the regression model was also tested. The assumptions of homoscedasticity, linear relationship between outcome and predictors, and independence of residuals were met. In addition, variance inflation factor and tolerance values revealed that no multicollinearity existed in analyzed data sets. Finally, the assumption of homogeneity of regression slopes was also met (Field, 2017).
To further investigate the aspects in which Koios users achieved higher scores than users of MicroworldsPro and Glossomatheia, an in-depth analysis of the end-test scores was performed. Figure 5 presents each group’s mean scores with respect to aspects of creating and executing a program.
Mean scores for each programming aspect per group.
Descriptive Statistics for the Six Programming Aspects.
More specifically, the mean for the Koios group was higher than that for the Glossomatheia and the MicroworldsPro groups on the following programming aspects: creation of variables, use of commands, correctness of program syntax, correctness of command order, executability of the program, and correctness of program execution (Table 6). As shown in Table 6 and according to Cohen’s (1988) conventions, the effect sizes observed for the difference of the mean scores for the different programming aspects between the Koios and the Glossomatheia groups and between the Koios and the MicroworldsPro groups each varied from small to very large. Therefore, although an advantage of Koios was observed for each of the six aspects and not confined to a subset, the largest advantage by far was in program correctness.
Discussion
We discuss the main contribution of the work that is presented in this article and potential extensions. Finally, we discuss limitations of our work and future research.
Koios Contribution to Research and Knowledge
The design and development of this new Greek programming tool, Koios, was motivated by our intention to facilitate Greek novice programmers’ learning in their introductory programming course and, thereby, to contribute to knowledge. Therefore, Koios’ design and implementation was guided by heuristics and rules of thumb that were derived from the combined principles of constructivism and CLT. Koios includes a PL and IDE in users’ native language that provides a minimum set of instructions for novices, which seems to have more natural-language-like syntax and semantics. Moreover, Koios supports visual creation of programs, both graphical and textual notations, constant guidance, feedback, and comprehensible error messages during program creation. These guarantee that programs in Koios are always in syntax-error-free state. In addition, Koios supports visualization and explanatory messages regarding program execution.
The combination of all these features supported by Koios constitute a major advantage for our tool against other Greek programming tools that only consider part of these features. Moreover, Koios’ design and implementation followed a user-focused approach, as developing a software tool that helps and supports its users in efficiently accomplishing tasks, which is particularly important especially for complex and challenging tasks like programming. Thus, Koios’ characteristics (e.g., textual code automatic generation, error prevention, informative feedback, and constant guidance) could support its users in overcoming common difficulties novices face and enable them to focus on the task of programming.
A second contribution of this research is establishing that Koios can effectively facilitate Greek novices in program creation. First, Koios significantly facilitated its users in creating programs, even when they created them on their own and without guidance or help from teachers. More specifically, the results of the end test revealed a significant effect of the quasi-experimentally controlled factor programming tool (Koios vs. MicroworldsPro and Glossomatheia) on end-test scores. In particular, Koios users performed significantly better regarding both syntax and nonsyntax items of the practical test than the users of the two other tools. Although a significant performance regarding syntax items from Koios users was anticipated because Koios does not allow syntax mistakes, the significant test results for nonsyntax items more generally reveal the effectiveness of Koios as a programming tool for Greek novices. Moreover, Koios users appear to perform better than users of standard software in the following six programming aspects: creation of variables, overall use of commands, correctness of syntax, correctness of command order, executability of the program, and correctness of program execution. Second, GPA was a significant independent predictor of students’ performance in the end test. Therefore, although novices’ overall academic ability predicts programming competence, programming tool explains additional variance in competence. Furthermore, because gender did not correlate significantly with the end-scores, this study corroborated the finding from previous studies that gender does not affect programming performance (Kahler, 2002; Lau & Yuen, 2011; Linn, 1985; Pillay & Jugoo, 2005).
These results are explained by the fact that the rules of thumb and heuristics that guided Koios’ design and implementation were primarily based on the combination of principles of constructivism and CLT, while the design of Glossomatheia and MicroworldsPro is not explicitly influenced, at least not to the extent that Koios is, by these two theories. Designing programming software based on principles of and heuristics inspired by constructivism and CLT can produce a programming tool capable of (a) reducing extraneous cognitive load, (b) providing a proper management of intrinsic cognitive load, (c) reallocating more working-memory resources as germane resources, (d) facilitating the understanding of programming concepts and reducing possible misconceptions, (e) actively engaging users in program creation and execution, (f) providing an easier and more comprehensible way of program creation, and (g) presenting a clearer view and supporting a better understanding of program execution. Thus, principles and heuristics that are based on these two theories should be used to inform the design of an effective and supportive learning environment for novice programmers.
Limitations and Future Work
Limitations
The main source of bias for this study could be the fact that the first author was the person who designed and developed Koios, was the teacher in three quasi-experimental groups and collected the end-test data. The evident conflict of interests and potential bias could genuinely affect the validity of this study. To moderate this bias, the marking process was made more objective by producing a marking scheme for the practical test and recruiting an external marker to mark a sample of students’ responses. The reliability of marking of the practical test was calculated using external marker’s assessment and was found to be high. Another limitation of this study was the use of a quasi-experimental design instead of an experimental design (Stefik & Hanenberg, 2014), which could not be used for constraints imposed by school administration.
Future work
In future research, we look forward to further evaluate Koios in comparison to Greek versions of modern and popular programming tools, like Scratch or Alice (Chih-Kai, 2014). We would also like to employ measurements that could reveal a more in-depth knowledge of programming concepts acquired by using these specific programming tools.
An important aspect for further improvement of Koios’ design and implementation could be the on-going refinement of the syntax and semantics supported by Koios based on new rigorous research findings, such as the ones reported by Stefik and Siebert (2013). Furthermore, proven techniques of CLT (see Section Theoretical Background of Koios) could be a considerable source of inspiration in further improving Koios. Moreover, the messages and comments during execution could be improved taking into account findings reported by Koulouri et al. (2014), Stefik et al. (2011), and D’Mello, Graesser, and King (2010). These improvements could aim at highlighting explanatory comments’ content provided by Koios during execution, making comments to be perceived as more natural by users, and help them in the debugging process. Another suggestion may involve displaying the graphical view together with the textual view during execution. This juxtaposition could further increase comprehension and allow users to refer to and focus on a more compact (graphical) and a more detailed view (textual) of the code simultaneously, which is supposed to help programmers better understand programs (Green & Petre, 1996). Moreover, during execution users might create more associations between the graphical and the textual view of the code as well as between these and programs’ functionality.
In addition, modifications in Koios’ design and development could be made to further improve its educational impact. For example, additional modes could be provided with respect to the execution of programs. Hence, programs could be executed in a step-by-step mode, could allow users to return to a previous state in execution, and overall provide simple features of professional debuggers. Similarly, these modes could be used to increase the visibility of the notional machine (du Boulay et al., 1999) and, more particularly, expose the intermediate states and steps during program execution.
Conclusion
In conclusion, we have designed and implemented a new highly interactive and visual Greek programming tool, namely Koios, based on combined principles of constructivism and CLT that guided our design. The benefits of Koios over widely used programming tools for novices in Greece were empirically demonstrated. We look forward to the application and further development of this work in programming instruction.
Appendix A. End Test Used in Empirical Study
The end test consisted of the following items and took approximately 30 minutes to complete. Students were asked to create the following three programs with the help of the programming tool they had been using during the study.
Program 1 (10 marks)
Create a program using Koios or Glossomatheia or MicroworldsPro that requests from the user to give (input) a name to the string variable NAME and an integer number to integer variable TIMES. The program prints to the screen the value of NAME as many times as the value of TIMES.
Program 2 (10 marks)
Use Koios/Glossomatheia/MicroworldsPro to create a program with a procedure named OPP. Procedure OPP takes an integer input parameter X and prints to the screen the opposite number of X, namely—X. The main procedure requests from the user to give (input) an integer number to the integer variable NUMBER, and with the use of OPP prints to screen the opposite value of NUMBER.
Note: MicroworldsPro does not support the expression -X, to calculate the opposite of X. Therefore, the following hint was given to MicroworldsPro users only.
Hint: Calculate –X using the formula: −X = −1 × X.
Program 3 (10 marks)
Create a program using Koios or Glossomatheia or MicroworldsPro that requests from the user to give (input) an integer number to the integer variable GRADE. If the value of GRADE is less than 10, then the string FAIL will appear on the screen, otherwise the string PASS will appear on the screen.
Appendix B. Marking Scheme for End Test
Marking Scheme for the First Program of End Test.
Marking Scheme for the Second Program of End Test.
Marking Scheme for the Third Program of End Test.
Footnotes
Acknowledgments
Many thanks to Dr. Aggeliki Pragiati for her overall help, support, and ideas with respect to this study. Special thanks to Vera Gerasimatou, Konstantina Plessa and Dr. Alexis Lazanas for facilitating and their overall help regarding the conduction of the empirical study.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
