Effectiveness of Feedback Based on Log File Analysis in Introductory Programming Courses

Challenges in Introductory Programming Courses

Programming is a useful skill, but it is difficult for beginners, and there is still a high dropout rate in the courses, although it has been thoroughly addressed (Medeiros et al., 2018; Watson & Li, 2014). Bennedsen and Caspersen (2019) emphasize that this is why we still face a major challenge in developing more inclusive and effective learning environments and in developing a methodology for teaching introductory courses. As programming education is popular, there are more and more courses with a lot of learners (Luxton-Reilly et al., 2018). Therefore, teaching methodology also needs to be adapted for large courses. Also, learners often have very different prior programming experiences and backgrounds. O'Malley and Aggarwal (2020) emphasize that the increasing diversity and size of introductory programming courses and non-major contexts create environments consisting of students with various educational needs. Studies have also suggested that previous programming experience influences students’ course outcomes (Leinonen et al., 2016; Veerasamy et al., 2018; Zhang et al., 2013). Holden and Weeden (2003) conclude that previous programming experience has a particular impact on the results of the very first programming course.

Qian and Lehman (2017) have found that students have various misconceptions and difficulties in terms of syntactic, conceptual and strategic knowledge. They say that these difficulties, in turn, are related to many different factors such as skills of syntax, natural language, and knowledge of math. There may be a lack of strategies, a good programming environment, the teacher’s knowledge, skills and the ability to instruct. Although a lot of attention has been paid to syntax errors (Altadmri & Brown, 2015; Hristova et al., 2003; McCall & Kölling, 2019), students often find and correct these errors relatively quickly (Qian & Lehman, 2017). However, some studies show that it can be difficult for beginners to find syntax errors in the program, especially at the beginning of the course (Denny et al., 2012; Marceau et al., 2011). Smith and Rixner (2019) observed that syntax errors are disproportionately distributed and that tutors may need to identify learners who are struggling with syntax and provide targeted instruction. The results of the Denny et al. (2012) study show that even among the most common errors, some take a long time to find. Denny et al. (2011) find that students struggle with syntax errors much more than might be expected, even if they write very short pieces of code. Although all types of errors can be severe at first, it takes more time to correct semantic errors. Bosse and Gerosa (2017) find that students are less accustomed to semantic errors and that their presence is more likely to force them to give up solving, because they immediately realize that it takes more time to find them. That is why it is very important to teach students how to find mistakes in their programs. Chmiel and Loui (2004) suggest that teaching and training to debug effectively helps students find and fix bugs in programs. Ahmadzadeh et al. (2005) have found that many learners with a good understanding of programming do not have the skills to debug programs. They recommend placing more emphasis on debugging skills when teaching programming. Raj et al. (2018) find that live coding is a good method that helps students learn the process of debugging because it allows seeing which methods instructors use. Deeb and Hickey (2021) created an extra feature for the programming environment they used, with students having to analyze their error messages and unit test results in addition to correcting their program. When students had to reflect on their debugging, they completed the task faster and made fewer mistakes. Thus, new methods that help learners find errors in programs and better understand how the programs work are undoubtedly important.

Programming Process Analysis

Since the last decade, an increasing number of systems have been created that log the programming process and thus allow for a more detailed analysis of various aspects of programming. As a result, there are many ways to use programming process analysis to understand and support programming learning. One way is to use this analysis to understand how beginners program and what differences can be found between different learners. Vihavainen et al. (2014b) analyzed the programming process using three different granularities of data: submissions, snapshots, and keystroke events. The data showed that about half of the students also solved tasks they never submitted. Moreover, some students continued to work on the tasks after they were submitted. They also found a significant difference in workload depending on whether or not the participant had previous programming experience. Blikstein (2011) used learning analytics and his focus was on finding out the differences between beginners and advanced programming styles. More experienced learners were more likely to try to debug and improve the program without outside help. Beginners, however, were more inclined to fix the program using many trial-and-error attempts. They also used more copying and customized code from external sources. He also found that learners need different help depending on their programming style and programming experience. It may be that more experienced students want to come up with solutions themselves and would not like to be helped to solve problems. However, beginners need more help and examples from where they can copy the code to modify it. This understanding is also supported by Vihavainen et al. (2014a), who examined how students program when they are just beginning to learn to program. Students copy and paste relatively often when they create programs in the first weeks. The results show that, on average, one-tenth of the learners' code base is pasted. However, they copy most of it from their previous works. Marceau et al. (2011) analyzed changes that students made after receiving error messages. In addition to programs and error messages, they recorded students' keystrokes after error messages until they resubmitted the program. They identified errors that were challenging to understand and also tasks with complex parts.

Hosseini et al. (2014) analyzed students' behavior patterns in solving paths. They found that students made a relatively large number of conceptual changes in the first quarter of solving the task, and then the number of changes dropped sharply, especially for complex problems. It also turned out that the number of deletions does not depend so much on the task as on the students' programming style. Most students add code gradually (77.63%), and some students build code gradually, but in the meantime, they have long downtimes when they make minor code changes and try to get the program to work. Some students first add a large amount of code and gradually reduce it until they reach the correct solution. They try to get results by lots of trial-error attempts. Blikstein (2011) took copying and pasting as the main differentiator between students’ styles. Meier et al. (2020) found that students who used the gradual approach tested the program during the programming process, used few or none of their previous solutions, and got a few error messages. They fixed bugs during the programming process. They also had not had many error messages. Others tested the program for the first time when most of the program had been written. They needed a lot of time and received many error messages. Some students tested the program when they had written most of it, but it was correct right away. The learners who solved tasks gradually were more likely to achieve better performance.

Quite a lot of attempts have been made to predict the results of introductory programming courses using student background data, high school grades (Costa et al., 2017; Khan et al., 2019; Sivasakthi, 2017), grades in related subjects (Arora et al., 2014), but predictions have also been made based on the programming process. Castro-Wunsch et al. (2017) developed a model that predicts which students need help. It was based on the number of times the solutions were submitted and the correctness of the code. Vihavainen (2013) relied on more fine-grained data on the programming process. They described student performance through characteristics that can be observed throughout the course. They used a system that records every instance of a student saving their work and measures the time left until the deadline, the time between saves, and the changes between them. They also analyzed the programs to see if the program worked, had any style errors, included copy-pasted code, etc. Based on this information, they predicted whether the student’s score would be “fail”, “pass”, or “excellent”. Those students who started working earlier were more likely to belong to the “excellent” group. Earlier start can be explained by enthusiasm or good planning skills. Students who began solving tasks late were more likely to fail. In addition, a positive indicator was small number of indentation errors in the code, which may indicate whether the student is paying attention to more minor details. Ahadi et al. (2015) recorded the programming process so that each keystroke and other related information, such as time and solution details, were collected. In addition, they collected students' background information, including gender, age, and previous programming experience. The results show that the information from the programming process is more relevant for predictions than the background information. High and low progress students can be distinguished from the first week of the programming process. This means that low-performing students can be supported from the second week onwards, and high-performing students can be given additional challenges. Figueiredo et al. (2019) monitored and evaluated student activities. Based on this information, they created a machine learning model that allows teachers to identify students who are more likely to fail. The model will enable teachers to devote more time to those students and use effective strategies to help improve their programming skills.

Feedback Related to the Programming Process

The programming process can be recorded and analyzed for various purposes. One goal is to give students feedback on aspects that are not covered in the analysis of the submitted program. Matsuzawa et al. (2013) developed a tool that allows learners to analyze their programming process after solving a task. It is possible to replay the programming process and see how much time was spent, the number of runs, the correctness of the program, etc. Zhi et al. (2019) created a system that automatically generates example-based feedback on the mistakes in students' programs. For example, such feedback can be shown to students when they request help. The idea was based on the insight that beginners need examples to learn. Marwan et al. (2020) worked out an immediate adaptive feedback system for a block-based programming environment that continuously checks student code and provides students feedback during the programming process. The system also has a progress panel to show the programming process and motivational pop-up messages provided when objectives are completed or when a student has not achieved any goals within a specific time. They found that the system increased students' engagement, and students thought that the system was fun and helpful and they were more focused. Systems have also been created where it is necessary to reflect on code or mistakes during the programming process, and write about them. Deeb and Hickey (2021) developed a system where students sometimes reflect on error messages or unit test failures during the programming process. They can continue programming after this reflection. Students had to describe the error and plan how to fix it. When students had to reflect on their debugging, they were quicker to solve the task and made fewer mistakes. At the same time, the number of those who gave up solving the problem was higher. The system allows the instructor to view each student’s coding history and see the comments students have written during the programming process. They think it can help understand students' knowledge level, which can be useful when working with a student individually. Dorodchi et al. (2020) introduce a cloud-based IDE developed for situations where there is a lack of in-person interactions. Once students have submitted their solution, they receive instant feedback from the IDE and will be able to compare their code with the solution provided by the instructor. Based on this comparison, students need to reflect on their work. Resources have also been created for use in the classroom where the student and the teacher interact. Yan et al. (2019) created a tool that visualizes the code creation process. The tool allows the teacher to analyze the students’ programming process, for example, to see where the students struggled with concepts or worked smoothly. The goal of the tool was for teachers to talk to students about problem-solving techniques. Their experience showed that using the tool in teaching improved results and task completion time. They also point out that using such a tool reduces plagiarism, as it is much harder to cheat if you need to present your programming process. Shrestha et al. (2022) created a tool visualizing the programming process to help instructors understand students’ problem-solving styles and indicate whether a student is struggling. They also said that the tool is useful for students' self-reflection.

Background Factors in Experiments

For an experiment, it is essential to consider possible background factors that may affect the results. In the case of programming, one of the factors is previous programming experience and its potential impact on the results. Based on 3 years of data, Veerasamy et al. (2018) found that prior programming knowledge affects the exam results of an introductory programming course. They experimented with students from different disciplines. The experiment by O'Malley and Aggarwal (2020) showed that exam scores for students with previous programming experience were higher on average. Davis (1991) and Hagan and Markham (2000) also showed that programming experience influences progress. Holden and Weeden (2003) concluded that previous programming experience influences the results of the very first programming course. Vihavainen et al. (2014b) also found that the differences between learners depend on whether they have previous programming experience. For example, the volume of work in solving tasks depends on this. The effect of gender on results has also been studied. Byrne and Lyons (2001) found that the progress of female students in an introductory programming course was slightly higher than average, but the difference was not statistically significant. The results may also be affected by the low interest of some students in the subject and the associated plagiarism. Alam (2004) found that 41% of students who use dishonest practices do so because they lack knowledge, understanding and interest in the subject. Park (2003) also emphasizes the lack of interest as a cause. Therefore, there may be differences between groups for whom the course is mandatory compared to those who have chosen it themselves.

Overview of the Course

The effectiveness of the feedback from the logs was examined for 2 years (2020 and 2021) in the University of Tartu course “Introduction to Programming”, intended for students who are not studying computer science as a major. In 2020, there were 128 participants in the course, of whom 105 completed the course; in 2021, the respective figures were 142 and 115. In some curricula, the course is compulsory, but it can be chosen by all students, regardless of the level of study. The course takes place every spring and lasts for 7 weeks. It is organized in Python, and the main topics are variables, conditional statement, loop, list, reading from a file, writing to a file, function, and a simple user interface. The course uses the flipped classroom method. For each new topic, the students first get acquainted with the material at home, solve programming tasks and a test, and then deepen their knowledge in a practical session under the guidance of a lecturer. In both years (2020 and 2021), there were a total of 7 practical session groups. Since in the 2 years when the experiment was conducted, there was a pandemic situation, the practical sessions took place in the classroom at the beginning of the course in 2020. After transition to distance learning in the second year, the practical sessions took place via Zoom. The course ends with an exam, which consists of two parts: a test in Moodle and a programming task. The test is focused on assessing how students can read the code and the programming task provides an opportunity to demonstrate their programming skills. The exam covers all the main topics, except the user interface. The total time for both parts is 100 minutes. In both years, we used the following rules for the exam: The test was in Moodle and had to be solved without communication, without the help of materials and any programming environment. The maximum time was 30 minutes. The programming task could be solved once the test had been submitted. It was compulsory to use Thonny’s programming environment during the second part and submit Thonny logs. The use of course materials, previous solutions, Google, etc., was permitted. In addition, the screen had to be recorded during both parts of the exam because everyone solved the exam exercises in their own home due to the pandemic situation. To pass the course, students had to collect 50% of the maximum points available both in the test and in the computer part.

Experiment

To find out how feedback based on Thonny logs affects exam results and how feedback based on logs affects task completion time, the number of runs, error messages, and pastes, an experiment was performed, and it was conducted similarly in 2020 and 2021. Some groups (1 group in 2020 and 2 groups in 2021) received additional feedback throughout the course based on the information from the logs; the rest of the groups (6 groups in 2020 and 5 groups in 2021) worked as usual. One lecturer worked with the experimental groups, and other lecturers with the other groups. In order to provide feedback based on the logs, the lecturer reviewed all the experimental group students' homework logs every week using Thonny’s functionality of replaying the programming process. This Thonny feature allows a letter-by-letter view of the students' programming processes from start to finish. For example, the viewer can see precisely in which order students built the program and when they erased some parts. The viewer also sees when they ran the program, what the result was, what they did in case of an error message, etc. If the logs showed ineffective programming techniques, suggestions were given on how to improve them. For example, if the log showed that the student tried to find solutions to the error message mainly by trial and error attempts, he was advised to use the Thonny debugger in the individual feedback. The debugger allows the learner to see how the program works step by step and thus more easily find errors and learn how the program works. Another example: if the student did not test the function separately but wrote an entire program at once, and it was difficult for him to find errors, it was suggested in the feedback to test the function separately. If the working methods were good, it was also mentioned in the feedback. Based on the information received from the logs, general recommendations were also offered in the practical session. For example, if it was clear from the logs that many students had difficulty understanding the loop while doing homework, it was explained in class in more detail how to use the print function to understand the loop and focus on the examples, which helped clarify the problems seen in the logs. Previous research (Matsuzawa et al., 2013; Shrestha et al., 2022; Yan et al., 2019) that has used feedback based on visualization of the programming process has shown its usefulness, especially in improving students' problem-solving style.

Sample, Data Collection and Analysis

We queried the Study Information System for the following background information about the students: gender, specialty, the field of study, and whether the subject is compulsory for them or not. We also received information about the mandatory nature of the subjects from the curricula. As research (Blikstein, 2011; Hagan & Markham, 2000; Holden & Weeden, 2003; O'Malley & Aggarwal, 2020; Veerasamy et al., 2018; Vihavainen et al., 2014b) has shown that the results can be influenced by previous programming experience, we surveyed this as well. Students could choose between three options in their answers: (1) I have not tried to program before this course, (2) I have tried to program a little before but have not come far, (3) I can program well, and I don’t think I’m learning much in this course. The number of students who took the exam and responded to the survey on programming experience was 57 in 2020 and 68 in 2021. Next, the groups were balanced according to gender, programming experience and whether the subject was compulsory or not. The Chi-Square test was used to check the balance. Students' distribution in groups after equilibration is shown in Table 1.

OPEN IN VIEWER

Table 1.

Students’ Distribution in Groups.

	Students	Beginners	Non-Beginners
Experimental group	25 (32% men)	13	12
Control group	55 (31% men)	28	27

Next, the data were analyzed. The Mann-Whitney U test was used to analyze whether the differences between the test result, programming task result, the solution time, the number of runs, error messages and pastes of the two groups were statistically significant or not. In addition, descriptive statistics were used. The features were selected based on the results of previous studies (see Table 2).

OPEN IN VIEWER

Table 2.

Selected Features and the Previous Studies.

Feature	Reference
Solution time	Matsuzawa et al., 2013; Vihavainen et al., 2014b
The number of runs	Blikstein, 2011; Hosseini et al., 2014; Matsuzawa et al., 2013; Meier et al., 2020
The number of error messages	Blikstein, 2011; Hosseini et al., 2014; Marceau, Fisler & Krishnamurthi, 2011Marceau et al., 2011; Matsuzawa et al., 2013; Meier et al., 2020
Copying-pasting	Blikstein, 2011; Meier et al., 2020; Vihavainen et al., 2014a

RQ1: How does feedback based on logs affect exam results (of the whole group, of beginners and non-beginners)?

To answer the first research question, we compared at first the numbers of exam test scores for both groups (see Figure 1 and Table 3). The Mann-Whitney U test result showed higher exam test results in the experimental group, and the difference was statistically significant (U = 432.5, p < 0.05, η2 = 0.091). Next, we compared the results of the programming task for both groups with the Mann-Whitney U test. The experimental group’s results were better but not significantly so (U = 648.5, p > 0.05, η2 = 0.002).

We also made a separate analysis of learners with and without previous programming experience (see Figures 2 and 3). The Mann-Whitney U test result showed higher exam test results for beginners from the experimental group, and the difference was statistically significant (U = 78, p < 0.05, η2 = 0.218). Programming task results were also better among beginners in the experimental group, but the difference was not statistically significant (U = 170, p > 0.05, η2 = 0.003). Experimental group beginners received an average of 22.0 points in the exam test and 21.5 in the programming task, while control group beginners scored 19.6 and 20.5 points, respectively. The students with previous programming experience in the experimental group did better than the control group on the test and programming task, but the differences were not statistically significant (Test: U = 130.5, p > 0.05, η2 = 0.025; Programming Task: U = 154.5, p > 0.05, η2 = 0.002). Experimental group non-beginners received an average of 21.4 points in the exam test and 23.3 in the programming task, while control group non-beginners scored 20.0 and 21.8 points, respectively.

OPEN IN VIEWER

Figure 1.

Scores of exam test and programming task (A – experimental group, B – control group).

OPEN IN VIEWER

Table 3.

Descriptive Statistics of Exam Test and Programming Task Scores.

		N	Min	Max	M	SD	Mdn	Sk	K
Test scores	Experimental group	25	16	24	21.7	2.4	22	−0.735	−0.339
	Control group	55	10.8	24	19.8	3.2	20	−0.841	0.583
Programming task scores	Experimental group	25	14	24	22.4	2.6	23.5	−2.045	3.865
	Control group	55	10	24	21.1	4.1	23.5	−1.395	0.813

N - Number of cases; Min - Minimum; Max - Maximum; M - Mean; SD - Standard Deviation; Mdn - Median; Sk - Skewness; K - Kurtosis.

OPEN IN VIEWER

Figure 2.

Beginners’ Scores of Exam Test and Programming Task (A – experimental group, B – control group).

OPEN IN VIEWER

Figure 3.

Non-beginners’ Scores of Exam Test and Programming Task (A – experimental group, B – control group).

RQ2: How does feedback based on logs affect task completion time, number of runs, error messages, and pastes (of the whole group, of beginners and non-beginners)?

To describe the solution of the programming task more precisely than only through exam scores, we analyzed solution time, the number of runs, error messages, and pastes. The descriptive statistics about those indicators are presented in Table 4.

The Mann-Whitney U test result showed that the experimental group solved the task faster than the control group. The difference was statistically significant (U = 458.5, p < 0.05, η2 = 0.072). The density plot (see Figure 4) shows the programming task solving time distribution. The experimental group also had fewer runs and error messages, but the difference was not statistically significant.

We also made a separate analysis of learners with and without previous programming experience. The Mann-Whitney U test result showed that the beginners from the experimental group solved the programming task faster, and the difference was statistically significant (U = 100.5, p < 0.05, η2 = 0.131). For experimental group beginners, the average task solving time was 53 minutes, while control group beginners spent 72.1 minutes on average on the task. The students with previous programming experience in the experimental group solved the programming task faster than the control group, but the difference was not statistically significant (U = 131, p > 0.05, η2 = 0.024). The average time taken by the experimental group’s non-beginners was 44.8 minutes, while the control group’s non-beginners spent 53.6 minutes on the task. The density plots (see Figures 5 and 6) show the programming task solving time distribution among beginners and non-beginners.

Students in the experimental group had fewer runs and error messages than the control group (see Figure 7), but the differences were not statistically significant (Runs: U = 625.5, p > 0.05, η2 = 0.005; Error messages: U = 543.5, p > 0.05, η2 = 0.028). The control group had fewer pastes than the experimental group, but the difference was not statistically significant (Pastes: U = 678, p > 0.05, η2 = 0.0001).

The Mann-Whitney U test result showed fewer runs and error messages for beginners from the experimental group, but the difference was not statistically significant (Runs: U = 155, p > 0.05, η2 = 0.014; Error Messages: U = 137.5, p > 0.05, η2 = 0.039). Experimental group beginners had an average of 22.92 runs and 9.54 error messages in the programming task, while the respective values for control group beginners were 27.96 and 13.96. Control group beginners had fewer pastes than the experimental group (accordingly 5.43 and 8.31), but the difference was not statistically significant (U = 153, p > 0.05, η2 = 0.017). Students with previous programming experience in the experimental group also had fewer runs, error messages and pastes than the control group. The differences were not statistically significant (Runs: U = 148, p > 0.05, η2 = 0.005; Error messages: U = 135.5, p > 0.05, η2 = 0.017; Pastes; U = 138.5, p > 0.05, η2 = 0.014). The experimental group non-beginners had an average of 21.33 runs, 8 error messages and 3.83 pastes in the programming task; the control group non-beginners had 26.89, 12.04 and 6.59, respectively. The graphs (see Figures 8 and 9) show the distributions of runs, error messages and pastes among beginners and non-beginners in the experimental and control groups.

OPEN IN VIEWER

Table 4.

Descriptive Statistics of Programming Task Solution Time, the Number of Runs, Error Messages and Pastes.

		N	Min	Max	M	SD	Mdn	Sk	K
Programming task solution time	Experimental group	25	22	95	49.1	21.3	46	0.378	0.922
	Control group	55	22	96	63	24.5	66	0.267	−1.322
Number of runs	Experimental group	25	2	45	22.2	13.6	19	0.366	−1.150
	Control group	55	2	83	27.4	21	19	1.220	0.971
Number of error messages	Experimental group	25	0	31	8.8	8.5	6	1.374	1.014
	Control group	55	0	65	13	12.4	10	1.905	5.233
Number of pastes	Experimental group	25	0	28	6.2	7.3	4	1.994	3.567
	Control group	55	0	34	6	7.5	4	2.504	6.535

N - Number of cases; Min - Minimum; Max - Maximum; M - Mean; SD - Standard Deviation; Mdn - Median; Sk - Skewness; K - Kurtosis.

OPEN IN VIEWER

Figure 4.

Time spent on solving the programming task (A – experimental group, B – control group).

OPEN IN VIEWER

Figure 5.

Time spent by beginners on solving the programming task (A – experimental group, B – control group).

OPEN IN VIEWER

Figure 6.

Time spent by non-beginners on solving the programming task (A – experimental group, B – control group).

OPEN IN VIEWER

Figure 7.

The number of runs, error messages and pastes (A – experimental group, B – control group).

OPEN IN VIEWER

Figure 8.

Beginners’ number of runs, error messages and pastes (A – experimental group, B – control group).

OPEN IN VIEWER

Figure 9.

Non-beginners’ number of runs, error messages and pastes (A – experimental group, B – control group).