Enhancing student experience in remote computer programming course practice: A case of the Java language

Abstract

The increasing availability of remote education opportunities enables students to access different types of study programs limited by the dimensions of a classroom. Remote learning has been accredited as an effective technique for delivering education to students and teachers globally. However, some academic disciplines such as Computer Science are abounding with many challenges in practicing programming tasks in remote settings. This research aims to understand the students’ existing setup for practicing JAVA programming in remote settings and proposes an intervention to help students get appropriate hints to fix compiler errors. The proposed solution has implications for both students and teachers, as it is under the teacher’s direct observation. The teacher can monitor the content according to students’ needs, which may potentially save them from teaching challenges. To formulate our study, we first surveyed to understand the challenges that students face in JAVA programming when trying to fix errors in a remote setting. Then, based on the results of the first survey and prior experience of conducting programming course practicals along with the existing literature, the proposed web portal was created and presented to the students to fix errors. An experiment was conducted to evaluate the proposed setup followed by a student attitude survey. The results show the portal’s leverage in reducing the time required to fix errors and the number of compilation attempts. The survey indicates a positive response.

Keywords

Remote learning Programming course practice JAVA Computer Science education

Introduction

With the growth and globalization of higher education, universities and institutions compete for ranking positions and reputations by perfecting curriculums, on-campus facilities, and partnerships. They aim to provide a successful student experience. To increase the enrollment of students in their institutions, universities are increasingly employing technology-based instructional approaches, such as blended learning (Kumar et al., 2021), remote learning (Fujs et al., 2022), and often with various methodologies and approaches. Also, remote learning takes the focus away from on-campus life and moves it toward the quality of information and teaching style (Settle and Settle, 2007). As a result, students can easily enroll in different types of online programs to gain enough technical skills as per their requirements while learning or doing a job (Kumar et al., 2021). The biggest advantage of enrolling in a remote learning program is the flexibility and comfort of studying at their own pace (Sun and Chen, 2016). It is considered to be a relatively cheaper mode of education in terms of the lower cost of transportation, accommodation, and the overall cost of institution-based learning (Sun and Chen, 2016).

With today’s ubiquitous reach of technology, the market has become increasingly competitive, with providers vying for attention from a broad set of prospective students. From 2011 to 2021, the number of learners reached by MOOCs (Massive Open Online Courses) increased from 300,000 to 220 million (Shah, 2021). Between 2012 and 2019, the number of hybrid and distance learning students at traditional universities increased by 36%, while the circumstances of the COVID-19 pandemic in 2020 rapidly accelerated that growth by an additional 92% (Davis et al., 2018).

However, remote learning has been accredited as an effective technique for the global delivery of education materials to suit learners who cannot participate in a physical classroom setting due to time or distance constraints (Kumar et al., 2021). Some academic disciplines require laboratory components to conduct practicals and face the challenges of supporting remote learning students in lab classes. Students from different backgrounds, especially novice students, learning a programming language for the first time, face several challenges in fixing errors during their programming course practice in the absence of teachers and peers (Brown, 2014). In Computer Science, there is a significant lab component for students to practice computer programming problems with the aid and under the supervision of a teacher or a lab instructor. There are many technical and social barriers, especially for students pursuing computer programming lab classes. Lack of physical infrastructure like the affordability of resources at the individual level (e.g., personal laptop or desktop, software, and a stable high-speed Internet connection), and accessibility of the Internet in rural areas are the major technical barriers (Nugroho et al., 2020; Sun and Chen, 2016). Besides, setting up a software and hardware environment on their own may be a daunting task for many, especially for novices. An institutional computing lab provides a ready-to-use setup. Face-to-face interactions with teachers, lab instructors, and peers in understanding and fixing the compiler-generated errors are also missing and may be affecting the well-being of students in the remote learning mode. Due to all these barriers, a student may lose interest in practicing programming problems on their own, which may lead to poor practical knowledge of the subject (Nugroho et al., 2020). It may further affect their career choices or readiness for a job in the information technology (IT) industry.

Teaching and learning computer programming subjects in the remote learning mode has not received the needed traction so far. Evidence highlighting difficulties faced by teachers and learners for computer programming subjects has been derived from physically collocated students and teachers i.e., in the lab setting interactions rather than remote learning mode (Albrecht and Grabowski, 2020; Chatterjee et al., 2020; Jackson et al., 2005; Mooney and Becker, 2016; Walker and Schach, 1996). The lack of research is also due to the emergence of the COVID-19 situation in an unprecedented manner that posed mobility restrictions (Leal Filho et al., 2021). In India, computer education traditionally in the distance mode has a regular lab component on weekends (Jana, 2018). However, the trend towards remote learning has been accelerated due to an increase in the ownership of computing devices, and the availability of Internet resources, especially for economically sound urban students. This background makes the need for research well-timed in this direction to investigate which challenges students face while learning programming on their own in the absence of a teacher or a peer group.

An understudied aspect in this context is what happens when a student gets error messages in their computer program while performing computer programming practicals on their own in an out-of-lab setting. There is no support, which is otherwise available in a university lab setting, in the form of assistance from teachers, lab instructors, or the peer group. While learning to program, students have to deal with several types of errors (Albrecht and Grabowski, 2020). How do they deal with errors in the new setting where they are alone and have access only to web resources or offline materials like books?

Previous research shows that students learning JAVA through distance learning programs were less satisfied in comparison to their peers attending a regular university program (Settle and Settle, 2017). Experts agree that programming is a difficult task to learn for a variety of reasons (Mooney and Becker, 2016). Particularly, novice programming students face several challenges and difficulties with programming languages. Error messages produced by language compilers are typically concise and many novice programming students find them difficult to understand (Chatterjee et al., 2020). Many compilers were designed for experts rather than beginners. Sometimes the students are unable to discern the actual error from an error message generated by the compiler. Generally, they do not accurately reflect the actual cause and hint to fix the error, and that is a big challenge to fix an error. Additionally, other issues related to hints, solutions, or suggestions available online are:

• Novice programming students may not know how to find solutions to error messages online (Jackson et al., 2005), as they need skills to form a query to explore suggestions on error messages.

• As per our past teaching experience, an available solution may be outdated, and may not be compatible with the environment being used by the student. E.g., the interface of JDK 7.0 is different from JDK 13.0 in some features.

• There may be many solutions available online, and a student may find it hard to choose the best one (Chatterjee et al., 2020).

In this context, this research focuses on understanding the challenges faced by remote learning students in practical programming subjects and proposing an intervention to support them. We propose, myCodingPal, a web portal to assist students in fixing their coding errors by providing hints on error messages with similar examples. We also evaluate the proposed solution both quantitatively and qualitatively. The proposed solution has implications for students as well as teachers. Affording remote learning students with an up-to-date resource that is under the direct observation of their teacher and is also appropriate for their needs may enhance their learning experience and save them from challenges, e.g., information overload (Chatterjee et al., 2020) of using online QA websites like stackoverflow.com. For a teacher, it is also hard to help fix errors even if they engage online with students for programming practice because it is overwhelming and time-consuming.

Thus, the following four research questions guide our research agenda:

RQ1. What kind of support do students have or want, and what challenges do they face while fixing errors in remote settings?

RQ2. What leverage does the portal, myCodingPal, provide to fix programming errors?

RQ3. What attitudes do students have towards the proposed portal?

RQ4. What enhancements do students suggest for the proposed portal?

In the research in this context, we chose to study students learning the JAVA programming language, as it is the most widely used language for teaching and learning introductory programming languages (Ahmadzadeh et al., 2005; Coull, 2008). To formulate our study, we contacted the students of an undergraduate JAVA course at a large and well-established state university and administered surveys to understand their challenges, usage of the proposed web portal, and its effectiveness. The present study provides significant contributions by attempting to fill up several gaps.

To the best of the authors' knowledge and through the search of peer-reviewed databases, no previous study has explored the challenges students face in fixing programming errors in the absence of an instructor. Firstly, the study understands the challenges students face while learning computer programming languages in an out-of-university lab setting in the absence of an instructor. Secondly, in the era of huge collections of online resources available, novice students may get frustrated due to overwhelming information. It seems difficult for novice programming students to choose the best resource among themselves (Chatterjee et al., 2020). The study proposed a web portal to scaffold students struggling to find appropriate hints to fix errors in the absence of a teacher or a peer group. Third, the study presents an empirical evaluation of the web portal to check its effectiveness.

Literature review

Recently, there have been many studies specifically focused on JAVA programming courses. For an in-depth analysis, we work on the identification of common JAVA coding errors and scaffolding tools for learning programming languages, including JAVA.

The concept of observing students’ behaviors and mistakes while writing programs has a long history in computer science education research. Research has indicated that students who can learn a second programming language independently need help learning new structures; otherwise, they keep on using the ones that they are familiar with (Walker and Schach, 1996). It has been stated (Settle and Settle, 2017) that students did not learn JAVA well in an online learning environment. Some studies are focused on the type of compiler they used for JAVA programming (Ben-Ari, 2007). The error messages produced by the compiler are terse, and many novices find it difficult to achieve even syntactically correct programs. Many of them recommended using the Sun SDK (System Development Kit) either directly or indirectly through a development environment like BlueJ (Jadud, 2006), DrJava (Allen et al., 2002), and Eclipse (Rigby and Thompson, 2005), because of their superior error messages and their support for identifying and correcting syntax errors (Nugroho et al., 2020).

Identification of common JAVA coding errors

Many of the related studies focus on the classification of compiler error messages (Dy and Rodrigo, 2010; Hristova et al., 2003; Jackson et al., 2005; McCall and Kölling, 2015; Mow, 2012). The studies identified the most frequent errors among novice programming students (Jackson et al., 2005). Hristova et al. (2003) have compiled a list of common JAVA errors committed by students, combining the data from teachers and students from different institutes to ask them for the most common JAVA mistakes. Their goal was to enhance the compiler error messages. The result indicated a reduction in errors. Mow (2012) collected the most common errors generated by the students and classified them in the same way as Hristova et al., (Hristova et al., 2003). The study (Dy and Rodrigo, 2010) explored ‘non-literal errors’ where the compiler error messages do not match the actual error committed by the student. They focused on improving the error messages and presented a list of the most common compiler error messages. Some researchers (McCall and Kölling, 2015) focused on improving compiler error messages by developing and validating error categories and found that the categorization scheme can achieve higher precision in describing errors. Ben-Ari (2007) lists the compiler error messages and describes typical mistakes that give rise to them as well as prevent ways of debugging to avoid these errors.

Scaffolding tools for learning programming languages including JAVA

McCall (2013) suggests that compiler error messages have an imperfect (many-to-many) mapping to student misconceptions. Jadud (2006) looked at the types of mistakes committed by students in JAVA and the ways they solved them. Some investigators (Ahmadzadeh et al., 2005) looked at student error frequencies and debugging behavior. Schorsch (1995) introduced CAP (Code Analyzer for Pascal) to provide students with user-friendly feedback on syntax, logic, and style errors. The feedback informs the student what is wrong, why it is wrong, and how to fix the problem. The study (Flowers et al., 2004) introduced Gauntlet, designed for the cadets at the US Military Academy, which augmented the existing compiler with more meaningful messages. An Eclipse plug-in called Gild (Groupware-enabled Integrated Learning and Development) was introduced in 2003 (Storey et al., 2003) and has been used in Canada (University of Victoria), in the USA (Berkeley, Virginia Tech, and the University of Washington), and possibly other institutions. A follow-up study (Rigby and Thompson, 2005) concluded that “Gild will be more effective, efficient, and satisfactory. Another learning support tool designed for JAVA, called SNOOPIE (Supporting Novices in an Object-Oriented Programming Integrated Environment) provided students with more detailed feedback on syntax errors and semantic mistakes. A study implemented (Denny et al., 2014) a web-based tool called CodeWrite to assist students in writting code for the body of methods whose headers had been provided. Studies conducted by (Becker, 2015, 2016b) tested a custom editor called Decaf, which enhanced 30 high-frequency CEMs while intercepting the standard JAVA Compiler Error Messages (CEMs). The study concluded that ECEMs can benefit novice programmers.

At the University of Kent, the BlueJ IDE experimented with CS1 students using JAVA (Jadud, 2006; Jadud and Dorn, 2015). The study introduced the error quotient (EQ) metric, a measure of the number of errors per session based on the compilation events of students. The EQ is a number between 0 and 1, where 0 denotes the highest level of error-free programming and closer to 1 denotes the highest level of difficulty with fundamental programming ideas. In the following study (Becker, 2016b), Becker, 2016a introduced a new metric, called the repeated error density (RED) to quantify repeated errors. Pettit et al. (2017) designed an automated assessment tool (AAT), named Athene, that used data from existing research concerning compiler error frequency and approaches used to address this issue. He analyzed the past data to examine the frequency of compiler error types, and to improve the compiler error messages students receive when submitting similar code. The improvement received mostly positive verbal feedback from both students and instructors. An Intelligent Tutoring Companion, Annete (Day et al., 2019) was proposed in 2019 to support students in JAVA programming with intelligent feedback generated by a neural network. It can be embedded into the Eclipse IDE.

The PACA-ITS (pedagogical agent-based cognitive architecture) online practical intelligent virtual laboratory (IVL) platform was developed by Munawar et al. (2019). With the help of a multi-agent system, students can learn and complete practical tasks (like programming courses) anywhere. It was a web-based experimental application that was proposed to overcome the problems by enhancing the physical laboratory. A recent study (Denny et al., 2020) proposed a web-based debugging tool for the C language, to allow the students to edit the code and submit it for feedback directly. Some investigators (Karvelas et al., 2020), studied compiler error messages and their accessibility in programming environments used by novices.

Research methodology

This section presents details about the research experiments, including a selection of participants, instruments, and experiment design. A detailed view of the research study is mentioned in Figure 1. To achieve the research agenda, we conducted two experiments: first, to understand the challenges faced by students while performing practicals of the JAVA programming subject on their own; and second, to evaluate the usage, effectiveness, and improvement of the proposed portal.

Figure 1.

The activity diagram of the research methodology.

Selecting participants

The participants in these experiments were students belonging to the 5^th semester of an undergraduate course at a state-level public university in northern India. The age group of students lies between 21-23 years old. All the students were learning the JAVA programming language for the first time in their undergraduate course and had no exposure to JAVA IDE. However, all students had exposure to other programming languages, like C and C++ in their previous semesters. The students have experienced both, in general, in-the-lab (in their past semesters) and out-of-the-lab or online or remote (institutes closed due to the COVID-19 pandemic) settings for programming practice. The teacher and researchers motivated the students to participate in this experiment. Because the researchers were not currently teaching JAVA programming to the students participating in the experiment. So, to convince the students to participate in this study, the researchers requested the teacher (who is currently teaching JAVA) to encourage their students to participate in this experiment. Both the teacher and the researchers made them understand the significance of this research and the importance of their contribution. They joined the study voluntarily, also knowing that their performance in this experiment would not affect their academic scores.

To understand student demographics, data from the students’ profiles related to their academic details was captured. To begin with, a total of 51 students participated in the study for the first experiment. However, in the second part, seven students withdrew. Thus, the second experiment was completed with 44 students.

The experiment tasks

In the first experiment, the students were given a survey form, which included 11 items. For details of the survey instrument, please refer to the section Results & Analysis.

In the second experiment, students had to remove errors from four different programs (Figure 2–5) presented to them. Errors (Ko and Myers, 2005) in the programs are listed below:

Figure 2.

Code with Array bug.

Figure 3.

Code with Gestalt bug.

Figure 4.

Code with Spurious bug.

Figure 5.

Code with data-type inconsistency.

T1: Array bug i.e., error in array index expression due to lack of understanding of language (Figure 2).

T2: Gestalt bug i.e., unforeseen side effects of commands due to failure to see the whole picture (Figure 3).

T3: Spurious bug i.e., an unnecessary program element (Figure 4).

T4: Data-type inconsistency bug i.e., error due to misunderstanding of data types (Figure 5).

The tasks were planned keeping in mind the concepts that had been covered in the class. Past teaching practices of the researchers, as well as the existing literature on common errors in JAVA programs (see the related work section), influenced choosing these errors, as it was observed that the majority of the students need help to fix these types of errors. The difficulty level of these tasks was the same, which did not influence their feedback while solving the tasks with either online web resources or the proposed intervention. Each program code carried only one error, and the code was documented sufficiently with a description added in the beginning.

The second experiment was conducted in two iterations. In the first iteration, they were asked to solve the first two tasks T1 and T2, using existing web resources to seek hints to fix errors. In the second iteration, the students were given T3 and T4 tasks with errors and asked to seek help from the proposed web portal, myCodingPal. The second experiment was followed by another survey form, which included 14 items to determine the attitude of the students towards the proposed web portal, myCodingPal.

The survey instruments

The primary research instrument for the study was a set of two questionnaires developed by the researchers. The first survey (identified as FS) form includes 11 items focused on their programming practice experiences both in-the-lab as well as out-of-the-lab settings. It asked about the kind of support they have, and to investigate the challenges they face when they get an error message while doing their practical assignments. It also asked which online or web resources the students refer to fix their errors, if they do not have access to any help from their teacher or the lab instructors.

Items 1 and 2 were designed to determine the environment for the practical programming assignments that the students prefer i.e., remote (out of lab setting) or offline (in a lab setting), and their skill level in JAVA programming. Items 3 and 4 sought to determine what the students felt when they got stuck in an error, and how to fix it. Items 5 and 6 were designed to determine whom they approached for help. Items 7 and 8 asked about their preferences for web resources. Item 9 was designed to understand their willingness to use a dedicated portal for scaffolding when they are coding alone. The last two items, 10 and 11, gauged their readiness to conduct programming practice on their own.

The second survey which is a feedback questionnaire (identified as SS) was conducted after the programming experiment session to investigate the role of different offline vs. online affordances in fixing errors. The feedback questionnaire contains 14 items. Table 2 in section 3 has the list of the questions asked. The items 1st to 13th were designed to determine their attitude, productivity, confidence, frustration, and interest in programming using the 5-point Likert scale; 1- Strongly Disagree, 2- Disagree, 3- Uncertain, 4- Agree, 5- Strongly Agree. The 14th item was an open-ended question. The survey instruments have Cronbach’s alpha reliability scores of 0.74 and 0.81, respectively, indicating their good quality.

The proposed portal

To formulate our research, we developed a web-based portal called myCodingPal, for which we collected 75 JAVA error messages based on our experience of students committing errors during lab sessions, along with research literature. The collected error messages were put on a dedicated web portal created using Google Sites. It also included the causes of errors and their corresponding solutions, along with relevant examples. The error messages were categorized under different pointer labels like arrays, strings, exception handling, constructors, inheritance, packages & interface, polymorphism, and JAVA input/output so that searching can be easy. The errors that are common in all topics were labeled as general errors. Some screenshots of error messages collected in myCodingPal are listed below in Figure 6. The web portal is directly handled by the teacher to update the portal according to the topics covered in class so that students can practice their programming assignments at a remote site on their own.

Figure 6.

Screenshots of the web portal myCodingPal.

Data analysis

In the first survey form, all responses were analyzed with the chi-square test to understand the significance of the results. MS Excel was used to calculate the chi-square score and the corresponding level of significance.

For the coding part of the second experiment, two metrics were operationalized based on time and effort spent in fixing errors in the tasks. Averages of the metric values were further analyzed to compare the performance of the students. The time and effort were calculated using a screen recorder application (Free Screen Recorder for Windows 10.8, developed by Thundersoft).

In the second survey form, 13 items (SS1 to SS13) were given ratings ranging from 1 to 5. The average rating with a mean score above 3 served as an indicator of a student’s positive attitude towards myCodingPal. The mean value equal to 3 acts as a neutral attitude. A mean score of less than 3 indicated students' negative attitudes towards myCodingPal. The 14th item was an open-ended question for suggestions and enhancements students expected in the myCodingPal. Its answers were analyzed using thematic analysis and open coding.

Results and analysis

This section reports the research questions RQ1, RQ2, RQ3, and RQ4 and their respective results.

Demographic analysis

Participants in this study were students from the 5th Semester of a five-year integrated course, and they were studying the JAVA language for the first time. A total of 51 students participated in the first part of the experiment. However, in the second part, only 44 students participated, as 7 of them withdrew after the first experiment. So, the response rate of the first survey was 100%, and 80% for the second one.

All of them were exposed to different programming languages together in the same conditions after they joined the university. Before this, as per their past educational qualifications, they studied Computer Science with a similar school curriculum. The majority of the students were trained in STEM fundamentals.

RQ1: What kind of support do students have or want, and what challenges do they face while fixing errors in remote settings?

In the first part of the experiment, the students responded to the first questionnaire. There were two questions asked regarding the configuration of the environment setup for performing computer practicals using the JAVA programming language. The first two items focused on the students’ perceptions of their lab and language needs.

FS1 Which lab environment do you prefer when you need to fix errors in your programs?

options

A. Remote

B. University Lab.

For the FS1 item, 13 students chose option A, compared to 38 who chose B. χ2 = 12.25; df = 1; p < .01. Therefore, the difference is statistically significant.

FS2 To what extent can you develop solutions in the JAVA programming language? options

A. I can write programs and fix errors as well.

B. I can write programs but need help to fix errors.

C. I can write programs with help.

In response to this item, 16 chose option A, 24 chose option B, and 11 chose option C i.e., a good number is not independent enough to fix errors. χ2 = 5.05; df = 2; p = .07.

The following items from FS3 to FS5 are for understanding students’ experience of fixing errors.

FS3 What is your first response when you get an error in your program?

options

A. I get confused and leave the problem.

B. I am confident most of the time to solve the error myself.

C. I accept it as a challenge and can seek the help I may need.

In response to this item, 6 feel confused, 21 feel confident, and 24 take it as a challenge and seek help i.e., in general students are willing to fix errors. χ2 = 18.83; df = 2; p < .01.

FS4 When you get an error in your program, how do you approach fixing it?

options

A. I first try a solution from my own experience and seek help only when I am unable to.

B. I look for help immediately.

For the FS4 item, 23 students chose A i.e., first take some time to do it themselves, and 28 wanted to ask for help immediately. χ2 = 0.49; df = 1; p = .48.

FS5. To whom would you prefer to ask for help in the lab to fix errors?

options

A. peers B. Teacher or instructor C. None

For the FS5 item (see Figure 7), 43 students appreciated the presence of a teacher or instructor who can help in solving errors, and 8 liked to seek help from their peers. None of them chose option C i.e., everyone looked for help and most of them were from an expert. χ2 = 44.5; df = 2; p < .01.

Figure 7.

Representation of question FS6.

FS6.Give your order of preference to seek help to solve an error in your program while in a lab setting.

Results in Figure 8 show that students’ first choice in a lab setting is the teacher, followed by peers, and then search online if the Internet is available. It is interesting to see that even if the online option is there, students find it more useful to explore their social group first to seek help. However, in the distance or remote mode, this social support with face-to-face interaction is absent.

The following items from FS7 to FS10 are for understanding students’ experience of fixing errors in an out-of-lab or remote setting.

Figure 8.

Priority rankings of different preferences.

FS7. What problems do you face while finding solutions to fix errors on your own on the Internet?

Choose all applicable to you.

options

A. long posts with difficult-to-understand explanations

B. available solution is not just appropriate

C. too many solutions

D. looking for online solutions is time-consuming

E. others

For the FS7 item, 33 students selected all; 9 selected B, C, and D; 6 of them chose A and D; 3 chose E. This means there are many challenges with online resources.

FS8. Select the website that you refer to the most to fix programming errors.

Options

A. www.geeksforgeeks.com

B. www.JAVApoint.com

C. www.stackoverflow.com

D. www.tutorialpoint.com

E. www.w3schools.org

F. Any Google search result

G. others

In this response (Table 1), students’ interest in the two websites is very clear. Comparatively, few are using w3schools.com and stackoverflow.com as well. χ2 = 15.4; df = 6; p < .01.

Table 1.

Preferences for Web resources.

Web resources	No. of students
A	15
B	3
C	5
D	12
E	8
F	8
G	Nil

FS9. Would you prefer a web resource that provides hints in your context and at your level of understanding in an out-of-lab or remote setting?

options

A. Strongly Agree

B. Agree

C. Uncertain

D. Disagree

E. Strongly Disagree

For the FS9 item, 26 students admitted strong agreement, 11 agreed, 4 were not sure, 6 did not agree, and 4 strongly disagreed with the solution. χ2 = 28.2; df = 4; p < .01. It shows students are satisfied with a customized solution.

The next two items are there to understand and check the students’ readiness for practical classes in an out-of-lab or remote setting.

FS10 Which of the following devices do you have of your own for programming practice at home?

a. Laptop or desktop, b. Smartphone or tablet, c. none

options

A. both a and b

B. only a

C. only b

D. c.

In this item, 44 students have access to multiple devices at home including a laptop or desktop, smartphone or tablet. Everyone has a smartphone or tablet, but 7 have only a smartphone or tablet to write programs. χ2 = 89.34; df = 3; p < .01.

FS11. Have you ever set up a programming environment on your device in the past?

options

A. Yes, many times

B. Yes, once

C. Never, but I can do

D. Never, and I don’t think I can do

Setting up a programming environment is a major step in learning a programming language. In this survey, many students are familiar with the task (n = 35), very few are confident though they have not ever done it (n = 7), and some are varied in doing it on their own (n = 9). χ2 = 11.02; df = 3; p < .01.

RQ2: What leverage does the portal, myCodingPal, provide to fix programming errors?

In this question, we aim to evaluate the web-based resource, myCodingPal, customized for JAVA learners to help them find hints to fix errors on their own. Stackoverflow.com is a great QA site in the programming world for seeking help, but novice programmers find the content overwhelming (Chatterjee et al., 2020). Therefore, they need some customized web resources where they can find relevant answers easily. To answer this research question, we set up the first part of the second experiment through Google Meet, in which students were asked to fix errors in four different tasks in two iterations without any help from the teachers, instructors, or peer group. To calculate the time used for compilation and the number of compilations per program, the students were asked to record their screens using the Screen recorder application (Free screen recorder 10.8 for Windows) and submit the recording after the experiment. In the first iteration, they were asked to solve the first two tasks using the web resources of their choice. In the second iteration, they were asked to use the web resource myCodingPal, customized for them.

In iteration 1, students completed the two tasks one by one. They typed the code using the Eclipse IDE and started compiling the code. After compilation, when the IDE displayed the error messages, we noticed two strategies that the students followed:

1. Read the error message, and spend some time understanding it.

This was a small group of students. Out of them, some were able to understand and solve it on their own. The remaining were shifted to the next step.

2. Copy and paste the error message into the Google search bar. After looking at the search results, their actions can be categorized as:

• Choose the first result.

• Look for a specific site (e.g., www.tutorialpoint.com in the search results).

• Only a few of them opened the link for Stackoverflow.com in the search results.

Considering the strategy adopted for finding solutions, we divided the students into two groups:

1. Fixing errors on their own

2. Fixing errors by looking for any web resource

The results are obtained by quantitative analysis of their activity operationalized through two metrics:

The average time to fix an error, measured in minutes, is calculated from the time taken by each individual to fix the error and make the program compile successfully.

An average number of compilations is calculated from the number of times each student changes the program and recompiles to fix the error until the program compiles successfully.

Table 2 shows the results for the metrics. The majority of the students use web resources for help when any help from teachers, instructors, or peers is not available. No one consulted any book or class notes. The students who have completed the tasks on their own without any help are considered as the “Self” group and the students who completed the tasks using web resources and myCodingPal are considered as “Web resource” and “myCodingPal” groups as well.

First, considering the lowest numbers for all the tasks, 18/44 (p < .01) of the students did not consult any web resources and fixed the error in each task on their own, which is a significant difference. Their average time to fix errors varies for each task but it is <= 2.5 minutes. The number of compilations is also around 2.

For the other lot, using any web resource consumes more time as well as a greater number of compilations as time is spent searching for and trying different solutions. However, students’ dependence upon external help while using the proposed portal myCodingPal, is also close to the students solving on their own in terms of time and effort. Their average time to fix errors is <= 3 minutes, and the average number of compilations is <= 2.5.

It may be argued that exploring more resources can help with better learning. However, not everyone has the patience to see a large number of resources, especially novice learners. They may get frustrated due to loss of productivity in comparison to their peers, and lose confidence due to overwhelming information. It is a very time-consuming process to search for a solution using web resources. In Table 2, we have identified that the students using the web resource spent 7.5 minutes to complete task T1 and 8.3 minutes to complete task T2, which was much bigger than the time spent on tasks T3 and T4 using myCodingPal. Also, the number of compilations of tasks T1 and T2 using web resources was greater than the number of compilations of tasks T3 and T4 using myCodingPal.

The next research question finds answers to see how the proposed affordance affects students’ attitudes.

Table 2.

A quantitative view of fixing errors in four tasks.

Task	Group	Average time to fix error in minutes	Average number of compilations
T1	Self (n = 19)	1.3	2.1
T1	Web resource (n= 25)	7.5	4.5
T2	Self (n = 18)	2.5	2
T2	Web resource (n= 26)	8.3	4
T3	Self (n = 20)	2.1	2.3
T3	myCodingPal (n = 24)	3	2.5
T4	Self (n = 21)	1.5	2
T4	myCodingPal (n = 23)	2.5	2.1

RQ3: What attitudes do students have towards the proposed portal?

The answer to the third research question is furnished by employing a feedback survey form for the students. The students who fixed errors on their own were also asked to explore the portal and give their feedback. In the 2nd part of the experiment (1^st part discussed in RQ2), four tasks were conducted with the students to expose them to different resources available online for fixing errors.

The research question in this section is, “What attitudes do students have toward myCodingPal?” This research question is to evaluate their attitudes related to the proposed affordance.

Among the fourteen items in this survey form, thirteen (SS1 to SS13) have a five-point Likert scale, i.e.: “Strongly disagree”, “Disagree”, “Neutral”, “Agree”, and “Strongly agree”. Item number 14 (SS14) is an open-ended question aimed at seeking suggestions for improving the portal.

The average rating score for 11 items out of 13 is above the neutral value of 3 (see Table 3); hence, students' attitude towards the use of myCodingPal is positive. The two items, FS5 & FS6, have an average rating score below the neutral value of 3, which indicates that students are still not ready to think about a lab setting independent of the expert guidance available in a physical setup.

Table 3.

Students’ attitude towards the use of myCodingPal (N = 44).

No.	Items	Mean	Std. dev.
1.	This resource is useful for providing suggestions to fix errors in program code.	3.49	.703
2.	This resource is better rather other resources.	3.84	.574
3.	This resource is helpful for online study.	3.74	.759
4.	This resource replaces the other offline resources.	3.53	.827
5.	This resource replaces the teacher.	2.47	.909
6.	This resource replaces the Lab staff.	2.70	.887
7.	This resource increases my confidence in programming.	3.77	.611
8.	This resource helps to reduce my frustration.	3.58	.906
9.	This resource helps to increase my productivity.	3.91	.684
10.	I feel more interested after using this resource.	3.77	.684
11.	It can make programming easier for me.	3.77	.684
12.	It turns my attitude positive toward programming subjects.	3.60	.660
13.	This technique used in learning should be expanded to include other subjects.	3.81	.664

RQ4: What enhancements do students suggest for the proposed portal?

The last research question to understand the suggestions for improvement of the portal is answered through qualitatively analyzing the response to FS14, which was an open-ended question: What enhancements do you suggest for the proposed portal?

In response to this question, students admitted their opinions according to their interests. Some students found it very useful. Some excerpts of their comments regarding the usefulness of the portal were: “Very useful straight-to-the-point solution…”, “Awesome no need to enhance this…”, and “No, it’s good”. Some students wanted more explanation for error messages. Some of their submitted responses were like: “The solutions need to be expanded.”, “The solutions should be in more detail “, “More Clarity on answers.”, and “Can also put the visual representation of the solution so that in case any student finds it difficult to understand the language, he/she could understand the solution more easily…”. Many of these answers have redundant content. Some of the students wanted more enhancements like: “Some more hard problems should also be added”, “more Questions”, and “A wider variety of errors should be available.”.

Both authors coded student responses employing thematic analysis with an interrater agreement of 78%. Five main themes emerged and they are:

1) add harder problems;

2) make solutions more explanatory;

3) add more error cases;

4) visual representation of the solution;

5) nothing i.e., no suggestion.

Discussion

In this study, we analyzed that the majority of the students seek help to fix errors. Only a few students were confident in solving the errors themselves. Everyone else needs some resources for help, such as teacher/instructor scaffolding or materials like class notes, books or searching online (from websites). They prefer to be put in a physical lab setup e.g., a university lab. A teacher or an expert is the most trusted fellow to seek answers. It was observed in the ensuing experiment that students hardly looked into books or notes at the time of fixing errors. The few, who did, took a lot of time to return to the exercise.

The study identified that searching for hints on error messages using online web resources is overwhelming, and time-consuming. Therefore, providing them with a dedicated portal for querying error messages, and getting hints or suggestions may enhance their programming experience, which myCodingPal has attempted to do successfully. A teacher knows the best about the topics students are struggling with in class, so it will be in their interest if material focused on those topics is available to students when they are not in contact with their teacher at the time of programming practice in a remote setting. Moreover, a teacher can add any number of questions to the portal and can monitor the content regularly. Additionally, the platform is also capable of supporting any programming language. This study demonstrates the JAVA language. A major problem with online resources is that content becomes outdated as new versions of compilers or libraries become available. All students cannot afford the infrastructure required to attempt programming practice on the Internet. However, this may not always be true, as this study does not generalize to different strata of students coming from different backgrounds. It seems more helpful in the case of our proposed web portal myCodingPal, in which the teacher uploads the content regularly based on the IDE version students are using.

Keeping in view the challenges of exploring the plethora of web resources for finding solutions to fix errors, this research proposed a web portal, myCodingPal. We have presented evidence that myCodingPal reduces the average time to fix errors as well as the average number of compilations. This study is comparable with several previous studies, for the same language. Some interventions in the form of scaffolding tools for programming language learners, e.g., CodeWrite (Denny et al., 2014) or Decaf (Becker et al., 2016) have also been suggested in the past. Decaf is a custom JAVA editor that generates helpful ECMs (Enhanced Compiler error messages) along with original CEMs (Compiler error messages) for 30 selected CEMs for the user. Although the results of the study have shown that Decaf would help students to identify and correct errors, Decaf does not enhance three of the 15 most frequent error messages: ‘else’ without ‘if’, the illegal start of type, and ‘.class’ expected. Decaf only presents an enhanced version of the first error message when more than one is present to reduce the confusion of the students with multiple messages. It looks inconsistent in the case myCodingPal does not provide an enhanced version of error messages, rather it provides error messages with their possible causes and solutions. Moreover, the myCodingPal portal assists with 75 types of JAVA error messages. The solutions provided to the students were based on topics covered in the class and assignments given to students for practice. Another perception was that myCodingPal looks more useful than Decaf in the case that Decaf is an editor that may support a specific version of JDK. With the change of version, rewriting the JAVA editor for the new JDK version is a very complex and time-consuming process. But on the other side, just updating the solutions on myCodingPal according to the new version is a very easy and fast process.

CodeWrite (Denny et al., 2014) is a web-based tool that helps learners write the body of a method in JAVA in which the header of the method is always provided. CodeWrite passes the first two compiler-generated error messages back to students. The feedback provided by the enhanced feedback module typically contains the line where the error occurs and a detailed explanation of what is most likely causing the syntax error. The findings of the study revealed that students in the intervention group did not pay much attention to the additional information in the enhanced error messages. The analysis of the data shows no significant (or practical) effect. Although CodeWrite reported a similar concern to myCodingPal in presenting the error message with a detailed explanation and likely causes of the error. We focused on providing one compiler error message with the most likely causes and possible solutions with code examples to novice students. It is possible that being presented with one error message with cause and solution reduces frustration in students, potentially resulting in fewer errors.

It is unreasonable to think that providing solutions to compiler error messages will completely alleviate the problems students have with them. It is true that in the real-world setting, it will evolve as more topics and their related errors will be added. We agree that such a platform can never be exhaustive at any point in time, but it is beneficial for students when they are exploring a language as a beginner. Our intention in designing the portal was to improve remote teaching and learning by assisting students in programming assignments in the absence of an instructor or peer group, However, it has been shown that myCodingPal reduced the number of compilations and the average time to fix errors and provided a positive learning experience. We must identify some threats to validity.

One major threat of this study is the limited generalization of results as data is collected from a mid-sized university in a tier II city in India, there is a need to study this research question by taking students from universities or colleges of various settings to create good resources for novice learners with diverse profiles. The second threat is that students in the dataset are novice JAVA programmers, but they have exposure to programming in other languages. So, results may be different for students doing programming for the first time. Thirdly, the study does not take into account any differences based on students’ attitudes or attitudes towards programming. Fourthly, another limitation could be that the experiment for remote learning was simulated. Observing real-life situations, there might be different responses. Conducting such an experiment in a remote setting would have required making students record their activity on their own. It may have other kinds of challenges like the availability and accessibility of good quality computing and communication devices.

In this study, students show a good attitude towards the proposed portal. They feel it reduces their frustration and increases their confidence in fixing errors independently. They also reported that after using myCodingPal, they found that learning programming in a remote setting is easier, and fixing compiler errors is not much of a barrier to learning programming in a remote setting. Quantitative analysis has also shown that myCodingPal reduced their time and effort searching for hints on error messages. However, it is not perceived as a replacement for their teacher or other advice available in the lab from peers.

Conclusions and future work

This paper identifies the challenges students face in a remote learning environment and proposes a web-based resource, myCodingPal, to provide an appropriate platform for novice programmers to fix coding errors in the JAVA language without a teacher or any peer support.

To evaluate the effectiveness of the web portal, a mixed-methods study with students of an undergraduate degree course was performed. The quantitative analysis of the students' usage of the portal indicates that it provides leverage to its users in terms of saving time and effort while fixing errors. The student attitude survey also indicates a positive attitude towards the web portal myCodingPal. They feel more confident and less overwhelmed towards fixing errors in the absence of the teacher.

Future work can be targeted at understanding challenges in remote settings from other points of view e.g., building confidence to fix errors on their own, and challenges of fixing logical errors. In the future, there is a need to develop some interventions that play a successful role in the remote practice of programming courses by student learners. More research is needed to develop independent skills for fixing errors and learning to code in isolated environments.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Harpreet Kaur

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors upon reasonable request.*

Author Biographies

Harpreet Kaur is a PhD student in the Computer Science department at the Guru Nanak Dev University, Amritsar, India. She received her Master of Computer Applications degree from Punjab Technical University, Kapurthala, India in 2009. She has cleared UGC-NET in 2019. She has worked as an Assistant professor in the Department of Computer Science at various colleges and universities. Her research interests include Remote education, educational data mining, programming performance analysis, and computational thinking. Contact her at harpreetcsc.rsh@gndu.ac.in.

Kuljit Kaur Chahal is a Professor and Head of the Department of Computer Science, at Guru Nanak Dev University, Amritsar, India. Her research interest lies in open-source software, distributed systems, intelligent systems, cyber security, and computing education. She has been awarded research and travel grants by IBM Research Fund, Google, University Grants Commission (India), MHRD (India), and Guru Nanak Dev University (India). Contact her at kuljitchahal.cse@gndu.ac.in.

Kamaljit Kaur is an Assistant Professor (Senior Grade), in the Department of Computer Engineering and Technology, Amritsar, India. She completed PhD degree in the field of Cloud Computing from Guru Nanak Dev University, Amritsar in 2020. Her research interests include Resource Provisioning in Cloud Computing, Resiliency in Cloud Computing and Distributed Systems, Big Data, and IoT. She has published and presented more than 50 papers in scientific journals and international conferences. Contact her at Kamaljit.dcse@gndu.ac.in.

References

Ahmadzadeh

Elliman

Higgins

(2005) An analysis of patterns of debugging among novice computer science students. ACM SIGCSE Bulletin 37(3): 84–88.

Albrecht

Grabowski

(2020) Sometimes it’s just sloppiness studying students’ programming errors and misconceptions. In: SIGCSE 2020 - Proceedings of the 51st ACM Technical Symposium on Computer Science Education, Portland, OR, USA, 26 February 2020, pp. 340–345.

Allen

Cartwright

Stoler

(2002) DrJava. In: Association for Computing Machinery (ACM), 27 February 2002, pp. 137–141.

Becker

(2015) An Exploration of the Effects of Enhanced Compiler Error Messages for Computer Programming Novices. PhD Thesis. Dublin: Technological University.

Becker

(2016a) A new metric to quantify repeated compiler errors for novice programmers. In: Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, Arequipa, Peru, USA, 11–13-July: 296–301.

Becker

(2016b) An effective approach to enhancing compiler error messages. In: SIGCSE 2016 - Proceedings of the 47th ACM Technical Symposium on Computing Science Education, Memphis, Tennessee, USA, March 2 - 5, 2016, 126–131.

Becker

Glanville

Iwashima

, et al. (2016) Effective compiler error message enhancement for novice programming students. Computer Science Education 26(2–3), 148–175.

Ben-Ari

(2007) Compile and runtime errors in JAVA. Available at: http://stwww.weizmann.ac.il/g-cs/benari/

Brown

NCC

Altadmri A (2014) Investigating novice programming mistakes: educator beliefs vs student data. In: ICER 2014 - Proceedings of the 10th Annual International Conference on International Computing Education Research, (Glasgow, Scotland, United Kingdom), August 11 - 13, 2014, 43–50.

10.

Chatterjee

Kong

Pollock

(2020) Finding help with programming errors: an exploratory study of novice software engineers’ focus in stack overflow posts. Journal of Systems and Software 159: 110454.

11.

Coull

(2008) SNOOPIE: Development of a Learning Support Tool for Novice Programmers within a Conceptual Framework. PhD Thesis, University of St. Andrews, Epub ahead of print 2008.

12.

Davis

Hauff

Houben

(2018) Evaluating crowdworkers as a proxy for online learners in video-based learning contexts. In: Proceedings of the ACM on Human-Computer Interaction 2(CSCW). Association for Computing Machinery.

13.

Day

Penumala

Gonzalez-Sanchez

(2019) Annete: an intelligent tutoring companion embedded into the eclipse IDE. In: Proceedings - 2019 IEEE 1st International Conference on Cognitive Machine Intelligence, CogMI 2019, Los Angeles, California, USA, December 12-14, 2019, 71–80. Institute of Electrical and Electronics Engineers Inc.

14.

Denny

Luxton-Reilly

Carpenter

(2014) Enhancing syntax error messages appears ineffectual. In: ITICSE 2014 - Proceedings of the 2014 Innovation and Technology in Computer Science Education Conference,Uppsala, Sweden, June 23-25, 2014, 273–278.

15.

Denny

Prather

Becker

(2020) Error message readability and novice debugging performance. In: Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 2020, Trondheim, Norway, June 15-19, 2020. Association for Computing Machinery, 480–486.

16.

Rodrigo

(2010) A detector for non-literal JAVA errors. In: 10th Koli Calling International Conference on Computing Education Research, New York, NY, USA, 2010, 118–122. Association for Computing Machinery.

17.

Flowers

Carver

Jackson

(2004) Empowering students and building confidence in novice programmers through gauntlet. In: Proceedings - Frontiers in Education Conference. FIE, Savannah, GA, USA, October 19-22, 2004.

18.

Fujs

Vrhovec

Žvanut

, et al. (2022) Improving the efficiency of remote conference tool use for distance learning in higher education: a kano based approach. In: Computers and Education. Amsterdam, Netherlands: Elsevier Ltd.

19.

Hristova

Misra

Rutter

, et al. (2003) Identifying and Correcting JAVA Programming Errors for Introductory Computer Science Students. Association for Computing Machinery (ACM). 153.

20.

Jackson

Cobb

Carver

(2005) Identifying top JAVA errors for novice programmers. In: Proceedings - Frontiers in Education Conference, TX, USA October 18-21, 2005.

21.

Jadud

(2006) An Exploration of Novice Compilation Behaviour in BlueJ. PhD Thesis. Canterbury: University of Kent, 1–250.

22.

Jadud

Dorn

(2015) Aggregate compilation behavior: findings and implications from 27,698 users. In: ICER 2015 - Proceedings of the 2015 ACM Conference on International Computing Education Research, 9 July 2015. Association for Computing Machinery, Inc, 131–140.

23.

Jana

(2018) How does the BCA practical happen in the distance learning program of IGNOU? Available at: https://www.quora.com/How-is-IGNOUs-distance-learning-programme-for-BCA (accessed 10 March 2024).

24.

Karvelas

Dillane

Becker

(2020) Compiler error messages: their content and accessibility in novice programming environments. In: SIGCSE 2020 - Proceedings of the 51st ACM Technical Symposium on Computer Science Education, Portland, OR, USA, March 11 - 14, 2020, 1310.

25.

Myers

(2005) A framework and methodology for studying the causes of software errors in programming systems. Journal of Visual Languages and Computing 16(1–2SPEC. ISS.): 41–84.

26.

Kumar

Krishnamurthi

Bhatia

, et al. (2021) Blended learning tools and practices: a comprehensive analysis. IEEE Access 9 . Institute of Electrical and Electronics Engineers Inc., 85151–85197.

27.

Leal Filho

Azul

Wall

, et al. (2021) COVID-19: the impact of a global crisis on sustainable development research. Sustainability Science 16(1): 85–99.

28.

McCall

(2013) Improving the Experience of Novice Programmers – Language and Tools Technical Report. UK: University of Kent.

29.

McCall

Kölling

(2015) Meaningful categorization of novice programmer errors. In: Proceedings - Frontiers in Education Conference, FIE, El Paso, Texas, USA, 21-24 October 2015.

30.

Mooney

Becker

(2016) Categorizing compiler error messages with principal component analysis. Available at: https://www.researchgate.net/publication/308888663.

31.

Mow

(2012) Analyses of student programming errors in JAVA programming courses. Journal of Emerging Trends in Computing and … 3(5): 739–749.

32.

Munawar

Toor

Aslam

, et al. (2019) PACA-ITS: a Multi-agent system for intelligent virtual laboratory courses. Applied Sciences (Switzerland) 9(23). MDPI AG.

33.

Nugroho

Basari

Suryaningtyas

, et al. (2020) University students’ perception of online learning in Covid-19 Pandemic: a case study in a translation course. In: Proceedings - 2020 International Seminar on Application for Technology of Information and Communication: IT Challenges for Sustainability, Scalability, and Security in the Age of Digital Disruption, iSemantic 2020, 19 September 2020. Institute of Electrical and Electronics Engineers Inc, 225–231.

34.

Pettit

Homer

Gee

(2017) Do enhanced compiler error messages help students? Results inconclusive. In: Proceedings of the Conference on Integrating Technology into Computer Science Education, ITiCSE, 8 March 2017. Association for Computing Machinery, 465–470.

35.

Rigby

Thompson

(2005) Study of novice programmers using Eclipse and Gild. In: Proceedings of the 2005 OOPSLA Workshop on Eclipse Technology eXchange, Eclipse’05, 105–109.

36.

Schorsch

(1995) CAP: an automated self-assessment tool to check Pascal programs for syntax, logic, and style errors. In: Proceedings of the Twenty-Sixth SIGCSE Technical Symposium on Computer Science Education, NY, USA, March 1995, 168–172. DOI: 10.1145/199688.199769.

37.

Settle

(2007) Distance learning and student satisfaction in JAVA programming courses. Journal of Universal Computer Science 13: 1270–1286.

38.

Shah

(2021) Article title. In: A Decade of MOOCs: A Review of Stats and Trends for Large-Scale Online Courses in 2021. Available at: https://www.edsurge.com/news/2021-12-28-a-decade-of-moocs-a-review-of-stats-and-trends-for-large-scale-online-courses-in-2021 (accessed 10 March 2024).

39.

Storey

M-A

Michaud

Mindel

, et al. (2003) Improving the usability of eclipse for novice programmers. In: Proceedings of the 2003 OOPSLA Workshop on Eclipse Technology eXchange, Anaheim, CA, USA, October 2003, 35–39. DOI: 10.1145/965660.965668.

40.

Sun

Chen

(2016) Online education and its effective practice: a research review. Journal of Information Technology Education: Research 15(2016): 157–190.

41.

Walker

Schach

(1996) Obstacles to learning a second programming language: an empirical study. International Journal of Phytoremediation 21(1): 1–20.