Promoting Students’ Programming Logic and Problem-Solving Awareness With Precision Feedback: A Two-Tier Test-Based Online Programming Training Approach

Abstract

Fostering students' computer programming skills has become an important educational issue in the globe. However, it remains a challenge for students to understand those abstract concepts when learning computer programming, implying the need to provide instant learning diagnosis and feedback in computer programming activities. In this study, a Two-Tier Test-Based Programming Training (T³PT) approach was proposed. Accordingly, an online learning system was developed to provide students with precision feedback for guiding them to identify misconceptions of computer programming to improve their computer programming learning achievement. In order to examine the effects of the proposed approach, a learning system was developed and a quasi-experiment was conducted. Two classes of 99 eighth-grade students from Taiwan were divided into an experimental group and a control group. The students in the experimental group used the learning system based on the T³PT approach, while the control group used the conventional learning system. The experimental results showed that the proposed approach was significantly superior to the conventional programming learning approach in terms of students' programming logic concepts, problem-solving awareness, technology acceptance, and satisfaction with the learning approach. Accordingly, discussion and suggestions are provided for future research.

Keywords

two-tier test problem-solving awareness technology acceptance computer programming online learning

Introduction

With the rapid development of information technology, the demand for software in various fields is increasing day by day, and programming ability is growing in importance. Wing (2006) believed that in the 21st century, like writing, reading, and arithmetic, computational thinking is also a fundamental skill that everyone should acquire, not only computer scientists but also children. The most important aspect of learning programming is to learn the thinking patterns often used in the field of computer science. By understanding algorithms and logic concepts, students learn how to precisely describe the problem-solving process and use the thinking patterns to solve similar problems. Scholars have pointed out that Computational Thinking can be used in many daily contexts. It can not only help students solve problems related to course content, but also solve problems in everyday life (Liu et al., 2021). Yildiz Durak (2020) pointed out that learning programming can cultivate students' problem-solving skills and higher-order thinking skills. Therefore, Computer Science Education (CSE) has been paid increasing attention in recent years, in which programming plays an important role (Romero et al., 2017). Therefore, many countries have been incorporating computer science into the K-12 curriculum and teaching programming knowledge and skills such as syntax, algorithms, and debugging (Price & Price-Mohr, 2018; Wu et al., 2020).

For beginners of computer programming, the abstract logic concepts and thinking patterns in programming can easily lead to students' learning difficulties (Anniroot & De Villiers, 2012; Medeiros et al., 2018). When students have insufficient prior knowledge and encounter syntax errors, many programs cannot be compiled successfully (Ullah et al., 2018). Besides, they may not be familiar with the text programming interface because they are used to the graphical environment and mouse operation (Rodríguez Corral et al., 2014), which also leads to learning difficulties for students. As a result, many research teams have developed easy-to-use, user-friendly programming software and systems to assist in learning computer programming using such software as Scratch, Alice, Kodu, and ToonTalk. These software and systems replace complex programming syntax with graphical programming puzzle blocks to reduce execution errors caused by students' unfamiliarity with programming syntax, thus making it easier for students to complete their programs (Chang, 2014). For example, Scratch is a well-recognized visual programming language with a user-friendly graphical interface (Alonso, 2020). Several previous studies have reported the potential of using such visual programming systems to foster novice learners’ computational thinking (Wu & Su, 2021) and creative thinking (Su et al., 2021; Resnick et al., 2009) as well as programming syntax (Meerbaum-Salant et al., 2013).

However, the reason for the difficulty of learning programming is not only the lack of prior knowledge of programming, but also the lack of problem-solving skills. Usually, engineers need to use logical thinking and problem-solving skills when writing programs (Psycharis & Kallia, 2017). Therefore, the structured graphical presentation of visual programming software can lower the threshold of entry into programming and, most importantly, help students to understand program architecture, execution processes, and programming logic through observation, decomposition, imitation, reorganization, testing, and debugging. It also helps develop students’ core competencies in computational thinking, such as decomposition, pattern recognition, pattern generalization and abstraction, and algorithm design. Scholars have pointed out that it is crucial to help students identify real logical problems during the programming process (McCall & Kölling, 2019; Milková & Hulkova, 2013). From the perspective of precision education, it is important to guide individual learners to practice and make improvements during the learning process based on the best available evidence (Cook et al., 2018; Hart, 2016). Therefore, providing accurate feedback has become an important challenge in the learning process (García-Dantas & Quested, 2015; Mirosław & Bujak, 2017).

The Two-tier Test is considered to be an effective tool (Treagust, 1988) for assessing students' understanding of programming concepts. It consists of two tiers of multiple-choice questions; the first tier is used to assess students' descriptive or factual knowledge, and the second tier is used to investigate the reasons for students' choices made in the first tier test to help them correct their misconceptions and improve their understanding of the learning concepts (Halverson et al., 2011). It is important to prevent students from giving up when they encounter various challenges and difficulties in the process of learning programming by providing immediate precision feedback, thus helping them learn the logic and algorithms of the program, and the skills to solve problems (Psycharis & Kallia, 2017).

Therefore, a Two-Tier Test-Based Programming Training (T³PT) approach learning system was developed to assess students' understanding of programming logic concepts and to provide precision feedback to enhance their understanding of the correct concepts of programming logic. At the same time, an experiment was conducted to answer the following research questions to test the efficacy of the proposed methodology.

(1) Can the T³PT approach enhance students' learning achievement compared to the conventional computer programming learning approach?

(2) Can the T³PT approach enhance students' problem-solving awareness compared to the conventional computer programming learning approach?

(3) Can the T³PT approach enhance students' technology acceptance compared to the conventional computer programming learning approach?

(4) Is the T³PT approach more satisfying to students than the conventional computer programming learning approach?

Based on the research questions, we proposed the following hypotheses:

H1: The students learning with the T³PT approach would show better learning achievement than those learning with the conventional computer programming learning approach.

H2: The students learning with the T³PT approach would show better problem-solving awareness than those learning with the conventional computer programming learning approach.

H3: The students learning with the T³PT approach would show higher technology acceptance than those learning with the conventional computer programming learning approach.

H4: The students learning with the T³PT approach would show higher learning satisfaction than those learning with the conventional computer programming learning approach.

Related Research

The two-tier test refers to the testing strategy that uses two tiers of questions to assess learners’ understanding of a specific topic: the first tier question is designed for assessing learners’ knowledge or comprehension regarding the topic, while the second tier is to examine learners’ reasons for giving the answer in the first tier (Treagust, 1988). Two-tier tests solve the problem that the general multiple-choice test cannot distinguish the reasons why students choose right or wrong answers (Çil, 2015), allowing a deeper understanding of students' understanding of programming concepts (Lin, 2016).

The two-tier test has been shown by many researchers to be an effective approach to investigating students’ prior knowledge or misunderstandings, especially in science education. Chang et al. (2020) proposed a non-immersive virtual reality (VR) guidance system combined with a two-tier strategy which improved the students' learning motivation and learning achievement in a natural science course. Kılıc and Saglam. (2014) used a two-tier test in a biology course to investigate students' understanding of genetic concepts and reasoning ability to help students understand the deeper learning content. Çil (2015) applied the Two-tier Test in a physics laboratory course as an effective tool to help students understand the concept of complex experimental variables, and also confirmed that it can improve students' learning achievement. In addition, Chu and Chang (2014) used the Two-tier Test as a learning guidance strategy to help students engage in high-order thinking and reflection. Lin (2016) developed a Two-tier Test system to improve students' responses to cognitive judgments and to reduce students' use of examination techniques, leading to better diagnostic outcomes. Yang et al. (2015) pointed out that the two-level test can provide precise guidance for the myths encountered by students in the learning process. Ullah et al. (2018) further pointed out that providing precise guidance helps improve students' programming skills. Many scholars have also stated that in the learning interaction of visual computer programming, it is very important to provide accurate feedback to address learning difficulties (Marwan et al., 2020; Yang & Lu, 2021). Through precision feedback, learners can understand the difference between their actual and expected learning outcomes and change their perceptions of the content (Yang et al., 2015).

Therefore, this study proposed the T³PT approach to developing online computer programming learning systems for beginners. It is expected that the proposed approach can help to assess students' understanding of programming concepts and to provide precision feedback to correct misconceptions and strengthen their understanding of programming concepts.

The Two-Tier Test-Based Programming Training (T³PT) System

The T³PT system was developed using the web programming languages HTML5, CSS3, JavaScript, and jQuery, and was designed with software-assisted systems such as Sublime Text three and Adobe graphics for presentation of the dynamic learning screen. The structure of the T³PT system is shown in Figure 1. The front end of the system includes a two-tier test subsystem and a precision feedback subsystem, and the back end is a database for data management and recording.

Figure 1.

Structure of the T³PT system.

The front-end two-tier test subsystem includes two-tier test modules, program animation modules, program building blocks, and animation modules. The front-end feedback subsystem includes feedback modules, knowledge bases, and program animation modules. The program animation module is used to help students clarify the difference between the correct answer and the incorrect answer in the dynamic execution result. In addition, the knowledge base is used to provide other supplementary materials to help students in their learning. In order to record the learning status of students, Firebase back-end database processing is used to manage and record the data of the learning process, including a test question bank, learning materials, the student learning process, and a component database.

The two-tier test helps teachers to assess students' mastery of programming concepts and to give them timely precision feedback to strengthen their understanding of programming concepts. In the two-tier test subsystem, each question is divided into two tiers. The first tier of questions assesses students' understanding of the programming concepts. The second tier of questions is based on the answer chosen in the first tier and is used to further understand one of the reasons why students chose the answers in the first tier, in order to detect whether students have misconceptions about the programming concepts. The first tier is multiple-choice, and the options are animated to present the results of the program execution. For example, in the first question, the code contains the logic concept of loops and functions, and in the first tier, students are asked to read the code to judge the result of execution and select the corresponding option. Figure 2 shows the interface of the first tier of the two-tier test.

Figure 2.

The interface of the first tier of the two-tier test.

In the second tier, students are asked to further consider the reasons for choosing this option in the tier and are asked to select the logical and conceptual reasons from the matching questions. So as to be similar to the Scratch software, it was designed as a drag-and-drop block to make a choice. Students can drag and drop the appropriate blocks from the right box and combine them into the code on the left. For example, if a student selects the correct execution result as option C in the first tier test, the second tier will display the student’s selected execution result at the top of the page. The option design is to break down the logic concept related to the execution result and simplify the code as options for the second tier questions. The second level of the two-level test is shown in Figure 3.

Figure 3.

The interface of the second tier of the two-tier test.

The feedback subsystem is a system that provides timely precision feedback after diagnosing the answers selected by the student on a test. If the diagnosis reveals that the student has chosen correctly on both tiers of the test, then the next learning objective is pursued. If an incorrect choice is made in a question, the learning system makes a judgment and gives precise feedback based on the different incorrect choices. If the student is unfamiliar with the programming concepts or has a wrong understanding of how they are used, and alternative concepts or misconceptions arise, the student is asked to think about the difference between the wrong choice and the correct choice to clarify the misconceptions. As shown in Figure 4, the left side of the figure shows the correct result of the execution of the topic code, while the right side shows the result of the execution of the option chosen by the student and the code content.

Figure 4.

The interface of the feedback subsystem of the two-tier test.

If the system diagnoses that a student is not clear about the meaning of a primary concept (e.g., a definition function), it provides the student with a knowledge base of the concept for study (as shown in Figure 5) and then asks the student to answer again. Such a mechanism ensures that students have a correct understanding of the relevant concepts.

Figure 5.

The interface of the knowledge base of the two-tier test.

Method

A mixed research design was adopted to investigate the effectiveness of the T³PT approach. The quantitative data from the tests and scales were collected to analyze the effects of the approach on students' learning achievement, problem-solving awareness, technology acceptance, and satisfaction with learning approaches; moreover, the qualitative data from randomly sampled interviews were collected to understand the students’ perceptions of the learning activities.

Participants

Two classes of eighth-graders participated in the study. Their average age was 14.2. One class was assigned to be the experimental group, and the other class was the control group. The T³PT approach was used for the experimental group with 49 students (21 females and 28 males), while the conventional computer programming approach was used for the control group with 50 students (24 female and 26 male). The learning content was Scratch programming in an eighth-grade computer science class. In order to ensure the validity of the research, the two classes were taught by the same teacher; moreover, the two groups of students were scheduled to learn the same content in the same length of time.

Data Gathering Tools

To assess the participants' computer programming learning achievement, a pre-test and post-test were designed, which were developed and reviewed by two experienced information technology teachers. The pre-test was designed to test students' prior knowledge of programming, including 20 multiple-choice questions on basic computer programming concepts, each worth five points, for a total of 100 points. The post-test was designed to examine students' programming knowledge and programming logic knowledge based on the content of the learning activities, including 15 multiple-choice questions on programming knowledge and 10 multiple-choice questions on programming logic, each worth four points, for a total of 100 points. Memory-type questions were used to evaluate students' understanding of each program instruction in Scratch programming, for example, “Which of the following can name the blocks and functions by itself, and improve the legibility of the program? (A) Repeated structure (B) Function (C) Variable (D) Broadcast”. Logical type questions were used to evaluate students' logic concepts and programming application ability, for example, “Draw a hexagon using coding blocks.” The KR20 values for the pre-test and post-test were 0.70 and 0.61, respectively, indicating acceptable internal consistency.

The problem-solving awareness scale was a self-reported questionnaire adapted from Heppner and Petersen (1982) to investigate the effects of peer assessment on students’ problem-solving awareness, which adopted a 5-point Likert scale (1 = totally disagree, 5 = totally agree). This scale included five items, for example, “I always try my best to solve the problems I encounter,” and its Cronbach’s α value was .85.

The technology acceptance scale was modified from the scale proposed by Davis (1989), which adopted a 5-point Likert scale (1 = totally disagree, 5 = totally agree). The technology acceptance scale consists of 13 questions on the cognitive usefulness and cognitive ease of use of the learning system, for example, “The learning approach helped me learn better.” The Cronbach’s alpha for cognitive usefulness was .94, and the Cronbach’s alpha for cognitive ease of use was 0.95. The technology acceptance scale was conducted at the end of the activity to examine the students' acceptance of technology during the learning activities.

The satisfaction with learning approach scale was modified from the scale proposed by Hwang et al. (2013), which adopted a 5-point Likert scale (1 = totally disagree, 5 = totally agree). This scale included seven items, such as “I quickly learned to use the learning system.” The Cronbach’s α value of the scale was .87. The satisfaction with learning approach scale was conducted at the end of the activity to explore students' satisfaction with the learning approach in the process of carrying out the learning activities.

The interview outline was modified from the outline proposed by Hwang et al. (2009). The interview aimed to investigate the strengths and weaknesses of the learning approaches from the students' perspectives. It was expected that the results of the interview could provide more evidence to explain the findings of this study as well as references for improving the approach in the future.

Experimental Process

The experimental procedure of this study is shown in Figure 6. The experimental group used the T³PT approach. On the other hand, the control group used the conventional computer programming learning approach, that is, the test system only displays multiple-choice questions about programming concepts. When students chose the correct choice, they entered the next learning goal test. When the students chose wrongly, the system gave them the teaching material of the relevant learning concept as feedback, so that they could understand the knowledge.

Figure 6.

Experimental process.

Before the learning activity, students in both the experimental and control groups were required to take a pre-test and fill out a pre-questionnaire to confirm their basic knowledge of Scratch programming and their ability to perform. Then, the two groups of students used the learning platform for 2 weeks of 90-minute programming learning. The only difference was that the experimental group learned with the two-tier test approach during the learning process, while the control group learned with the conventional computer programming learning approach. That is, after answering the first tier question, the experimental group students were guided to answer the second tier question, while the control group received direct feedback (correctness of answers) and explanations (relevant information and explanations of the questions). The difference between the experimental group and control group is shown in Figure 7.

Figure 7.

Difference between the experimental group and the control group.

After the learning activities, students were asked to complete a computer programming learning achievement post-test (45 minutes), as well as a post-questionnaire of problem-solving awareness, technology acceptance, and satisfaction with the learning approach (30 minutes). Finally, three students from each class were randomly selected for a 10-minute group interview.

Results

Learning Achievement

In this study, analysis of covariance (ANCOVA) was used to exclude the differences in pre-test scores between the experimental group and the control group, and to compare whether there were significant differences in the two groups’ post-test scores. The programming post-test included memory-type questions and logic-type questions, which are discussed in the following two parts. The programming post-test included memory-type questions and logic-type questions, so the discussion is divided into two parts.

Firstly, one-way ANCOVA was performed on the memory-type programming questions. According to Levene’s homogeneity test, the variance of learning achievement from memory-type questions between the two groups was similar (F = 0.08, p > 0.05), indicating that there was no significant difference in the dispersion of the two groups of samples. The results of the homogeneity test of regression coefficients within the group showed that there was no interaction between the pre-test and post-test of learning achievement in terms of the memory-type questions (F = 3.84, p > 0.05), indicating that the hypothesis of homogeneity of the regression coefficient in the group was confirmed. The Shapiro–Wilk test was employed to examine the normality of the data. The value of this test was 0.90, p = 0.23, indicating that the data had normal distribution. Therefore, ANCOVA could be performed. As shown in Table 1, the results of ANCOVA measured after programming memory-type questions showed that there was no significant difference (F = 1.50, p= > 0.05).

Table 1.

Results of ANCOVA of learning achievement (memory-type questions).

Groups	N	Mean	SD	Adjusted mean	Std. error	F	η2
Experimental	49	41.31	11.00	41.17	1.25	1.50	0.02
Control	50	38.88	9.36	39.01	1.24

Secondly, according to the results of Levene’s homogeneity test, there was a significant difference in the variance of the two groups of students (F = 5.99, p < 0.05), which violated the homogeneity assumption and so could not be used for ANCOVA. Therefore, an independent sample t test was used. The independent sample t test was conducted on the pre-test of the basic knowledge of computer programs of the two groups of students. The results showed that there was no significant difference in the two groups of students’ basic knowledge of computer programs before the learning activities (t = 0.25, p > 0.05). The Shapiro–Wilk test was employed to examine the normality of the post-test. The value of this test was 0.74, p = 0.46, indicating that the post-test had normal distribution. Then, an independent sample t test was conducted on the learning achievement post-test of the logic-type questions, and the results showed that there was a significant difference in the learning achievement of the two groups of students (t = 2.25, p = 0.026 < 0.05), as shown in Table 2. The effect size was 0.46 (Cohen’s d = 0.46), indicating a moderate correlation between the explanatory power of the independent and dependent variables. The results showed that the T³PT approach was helpful for improving students' learning of programming logic concepts.

Table 2.

Results of the t test on learning achievement (logic-type questions).

Groups	N	Mean	SD	t	d
Experimental	49	21.71	8.41	2.25^a	0.46
Control	50	18.32	6.47

^ap < .05.

Problem-Solving Awareness

Problem-solving awareness was analyzed using ANCOVA, with the problem-solving awareness pre-questionnaire as the covariate and the problem-solving awareness post-questionnaire as the variable. According to Levene’s homogeneity test, the variance between the two groups was similar (F = 0.98, p > 0.05), indicating that there was no significant difference in the discretization of the two groups of samples. According to the homogeneity test results of the regression coefficient within the group, there was no interaction between the pre- and post-questionnaires of problem-solving awareness (F = 2.31, p > 0.05), indicating that the hypothesis of homogeneity of the regression coefficient in the group was confirmed. The Shapiro–Wilk test was employed to examine the normality of the data. The value of this test was 0.81, p = 0.33, indicating that the data had normal distribution. Therefore, ANCOVA could be performed. As shown in Table 3, according to the analysis results, the average score of the experimental group using the T³PT approach was 4.07, and the adjusted average score was 4.05. The average score of students in the control group was 3.67, and the adjusted average score was 3.70. After excluding the influence of the former questionnaire, the results showed that the students in the experimental group and the control group had a significant difference in problem-solving awareness (F = 7.36, p=0.008 < 0.01). The effect size was 0.07 (η² = 0.07), indicating that the explanatory power of the independent variable and dependent variable is moderately correlated. In other words, after using the T³PT approach, students in the experimental group showed significantly higher problem-solving awareness than those in the control group.

Table 3.

Results of ANCOVA of problem-solving awareness.

Groups	N	Mean	SD	Adjusted mean	Std. error	F	η2
Experimental	49	4.07	0.76	4.05	0.09	7.36^a	0.07
Control	50	3.67	0.74	3.70	0.09

^ap < .01.

Technology Acceptance

In order to understand whether there were differences in students’ technology acceptance, a technology acceptance questionnaire was used, which was divided into two aspects: perceived usefulness and perceived ease of use. After the learning activity, the students' technology acceptance data were collected. The Shapiro–Wilk test was employed to examine the normality of the data. The value of this test was 0.87, p = 0.18, indicating that the data had normal distribution. The independent sample t test was used for analysis to determine whether there was a significant difference between the two groups of students in the two aspects of technology acceptance after completion of the learning activities. Table 4 shows the results of the independent sample t test of the two groups of students’ technology acceptance. In terms of perceived usefulness, the average score of students in the experimental group was 3.60, and that of students in the control group was 3.26. The statistical results showed that there was a significant difference in the two groups’ perceived usefulness (t = 2.06, p < 0.05; Cohen’s d = 0.42), indicating that the explanatory power of the independent variable and dependent variable was moderately correlated. In terms of perceived ease of use, the average score of students in the experimental group was 3.38, while that of students in the control group was 3.19. Statistical results showed that there was no significant difference in the two groups’ perceived ease of use (t = 1.15, p > 0.05).

Table 4.

Results of the t test of technology acceptance.

Dimension	Groups	N	Mean	SD	t	D
Usefulness	Experimental	49	3.60	0.80	2.06^a	0.42
Usefulness	Control	50	3.26	0.82
Ease of use	Experimental	49	3.38	0.83	1.15	0.23
Ease of use	Control	50	3.19	0.79

^ap < .05.

According to the above-mentioned independent sample t test analysis results, it can be seen that the usefulness of the learning approach that combined the two-tier test and precision feedback mechanism was significantly different from that of the general program learning approach; and the ease of use for the two groups of students was not significantly different, but the average score of ease of use of the experimental group was higher than that of the control group.

Satisfaction With the Learning Approach

In order to find out whether there was a difference in students' satisfaction with the learning approaches after the learning activities, data on the students' satisfaction with the learning approaches were collected. The Shapiro–Wilk test was employed to examine the normality of the data. The value of this test was 0.87, p = 0.18, indicating that the data had normal distribution. The independent sample t test was used for analysis to determine whether there was a significant difference between the two groups of students in the two aspects of satisfaction with the learning approach after completion of the learning activities. Table 5 shows the results. The mean score for the experimental group was 3.58 and the mean score for the control group was 3.21. The statistical results showed that the two groups of students reached a significant difference in their satisfaction with the learning approach (t = 2.27, p = 0.025 < 0.05). The effect size, Cohen’s d = 0.42, indicated that the explanatory power of the independent variable was moderately correlated with the dependent variable.

Table 5.

Results of the t test of satisfaction with the learning approach.

Groups	N	Mean	SD	t	d
Experimental	49	3.58	0.80	2.27^a	0.46
Control	50	3.21	0.81

^ap < .05

Results of the Interview

After the learning activities, three students from each class were randomly selected, and a total of 12 students were interviewed for 10 minutes to find out how they felt about the learning activities and how they responded to the learning approaches, including the test mode, feedback mechanism, and other suggestions.

Experimental Group

In terms of the test mode, students in the experimental group reported that the two levels of questions reduced the use of guesswork and encouraged active thinking. For example, student A05 said, “For multiple-choice questions, some people will guess and succeed, but because there are two stages in this test, it is not easy to complete the test by guessing, and it is faster to find the answer by reading the code carefully and then answering.” Another student, A03, pointed out that “the two-level test is more thoughtful, because usually the teacher will tell us what to do directly, and most of the answers are intuitive, but with this test, we have to think more on our own.” Another student, A06, said, “The two-layer approach feels like we are afraid to use guesswork, so this way is somewhat helpful and will make you think.” It can be seen that the two-tier test mechanism can help students in the experimental group reduce the use of guesswork in their responses so that the diagnostic results are closer to the students' real state and teachers can understand the students' level more easily.

In terms of the feedback mechanism, the experimental group mentioned that the dynamic feedback mechanism allows students to know precisely which aspect of the wrong option they chose is different from the original question, which helps them understand how to use the blocks. Student A01 said, “I can tell which aspect of the original question is different from the one I got wrong”; student A03 said, “It (the system) will tell you exactly where you got wrong, so you can show it to people who are not clear about the usage of the blocks.” It is clear that the combination of feedback mechanisms can help give students precise and timely learning aids during the testing process to help them clarify their mistakes.

In other suggestions, the students in the experimental group mentioned that “the learning method of the two-level test is similar to the test mode, and it would be better if it can be presented more interestingly.” In the future, we can try to combine different teaching strategies. For example, game-based learning or cooperative learning could make students more engaged in the learning process, improve their learning motivation, and lead to better performance in learning.

Control Group

In the interviews, students mentioned that in terms of the test mode, the dynamic presentation of the content of the questions could help them solve the questions. Student B05 commented: “The animation option is better than the paper and pencil test.” Another student, B04, said, “Some people don't understand the meaning of (the question stem) if there are only text explanations, and it is better to answer with animations, so I will read the code first and then look at the relationship between the code and the animation.” Some students thought that “the dynamic presentation of the test was quite helpful.” However, some students felt that the multiple-choice test was not sufficiently helpful in terms of learning. Student B07 said, “Because it is easy to guess with multiple-choice questions, I don't know if I am wrong or right, or maybe I am wrong.” This shows that this is a problem that is more difficult to solve with traditional multiple-choice questions.

In terms of the feedback mechanism, many students in the control group said that because the number of questions and options are simple in the answers, they will be solved directly by guessing when they encounter questions that they do not know, and they are less likely to study the wrong choices. Student B07 mentioned, “There is no way to directly understand personal mistakes, so (the learning materials) can only give a little help."

In terms of other suggestions, several students in the control group indicated that it was quite possible to correctly answer the multiple-choice questions by lucky guess if the number of the candidate answers was small; therefore, they suggested that the number of test items might need to be increased to reduce the impacts of some lucky guesses on the final scores. In addition, student B01 suggested that “I hope I can operate and drag that thing (blocks) by myself. Yes, it will feel like we are doing our work in Scratch."

It can be known from the interview results that both the control group and the experimental group students believed that the way of dynamically presenting the test content can help them have better results in answering. Students in the experimental group were more active in thinking, and combined with the feedback mechanism, it can help them to accurately learn auxiliary materials during the test so that they can clarify their mistakes. Because the students in the control group studied with traditional multiple-choice questions combined with general learning materials as feedback, the approach could not accurately give students appropriate supplementary materials when they answered incorrectly. It can be seen that the programming training method based on the two-level test proposed in this study is indeed a great help for students to learn programming logic.

Discussion and Conclusions

In terms of students' learning achievement, this study analyzed the scores of memory-type questions and logic-type questions in the programming test. This means that the T³PT approach is not significantly different from the conventional programming approach. In other words, there was no significant difference between the two groups' learning performance on the memory-type questions when using the conventional computer programming approach. The reason for this result is that Scratch is graphical programming software, and each block has a specific way to be used, not just the functions related to writing programs, such as repetitive loops, and so on. Scratch blocks also contain other functions which have special effects, such as changing the size by 10 times. However, the two-tier test focused on helping students understand the basic concepts of programming logic, and less on how to use each block of Scratch, so there was no significant difference between the two groups in terms of the memorization test.

As for the logic-type questions, the T³PT approach is significantly better than the conventional computer programming learning approach. This study is in line with the scholarly findings of Yang et al. (2015) that two-tier tests can enhance students' programming skills and learning achievement. From the student interviews, it was found that students were prompted to rethink the same logic concept during the two-tier test process. Providing precision feedback immediately after a mistake is the most effective teaching behavior (Gálvez et al., 2009), and the precision feedback mechanism can help students clarify their misconceptions and better understand the application of logic concepts in a timely and precise manner. In addition, some students believed that the test can test program logic and self-organization skills. The use of the T³PT approach has been hypothesized to enhance students' learning of programming logic concepts and to help them think at a high level in order to learn to understand and apply them.

The analysis results of the problem-solving awareness showed that students who used the T³PT approach had significantly higher problem-solving awareness than those who used the conventional computer programming approach (F = 7.36, p = 0.008 < 0.01). In other words, the T³PT approach in the experimental group can achieve significantly better problem-solving awareness in classroom activities. The reason for this may be due to the use of the T³PT approach, which requires students to think deeply and repeatedly about the reasons for choosing the answer to the same question during the two-tier test, and the precision feedback mechanism that allows students to compare the differences among the options to confirm whether their choices are correct. The results of this study echo the research of Chu and Chang (2014), who found that the learning activities of the two-tier test enhanced students' learning motivation, and students showed better learning outcomes in problem-solving (Chang et al., 2020).

The results of this study using t test analysis showed that students in the experimental group using the T³PT approach were significantly more satisfied with the learning approach than students in the control group (t = 2.27, p = 0.025 < 0.05). The reason for this is that beginners often face various challenges and difficulties in learning programming, and are prone to lose motivation (Anniroot & De Villiers, 2012). The T³PT approach can foster students' active thinking and provide them with more accurate learning aids and positive assistance in misconceptions clarification than the general programming learning approach. Analyzing the results of the qualitative interviews with the students, most of the students in the experimental group agreed that “this approach makes them think about the problems seriously”; “it is easier to understand the problems with the help of animations”; and “there is feedback to tell them what they did wrong when they answered the problems wrongly.” In other words, students in the experimental group thought that the T³PT approach was more helpful for them to learn computer programming. However, most of the students in the control group thought that “there were not many options,” and when they encountered a question they did not know, they “would answer it by guessing” and “the feedback would only be skimmed over,” although some students also thought that “it was practical and the animations were good for the answers.” In other words, the students in the control group thought that the traditional computer programming learning approach was not very helpful to them for learning programming. Therefore, it is inferred that there was a difference in the satisfaction level of learning approaches between the two groups of students.

In summary, compared with the related studies in the past, the contribution of this study is that it proposes a two-tier test-based visualized computer programming approach with precision feedback. The approach provides an opportunity to assess students' mastery of programming logic concepts and to incorporate a precision feedback mechanism to enhance students' correct understanding of programming logic concepts. It initially solves a major problem in programming learning by helping students grasp the correct programming logic concepts and avoid the generation of misconceptions. In addition, the approach enhances students' problem-solving awareness. The results of this study provide a reference for researchers to address the learning of logic concepts in programming learning and for school teachers to promote students' learning achievement and problem-solving awareness in computer programming learning. Yang et al. (2015) used a two-level test system to help university students learn the Java language, and this study serves as a follow-up study to explore different participants, different programming languages, and different topics. The two-tier test learning system helps secondary school students learn a graphical programming language (Scratch) and develop their problem-solving awareness. The precise feedback from the two-tier tests not only promotes students' programming logic, but also reduces the burden on teachers.

There are some limitations to this study. Firstly, the study was conducted on Taiwanese junior high school students who were learning the content of Scratch programming. Without further research, the results of this study may not be generalizable to other samples and courses. Secondly, the sample size of this study was not large enough and the duration of the experiment was not long enough. Based on this, suggestions for further research are proposed. First, a two-tier test-based visualized computer programming approach can be applied to computer programming learning at different ages, and the sample size of the study can be expanded to extend the duration of the experiment. Second, this study provides preliminary evidence of the impact of the T³PT approach on students' learning achievement, which focuses on the test and can be incorporated into different learning approaches in the future, such as game-based learning or cooperative learning. It is also possible to further investigate the learning process of students when they use the two-tier test to learn the content of logic concepts or higher-order thinking, such as critical thinking skills and computational thinking skills.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported in part by the Ministry of Science and Technology of Taiwan under contract number (MOST-109-2511-H-011-002-MY3 and MOST-108-2511-H-011-005-MY3). This study is also supported by the National Educational Science Plan, the People’s Republic of China. The National Educational Science Plan “13th Five-Year Plan” in 2019. Project name: Big data-driven junior high school students’ academic development monitoring and precision intervention research. Contract number (BCA190089).

ORCID iDs

Gwo-Jen Hwang

Jian-Wen Fang

References

Alonso

J. M.

(2020). Teaching explainable artificial intelligence to high school students. International Journal of Computational Intelligence Systems, 13(1), 974–987. https://doi.org/10.2991/ijcis.d.200715.003.

Anniroot

De Villiers

M. R.

(2012). Study of Alice: A visual environment for teaching object-oriented programming. In Paper presented at IADIS International Conference on Information Systems, Berlin, Germany, March 10-12.

Chang

C.-K.

(2014). Effects of using Alice and scratch in an introductory programming course for corrective instruction. Journal of Educational Computing Research, 51(2), 185–204. https://doi.org/10.2190/EC.51.2.c.

Chang

S. C.

Hsu

T. C.

Kuo

W. C.

Jong

M. S. Y.

(2020). Effects of applying a VR-based two-tier test strategy to promote elementary students’ learning performance in a geology class. British Journal of Educational Technology, 51(1), 148–165. https://doi.org/10.1111/bjet.12790.

Chu

H.-C.

Chang

S.-C.

(2014). Developing an educational computer game for migratory bird identification based on a two-tier test approach. Educational Technology Research & Development, 62(2), 147–161. https://doi.org/10.1007/s11423-013-9323-4.

Çil

(2015). Effect of two-tier diagnostic tests on promoting learners’ conceptual understanding of variables in conducting scientific experiments. Applied Measurement in Education, 28(4), 253–273. https://doi.org/10.1080/08957347.2015.1064124.

Cook

Kilgus

Burns

(2018). Advancing the science and practice of precision education to enhance student outcomes. Journal of School Psychology, 66, 4–10. https://doi.org/10.1016/j.jsp.2017.11.004.

Davis

F. D.

(1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 13(3), 319–340. https://doi.org/10.2307/249008.

Gálvez

Guzmán

Conejo

(2009). A blended E-learning experience in a course of object oriented programming fundamentals. Knowledge-Based Systems, 22(4), 279–286. https://doi.org/10.1016/j.knosys.2009.01.004.

10.

García-Dantas

Quested

(2015). The effect of manipulated and accurate assessment feedback on the self-efficacy of dance students. Journal of Dance Medicine & Science, 19(1), 22–30. https://doi.org/10.12678/1089-313X.19.1.22.

11.

Halverson

K. L.

Pires

C. J.

Abell

S. K.

(2011). Exploring the complexity of tree thinking expertise in an undergraduate systematics course. Science Education, 95(5), 794–823. https://doi.org/10.1002/sce.20436.

12.

Hart

S. A.

(2016). Precision education initiative. Moving toward personalized education, 10(4), 209–211. https://doi.org/10.1111/mbe.12109.

13.

Heppner

P. P.

Petersen

C. H.

(1982). The development and implications of a personal problem-solving inventory. Journal of Counseling Psychology, 29(1), 66–75. https://doi.org/10.1037/0022-0167.29.1.66.

14.

Hwang

G.-J.

Yang

T.-C.

Tsai

C.-C.

Yang

S. J. H.

(2009). A context-aware ubiquitous learning environment for conducting complex science experiments. Computers & Education, 53(2), 402–413. https://doi.org/10.1016/j.compedu.2009.02.016.

15.

Hwang

G.-J.

Yang

L.-H.

Wang

S.-Y.

(2013). A concept map-embedded educational computer game for improving students’ learning performance in natural science courses. Computers & Education, 69, 121–130. https://doi.org/10.1016/j.compedu.2013.07.008.

16.

Kılıc

Saglam

(2014). Students’ understanding of genetics concepts: the effect of reasoning ability and learning approaches. Journal of Biological Education, 48(2), 63–70. https://doi.org/10.1080/00219266.2013.837402.

17.

Lin

J.-W.

(2016). Development and evaluation of the diagnostic power for a computer-based two-tier assessment. Journal of Science Education and Technology, 25(3), 497–511. https://doi.org/10.1007/s10956-016-9609-5.

18.

Liu

Y. C.

Huang

T. H.

Sung

C. L.

(2021). The determinants of impact of personal traits on computational thinking with programming instruction. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.1983610.

19.

Marwan

Gao

Fisk

Price

T. W.

Barnes

(2020). Adaptive immediate feedback can improve novice programming engagement and intention to persist in computer science. In Paper presented at the 2020 ACM conference on international computing education research, virtual event, NZ, August 7-11, 2022, Lugano, Switzerland. https://doi.org/10.1145/3372782.3406264.

20.

McCall

Kölling

(2019). A new look at novice programmer errors. ACM Transactions of Computing Education, 19(4), 1–30. https://doi.org/10.1145/3335814.

21.

Medeiros

R. P.

Ramalho

G. L.

Falcão

T. P.

(2018). A systematic literature review on teaching and learning introductory programming in higher education. IEEE Transactions on Education, 62(2), 77–90. https://doi.org/10.1109/TE.2018.2864133.

22.

Meerbaum-Salant

Armoni

Ben-Ari

(2013). Learning computer science concepts with scratch. Computer Science Education, 23(3), 239–264. https://doi.org/10.1080/08993408.2013.832022.

23.

Milkova

Hulkova

(2013). Algorithmic and logical thinking development: base of programming skills. WSEAS Transactions on Computers, 12(2), 41–51.

24.

Mirosław

Bujak

(2017). Precision of verbal feedback affects accuracy of motor skill performance. The Journal of Sports Medicine and Physical Fitness, 58(4), 435–441. https://doi.org/10.23736/s0022-4707.17.06749-4.

25.

Price

C. B.

Price-Mohr

R. M.

(2018). An evaluation of primary school children coding using a text-based language (Java). Computers in the Schools, 35(4), 284–301. https://doi.org/10.1080/07380569.2018.1531613.

26.

Psycharis

Kallia

(2017). The effects of computer programming on high school students’ reasoning skills and mathematical self-efficacy and problem solving. Instructional Science, 45(5), 583–602. https://doi.org/10.1007/s11251-017-9421-5.

27.

Resnick

Maloney

Monroy-Hernández

, et al. (2009). Scratch: programming for all. Communications of the ACM, 52(11), 60–67. https://doi.org/10.1145/1592761.1592779.

28.

Rodríguez Corral

J. M.

Civit Balcells

Morgado Estévez

Jiménez Moreno

Ferreiro Ramos

M. J.

(2014). A game-based approach to the teaching of object-oriented programming languages. Computers & Education, 73, 83–92. https://doi.org/10.1016/j.compedu.2013.12.013.

29.

Romero

Lepage

Lille

(2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(1), 1–15. https://doi.org/10.1186/s41239-017-0080-z.

30.

Y. S.

Shao

Zhao

(2021). Effect of mind mapping on creative thinking of children in scratch visual programming education. Journal of Educational Computing Research. https://doi.org/10.1177/07356331211053383.

31.

Treagust

D. F.

(1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. International Journal of Science Education, 10(2), 159–169. https://doi.org/10.1080/0950069880100204.

32.

Ullah

Lajis

Jamjoom

Altalhi

Al-Ghamdi

Saleem

(2018). The effect of automatic assessment on novice programming: strengths and limitations of existing systems. Computer Applications in Engineering Education, 26(6), 2328–2341. https://doi.org/10.1002/cae.21974.

33.

Wing

J. M.

(2006). Computational Thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215.

34.

Looi

C.-K.

Multisilta

, et al. (2020). Teacher’s perceptions and readiness to teach coding skills: a comparative study between Finland, mainland China, Singapore, taiwan, and South Korea. The Asia-Pacific Education Researcher, 29(1), 21–34. https://doi.org/10.1007/s40299-019-00485-x.

35.

S.-Y.

Y.-S.

(2021). Visual programming environments and computational thinking performance of fifth-and sixth-grade students. Journal of Educational Computing Research, 59(6), 1075–1092. https://doi.org/10.1177/0735633120988807.

36.

Yang

T. C.

Hwang

G. J.

Yang

S. J.

Hwang

G. H.

(2015). A two-tier test-based approach to improving students’ computer-programming skills in a web-based learning environment. Education Technology & Society, 18(1), 198–210.

37.

Yang

K. H.

B. C.

(2021). Towards the successful game-based learning: detection and feedback to misconceptions is the key. Computers & Education, 160. https://doi.org/10.1016/j.compedu.2020.104033.

38.

Yildiz Durak

(2020). The effects of using different tools in programming teaching of secondary school students on engagement, computational thinking and reflective thinking skills for problem solving. Technology, Knowledge and Learning, 25(1), 179–195. https://doi.org/10.1007/s10758-018-9391-y.

Promoting Students’ Programming Logic and Problem-Solving Awareness With Precision Feedback: A Two-Tier Test-Based Online Programming Training Approach

Abstract

Keywords

Introduction

Related Research

The Two-Tier Test-Based Programming Training (T3PT) System

Method

Participants

Data Gathering Tools

Experimental Process

Results

Learning Achievement

Problem-Solving Awareness

Technology Acceptance

Satisfaction With the Learning Approach

Results of the Interview

Experimental Group

Control Group

Discussion and Conclusions

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

References

The Two-Tier Test-Based Programming Training (T³PT) System