A Valid and Reliable Scale for Developing Programming-Oriented Computational Thinking

Abstract

In this research, a scale was developed to determine the programming-oriented computational thinking skills of university students. The participants were 360 students studying in various departments at different universities in Turkey for computer programming. The scale consists of 33 items under conceptual knowledge, algorithmic thinking, and evaluation subscale. While there was no significant difference between the students’ conceptual knowledge and algorithmic thinking skills, the evaluation skills of male students differed significantly compared to females. Programming experience has a significant effect on conceptual knowledge, algorithmic thinking, and evaluation. The algorithmic thinking skills of the students who have low, middle, and high-level programming experience differed significantly. In terms of the development of conceptual knowledge and evaluation skills, it was observed that students should have at least one year of programming experience, but this experience will not make a significant difference if it is four years or more. It is thought that this scale, which is structured for different applications (e.g., web, game, robot) and learning environments (e.g., text, block) within the framework of its programming capabilities (conceptual, semantic, strategic knowledge), will contribute significantly to the evaluation of computational thinking as programming oriented.

Keywords

computational thinking computer programming and programming language algorithmic thinking evaluation conceptual knowledge CTS scale development

The widespread use of information technologies in social, cultural, and economic fields changed the competencies expected from students. It is important for students to be technology literate. Students should have some skills such as using digital technologies effectively, problem-solving, creative thinking, and computational thinking in the context of 21st-century skills (International Society for Technology in Education [ISTE], 2020). As computers become more used in the processing, editing, and presentation of all kinds of data, instructors should support students for the development of computational thinking skills (CTS), which are important for the efficient operation of these processes. Wing (2006) defines CTS as transforming complex problems into simpler solvable problems by formulating and saving them from unnecessary details. Computing At School (2019) defines CTS, expanding the definition of Wing, as a logical investigation and critical thinking process on problems and conditions, considering the contribution of information processing tools to solve problems or fulfill some requirements. According to these definitions, CTS is a problem-solving approach and is necessary to effectively use information processing tools in the problem-solving process.

In recent years, different tools were used for the development of CTS such as coding without computers, programming by designing games, robot programming, and interdisciplinary applications like STEM (Lee et al., 2011; Weinberg, 2013). It is known that programming activities can develop the students’ CTS (Denner et al., 2012; Denning, 2007; 2009; Riley & Hunt, 2014; Weintrop et al., 2016). Programming is considered a reflection of the problem-solving process and students use their CTS while coding (Rodríguez-Martínez et al., 2020). Various computer science concepts such as system design (Denning, 2007; Wing, 2008), automation (Barr & Stephenson, 2011), self-iteration (Brennan & Resnick, 2012), parallel processing (Computer Science Teachers Association [CSTA] & ISTE, 2011), and debugging (Brennan & Resnick, 2012; NRC, 2010) are used to explain CTS. Students can work both individually and with a group when designing software. In programming activities, students first generate specific algorithms using syntax structures (Oliver, 1993). Then, these algorithms are used to contribute to the solution of the problems (Bayman & Mayer, 1988). Students behave progressively and systematically from the beginning, and eventually, their algorithmic thinking skills develop through this process (Selby & Woollard, 2013). Students try to extract the errors occurring in the coding process individually or debug recursively with the contribution of the group (Brennan & Resnick, 2012). Accordingly, Sysło and Kwiatkowska (2013) expressed that computational thinking is a thinking skill that can develop through programming. Bocconi et al. (2016) point out that programming activities can be an important tool to embody and develop CTS.

The report published by the members of the Italian National Research Council and the European Commission emphasized that students who participate in activities aimed at developing programming and CTS will make significant economic contributions to the country in the future (Bocconi et al., 2016). While many countries integrate the concepts of CTS in higher education programs, large-scale projects are conducted to improve the CTS of the K-12 students in the USA and the UK. Programming instruction is considered important for high school students in the USA to participate in CTS-based activities and to develop these skills (Garcia et al., 2015). In addition to educational institutions, many web-based platforms focus on the development of programming skills. Code.org (2020), one of these initiatives, provides programming training to 10% of all students around the world with Coding Time activities. Similarly, with more than 508 members, the Computer Science for All Project provides a learning environment suitable for programming for teachers and K-12 students (Csforall.org, 2020).

Assessment of CTS Development in Programming Activities

While computer programming activities are used more for the development of CTS, some limitations emerged in measuring these skills. For example, it is difficult to evaluate the improvement of CTS individually in the programming courses conducted with a large number of students. The fact that CTS has different sub-components (Kalelioglu et al., 2016) and a complex structure associated with different areas (CSTA & ISTE, 2011) also constitute differences in the evaluation of these skills (Yeni, 2018). Researchers focused on formative assessments to identify student performances and progressive developments at the end of the programming teaching process. The development of CTS for programming is measured using coding, interpreting blocks of code, asking open-ended questions for coding, achievement tests, and automated assessment tools (e.g., Dr. Scratch, Scrape) (Yeni, 2018). Besides, different methods are used for evaluation, too (Basawapatna et al., 2011; Brennan & Resnick, 2012; Koh et al., 2010; Seiter & Foreman, 2013). When these models are examined, it is seen that the games designed by the students or programming tasks are evaluated within the framework of programming concepts and problem-solving strategies. Multiple-choice tests (Chen et al., 2017; Román-González, 2015), open-ended questions (Atmatzidou & Demetriadis, 2016; Chen et al., 2017) and scales (Gülbahar et al., 2019; Korkmaz et al., 2017; Kukul & Karataş, 2019; Yağcı, 2019) are the other tools commonly used for evaluation to determine the skill levels of the students.

Scales Developed for CTS

There are various scales for evaluating the development of students’ CTS. Gülbahar et al. (2019) developed the computational thinking self-efficacy perception scale, which consists of self-confidence competence, algorithm design competence, basic programming competence, data processing competency, and problem-solving competence subscales. Kukul and Karataş (2019) developed a computational thinking self-efficacy scale within the framework of reasoning, abstraction, decomposition, and generalization dimensions. These scales are intended to determine the self-efficacy perception of secondary school students. In the framework of the creativity, algorithmic thinking, collaboration, critical thinking, and problem-solving dimensions of computational thinking, which presented by ISTE (2015), Yağcı (2019) developed the CTS scale for high school students and Korkmaz et al. (2017) developed another scale for university students. This scale, developed by Korkmaz et al. (2017), is aiming to measure the CTS development and is widely used in research (Durak, 2020; Pérez-Marín et al., 2020; Tang et al., 2020). However, the sub-dimensions used in the scale were not specifically structured to address concepts directly related to programming. It is structured more generally to be used for interdisciplinary applications. ISTE (2015) especially mentioned that five sub-skills of CTS can improve with real life problems and STEM applications. Students experience change and development in their conceptual knowledge (Bayman & Mayer, 1988; Brennan & Resnick, 2012), semantic knowledge (Oliver, 1993), and strategic knowledge (Bayman & Mayer, 1988; Oliver, 1993) while solving programming problems. For this reason, it gains importance to evaluate the development of students’ CTS within the context of these concepts (Brennan & Resnick, 2012; Seiter & Foreman, 2013).

In summary, these valid and reliable scales, and their subscales are not specifically structured to measure the CTS, which emerges directly from programming activities. Therefore, a valid and reliable scale including the concepts of various programming tools (text, block, robot programming, and game-design programming) is required to evaluate the development of the CTS of university students.

Background

Computational Thinking

Wing (2006) defined CTS as a skill that everyone should have, such as reading, writing, and arithmetic (p. 33). In addition, Wing defined CTS as the ability to solve problems, design systems and understand human behaviors by using the concepts of computer science. With the use of CTS together with the concepts of “computer science” and “a skill that everyone should have” by Wing, it has increased its emphasis on computer and daily life. The relationship between computer and daily life of CTS has been increasingly emphasized. By examining the definitions for CTS, Mannila et al. (2014) expressed CTS as an effective use of computer sciences to formulate the problems arising in different fields and to produce solutions to these problems. CSTA and ISTE (2011) also defined CTS as formulating problems using computers and other tools, analyzing and organizing data logically, automating solutions through algorithmic thinking, defining possible solutions, analyzing and applying them, generalizing the solutions. In the definitions above, the computer science concepts are emphasized mostly for CTS. Figure 1 visualizes the importance of programming concepts in the developing of the CTS framework.

Figure 1.
The programming concepts used in CTS (Bocconi et al., 2016).

Relationship Between Programming and CTS

Computer programming has been called “a creative endeavor requiring planning, precision in the use of language, the generation and testing of hypotheses, the ability to identify action sequences, and a variety of other skills that seem to reflect what thinking is all about” (Nickerson 1982, p. 42). In this definition, it is emphasized that programming is not just a set of algorithms, created by bringing together some structures. Several high-level thinking skills such as analyzing, planning, hypothesizing, and debugging are also employed in this process. Bayman and Mayer (1988) indicates that individuals’ competencies in programming knowledge should be evaluated within the framework of conceptual and strategic knowledge. While conceptual knowledge refers to the understanding of programming concepts, strategic knowledge includes solution strategies for defining and decomposing problems. Oliver (1993) progressively categorized programming competencies as Type I, II, and III. Type I includes conceptual structures for programming. Type II involves understanding the algorithmic process developed by conceptual structures. Type III covers strategies to solve problems with meaningful patterns created by using both concepts together. In the assessment model of CTS developed by Brennan and Resnick (2012), they focus firstly on programming concepts, secondly on debugging and correcting errors caused by using these concepts, and lastly on problem-solving through collaboration and communication. The knowledge (conceptual, semantic, strategic knowledge) required in solving programming-related problems and some skills (e.g., concepts, testing, debugging, and questioning) revealing the development of CTS are parallel and support each other. For example, students’ programming concepts knowledge and their effective use are regarded as the main competence in solving programming problems (Brennan & Resnick, 2012; Kazakoff et al., 2013; Touretzky et al., 2013). The use of programming concepts, testing and debugging the algorithms, evaluating the solution by making some queries are skills that can be taken as a reference in measuring CTS. Associating CTS with computer programming concepts changed the scope of studies to ensure students’ CTS development via programming activities (Sysło & Kwiatkowska, 2013). Conceptual knowledge, algorithmic thinking, and evaluation were included in some studies examining the development of CTS through programming (Table 1). Therefore, we structured the programming-oriented CTS (P-CTS) scale based on these three components.

Table 1.
CTS Developed by Programming.

Conceptual knowledge Algorithmic thinking Evaluation (testing and debugging, expressing, questioning)

Atmatzidou & Demetriadis (2016) ✓

Berland & Lee (2011) ✓ ✓

Bers et al. (2014) ✓

Brennan & Resnick (2012) ✓ ✓

Grover (2011) ✓ ✓

Chen et al. (2017) ✓ ✓

Csizmadia et al. (2015) ✓ ✓

Eguchi (2014) ✓

Eguchi (2016) ✓ ✓

Kazakoff et al. (2013) ✓

Lin et al. (2018) ✓

Oluk et al. (2018) ✓

Pala & Mıhçı Türker (2019) ✓

Penmetcha (2012) ✓ ✓

Pérez-Marín et al., (2020) ✓ ✓

Rodríguez-Martínez et al. (2020) ✓

Seiter & Foreman (2013) ✓ ✓

Selby & Woollard (2013) ✓ ✓

Touretzky et al. (2013) ✓

When the studies in the table are examined, it is seen that programming (e.g., robotic, game) activities are mostly implemented with K-8 students (Atmatzidou & Demetriadis, 2016; Brennan & Resnick, 2012; Chen et al., 2017; Touretzky et al., 2013). In these studies, positive results were obtained for the development of students’ conceptual knowledge (Pérez-Marín et al. 2020; Rodríguez-Martínez et al., 2020), algorithmic thinking (Chen et al., 2017; Oluk et al., 2018) and evaluation skills (Csizmadia et al., 2015; Selby & Woollard, 2013).

Except for the sub-skills specified in the studies in Table 1, abstraction, decomposition, and generalization are also considered for evaluating CTS (Atmatzidou & Demetriadis, 2016; Grover, 2011; Selby & Woollard, 2013). In the process of solving programming problems, abstracting problems from unnecessary details (Curzon et al., 2014; Denning & Staff, 2007) and decomposition them into smaller parts (Csizmadia et al., 2015; Curzon et al., 2014) are utilized as important skills. When the students separate a program into data entry (variable definition, value assignment, etc), data processing (loops, conditions, mathematical operations, etc), and data output, they can abstract the program from unnecessary details by deciding which structures should use in these sections. Aharoni (2000) stated that students go through three abstraction levels in the programming process. The first level includes the use of programming concepts belonging to a language, the second level the integration of the concepts used, and the third level includes the use of various problem-solving strategies beyond concepts. Effective use of conditional structures, another component of programming, can be seen as a skill resulting from the interrelated use of CTS’ abstraction and decomposition components. The next stage of the programming process is the creation of the operation steps and their sequencing (Oliver, 1993). By the way, students’ algorithmic thinking skills can be improved (Berland & Lee, 2011; Grover, 2011; Linn & Dalbey, 1985). It is stated that different abstractions are realized also in the development process of algorithmic thinking while programming (Perrenet et al., 2005; Perrenet & Kaasenbrood, 2006). The last step of programming involves processes for defining and correcting errors encountered in the programming process (Berland & Lee, 2011; Grover, 2011), interpreting the effectiveness of algorithms (Curzon et al., 2014), and making inferences about possible results (Csizmadia et al., 2015). In this process, students’ evaluation skills can be developed (Csizmadia et al., 2015; Selby & Woollard, 2013). Evaluating the functionality of a program can be expressed as generalizing the proposed algorithmic approach to different programs with a similar structure and associating it with similar activities (Yadav et al., 2018). In summary, in the development process of CTS sub-skills, we focused on conceptual knowledge, algorithmic thinking, and evaluation. But, other skills such as abstraction, decomposition, and generalization can also be used for evaluating CTS. For this reason, we generated some items in the subscales of the P-CTS by considering the abstraction, decomposition, and generalization skills.

Conceptual Knowledge

Conceptual knowledge includes syntactic, semantic, and schematic knowledge, which are widely used in programming (Wong & Cheung, 2020). In addition, conceptual knowledge is used with terms such as scripts, syntax, self-iteration, and variables (Liao & Bright, 1991). Basic structures in programming are sequences, loops (for, while), parallelism, events, conditionals (if-else, switch), variable (int, string), operators (=, >, <, &), and data concepts (Armstrong et al., 1996; Brennan & Resnick, 2012). The ability to use these concepts is considered as the core competence of programming (Oliver, 1993). Since the components of CTS, such as decomposition and abstraction, include programming concepts and skills for the use of commands, it can be addressed within the framework of CTS’ conceptual structures (García-Peñalvo & Mendes, 2018; Grover & Pea, 2013).

Algorithmic Thinking

Algorithmic thinking is a design skill that occurs during the application of programming concepts to perform the tasks (Linn & Dalbey, 1985). It is a clear definition of the steps to solve the problem and put them in a certain order (Curzon et al., 2014). This enables students to prepare a plan to solve the problem and follow it to see if the necessary solution is available (ISTE, 2017). Algorithmic thinking is an important component of CTS because it develops abstractions that can be reused to solve similar problems (Berland & Lee, 2011). In basic programming training, educators primarily teach students pseudo-codes and algorithmic flow charts. Thus, this skill is considered as one of the main competencies of programming.

Evaluation

Evaluation is the process of evaluating simplicity, effectiveness, efficiency, and understanding of algorithms (Schneider & Gersting, 2018; Sedgewick & Wayne, 2011). Algorithms are evaluated in terms of their accuracy, economical use of resources, and ease of use (Curzon et al., 2014). This step is widely used in the development of CTS and it is crucial to identify the best solution for the problem and evaluate its suitability. It is a strategic skill for programmers to recognize errors while coding and identify the sources of these errors (Bayman & Mayer, 1988). Evaluation is also described as evaluating the functionality of a program, comparing it with different programs that perform similar tasks, associating it with similar actions, questioning the usefulness of the method used, and interpreting the correctness of the results (Yadav et al., 2018). In this context, generalization skill, which is expressed in the generalization of the problem-solving process in programming and its transfer to a wide variety of problems (Barr et al., 2011), was examined under the evaluation.

In this study, a scale was developed to measure the development of P-CTS for university students. In addition to developing the P-CTS scale, this study also attempted to explore whether gender and programming experience variables affect the development of students’ CTS based on programming. The main research objectives of the study can be summarized as follow:
To develop valid and reliable scale for measuring P-CTS.

To reveal the effect of gender on P-CTS.

To reveal the effect of programming experience on P-CTS.

Method

Sample

The sample of the study is 360 university students studying diverse programs related to computer programming at three universities in Turkey. The scale was implemented to 395 students using the paper-pencil method, but 35 students did not participate in the study. 360 students voluntarily participated in the study. 80.6% of the sample were undergraduate students, and 19.4% of sample were vocational school students. 26.1% of the students have low-level computer programming experience (1 year and less), 65.6% of them have middle (2–4 years), and 8.3% of them have high-level programming experience (5 years and above). Table 2 shows the demographics of the sample.

Table 2.
Demographics of the Sample.

Categories Sub-categories Female Male Total

Undergraduate Computer Education & Instructional Technologies 23 21 44

Computer Engineering 38 36 74

Electrical and Electronics Engineering 46 105 151

Industrial Engineering 7 14 21

Vocational School Computer Programming 17 53 70

Grades 1 12 35 47

2 18 33 51

3 51 74 125

4 50 87 137

Programming Experience Low 29 65 94

Middle 94 142 236

High 8 22 30

Development Process of the Scale

During the development process, we followed the scale development steps recommended by DeVellis (2003). DeVellis sorted these steps as a) deciding what to measure, b) creation of the item pool, c) reviewing the literature, d) examining factors and items in similar studies, e) expert opinion, f) determine the Likert type. Figure 2 summarizes the development process of the scale.

Figure 2.
Development process of the scale.

As shown in Figure 2, we followed these steps:
Deciding what to measure: We reviewed the studies related to the CTS to determine the structure of our scale. We searched (i) frequently used definitions for CTS in the literature (Brennan & Resnick, 2012; CSTA & ISTE, 2011; ISTE, 2015; Selby & Woollard, 2013; Wing, 2006; 2008), (ii) the studies aiming to improve CTS by programming (robotics, game, etc.), and (iii) the studies evaluating the sub-dimensions of CTS (Basawapatna et al., 2011; Brennan & Resnick, 2012; Csizmadia et al., 2015; Curzon et al., 2014; Seiter & Foreman, 2013).

Creation of the item pool: The common factors, frequently used for the development of CTS with computer programming, were listed in Table 1. Within the framework of these studies, we determined 45 items.

Reviewing the literature: We investigate the existing scales for measuring the CTS (Gülbahar et al., 2019; Korkmaz et al., 2017; Kukul & Karataş, 2019; Yağcı, 2019).

Examining factors and items in similar studies: As a result of our review, we noticed that CTS is explained with various sub-dimensions and there is no specific scale that directly measures the development of CTS by computer programming. For this reason, we revealed the subscales related to the development of the CTS through programming.

Expert opinion: Three professionals from the Department of Computer Education and Instructional Technology reviewed the items. Based on experts’ critics and recommendations, we excluded 6 items from the scale and so, there are 18 items for conceptual programming, 7 items for algorithmic thinking, and 14 items for evaluation.

Determine the Likert type: We applied the prior version of the scale to 70 students studying at the associate and undergraduate level for understanding the items by the target population correctly. Some arrangements were made on two items for spelling and the final version of the scale was structured as a five-point Likert-type scale, on which 1 means “absolutely disagree” and 5 means “absolutely agree”. A sample scale item and scoring instruction are shown in Figure 3.
Figure 3.
Sample item of a Likert-type scale.

Data Analysis

The validity of the scale was tested using exploratory factor analysis (EFA). EFA gathers highly related variables in the scale and creates conceptually meaningful structures among them (Stevens, 2009). EFA was implemented using principal components analysis. The principal component analysis is recommended to reduce a large number of variables and gather them under a small factor set (Tabachnick & Fidell, 2007). Kaiser-Meyer Olkin (KMO) and Bartlett Sphericity tests were conducted to ensure the appropriateness of the data with principal components analysis. After determining the data are suitable for the analysis, principal components analysis was used with varimax orthogonal rotation. The varimax orthogonal rotation is recommended in principal components analysis to determine the independent sub-factors as it provides the most sensitive distinction among the factors and is one of the commonly used rotation methods (Ho, 2006). EFA results provided the items in each factor according to factor loadings of each item. In the analysis, it was considered that the item factor load values were 0.40 and higher, and factor loading difference between two or more factors was required to be minimum 0.10 (Field, 2017; Kline, 2011). Cronbach Alpha (α) reliability coefficient and item-total correlations were used to determine the reliability of the scale. It was assumed that the Cronbach alpha value greater than 0.70 indicates perfect reliability (Cortina, 1993). The item-total correlations greater than 0.30 indicate that items discriminate participants in terms of the measured construct (Field, 2017).

The data was also used to evaluate the P-CTS of the students. In this context, we examined the effects of gender and programming experiences on P-CTS.

Findings

Scale Development

Validity Findings

The construct validity of the scale was investigated through EFA. KMO coefficient, used to determine whether the data is suitable for EFA, was obtained as 0.973 and Bartlett’s test produced significant results (χ² = 10287.625; df = 741; p = 0.000). The findings implied that the collected data were appropriate to conduct factor analysis (Field, 2017; Leech, Barrett, & Morgan, 2005).

The analyses showed that the scale consisted of 3 factors explained 59.084% of the total variance and eigenvalue greater than 1.0. The six items (I17, I10, I13, I16, I8, I19) with the factor loadings less than 0.40 and the overlapping items were removed from the scale by Rotated Component Matrix, respectively. After the repeated analysis, the KMO coefficient obtained 0.971 and Bartlett’s test produced significant results (χ² = 8318.099; df = 528; p = 0.000). The three-factor structure with 33 remaining items explains 59.941% of the total variance. The rotated factor loadings of the items range from 0.456 and 0.844. These obtained results were adopted as satisfactory (Kline, 2011). Table 3 shows the variance values obtained from the EFA.

Table 3.
Factor Analysis Results.

Factors Items Common F1 F2 F3

Evaluation I34 I can detect errors related to coding on a subject I know. 0.654 0.741

I35 I can determine which algorithm is more effective in solving a problem. 0.658 0.700

I39 I can detect and clean unnecessary code structures in a program. 0.569 0.680

I29 I can detect the similarities and differences between two different programs. 0.666 0.679

I32 I can estimate the results of an algorithm or encoding. 0.653 0.671

I37 I can understand how the result changes when different values assigned to the variables in the program. 0.591 0.668

I30 I can continue the coding on a subject where I left off. 0.548 0.665

I38 I can evaluate whether the interface of a program can be easily used by users. 0.536 0.664

I31 I can interpret the causes of the errors I encountered during the coding process. 0.585 0.664

I33 I can detect errors in syntax (if, for, operators, etc.). 0.616 0.590

I27 I can decide on the operations that will create my code structures or code blocks in the program. 0.495 0.587

I26 I can decide whether the solution applied for a program makes logic. 0.558 0.578

I28 I can create the algorithm for a program in a way that everyone can understand. 0.566 0.565

I36 I can fix errors in the mathematical operations of the program. 0.524 0.561

Conceptual knowledge I2 I can use the loop structures (for, while, etc.) appropriately. 0.799 0.844

I3 I can use the decision structure (if-else, switch-case) appropriately. 0.747 0.803

I4 I can determine the suitable data type (string, int, char, float, etc.) for a variable. 0.596 0.696

I1 I can use mathematical (<, >, <=, >=, == etc.) and logical operators (and, or, etc.) appropriately. 0.574 0.691

I5 I can use general methods of programming languages [write(), read(), wait(), move(), scanf(), printf(), etc.]. 0.614 0.685

I9 I can code programs in which decisions (if-else, switch-case) and loops (for, while) can be used together. 0.674 0.680

I6 I can make sections that require mathematical operations in the program. 0.546 0.615

I15 I can determine which of the similar structures (if-switch, for-while) would be more appropriate. 0.627 0.602

I7 I can write a program that can do more than one operation at the same time. 0.623 0.591

I14 I know which variables would use before I start coding. 0.570 0.531

I11 I can determine the code structures for data entry, data processing, and data output of a program. 0.538 0.491

I12 I can decide how to split complex programs into smaller pieces. 0.430 0.481

Algorithmic thinking I21 I can schematize the pseudo-codes of a program with algorithmic symbols. 0.646 0.761

I22 I can visualize the processing steps of the program in my mind before start coding. 0.677 0.688

I24 I can sort the solution steps of the complex program appropriately. 0.685 0.665

I20 I can express the steps of a program with pseudo-codes. 0.620 0.641

I23 I can develop different algorithms for the solution of the same problem. 0.592 0.610

I25 I can separate the codes of a program into algorithmic process steps as input, process, decision, and output. 0.617 0.577

I18 I can create simple algorithms without using unnecessary codes in the program. 0.383 0.456

Eigenvalue 7.963 6.961 4.857

Explained variance 24.129 21.095 14.717

As shown in Table 3, three factors emerged as a result of the EFA were named “Evaluation (F1)”, “Conceptual knowledge (F2)”, and “Algorithmic Thinking (F3)” based on the relevant literature and the items. There are fourteen items under the Evaluation factor that evaluate the behaviors such as to realize the errors in programming concepts and algorithms structure, comment on the results of coding, and compare the results. The factor loadings of the items range from 0.561 to 0.741. The eigenvalue is 7.963 and it explains 24.129% of the total variance. Conceptual Knowledge factor consists of twelve items that evaluate individuals’ knowledge for using programming structures (variables, loops, conditions, operators, process, etc.). The factor loadings of these items are between 0.481 and 0.844. The factor explains 21.095% of the total variance, and its eigenvalue is 6.961. Algorithmic Thinking factor involves seven items that measure individuals’ skills to develop algorithms. The eigenvalue of the factor is 4.857, and it explains 14.717% of the total variance.

The scree plot about the three-factor structure obtained as a result of the EFA was presented in Figure 4. When we analyzed the diagram, we saw that the slope prominently plateaus after component number three. This point is also considered as a threshold to decide the number of factors in the model (Çokluk et al., 2016).

Figure 4.
Scree plot diagram.

The relationship between the three factors examined using the correlation coefficient. Table 4 shows the analysis results. In Table 4, it is seen that there is a positive and significant correlation between the three factors.

Table 4.
Correlations Between the Factors.

Factors Conceptual knowledge Algorithmic thinking Evaluation

Conceptual knowledge 1 0.763 0.793

Algorithmic thinking 1 0.794

Evaluation 1

p = 0.01.

Reliability Findings

The reliability of the scale was examined through Cronbach Alpha and item-total correlations. The gathered values were presented in Table 5. The Cronbach Alpha value for the 33 items was found as 0.970. The reliability coefficient for each factor ranges from 0.887 and 0.943. The internal consistency coefficients for both the factors and the total scale are assumed as an acceptable level (Cortina, 1993). The item-total correlations for the items indicate that the items are satisfactory for the item discrimination in terms of the measured construct (Büyüköztürk, 2007; Field, 2017).

Table 5.
Reliability Results.

Conceptual knowledge
Algorithmic thinking
Evaluation

Item Item-total correlation Cronbach Alpha Item Item-total correlation Cronbach Alpha Item Item-total correlation Cronbach Alpha

I2 0.812 0.937 I21 0.650 0.886 I34 0.746 0.943

I3 0.787 I22 0.741 I35 0.757

I4 0.702 I24 0.745 I39 0.695

I1 0.662 I20 0.707 I29 0.786

I5 0.725 I23 0.682 I32 0.777

I9 0.787 I25 0.705 I37 0.722

I6 0.671 I18 0.533 I30 0.682

I15 0.749 I38 0.667

I7 0.746 I31 0.718

I14 0.711 I33 0.724

I11 0.676 I27 0.654

I12 0.603 I26 0.703

I28 0.693

I36 0.682

Student Scores on the Scale

Overall Students’ Subscales Scores

This study further explored the P-CTS scores of the sample of this study. Table 6 shows the descriptive statistics on students’ responses in the development process of the scale. The students’ highest item mean was on the Conceptual Knowledge (3.57) followed by the Algorithmic Thinking (3.46), and the Evaluation (3.44) subscales. During programming teaching, a progressive process was followed for programming concepts, algorithms, and evaluation of the program. Therefore, it can be said that the programming teaching process is effective on the students’ mean scores of the subscales. Also, the students’ overall score is above the average (3.00). This result indicates that the programming skills of students studying in departments where more computer programming courses are teaching are above average.

Table 6.
Students’ Scores of the Subscales.

Subscales Items Min Max Mean SD

Conceptual knowledge 12 1.25 5 3.57 0.87

Algorithmic thinking 7 1.29 5 3.46 0.83

Evaluation 14 1.29 5 3.44 0.80

The differences between the average scores of the subscales are elaborated in Table 7. Paired-samples t test was used to examine the differences between students’ scores on the subscales. Table 7 shows that the differences between the average scores are significant except for the evaluation and algorithmic thinking. Cohen’s effect size was calculated to determine the magnitude of these differences. The results show that the effect sizes are small (Cohen, 1988). This may be due to the fact that all subscales concern programming. Oliver (1993) and Brennan and Resnick (2012) categorized programming competencies in hierarchic levels. They listed these levels as concept learning, algorithmic processes, and problem-solving strategies development. So, a good programmer who is good in one subscale could be good in other subscales as well.

Table 7.
The Differences between the Students’ Scores of the Subscales.

Mean differences SD df Paired t R²

Evaluation – Conceptual knowledge –0.13 0.54 359 –4.59* 0.02

Evaluation – Algorithmic thinking –0.02 0.52 359 –0.77 0.00

Conceptual Knowledge – Algorithmic thinking 0.11 0.58 359 3.54* 0.01

*p = 0.00.

Gender Differences

We further explored if there is any gender difference in the P-CTS scores collected in this study. The number of male and female students in the sample are different. However, there are close to female and male students at each level in terms of programming experience. Accordingly, the effect of the gender on the overall scale, and the subscale was examined using the student’s t-test.

Table 8 shows that the significant difference in terms of gender is only in the Evaluation subscale (t = –1.99; p < 0.05). There isn’t any significant difference in terms of gender in the other subscales. According to these results, male students are more successful in the Evaluation factor than females (Male = 3.50; Female = 3.32). When the whole scale items were analyzed, no significant difference was found in terms of female and male students, similar to the subscales (t = –1.26; p = 0.20).

Table 8.
The Effect of Gender.

Female (N = 131)
Male (N = 229)

Mean SD Mean SD t p

Conceptual knowledge 3.55 0.82 3.58 0.89 –0.35 0.72

Algorithmic thinking 3.40 0.81 3.49 0.84 –1.05 0.29

Evaluation 3.32 0.77 3.50 0.82 –1.99 0.04

Total 3.42 0.73 3.53 0.79 –1.26 0.20

***p < 0.05.

The Role of Programming Experience

In this study, the relationships between students’ programming experiences and their programming-oriented computational thinking skills were also examined. We divided students’ programming experiences into 3 groups as Low (1 year and less), Middle (2 to 4 years), and High (5 years and above). The effect of the programming experience on the P-CTS was analyzed using ANOVA (see Table 9). The Post-Hoc test was used to determine the significance of the differences between the groups (see Table 10).

Table 9.
The Effect of Programming Experience.

Low experience(N = 94)
Middle experience(N = 236)
High experience (N = 30) F p Result

Subscale Mean SD Mean SD Mean SD

Conceptual knowledge 2.96 0.74 3.74 0.80 4.10 0.79 39.709 0.000 1 < 2 < 3

Algorithmic thinking 3.03 0.75 3.56 0.80 3.97 0.73 21.966 0.000 1 < 2 < 3

Evaluation 2.93 0.71 3.58 0.75 3.91 0.76 32.059 0.000 1 < 2 < 3

Table 10.
Post–Hoc Results.

(I) (J)
Conceptual knowledge
Algorithmic thinking
Evaluation

MD (I - J) SE p MD (I - J) SE p MD (I - J) SE p

Low Middle –0.77591* 0.09641 0.000 –0.52617* 0.09613 0.000 –0.64814* 0.09109 0.000

High –1.13191* 0.16576 0.000 –0.93820* 0.16528 0.000 –0.98668* 0.15662 0.000

Middle Low 0.77591* 0.09641 0.000 0.52617* 0.09613 0.000 0.64814* 0.09109 0.000

High –0.35600 0.15322 0.054 –0.41203* 0.15278 0.020 –0.33854 0.14477 0.052

High Low 1.13191* 0.16576 0.000 0.93820* 0.16528 0.000 0.98668* 0.15662 0.000

Middle 0.35600 0.15322 0.054 0.41203* 0.15278 0.020 0.33854 0.14477 0.052

*p < 0.05.

In Table 9, programming experience has a significant effect on conceptual knowledge, algorithmic thinking, and evaluation subscales. As students’ programming experience increases, their knowledge and skill levels for all three subscales are enhanced. Considering the programming experience as duration, although there is a significant difference between the low-level students and the others (middle- and high-level) in terms of assessment skills, there is no significant difference between middle- and high-level students. In this case, it can be said that students need to have at least more than one-year programming experience for the development of evaluation skills. However, this development will not be significant when this experience is developed for 5 years or above. Similarly, programming experience has a significant effect on conceptual knowledge. When the Post-Hoc test is examined, it can be said that at least one year of programming is essential for the development of conceptual knowledge. When the effect of programming experience on algorithmic thinking is analyzed, a significant difference between all three groups is determined. As students’ programming experiences increase, their algorithmic thinking skills increase in direct proportion. The algorithmic thinking skills of low, middle, and high-level students differ significantly. In sum, it can be said that it would be beneficial for students to have more programming experience to develop their algorithmic thinking skills.

Discussion

In this study, a scale was developed to determine the CTS development levels of university students studying in departments for computer programming. P-CTS consists of 33 items under 3 subscales named conceptual knowledge, algorithmic thinking, and evaluation. The scale factors developed by Korkmaz et al. (2017) and Yağcı (2019) are structured within the framework of the sub-dimensions of the CTS suggested by ISTE (2015). The subscales of the P-CTS are based on the results of the studies on programming in the literature. ISTE (2015) revealed the subscales of the CTS by focusing on real-life problems and STEM applications. Therefore, the scales developed in this context address more general CTS such as problem-solving, critical thinking, and cooperation. In this respect, it differs from other scales in terms of evaluating the developed CTS as programming oriented. In Computer Programming Self-Efficacy Scale (CPSES) developed by Tsai et al. (2019), based on a computational thinking framework, included five subscales: Logical Thinking, Algorithm, Debug, Control, and Cooperation. There are similarities between CPSES and P-CTS. Some items under logical thinking, algorithm, and debug subscales in CPSES are similar to the items under conceptual knowledge, algorithm, and evaluation in the P-CTS scale. We can say that the items in the P-CTS scale are suitable for measuring programming-oriented computational thinking.

Conceptual Knowledge

Conceptual knowledge refers to understanding and using programming structures (Bayman & Mayer, 1988). Within the conceptual knowledge subscale, items related to syntax structures (loops, conditionals, operators, variables, events, parallelism) of programming languages were included. It was emphasized that the concepts of programming should be included in the framework created by Brennan and Resnick (2012) for the evaluation of CTS. Moreno-León et al. (2015) evaluated coding activities such as the game, robot, and coding missions with Dr. Scratch software. In this evaluation process, they determined the CTS development levels of the students depending on the use of conceptual structures. Similarly, the conceptual knowledge subscale in the P-CTS scale measures the CTS development of students depending on the use of syntax structures. The level of knowing and using concepts also affects the development of algorithmic thinking and problem-solving strategy (Brennan & Resnick, 2012; Kazakoff et al., 2013; Touretzky et al, 2013). Under this subscale, there are also items for separating problems and abstraction of unnecessary details over syntax. Decomposition and abstraction skills considered as separate subscales in CTS studies are combined under conceptual knowledge when evaluated as programming based. This situation can be interpreted as breaking down the problems faced by students into meaningful parts while programming and identifying processes of focusing on key features instead of unnecessary details with the use of programming concepts. As a matter of fact, situations, where more than one operation is required at the same time, can be realized by breaking into pieces with parallelism. With loop structures, situations that need to be repeated many times can be done at once, and with conditionals, it is possible to focus on basic features instead of unnecessary details.

Algorithmic Thinking

The items under the algorithm thinking subscale are intended for problem-solving steps, schematizing the processes, and developing different algorithms for the same solution. Algorithmic thinking skill is a design skill providing meaningful structures in solving problems by using programming concepts (Linn & Dalbey, 1985). The ability to understand, apply, evaluate, and produce the algorithms created in this process are indicators of algorithmic thinking skills (Brown, 2015). Algorithmic thinking is generally situated in the scales developed for measuring the self-efficacy perception of the students towards CTS (Gülbahar et al., 2019; Kukul & Karataş, 2019; Tsai et al., 2019). It is considered as an important component of the CTS (Berland & Lee, 2011). Moreover, algorithmic thinking can be improved by programming activities. Therefore, the algorithmic thinking subscale should be included in the scales related to CTS.

Evaluation

The evaluation subscale involves items such as debugging, determining the effectiveness of algorithms, and predicting the results. Evaluation is the process of addressing the suitability of the algorithms, coding, programs, and systems (Schneider & Gersting, 2018). Making evaluations about coding and algorithmic errors encountered in this process (Bayman & Mayer, 1988) and interpretation of the possible results (L’Heureux et al., 2012) improve evaluation skills. Besides, statements about generalization skills were also included in the evaluation factor. Barr et al. (2011) stated that students experience an evaluation process while transferring the algorithmic solutions for a problem to different problems. Items in the debug subscale of the CPSES developed by Tsai et al. (2019) are similar to the items in the evaluation subscale of the P-CTS. Evaluation is an essential skill for programming and CTS as well as the algorithmic thinking.

Differences in Gender Roles

When CTS development of the students based on the program is examined; while there is no difference between females and males in terms of conceptual knowledge and algorithmic thinking, there is a significant difference between evaluation skills. While there is no research in the literature about the direct effect of gender factor on evaluation, there are studies on students’ self-efficacy towards computer programming. In one of these studies, Tsai et al. (2019) found that male students are more sufficient to detect and correct errors than female students. Similarly, the studies conducted by Askar and Davenport (2009) and Özyurt and Özyurt (2015) revealed that male students have higher programming self-efficacy than female students. When looking at the overall scale, there was no difference between the two groups in terms of CTS development. This result confirms the literature (Atmatzidou & Demetriadis, 2016).

Differences in Programming Experiences

We determined that programming experience has a significant effect on students’ CTS development. Having more than 1 year of programming experience is necessary for the development of the students’ conceptual knowledge, algorithmic thinking, and evaluation skills. In parallel with these results, it is stated that increasing programming experience contributes to students’ logical thinking, algorithm, control, debugging, and collaboration competencies (Tsai et al., 2019). On the other hand, we determined that while students having a programming experience of five years or more does not have a meaningful effect on their development of conceptual knowledge and evaluation skills, it contributes significantly to their algorithmic thinking skills development. Algorithmic thinking skill has been developing in a meaningful way in direct proportion to the programming experience. It can be said that algorithmic thinking skills will improve as the students receive programming training and conduct programming activities outside of the courses.

Conclusion

With this study, a scale examining the CTS that may occur after programming education of undergraduates in three subscales within the framework of conceptual knowledge, algorithmic thinking, and evaluation skills were developed. The creation of scale items according to the structures of different programming languages as block-based and text-based, allows the scale to be used in different applications (e.g., web, desktop, game, robot programming). CTS has gained importance as a concept that has been repeatedly emphasized in terms of learner characteristics in recent years. Research emphasized that CTS consists of different sub-components and problem-solving skill is at the base. As it turns out that CTS is a holistic skill consisting of different components, how to evaluate this skill has started to be among the important problems to be answered. Although various scales for the assessment of CTS have been developed, valid, and reliable measurement tools for assessment of CTS over programming skills are not found. It is thought that the P-CTS scale will contribute to the field by filling this gap.

In this study, we developed the P-CTS scale for university students. In future studies, this scale can be adopted for different educations levels such as K-12 or K-8. P-CTS scale can also be implemented to interdisciplinary-integrated curricula such as STEM education involving computer programming tasks (e.g., robotics). This study confirmed the significant effect of programming experience on conceptual knowledge, algorithmic thinking, and evaluation. On the other hand, this experience does not make a significant difference if it is four years or more. Future studies can further explore why this result may be caused.

	Conceptual knowledge	Algorithmic thinking	Evaluation (testing and debugging, expressing, questioning)
Atmatzidou & Demetriadis (2016)		✓
Berland & Lee (2011)		✓	✓
Bers et al. (2014)			✓
Brennan & Resnick (2012)	✓		✓
Grover (2011)		✓	✓
Chen et al. (2017)	✓	✓
Csizmadia et al. (2015)		✓	✓
Eguchi (2014)			✓
Eguchi (2016)		✓	✓
Kazakoff et al. (2013)	✓
Lin et al. (2018)			✓
Oluk et al. (2018)		✓
Pala & Mıhçı Türker (2019)		✓
Penmetcha (2012)	✓	✓
Pérez-Marín et al., (2020)	✓		✓
Rodríguez-Martínez et al. (2020)	✓
Seiter & Foreman (2013)	✓	✓
Selby & Woollard (2013)		✓	✓
Touretzky et al. (2013)	✓

Categories	Sub-categories	Female	Male	Total
Undergraduate	Computer Education & Instructional Technologies	23	21	44
	Computer Engineering	38	36	74
	Electrical and Electronics Engineering	46	105	151
	Industrial Engineering	7	14	21
Vocational School	Computer Programming	17	53	70
Grades	1	12	35	47
	2	18	33	51
	3	51	74	125
	4	50	87	137
Programming Experience	Low	29	65	94
	Middle	94	142	236
	High	8	22	30

Factors	Items		Common	F1	F2	F3
Evaluation	I34	I can detect errors related to coding on a subject I know.	0.654	0.741
I35	I can determine which algorithm is more effective in solving a problem.	0.658	0.700
I39	I can detect and clean unnecessary code structures in a program.	0.569	0.680
I29	I can detect the similarities and differences between two different programs.	0.666	0.679
I32	I can estimate the results of an algorithm or encoding.	0.653	0.671
I37	I can understand how the result changes when different values assigned to the variables in the program.	0.591	0.668
I30	I can continue the coding on a subject where I left off.	0.548	0.665
I38	I can evaluate whether the interface of a program can be easily used by users.	0.536	0.664
I31	I can interpret the causes of the errors I encountered during the coding process.	0.585	0.664
I33	I can detect errors in syntax (if, for, operators, etc.).	0.616	0.590
I27	I can decide on the operations that will create my code structures or code blocks in the program.	0.495	0.587
I26	I can decide whether the solution applied for a program makes logic.	0.558	0.578
I28	I can create the algorithm for a program in a way that everyone can understand.	0.566	0.565
I36	I can fix errors in the mathematical operations of the program.	0.524	0.561
Conceptual knowledge	I2	I can use the loop structures (for, while, etc.) appropriately.	0.799		0.844
I3	I can use the decision structure (if-else, switch-case) appropriately.	0.747		0.803
I4	I can determine the suitable data type (string, int, char, float, etc.) for a variable.	0.596		0.696
I1	I can use mathematical (<, >, <=, >=, == etc.) and logical operators (and, or, etc.) appropriately.	0.574		0.691
I5	I can use general methods of programming languages [write(), read(), wait(), move(), scanf(), printf(), etc.].	0.614		0.685
I9	I can code programs in which decisions (if-else, switch-case) and loops (for, while) can be used together.	0.674		0.680
I6	I can make sections that require mathematical operations in the program.	0.546		0.615
I15	I can determine which of the similar structures (if-switch, for-while) would be more appropriate.	0.627		0.602
I7	I can write a program that can do more than one operation at the same time.	0.623		0.591
I14	I know which variables would use before I start coding.	0.570		0.531
I11	I can determine the code structures for data entry, data processing, and data output of a program.	0.538		0.491
I12	I can decide how to split complex programs into smaller pieces.	0.430		0.481
Algorithmic thinking	I21	I can schematize the pseudo-codes of a program with algorithmic symbols.	0.646			0.761
I22	I can visualize the processing steps of the program in my mind before start coding.	0.677			0.688
I24	I can sort the solution steps of the complex program appropriately.	0.685			0.665
I20	I can express the steps of a program with pseudo-codes.	0.620			0.641
I23	I can develop different algorithms for the solution of the same problem.	0.592			0.610
I25	I can separate the codes of a program into algorithmic process steps as input, process, decision, and output.	0.617			0.577
I18	I can create simple algorithms without using unnecessary codes in the program.	0.383			0.456
Eigenvalue	7.963	6.961	4.857
Explained variance	24.129	21.095	14.717

Factors	Conceptual knowledge	Algorithmic thinking	Evaluation
Conceptual knowledge	1	0.763**	0.793**
Algorithmic thinking		1	0.794**
Evaluation			1

Conceptual knowledge	Algorithmic thinking	Evaluation
I2	0.812	0.937	I21	0.650	0.886	I34	0.746	0.943
I3	0.787		I22	0.741		I35	0.757
I4	0.702		I24	0.745		I39	0.695
I1	0.662		I20	0.707		I29	0.786
I5	0.725		I23	0.682		I32	0.777
I9	0.787		I25	0.705		I37	0.722
I6	0.671		I18	0.533		I30	0.682
I15	0.749					I38	0.667
I7	0.746					I31	0.718
I14	0.711					I33	0.724
I11	0.676					I27	0.654
I12	0.603					I26	0.703
						I28	0.693
						I36	0.682

Subscales	Items	Min	Max	Mean	SD
Conceptual knowledge	12	1.25	5	3.57	0.87
Algorithmic thinking	7	1.29	5	3.46	0.83
Evaluation	14	1.29	5	3.44	0.80

	Mean differences	SD	df	Paired t	R²
Evaluation – Conceptual knowledge	–0.13	0.54	359	–4.59***	0.02
Evaluation – Algorithmic thinking	–0.02	0.52	359	–0.77	0.00
Conceptual Knowledge – Algorithmic thinking	0.11	0.58	359	3.54***	0.01

	Female (N = 131)	Male (N = 229)
Conceptual knowledge	3.55	0.82	3.58	0.89	–0.35	0.72
Algorithmic thinking	3.40	0.81	3.49	0.84	–1.05	0.29
Evaluation	3.32	0.77	3.50	0.82	–1.99	0.04**
Total	3.42	0.73	3.53	0.79	–1.26	0.20

	Low experience(N = 94)	Middle experience(N = 236)	High experience (N = 30)	F	p	Result
Conceptual knowledge	2.96	0.74	3.74	0.80	4.10	0.79	39.709	1 < 2 < 3
Algorithmic thinking	3.03	0.75	3.56	0.80	3.97	0.73	21.966	1 < 2 < 3
Evaluation	2.93	0.71	3.58	0.75	3.91	0.76	32.059	1 < 2 < 3

(I)	(J)	Conceptual knowledge	Algorithmic thinking	Evaluation
Low	Middle	–0.77591*	0.09641	0.000	–0.52617*	0.09613	0.000	–0.64814*	0.09109	0.000
	High	–1.13191*	0.16576	0.000	–0.93820*	0.16528	0.000	–0.98668*	0.15662	0.000
Middle	Low	0.77591*	0.09641	0.000	0.52617*	0.09613	0.000	0.64814*	0.09109	0.000
	High	–0.35600	0.15322	0.054	–0.41203*	0.15278	0.020	–0.33854	0.14477	0.052
High	Low	1.13191*	0.16576	0.000	0.93820*	0.16528	0.000	0.98668*	0.15662	0.000
	Middle	0.35600	0.15322	0.054	0.41203*	0.15278	0.020	0.33854	0.14477	0.052

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Servet Kılıç, PhD, is an instructor at Ordu University. His academic specialty is instructional technologies, distance learning, teacher training, programming and robotics education, and problem solving strategies.

Seyfullah Gökoğlu is a PhD at the Computer Technologies Department of Cide Rıfat Ilgaz Vocational School, Kastamonu University, Turkey. Before joining Kastamonu University, he also was an Information Technologies Teacher at the Ministry of National Education, beginning in 2007. His research interests include programming education, teacher training, technology integration, and virtual reality.

Mücahit Öztürk is currently an assistant professor at the Department of Computer and Instructional Technologies at Aksaray University. His academic qualifications were in Educational Technologies (PhD). He teaches computer programming, distance education, and other IT related courses. His research interests include online learning, flipped learning, self-regulated learning, programming learning.

ORCID iDs

Servet Kılıç

Seyfullah Gökoğlu

Mücahit Öztürk

References

Aharoni, D. (2000). Cogito, Ergo sum! Cognitive processes of students dealing with data structures. ACM SIGCSE Bulletin, 32(1), 26–30.

Armstrong

Virding

Wikström

Williams

(1996). Concurrent programming in Erlang. Prentice Hall.

Askar

Davenport

(2009). An investigation of factors related to self-efficacy for java programming among engineering students. The Turkish Online Journal of Educational Technology, 8(1), 26–32.

Atmatzidou

Demetriadis

(2016). Advancing students’ computational thinking skills through educational robotics: A study on age and gender relevant differences. Robotics and Autonomous Systems, 75, 661–670. https://doi.org/10.1016/j.robot.2015.10.008

Barr

Harrison

Conery

(2011). Computational thinking: A digital age skill for everyone. Learning & Leading with Technology, 38(6), 20–23.

Barr

Stephenson

(2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community. ACM Inroads, 2(1), 48–54. https://doi.org/10.1145/1929887.1929905

Basawapatna

Koh

K. H.

Repenning

Webb

D. C.

Marshall

K. S.

(2011, March 17–21). Recognizing computational thinking patterns [Paper presentation]. The 42nd ACM Technical Symposium on Computer Science Education, Dallas, USA.

Bayman

Mayer

R. E.

(1988). Using conceptual models to teach BASIC computer programming. Journal of Educational Psychology, 80(3), 291–298. https://doi.org/10.1037/0022-0663.80.3.291

Berland

Lee

V. R.

(2011). Collaborative strategic board games as a site for distributed computational thinking. International Journal of Game-Based Learning (IJGBL), 1(2), 65–81. https://doi.org/10.4018/ijgbl.2011040105

10.

Bers

M. U.

Flannery

Kazakoff

E. R.

Sullivan

(2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum. Computers & Education, 72, 145–157. https://doi.org/10.1016/j.compedu.2013.10.020

11.

Bocconi

Chioccariello

Dettori

Ferrari

Engelhardt

Kampylis

Punie

(2016). Developing computational thinking in compulsory education [Policy Report]. European Commission.

12.

Brennan

Resnick

(2012, April 13–17). New frameworks for studying and assessing the development of computational thinking [paper presentation]. 2012 Annual Meeting of the American Educational Research Association, The University of British Columbia, Vancouver.

13.

Brown

(2015, May 10). Introduction to algorithmic thinking. www.cs4fn.com/algoritmicthinking.php

14.

Büyüköztürk

Ş.

(2007). Manual of data analysis for social sciences (8th ed.). Pegem Academy Publishing.

15.

Chen

Shen

Barth-Cohen

Jiang

Huang

Eltoukhy

(2017). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162–175. https://doi.org/10.1016/j.compedu.2017.03.001

16.

Code.org. (2020, June 5). Hour of code. https://code.org/international/about

17.

Cohen

(1988). The t test for means. Statistical power analysis for the behavioural sciences (2nd ed.). Erlbaum.

18.

Çokluk

Ö.

Şekercioğlu

Büyüköztürk

Ş.

(2016). Multivariate statistics for social sciences: SPSS and LISREL applications. Pegem Academy.

19.

Computer Science Teachers Association & International Society for Technology in Education. (2011). Computational thinking in K–12 education leadership toolkit. https://id.iste.org/docs/ct-documents/ct-leadershipt-toolkit.pdf?sfvrsn=4

20.

Computing At School. (2019). Computational thinking: How do we think about problems so that computers can help? https://community.computingatschool.org.uk/files/8221/original.pdf

21.

Cortina

J. M.

(1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98

22.

Csforall.org. (2020, May 8). Computer science for all. https://www.csforall.org/about/

23.

Csizmadia

Curzon

Dorling

Humphreys

Selby

Woollard

(2015). Computational thinking—A guide for teachers. Computing At School. http://eprints.soton.ac.uk/id/eprint/424545

24.

Curzon

Dorling

Selby

Woollard

(2014). Developing computational thinking in the classroom: A framework. https://eprints.soton.ac.uk/369594/1/DevelopingComputationalThinkingInTheClassroomaFramework.pdf

25.

Denner

Werner

Ortiz

(2012). Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts? Computers & Education, 58(1), 240–249. https://doi.org/10.1016/j.compedu.2011.08.006

26.

Denning

P. J.

(2007). Computing is a natural science. Communications of the ACM, 50(7), 13–18. https://doi.org/10.1145/1272516.1272529

27.

Denning

P. J.

(2009). The profession of IT beyond computational thinking. Communications of the ACM, 52(6), 28–30. https://doi.org/10.1145/1516046.1516054

28.

Denning

P. J.

Staff

(2007). An interview with peter denning on the great principles of computing. Ubiquity, 1–9.

29.

DeVellis

R. F.

(2003). Scale development: Theory and applications (2nd ed.). SAGE Publications.

30.

Durak

H. Y.

(2020). The effects of using different tools in programming teaching of secondary school students on engagement, computational thinking and reflective thinking skills for problem solving. Technology, Knowledge and Learning, 25(1), 179–195. https://doi.org/10.1007/s10758-018-9391-y

31.

Eguchi

(2014). Educational robotics for promoting 21st century skills. Journal of Automation, Mobile Robotics & Intelligent Systems, 8(1), 5–11. https://doi.org/10.14313/JAMRIS_1-2014/1

32.

Eguchi, A. (2016). Educational robotics as a learning tool for promoting rich environments for active learning (REALs). In P. Zaphiris & C. S. Ang (Eds.), Human-computer Interaction: Concepts, methodologies, tools, and applications (pp. 740–767). IGI Global.

33.

Field

(2017). Discovering statistics using IBM SPSS statistics (5th ed.). Sage Publications.

34.

Garcia

Harvey

Barnes

(2015). The beauty and joy of computing. ACM Inroads, 6(4), 71–79. https://doi.org/10.1145/2835184

35.

García-Peñalvo

F. J.

Mendes

J. A.

(2018). Exploring the computational thinking effects in pre-university education. Computers in Human Behavior, 80, 407–411. https://doi.org/10.1016/j.chb.2017.12.005

36.

Grover

(2011, April 7–11). Robotics and engineering for middle and high school students to develop computational thinking [paper presentation]. Annual Meeting of the American Educational Research Association. New Orleans, USA.

37.

Grover

Pea

(2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051

38.

Gülbahar

Kert

S. B.

Kalelioğlu

(2019). The self-efficacy perception scale for computational thinking skill: Validity and reliability study. Turkish Journal of Computer and Mathematics Education, 10(1), 1–29.

39.

(2006). Handbook of univariate and multivariate data analysis and interpretation with SPSS. Chapman & Hall/CRC.

40.

International Society for Technology in Education. (2015). CT leadership toolkit. https://id.iste.org/docs/ct-documents/ct-leadershipt-toolkit.pdf?sfvrsn=4

41.

International Society for Technology in Education. (2017). Turn coders into computational thinkers. https://www.iste.org/explore/Innovator-solutions/Turn-coders-into-computational-thinkers

42.

International Society for Technology in Education. (2020). Standards for students . https://www.iste.org/standards/standards/for-students

43.

Kalelioglu

Gulbahar

Kukul

(2016). A framework for computational thinking based on a systematic research review. Baltic Journal of Modern Computing, 4(3), 583–598.

44.

Kazakoff

E. R.

Sullivan

Bers

M. U.

(2013). The effect of a classroom-based intensive robotics and programming workshop on sequencing ability in early childhood. Early Childhood Education Journal, 41(4), 245–255. https://doi.org/10.1007/s10643-012-0554-5

45.

Kline

R. B.

(2011). Principles and practice of structural equation modeling (3rd ed.). The Guilford Press.

46.

Koh

K. H.

Basawapatna

Bennett

Repenning

(2010, September). Towards the automatic recognition of computational thinking for adaptive visual language learning. In 2010 IEEE symposium on visual languages and human-centric computing (pp. 59–66). IEEE.

47.

Korkmaz

Ö.

Cakir

Özden

M. Y.

(2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005

48.

Kukul

Karata

(2019). Computational thinking Self-Efficacy scale: Development, validity and reliability. Informatics in Education, 18(1), 151–164.

49.

L’Heureux

Boisvert

Cohen

Sanghera

(2012, October 7–9). It problem solving. An implementation of computational thinking in information technology. In The 13th annual conference on information technology education (pp.183–188). ACM. doi:10.1145/2380552.2380606

50.

Lee

Martin

Denner

Coulter

Allan

Erickson

Malyn-Smith

Werner

(2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32–37. https://doi.org/10.1145/1929887.1929902

51.

Leech

N. L.

Barrett

K. C.

Morgan

G. A.

(2005). SPSS for intermediate statistics: Use and interpretation (2nd ed.). Lawrence Erlbaum Associates.

52.

Liao

Y.-K.

Bright

(1991). Effects of computer-assisted instruction and computer programming on cognitive outcomes: A meta-analysis. Journal of Educational Computing Research, 7(3), 251–268.

53.

Lin

C. C.

Chao

P. Y.

Lin

E. T.

Tzeng

H. L.

(2018, August 7–9). Exploring the role of visual programming activities in computational thinking. In 2018 1st International cognitive cities conference (IC3) (pp. 135–138). IEEE.

54.

Linn

M. C.

Dalbey

(1985). Cognitive consequences of programming instruction: Instruction, access, and ability. Educational Psychologist, 20(4), 191–206.

55.

Mannila

Dagiene

Demo

Grgurina

Mirolo

Rolandsson

Settle

(2014, June 21–24). Computational thinking in K-9 education. In 2014 Innovation & technology in computer science education conference (pp. 1–29). New York.

56.

Moreno-León

Robles

Román-González

(2015). Dr. Scratch: Automatic analysis of scratch projects to assess and foster computational thinking. Revista de Educación a Distancia, (46), 1–23.

57.

National Research Council. (2010). Report of a workshop on the scope and nature of computational thinking. http://www.nap.edu/catalog.php?record_id=12840

58.

Nickerson

R. S.

(1982). Computer programming as a vehicle for teaching thinking skills. Thinking: The Journal of Philosophy for Children, 4, 42–48. https://doi.org/10.5840/thinking19834310

59.

Oliver

(1993). Measuring hierarchical levels of programming knowledge. Journal of Educational Computing Research, 9(3), 299–312. https://doi.org/10.2190/0LGX-M45X-2WBK-B7A6

60.

Oluk

Korkmaz

Ö.

Oluk

H. A.

(2018). Effect of scratch on 5th graders’ algorithm development and computational thinking skills. Turkish Journal of Computer and Mathematics Education, 9(1), 54–71. https://dx.doi.org/10.16949/turkbilmat.399588

61.

Özyurt

Ö.

Özyurt

(2015). A study for determining computer programming students’ attitudes towards programming and their programming self-efficacy. Journal of Theory and Practice in Education, 11(1), 51–67.

62.

Pala

F. K.

Mıhçı Türker

(2019). The effects of different programming trainings on the computational thinking skills. Interactive Learning Environments, 1–11. https://doi.org/10.1080/10494820.2019.1635495

63.

Penmetcha

M. R.

(2012). Exploring the effectiveness of robotics as a vehicle for computational thinking (Unpublished doctoral dissertation). Purdue University, USA.

64.

Pérez-Marín

Hijón-Neira

Bacelo

Pizarro

(2020). Can computational thinking be improved by using a methodology based on metaphors and scratch to teach computer programming to children? Computers in Human Behavior, 105, 105849. https://doi.org/10.1016/j.chb.2018.12.027

65.

Perrenet

Groote

J. F.

Kaasenbrood

(2005). Exploring students' understanding of the concept of algorithm: Levels of abstraction. ACM SIGCSE Bulletin, 37(3), 64–68.

66.

Perrenet

Kaasenbrood

(2006). Levels of abstraction in students' understanding of the concept of algorithm: The qualitative perspective. ACM SIGCSE Bulletin, 38(3), 270–274.

67.

Riley

D. D.

Hunt

K. A.

(2014). Computational thinking for the modern problem solver. CRC Press.

68.

Rodríguez-Martínez

J. A.

González-Calero

J. A.

Sáez-López

J. M.

(2020). Computational thinking and mathematics using scratch: An experiment with sixth-grade students. Interactive Learning Environments, 28(3), 316–327. https://doi.org/10.1080/10494820.2019.1612448

69.

Román-González, M. (2015, July 6á8). Computational thinking test: Design guidelines and content validation. In Proceedings of the 7th Annual International Conference on Education and New Learning Technologies (EDULEARN15) (pp. 2436–2444). Barcelona, Spain. https://doi.org/10.13140/RG.2.1.4203.4329

70.

Schneider

G. M.

Gersting

(2018). Invitation to computer science. Cengage Learning.

71.

Sedgewick

Wayne

(2011). Algorithms (4th ed.). Addison-Wesley professional.

72.

Seiter

Foreman

(2013, September 9–11). Modeling the learning progressions of computational thinking of primary grade students. In Ninth annual international ACM conference on international computing education research (pp. 59–66). Association for Computing Machinery.

73.

Selby

Woollard

(2013). Computational thinking: The developing definition . https://eprints.soton.ac.uk/356481/

74.

Stevens

J. P.

(2009). Applied multivariate statistics for the social sciences. Taylor & Francis.

75.

Sysło

M. M.

Kwiatkowska

A. B.

(2013, February 26–March 2). Informatics for all high school students. In International conference on informatics in schools: Situation, evolution, and perspectives (pp. 43–56). Springer.

76.

Tabachnick

B. G.

Fidell

L. S.

(2007). Using multivariate statistics (5th ed.). Boston: Allyn & Bacon.

77.

Tang

Yin

Lin

Hadad

Zhai

(2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798

78.

Touretzky

D. S.

Marghitu

Ludi

Bernstein

(2013, March 6–9). Accelerating K-12 computational thinking using scaffolding, staging, and abstraction. In 44th ACM technical symposium on computer science education (pp. 609–614). Association for Computing Machinery.

79.

Tsai

M. J.

Wang

C. Y.

Hsu

P. F.

(2019). Developing the computer programming self-efficacy scale for computer literacy education. Journal of Educational Computing Research, 56(8), 1345–1360. https://doi.org/10.1177/0735633117746747

80.

Weinberg

A. E.

(2013). Computational thinking: An investigation of the existing scholarship and research (Unpublished doctoral dissertation). Colorado State University, USA.

81.

Weintrop

Beheshti

Horn

Orton

Jona

Trouille

Wilensky

(2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5

82.

Wing

J. M.

(2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215

83.

Wing

J. M.

(2008). Computational thinking and thinking about computing. Philosophical Transactions: Series A, Mathematical, Physical, and Engineering Sciences, 366(1881), 3717–3725. https://doi.org/10.1098/rsta.2008.0118

84.

Wong

G. K. W.

Cheung

H. Y.

(2020). Exploring children’s perceptions of developing twenty-first century skills through computational thinking and programming. Interactive Learning Environments, 28(4), 438–413. https://doi.org/10.1080/10494820.2018.1534245

85.

Yadav

Krist

Good

Caeli

E. N.

(2018). Computational thinking in elementary classrooms: Measuring teacher understanding of computational ideas for teaching science. Computer Science Education, 28(4), 371–400. https://doi.org/10.1080/08993408.2018.1560550

86.

Yağcı

(2019). A valid and reliable tool for examining computational thinking skills. Education and Information Technologies, 24(1), 929–951. https://doi.org/10.1007/s10639-018-9801-8

87.

Yeni

(2018). How is computational thinking skill evaluated? Ankara: Pegem Academy.

Conceptual knowledge			Algorithmic thinking			Evaluation
Item	Item-total correlation	Cronbach Alpha	Item	Item-total correlation	Cronbach Alpha	Item	Item-total correlation	Cronbach Alpha
I2	0.812	0.937	I21	0.650	0.886	I34	0.746	0.943
I3	0.787		I22	0.741		I35	0.757
I4	0.702		I24	0.745		I39	0.695
I1	0.662		I20	0.707		I29	0.786
I5	0.725		I23	0.682		I32	0.777
I9	0.787		I25	0.705		I37	0.722
I6	0.671		I18	0.533		I30	0.682
I15	0.749					I38	0.667
I7	0.746					I31	0.718
I14	0.711					I33	0.724
I11	0.676					I27	0.654
I12	0.603					I26	0.703
						I28	0.693
						I36	0.682

	Low experience(N = 94)		Middle experience(N = 236)		High experience (N = 30)		F	p	Result
Subscale	Mean	SD	Mean	SD	Mean	SD	F	p	Result
Conceptual knowledge	2.96	0.74	3.74	0.80	4.10	0.79	39.709	0.000	1 < 2 < 3
Algorithmic thinking	3.03	0.75	3.56	0.80	3.97	0.73	21.966	0.000	1 < 2 < 3
Evaluation	2.93	0.71	3.58	0.75	3.91	0.76	32.059	0.000	1 < 2 < 3

(I)	(J)	Conceptual knowledge			Algorithmic thinking			Evaluation
(I)	(J)	MD (I - J)	SE	p	MD (I - J)	SE	p	MD (I - J)	SE	p
Low	Middle	–0.77591*	0.09641	0.000	–0.52617*	0.09613	0.000	–0.64814*	0.09109	0.000
	High	–1.13191*	0.16576	0.000	–0.93820*	0.16528	0.000	–0.98668*	0.15662	0.000
Middle	Low	0.77591*	0.09641	0.000	0.52617*	0.09613	0.000	0.64814*	0.09109	0.000
	High	–0.35600	0.15322	0.054	–0.41203*	0.15278	0.020	–0.33854	0.14477	0.052
High	Low	1.13191*	0.16576	0.000	0.93820*	0.16528	0.000	0.98668*	0.15662	0.000
	Middle	0.35600	0.15322	0.054	0.41203*	0.15278	0.020	0.33854	0.14477	0.052