The Development of an Assessment Scale for Computational Thinking Competence of In-Service Primary School Teachers

Abstract

In the context of digital technologies, computational thinking (CT) is considered a key competence of primary school teachers. However, assessment tools to measure in-service teachers’ CT competence are rare. The current study explains the design, development, and validation of a scale to determine the in-service teachers’ mastery level of their Computational Thinking Competence (CTC) in the Chinese context. The validity and reliability of the scale have been studied by conducting exploratory factor analysis (EFA), confirmatory factor analysis (CFA) and calculating internal consistency coefficients. The sample of this study consisted of 631 in-service teachers from different Chinese primary schools (EFA N = 189; CFA N = 442). The 31-item scale mirrors five factors: cognitive knowledge of CT, practices and skills of CT, attitudes and perspectives of CT, CT teaching design, and CT teaching implementation. The discussion of the results looks at the future use of the CT scale in Chinese educational settings that mirror a specific national education policy, talent demand, and educational culture.

Keywords

computational thinking competence assessment scale primary school in-service teachers

Introduction

In a digital-driven and complex society, computational thinking (CT) is considered a key competence of teachers in their daily lives and teaching practices. Despite the emphasis on ‘computational’, the competence plays a role independent of technology use. CT is therefore defined as a thinking skill or process that refers to the use of analytic and algorithmic approaches to formulate, analyze, and solve problems (Bocconi et al., 2016). The development of CT in education shows a shift from its adoption in information technology-related education toward discipline-related education (Kafai & Burke, 2013). Also, Barr and Stephenson (2011) suggested that not only STEM subjects but also social sciences are relevant subjects to be linked to CT skills. In these subject areas, teachers themselves use CT to pursue specific subject-related objectives and to improve their curriculum content (Yadav et al., 2016) also in primary schools (Grover & Pea, 2013). CT education in primary schools is becoming a focus of interest for various discipline teachers to meet quality teaching standards. This results in a growing diversity in the use and adoption of CT (Lai, 2022). The study of this diversity in CT approaches by teachers has resulted in a range of instruments, tools, and research designs—both quantitative and qualitative in nature—to map teachers’ approaches towards CT. For example, the exploratory study of Rich et al. (2019) mapped six CT components: abstraction, algorithmic thinking, automation, debugging, decomposition, and generalization. The study linked these components to the teaching of mathematics and science. The study shows teachers' ideas about CT are a useful resource (Rich et al., 2019). Teachers need to recognize how the CT teaching relate to their own instructional practices. According to Kale et al. (2018, p. 575), teaching CT should involve “entail the knowledge of using computational thinking tools (technology), knowing which instructional strategies to use to teach computational thinking and the subject matter (pedagogy), and understanding of computational thinking and the subject matter (content)”. Moreover, incorporating CT into subject-specific classrooms can offer teachers opportunities to collaboratively tackle problems, integrate design methodologies, and utilize technology effectively (Shute et al., 2017; Weintrop et al., 2016). Focusing on pre-service teachers in Singapore, other researchers focused on their CT knowledge, confidence, and use of CT sources (Huang et al., 2022). CT-infused activities enabled teachers to integrate technology in innovative and meaningful ways (Ketelhut et al., 2020). Other studies opted to characterize cognitive CT skills as derived from programming or computing concepts (Tang et al., 2020). However, primary school teachers are aware that CT goes beyond coding or information technology (Corradini et al., 2017).

Since teachers’ cognitions about CT affect the nature and quality of their teaching (Yadav et al., 2014), it seems critical to assess (a) teachers' focus on CT in the curriculum and (b) their abilities to teach CT by adopting specific instructional strategies. It is important to assess teachers' CT knowledge and skills to ensure that they can effectively teach CT-integrated subjects. The literature about CT-related assessment needs to be improved and challenged by the lack of a clear and comprehensive framework to map teachers’ CT competences in life and professional applications. Previous studies have used surveys to assess the perceptions of in-service teachers regarding the components of CT. For instance, Sands et al. (2018) focused on differences between teachers' perceptions of CT that seemed unaffected by the subject matter (STEM vs. non-STEM) or grade level (primary vs. secondary). The authors mentioned that teachers supported statements defining CT as problem-solving, logical thinking, and algorithmic thinking, they also tended to perceive CT as involving mathematics, computer use, and online gaming. Corradini et al. (2017) investigated Italian primary teachers' understanding of CT. The authors classified CT components into four categories: mental processes (algorithmic thinking), methods (automation), practices (testing and debugging), and transversal skills (creating). The survey analysis indicated that most teachers did not conceptualize CT within the four categories mentioned. They also expressed feeling unprepared to foster CT competencies in students. Some previous studies focus on teachers' perceptions of the opportunities and obstacles in CT teaching. There is a study focused on teachers’ observational skills to identify CT teaching opportunities in classrooms (Saito-Stehberger et al., 2021). Israel et al. (2022) carried out teacher interviews and surveys to know barriers to CT integration in the primary grades and results showed common challenges such as limited CT teaching competences, a lack of CT integration time, a lack of CT assessment tools, and a lack of understanding to meet students' diverse instructional needs, limited teacher commitment to teaching CT. Some European teachers are already engaged in classroom activities and have covered several competencies that show high potential for introducing CT components with less burden on teachers (Mannila et al., 2014). Garvin et al. (2019) developed a survey to study the impact of a CT training module on in-service primary teachers.

However, available research about the assessment of subject teachers' CT competence remains limited. Research gaps are related to the operational assessment of teachers’ CT cognitions and their CT integration into their classrooms when focusing on a wide range of school subjects (STEM, social sciences, and humanities). Limited research exists on how teachers comprehend CT, particularly in relation to how they consider integrating it into their own classrooms in primary schools (Yadav et al., 2018). Also, available research hardly reports about the reliability and validity of the scales or tools (Kong et al., 2020). Lastly, most research has been set up in Europe or North America and needs to include other countries. The above explains the focus of the present study to develop a reliable and valid scale to determine the primary school teachers’ CT competences in the Chinese context. Before describing the empirical study, we define the concept of CT. Next, we examine how CT has been measured in previous educational research. In this background, the questionnaire instrument in this study is designed to gather statistical information about the characteristics of in-service teachers in the Chinese context. By doing so, this study can help us understand teachers' cognitive CT competence and identify key dimensions to assess in-service primary school teachers' CT competence.

Conceptual and Theoretical Framework

Definitions of CT

Computational thinking definitions have evolved over time. In the initial phase, CT definitions emphasized specialization, systematization, and programming activities as derived from computer science. For instance, Wing (2006) emphasized that CT involves “solving problems, designing systems, and understanding human behavior, by drawing on the concepts fundamental to computer science (p. 33)”. The description of CT at this phase remained relatively abstract, emphasizing the construction, understanding, imitation, interpretation, and reproduction of computer problem-solving processing models. A first educational contextualization could be found in Papert's (1996) CT approach as linked to mathematics education. He pointed out that computers can help solve problems to allow people to analyze, solve problems, find solutions, and understand the relationship between them. This mirrored a CT conceptualization that thought processes could be defined as sequences of computational steps and algorithms (Aho, 2012).

Along with the expanding range of CT applications, CT is linked to processing information from different disciplines (Barr & Stephenson, 2011). This transition marked a departure from the traditional emphasis on the computing domain, by expanding its application field to information science and psychology. This resulted in attempts to define CT as an information-processing activity in the problem-solving mind (CSTA & ISTE, 2011). In related CT competency frameworks, researchers incorporated computer information processing mechanisms, such as algorithms, decomposition, abstraction, generalization, models, automation, debugging, and knowledge representations (Grover & Pea, 2013) and can be used in various subjects curricula. Next to these developments, other researchers linked CT conceptualizations to CT practices and CT perspectives (Brennan & Resnick, 2012). Reasons for this development are related to the need to link CT operational and practical research (Fang et al., 2021). Meanwhile, CT embodies an approach that can be applied broadly across a wide range of disciplines.

As researchers advanced their understanding of the connotation and functions of CT, they began to focus on CT’s participation, integration, popularization, and internalization in the educational area. For instance, Jocius et al. (2020) position CT in the context of discipline-specific and interdisciplinary learning settings; far away from its former isolated conceptualization in the computer sciences field. In this period, definitions link CT to creativity, critical thinking, cooperation, and communication (Korkmaz et al., 2017). Teacher-related discussions emphasize CT to inspire instructional practices (Kale et al., 2018), specific professions, and careers (Ertugrul-Akyol, 2019). CT is as such seen as a problem solving tool for many subject fields (Tsarava et al., 2022). The goal of CT education is to help students extend computer-related problem-solving techniques to general situations (Fang et al., 2021). State-of-the-art CT definitions emphasize cognitive knowledge, practices and skills, attitudes and perspectives to solve problems in real life by adopting computer science principles with or without the use of digital tools.

Assessment of Primary Teachers’ CT

The growing adoption of CT in education questions the status of teachers’ CT teaching competences. This goes hand in hand with a focus on an adequate assessment of teachers’ integration of CT into their instructional practices. Initially, CT assessment mirrored traditional approaches, such as tests, attitude questionnaires, or assessments based on projects and assignments (Tang et al., 2020; Yadav et al., 2017). Later CT conceptualizations adopted standardized assessment approaches, focusing on a cognitive, attitudinal, and behavioral assessment by using questionnaires, interviews, interventions, and classroom observations. These assessment approaches studied both the CT teaching process and CT learning outcomes (Kim et al., 2015; Mason & Rich, 2019). The increasing integration of CT in subject teaching - e.g., biology, physics, social sciences, (digital) humanities - required a further shift in assessment approaches (Jacob et al., 2018). Next to classroom based assessment of CT curriculum integration, also assessment in the context of teacher education and training was addressed (El-Hamamsy et al., 2022; Fessakis & Prantsoudi, 2019). In an experiment to assess the impact of CT modules on in-service teachers, a statewide survey of primary Maryland teachers is conducted to understand how to integrate CT into content lessons. The survey asked teachers about their conceptualization of CT, CT resources they rely on, CT integration, and their comfort levels in providing effective CT instruction for their students (Garvin et al., 2019). With the growing emphasis on CT integration, assessment approaches check teacher’s CT conceptualizations (e.g., problem-solving, logic, algorithms, use of technology/computers, and critical thinking) the CT resources being used, teachers’ confidence in providing CT instruction, teachers’ attitudes towards CT (Yadav et al., 2014). Moreover, evaluation systems in CT are being developed to reflect elements like creativity, cooperation, and communication in the form of subjective evaluation (Fang et al., 2021). This emphasizes the potential of CT education for fostering a diverse range of skills and abilities.

Nevertheless, a number of gaps are observed in currently available assessment approaches. Firstly, CT assessment is mainly based on European or North American to measure CT approaches and implementations. These CT assessment approaches do not meet culture-specific demands e.g., those found in the Chinese context. (Fang et al., 2021). Secondly, the number of assessment approaches involving in-service teachers is low (Tang et al., 2020). Thirdly, CT assessments partly neglect CT's abilities, attitudes, or dispositions (Jong et al., 2020). Fourthly, little consideration is being given to the instructional dimension while evaluating CT instruction, with most attention being paid to elements of CT. In the primary school context, CT assessment hardly considers the teaching dimension when tackling a specific subject in a non-programming context. The above analysis helps to delineate the focus of the current paper on the development of a CT scale appropriate for primary school teachers, considering the Chinese context and CT implementations that go beyond the realm of STEM subjects.

Rationale for the CT Competence Scale Construction

CT scale construction as presented in this article, adopts a new taxonomy of educational objectives as a backbone. Marzano and Kendall (2007) presented it and improved the clarity of the logical formulation as a foundation for the thinking skills curriculum and the framework for assessment design. The new taxonomy of educational objectives is a system constructed with a strong psychological dimension that involves the domain of knowledge and the level of processing (Marzano & Kendall, 2007). Specifically, the three domains of knowledge include information (the 'what' of terms, facts, principles, etc.), mental procedures (the 'how' of skills, intellectual skills, intellectual processes, algorithms, rules, etc.), and psychomotor procedures (action and thinking at the level of psychological skills and psychological processes). Any knowledge domain can be a different combination of these three specific knowledge categories. The levels of processing were divided into hierarchical three systems with six levels, including cognitive system (retrieval, comprehension, analysis, and knowledge utilization), metacognitive system (metacognition), and self-system (self-system thinking) (Marzano & Kendall, 2007). The task was finished by applying the cognitive system's fundamental abilities. The metacognitive system was used to generate problem solving strategies and methods, and the basic cognitive system was invoked. The value judgment for the task was produced by using the self-system. The effective functioning of these three systems relied on the foundational support provided by the knowledge domain. The new taxonomy of educational objectives is more suitable when it comes to thinking teaching and assessment. CT is present in the curriculum in the form of knowledge contents and thinking procedures, allowing the theory application for a new taxonomy of Marzano's educational goals to explore CT assessment methods. In other words, it is a coherent and integrated approach for fostering thinking and goal orientation.

The assessment model of this study was conducted based on the above analysis. The theory in this research is mainly based on the new taxonomy of educational objectives to assess teachers’ CT competence from daily life and professional practice applications in Chinese educational settings. This study incorporated the definition of CT from previous studies (Brennan & Resnick, 2012; CSTA & ISTE, 2011) and built a conceptual framework for its evaluation based on the categorization framework of Marzano’s educational goals (Marzano & Kendall, 2007). In previous studies, CT is defined as using conceptual knowledge in the problem-solving practice process and supported by attitudes. To begin with, we identified that CT can be embedded into the knowledge domains. The new taxonomy of educational objectives classification of thinking systems can provide a theoretical and coherent basis for the CT assessment (Sun & Wang, 2021). The three-dimensional framework for CT (computational concepts, computational practices, and computational perspectives) proposed by Brennan and Resnick (2012) maps to the classification of knowledge domains in the new taxonomy of educational objectives (information, mental procedures, psychomotor procedures) (Sun et al., 2022). Specifically, the information field emphasizes declarative knowledge, which is consistent with what CT concepts are defined. The mental procedures are mainly constructed from procedural information about “how to do,” which is in line with CT practices and skills. The psychomotor procedures are comprised of physical actions that an individual employs to navigate daily life and participate in complex physical activities for work, which also connect to the CT perspectives. Meanwhile, the division of processing levels provides ideas and directions for exploring the learning and assessment of CT. Specifically, CT assessment is classified as an evaluation of the cognitive system, metacognitive system, and self-system (Sun et al., 2022). At the level of each system, we focus on primary school teachers’ CT knowledge domains related to cognitive knowledge, practices and skills, attitudes and perspectives. Therefore, we finally focused on the three systems with CT knowledge domains to distinguish.

Assessment of the cognitive system focuses on the 'conceptual dimension’, which is the fundamental dimension of CT cognition to complete tasks and solve problems. Problem conceptualization of CT recognizes that problems need to be understood and presented so that algorithmic thinking may help in the solution-development process. Specifically, it includes problem formulation and analysis as well as data collection, analysis, and representation (Fraillon et al., 2018). The process of formulating a problem involves breaking down a complicated problem into smaller parts and specifying the requirements for creating a computational solution, often with the use of digital devices or resources (Fang et al., 2021; Fraillon et al., 2018). Analysis is the process of making connections between the characteristics and solutions of both experienced and new situations to provide a conceptual framework for problem formulation. Besides, it is vital to gather and interpret data from the system to make wise decisions on problem solving (Barr & Stephenson, 2011; Fraillon et al., 2018). Effective data collection and representation rely on a thorough understanding of the data's characteristics and the available mechanisms for collecting, organizing, and representing it for analysis (Fraillon et al., 2018). This may entail the creation or utilization of a simulation of a complex system to generate data that could reveal patterns or behavioral characteristics not easily discernible when viewed at an abstract system level (Fraillon et al., 2018). Halpern (2014) defined critical thinking as “the use of those cognitive skills or strategies that increase the probability of a desirable outcome”. Critical thinking may be seen as a crucial element of CT when it is used to address problems (Doleck et al., 2017; Korkmaz et al., 2017). Critical thinking in CT is expressed in the tolerance of ambiguity and the judgement of diverse problem-solving approaches (Sun et al., 2022). Therefore, cognitive knowledge of CT in this study can be observed in three aspects: problem formulation and analysis; data collection, analysis, and representation; and critical thinking for problem-solving.

Assessment of the metacognitive system considers the ‘process dimension’ of learners' use of CT to generate solutions and problem-solving strategies in practice. The metacognitive system primarily serves to establish goals and monitor the accuracy and clarity of the thinking process (Marzano & Kendall, 2007). Problem planning, evaluation, and solutions focus on the process of establishing, monitoring, and adjusting the problem planning, which involve developing functional specifications or requirements aligned with user needs and desired outcomes and with a view to designing implementing, and evaluating the key features of a solution (Fraillon et al., 2018). It is significant to possess the ability to plan and evaluate solutions from various perspectives, while comprehending the advantages, disadvantages, and impacts on stakeholders associated with alternative solutions (Fang et al., 2021; Fraillon et al., 2018). Cooperativity can enhance people's understanding and ability in cooperative problem-solving, as collaboration among individuals with diverse skills for addressing complex problems is essential in the twenty-first century (Korkmaz et al., 2017). When cooperation is needed to work towards a solution, it is required of the project members in CT to communicate effectively (Korkmaz et al., 2017). That means the critical CT skills also include cooperativity and communication. The cooperativity and communication within CT is reflected in the collaboration and communication with others to achieve common goals and solutions (Sun et al., 2022). Therefore, the practices and skills of CT in this study emphasize two aspects: problem planning, evaluating, and solution; communication and cooperativity.

Assessment of the self-system considers the 'thought dimension' and looks at the internal drives and the construction of thinking about real-life problems using CT. The self-system is composed of interconnected attitudes, beliefs, and emotions (Marzano & Kendall, 2007). Computational perspectives are ideas developed about the world around us and ourselves. Specifically, CT attitudes and perspectives can be seen as expressing (perception of computation as a way of expression and creation), connecting (perception of computation as a way of interacting and working with others), questioning (raising questions and using technology to solve real-life problems), and confidence (confidence in dealing with complexity) (Brennan & Resnick, 2012; CSTA & ISTE, 2011). For example, Brennan and Resnick (2012) proposed “the value of creating with others and the value of creating for others” as connecting that CT acquisition can be supported through creating work and is enriched by interactions with others. Therefore, this study focuses on four aspects of attitudes and perspectives of CT: expressing, connecting, questioning, and confidence.

Besides, building on the two foci when teachers consider CT, we further look at their instructional design competence to incorporate CT into the curriculum and their actual mastery of CT-related instructional implementation (Fang et al., 2021; Ministry of Education, 2022). As for CT teaching design, it discussed CT integrated into the curriculum teaching design of content and activities. It can be observed by the ability, process, and attitudes to integrate CT into subject knowledge content and activities for teaching design (Fang et al., 2021). As for CT teaching implementation, it mentioned CT integrated into the curriculum teaching implementation of processes and strategies. It can be observed by the ability and attitude to organize, guide, and intervene during CT teaching activities using strategies and resources (Fang et al., 2021). Figure 1 depicts the comprehensive assessment model of teachers' CT competences, serving as the theoretical foundation for the development of the instrument.

Figure 1.

Assessment model of teachers' computational thinking competence.

Method

Data Collection and Analysis

To assess the psychometric properties of the CT competence scale, responses from the teacher questionnaire were divided into two samples by separately conducting exploratory factor analysis (EFA) and confirmatory factor analyses (CFA). Both samples consider their personal information included the participant’s gender, age, teaching experience, teaching subjects, final degree, and school type in China. The samples are from in-service primary school teachers in China, covering the main subjects of primary education. Digital questionnaires were administered and voluntarily completed by teachers. Sample 1 (N = 189) were subjected to an EFA on the CT competence items. EFA was carried out with the maximum likelihood estimation (Finch & West, 1997), varimax rotation, and principal component analysis by SPSS statistics 27.0.

CFA was conducted on data from the Sample 2 (N = 442) to validate the identified factor structure. The in-service teachers comprised of 64 male (14.48%) and 378 female teachers (85.52%) with most of the respondents within the 31–50 age range (66.29%). Most of the teaching experience is concentrated in the range of 4–30 years (78.51%), which shows that most of the respondents are more experienced in teaching. Most respondents (95.02%) have undergraduate and master's degrees. In terms of school type, over 90% of the sample come from public schools, which is in line with the current trend in the proportion of primary schools’ type in China. The teachers’ teaching subjects are Chinese (26.70%), English (20.59%), mathematics (21.49%), science (9.73%), information technology (7.01%), arts (4.98%), music (4.07%), ethics and law (3.17%), sport and health (2.26%). Maximum likelihood estimation was used to test CFA models and were conducted using AMOS 26.0. Lastly, the reliability of the scales was determined.

The Development Process of the Scale and Instrument

The item pool was developed in three stages. In the first stage, a literature review was conducted to identify and examine existing scales designed to measure the same purpose. Based on the literature (Brennan & Resnick, 2012; CSTA & ISTE, 2011; Fraillon et al., 2018), CT competence was developed as cognitive knowledge, practices and skills, perspective and attitudes. The design of the questionnaire structure for assessing CT competences from both daily life and professional applications considered the theory of the new taxonomy of educational objectives, curriculum standards, and teaching practices in the Chinese context. Appropriate items were selected and developed from scales titled “computational thinking scales” (Korkmaz et al., 2017), “the computational thinking assessment scale for pre-service teacher based on teaching competencies” (Fang et al., 2021).

In the second stage, a team of 11 teachers and experts from universities, primary schools and computer science education institutions were consulted to evaluate the indicators and items. During the three round of consultation process, the expert is anonymous and free to make his or her own judgement about the consultation. The first round of experts’ interview consultations focused on suggesting conceptual definition, theoretical foundation, and overall construction of the assessment framework. The experts reached a consensus on the conceptual definitions, theoretical basis, and overall construction of the assessment framework in the scale. In the second round, experts specifically reviewed the descriptions of the indicators and item pool. We designed a consultation questionnaire for experts to provide their revision suggestions on each indicator and item. Experts filled in the questionnaire whether they agreed with the presence of indicators and items in the scale based on the theoretical basis and educational context in China. They removed, modified, improved, and added to some of the indicators and related items. Finally, items with repetitive, irrelevant, or unclear expressions were deleted and improved. The content and expression of items were further optimized and accurately described. The option content of items and the Likert scale were improved. After that, the expert consultation by interview in the third round was conducted for general instructions and supporting resources. In the third round, general instructions and supporting resources for the questionnaire were optimized and modified by experts. Relevant videos and pictures were added to the digital questionnaire to facilitate a better understanding of the questionnaire content by the participants from their daily lives and professional teaching. Meanwhile, the questionnaire instrument is written in Chinese to facilitate completion by in-service teachers in China (see Appendix). After three rounds of expert consultation, there was a gradual convergence of agreement among experts on the assessment scale and questionnaire instrument.

In the third stage, all the items were examined and determined by researchers and in-service teachers. To enhance the scale's clarity and suitability for teachers, the initial draft was reviewed by 27 primary school teachers who did not participate in the subsequent survey rounds. The researchers subsequently revised several items based on suggestions from in-service primary school teachers to enhance questionnaire clarity and comprehensibility for teachers.

Finally, considering the application of CT by in-service teachers in daily life and professional teaching, this CT competence scale is organized and further designed from five perspectives: CT cognitive knowledge, CT practices and skills, CT attitudes and perspectives, teaching design for CT into curriculum content and activities integration, and teaching implementation for CT into curriculum processes and strategies integration. The assessment scale for CT competence is a six-point Likert type scale used to determine the levels of in-service teachers expressed in the items. Response options were “(1) strongly disagree”, “(2) disagree”, “(3) slightly disagree”, “(4) slightly agree”, “(5) agree”, “(6) strongly agree”. This item pool consists of 75 items in total were subjected to an EFA. This item pool included 32 items related to cognitive knowledge of CT (CKCT), 12 items related to practices and skills of CT (PSCT), 15 items related to attitudes and perspectives of CT (APCT), 6 items related to design for contents and activities integration (DCAI), and 10 items related to implementation for processes and strategies integration (IPSI).

Results

Exploratory Factor Analysis

The 189 samples containing 75 items were analyzed for EFA. Regarding EFA, Kaiser-Meyer-Oklin (KMO) and Bartlett's test of sphericity analyzes have been detected whether factor analysis could be conducted. The corresponding p-value is less than .05, indicating that the corresponding item can be used for EFA. Based on the results, the Kaiser–Meyer–Olkin measure of sampling adequacy (KMO) was .937, showing that factor analysis can produce distinct and reliable factors (Hutcheson & Sofroniou, 1999). The significant results of Bartlett's test of sphericity ( $χ^{2}$ = 5118.745, df = 465, p = .000) indicated that the correlations between variables considerably differed from zero.

Based on the remaining 75 items, EFA was conducted on Sample 1. Principal component analysis method was used to extract factors, and varimax method was used to rotate component. All factor loadings coefficients less than .50 were suppressed (Hair et al., 2010). In terms of items selection, a comprehensive analysis was carried out in combination with load distribution, factor interpretation, and communalities analysis. Following that, 31 items were retained and 5 principal component factors were extracted, with the scope of the scale explain 72.2% of the cumulative total variance after testing. Table 1 shows the rotated component matrix, where the corresponding factor in each of the items included in the rotated principal components is higher than .50. Factor loadings of the 31 remaining items vary between .504 and .840.

Table 1.

Rotated Component Matrix.

Factors	Items	Com. Fact	Component
Factors	Items	Com. Fact	1	2	3	4	5
Cognitive knowledge of CT	1-1 I can understand information and describe or demonstrate daily life problems with appropriate diagrams, graphs, words and images.	.792	.840
	1-2 I can find and identify useful information from the data.	.758	.813
	1-3 I can use digital tools to collect and store data information.	.797	.788
	1-4 I can evaluate and update data information to meet the needs of the task.	.772	.778
	1-5 I can use mathematical concepts (e.g. permutations, probability, etc.) to characterize problems and solutions encountered in life.	.655	.754
	1-6 I can analyze and present data information to help draw conclusions and inform subsequent task planning.	.753	.749
	1-7 I can clearly explain the reasoning process and outcome of problem solving.	.734	.724
	1-8 I am good at preparing regular plans regarding the solution of the complex problems.	.718	.672
	1-9 I can understand some algorithmic rules or computer instructions with the help of numbers, symbols and concepts.	.608	.664
	1-10 I can make use of a systematic method while comparing the options at my hand and while reaching a decision.	.605	.586
Practices and skills of CT	2-1 I can quickly adapt to my individual role in the collaboration and fulfil the responsibilities undertaken.	.746		.816
	2-2 In previous collaborations, I have a clear understanding of the responsibilities of other members and have been able to obtain effective help from others.	.687		.791
	2-3 I can generate more ideas in communication and collaborative learning.	.686		.784
	2-4 I can make timely adjustments and improvements when I find bugs or wrong steps in the implementation plan.	.727		.655
	2-5 I can describe the solution and explain why it is the best solution among many.	.728		.635
	2-6 I can compare the relative advantages and disadvantages of one solution to others.	.616		.612
	2-7 I can identify plans, methods, and steps for new problem solutions by reflecting on and analyzing solutions to similar problems.	.564		.569
Attitudes and perspectives of CT	3-1 I can complete tasks and interact with others to expression through the use of digital technology.	.752			.730
	3-2 I can create work for others for entertainment, participation, support or educational purposes.	.721			.717
	3-3 I can raise questions that may arise in the use of digital technology.	.739			.681
	3-4 I have the confidence to deal with complex issues.	.627			.514
	3-5 I can question the results of tasks with redesign, revise and explain the content of tasks that are questioned.	.683			.504
Design for contents and activities integration	4-1 I can create contexts and activities that involve applications of computational thinking relevant to the teaching of the subject curriculum.	.803				.813
	4-2 I can experiment with the selection of different interdisciplinary themes to design course content in order to achieve or optimize the integration of computational thinking with disciplinary course content.	.808				.809
	4-3 I can create the basic conditions and foundational framework for helping others to use computational thinking to solve problems.	.813				.804
	4-4 I am looking forward to and confident in my future participation in information technology teaching demonstrations and competitions.	.651				.691
	4-5 I can find a lot of content and knowledge points related to computational thinking in the curriculum I teach.	.746				.654
Implementation for processes and strategies integration	5-1 I am skilled in guiding others to summarize the thinking and experience developed in the problem-solving process.	.853					.731
	5-2 I can judge the mental state of others by the way they behave in the problem-solving process (words, gestures, manners, expressions, etc.).	.779					.723
	5-3 I am good at organizing high quality and efficient group activities.	.696					.630
	5-4 I can understand or solve real-world problems through simulation or modelling.	.758					.587
Eigenvalue			15.31	2.55	2.35	1.16	1.00
Explained variance			49.4%	8.2%	7.6%	3.7%	3.2%

Finally, the items’ contents in factors have been examined and named. The item loadings for the 31 remaining items in the scale across factors are presented in Table 1, along with eigenvalues and explained variance. A first dimension was identified as “cognitive knowledge of CT (CKCT)” containing 10 items and the factor loading varying between .586 and .840. A second dimension was identified as “practices and skills of CT (PSCT)” containing 7 items with factor loading between .569 and .816. A third dimension was “attitudes and perspectives of CT (APCT)” containing 5 items with factor loading between .504 and .730. A fourth dimension “design for contents and activities integration (DCAI)” containing 5 items with factor loading between .654 and .813. A fifth dimension “implementation for processes and strategies integration (IPSI)” contains 4 items with factor loading between .587 and .731. The five factors’ eigenvalues are 15.31, 2.55, 2.35, 1.16 and 1.00 respectively. The first factor, CKCT explained 49.4% of the total variances. Other factors explained variances from 3.2% to 8.2%.

Confirmatory Factor Analysis

In a following stage, CFA is employed to empirically validate the factor structure identified in EFA. CFA was conducted on Sample 2, consisting of 442 teachers. CFA analysis on the five-factor with 31 items correlated factors model was run and the results were shown in Figure 2. The factor loadings for CKCT, PSCT, APCT, DCAI and IPSI are all higher than .70, indicating that each latent variable is highly representative of the corresponding items. The Average Variance Extracted (AVE) for each latent variable is higher than .50 and the Composite Reliability (CR) value exceeds .90.

Figure 2.

The confirmatory factor analysis diagram of the scale.

The fitness of this model is shown in Table 2. To evaluate the goodness of the model fit, several fit indices were calculated (χ² = 1200.564, p < .001, df = 416, χ²/df = 2.886, RMSEA = .065, SRMR = .044, CFI= .941, TLI = .934, IFI = .941, RMR = .038, NFI = .913). These values indicate that the model fit present an acceptable goodness (Kline, 2016). In other words, the obtained model demonstrates that the factors are confirmed and supported by the data. Table 3 presents the correlation of the principal component factor. Figure 2 shows the scale's factorial model and its values for the factor-item relationship.

Table 2.

Model Fit Summary.

Model Fit Indices	Fit Value	Acceptable Fit
χ²	1200.564
df	416
χ²/df	2.886	<3 (Kline, 2016)
p value	<.001	<.05 (Kline, 2016)
Root mean square residual (RMR)	.038	<.05 (McDonald & Moon-Ho, 2002)
Standardized root mean square residual (SRMR)	.044	≤.10 (Kline, 2016)
Root mean square error of approximation (RMSEA)	.065	<.08 (Brown, 2015)
Normed fit Index (NFI)	.913	≥.90 (Bentler & Bonett, 1980)
Comparative fit Index (CFI)	.941	≥.90 (Bentler, 1990)
Tucker-Lewis index (TLI)	.934	≥.90 (Bentler, 1990)
Incremental fit Index (IFI)	.941	≥.90 (Bollen, 1989)

Table 3.

The Correlation of Principal Component Factor.

Factors	CKCT	PSCT	APCT	DCAI	IPSI
CKCT	-	.784	.784	.731	.708
PSCT	.784	-	.830	.736	.772
APCT	.784	.830	-	.857	.853
DCAI	.731	.736	.857	-	.850
IPSI	.708	.772	.853	.850	-

Based on the theory and the results of the first-order CFA analysis, the correlation coefficients of the five latent variables ranged from .708 to .857, suggesting that the model may have a higher level of factor structure. To provide a more comprehensive understanding of the scale's construction, the study developed a second-order model. Therefore, the study constructed a second-order five-factor CFA model of in-service teachers' CT competence. The model assumptions are as follows: (1) The CT competence of in-service teachers can be explained by five first-order factors and one second-order common factors. (2) There is no cross-factorization of each item, i.e., each item falls on the single factor constructed. Path diagrams of the second-order five-factor CFA model using AMOS 26. The results indicated that the model fit indices for the second-order hierarchical model in Table 4 (i.e., X² = 1243.492, p < .001, df = 421, X²/df = 2.954, RMSEA = .067, SRMR = .048, CFI= .938, TLI = .932, IFI = .939, RMR = .041, NFI = .910) and showed that the model fit indices for both models were comparable. In other words, the obtained model demonstrates that the factors are confirmed and supported by data.

Table 4.

Model Fit Summary.

Model Fit Indices	Fit Value	Acceptable Fit
χ²	1243.492
df	421
χ²/df	2.954	<3 (Kline, 2016)
p value	<0.001	<0.05 (Kline, 2016)
Root mean square residual (RMR)	.041	<0.05 (McDonald & Moon-Ho, 2002)
Standardized root mean square residual (SRMR)	.048	≤0.10 (Kline, 2016)
Root mean square error of approximation (RMSEA)	.067	<0.08 (Brown, 2015)
Normed fit Index (NFI)	.910	≥0.9 (Bentler & Bonett, 1980)
Comparative fit Index (CFI)	.938	≥0.9 (Bentler, 1990)
Tucker-Lewis index (TLI)	.932	≥0.9 (Bentler, 1990)
Incremental fit Index (IFI)	.939	≥0.9 (Bollen, 1989)

The factor loadings of each latent variable for CKCT, PSCT, APCT, DCAI and IPSI are all higher than .70, indicating that strong representation of each latent variable by its corresponding items. The AVE for each latent variable is higher than .50 and the CR value exceeds .90. Figure 3 illustrates the scale's factorial model and the values for the second-order factor-item relationship.

Figure 3.

The second-order confirmatory factor analysis diagram of the scale.

Reliability Analysis

The results regarding the reliability of the scale are presented below through internal consistency analyses. The scale's psychometric quality according to the factors and has been calculated using Cronbach alpha coefficient (Cronbach, 1951). The accepted indicator for scale’s reliability coefficient is .70 and above (Field, 2009). In this study, the high item-scale correlations were found in both two samples, with Cronbach alpha values ranging from .881 to .967 for the 189 samples and .899 to .957 for the 442 samples. The overall internal consistency was high, with a general alpha coefficient of .982 and .976, respectively. The results about the perceptions of in-service primary school teachers’ CT competence (N = 442) shows the five subscales’ mean scores, including 4.400 (SD = .850) for CKCT, 4.691 (SD = .735) for PSCT, 4.516 (SD = .812) for APCT, 4.516 (SD = .818) for DCAI, and 4.524 (SD = .780) for IPSI. By analyzing the internal consistency of the sample, the alpha coefficients of scales all reached above .80, indicating that this assessment tool has good reliability.

Discussion

The aim of the study was to construct the Assessment Scale of Teachers' Computational Thinking Competences as a tool for assessment of teachers' CT in both life and professional application. This scale can be used as a pre- and post-test to assess teachers' cognitions and teaching effectiveness regarding CT in classrooms and educational training settings. It comprises the introduction and items as two main sections. The introduction clarifies the CT concept, answer requirements and items. This study tries to determine how much various variables explain teachers’ CT competence using the computational thinking competence (CTC) instruments. Based on the traditional scale research paradigm, this study rigorously constructed the evaluation model and assessed primary teachers' CT competences using a valid and reliable instrument across different disciplines. Overall, primary school teachers have positive perceptions of CT competences. Primary school teachers are highly competent in practices and skills of CT and its teaching implementation process and strategies for curriculum integration. However, teachers have difficulties with their cognitive knowledge of CT. It is important to further improve teachers' competences to design content and activities for curriculum integration, and to guide teachers to have the right attitudes and perspectives for integrated CT into subject curriculum.

This study's evaluation model is different from existing CT scales in its unique construction. Firstly, the scale systematically organizes the development of CT and combines the theory of the new taxonomy of educational objectives (Sun & Wang, 2021) to construct a comprehensive evaluation of cognitive, practical, internal, and pedagogical CT competences. This concept-theory-dimensional evaluation modeling path can better integrate concepts into the theoretical system and improve the scientific nature of the evaluation model. Secondly, the scale focuses on the needs of teachers' professional competence. Teachers need to be demonstrated how CT can improve their subject teaching and how CT relates to their teaching (Kale et al., 2018). It is crucial for teachers to consider ways for integrating CT concepts and ideas into their curricula and instructional practice (Yadav et al., 2016). Therefore, the scale in this study evaluates teachers' CT competence with the new taxonomy as a framework considering both pedagogical and real-life perspectives. The dimensions of core evaluation components have been further determined by combining the practical experience of teachers' CT teaching training programs and Chinese policy documents, and finally formed a multi-dimensional evaluation system that distinguishes other similar studies. Thirdly, the evaluation model specifies the components of each dimension with the mental activities and behavioral performance. These components are mostly derived from the collation and analysis of researches on the development of CT scales. By presenting a model of evaluation, we hope to help users understand the different components of CT that are the focus of this scale's items.

Computational thinking is a relatively young instructional field, and there is limited research available to guide the integration of CT, particularly in the context of diverse academic, linguistic, and cultural instruction (Israel et al., 2022). In view of the multi-dimensional evaluation system and multi-scenario application, the strict standard control was carried out based on comprehensive assessment. The data set was tested which including missing data treatment, removal of abnormal data, and normality testing. The validity of the scale has been examined through factor analysis, ensuring that excluded or retained items were justified. The construct validity of scale was established through factor loadings, eigenvalues of the factors, and explained variance rates. EFA was conducted on the sample and a comprehensive judgement was made on the items. CFA has been carried out to confirm the factor structures of the scale that the EFA detected to be composed of five factors as the result. As a result of the CFA, the observed values of the scale model reveal that the data show conformity, which means that the obtained model has been validated by the data. Based on the achieved values, it has been determined that each item and factor in the scale serves the aim of assessing the property to be measured and the entire scale at meaningful level. The scale's internal consistency coefficients have been calculated and the scale can make reliable measurements. In addition, the fitted indicators all conform to accepted standards for scale development and have good construct validity.

Conclusion and Suggestions

Computational thinking competence is a statistically valid and reliable instrument that could be used to identify levels of teachers' CT competences. Given the lack of a reliable and valid measurement instrument aimed at assessing CT levels for in-service teachers in the literature, it could be thought that this measurement tool could make significant contributions to the field. To use the scale for various groups and purposes, it is important to consider that each scale can be reliable and valid for its own group. When it is intended to be employed in the groups covered by the research, it is advised that the validity and reliability analysis be conducted once again. The five factor point obtained from the scale might be used separately or as a total point. The theoretical model of CT competences has developed in exploration along with changes in the CT connotation. The focus of the evaluation is mainly on the unique attributes and components of CT itself, and there are few discussions from professional needs, which inevitably loses the practical significance and value. Collaboration between computer science educators, teacher educators, and educational technology faculty is imperative to work with teachers in developing activities that highlight the intrinsic connections of CT concepts and practices with subject content (Yadav et al., 2016). The fundamental goal of this study is to promote CT and subject education, improve teachers' awareness and competences of CT education, and promote the integration of CT into multidisciplinary and interdisciplinary teaching. Moreover, the idea about popularization and participation of CT are integrated into the evaluation in teachers' life and professional application, and a standardized evaluation scale is generated. Given the cultural and contextual differences, it is vital to develop a CT assessment scale that is appropriate to the Chinese context, thereby enhancing the cultural applicability and practical value of this study.

However, this study is limited to the assessment of primary school teachers. The adaptation study should be conducted, if it is desired to apply this study to teachers in higher grades or in lower grades. Besides, it is necessary to consider the issue of gender imbalance among male and female teachers in future research in different educational contexts. Furthermore, CT competences is limited to the sub-competences defined in this scale, as well as the basic items in the scales developed separately for these competences. In this study, subject teachers lacking sufficient CT competences can consider collaborating with other subject teachers such as mathematics, information technology, science, art, and music who possess stronger CT competences in order to facilitate interdisciplinary curriculum teaching. It is suggested that curricula are scrutinized in accordance with the specific subjects taught by teachers, while also considering interdisciplinary cooperation experiences in the design of studies that aim to impart CT competencies with a validated model. Additionally, other factors such as teaching experience, teaching subject, and gender that influence teachers' CT competences and curriculum teaching integration should be further considered.

Footnotes

Author Contributions

Conceptualization: Xinlei Li; Methodology: Xinlei Li; Formal analysis and investigation: Xinlei Li, Guoyuan Sang and Johan van Braak; Writing - original draft preparation: Xinlei Li; Writing - review and editing: Xinlei Li, Guoyuan Sang, Martin Valcke and Johan van Braak; Resources: Xinlei Li, Guoyuan Sang, Martin Valcke and Johan van Braak; Supervision: Johan van Braak. All authors have read and agreed to the published version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the China Scholarship Council (CSC), grant number CSC201906040214.

ORCID iD

Xinlei Li

Appendix

Author Biographies

Xinlei Li is a PhD student at the Department of Educational Studies at Ghent University in Belgium. Her research focuses on computational thinking and curriculum integration in primary schools. She also has a keen interest in interdisciplinary education, pedagogical innovations, technology-enhanced learning and teaching. ORCID: .

Guoyuan Sang is a full professor at the Faculty of Education at Beijing Normal University in China. He specializes in teacher education, project-based learning, school culture improvement, curriculum and pedagogy. His expertise in setting up research projects within the local Chinese context provides valuable methodological and practical guidance. His research primarily focuses on Chinese curriculum and pedagogy studies with educational technology.

Martin Valcke is a full professor at the Department of Educational Studies at Ghent University in Belgium. His specialization field of research focuses mainly on teacher education, Information and Communication Technologies (ICT) in primary schools and the innovation of higher education. This is translated into a focus on studies about learning management systems, design studies, curriculum development, professional development of staff, university management, and policy development. His research field is multi-disciplinary and is mainly set up in the educational context in developing countries. ORCID: .

Johan van Braak is a full professor at the Department of Educational Studies at Ghent University in Belgium and head of the METiS (Measuring Educational Effectiveness and Technology) research group. His research interests include educational innovation, curriculum and school effectiveness, e.g. the assessment of digital competences of students and teachers in compulsory education. ORCID: .

References

Aho

A. V.

(2012). Computation and computational thinking. The Computer Journal, 55(7), 832–835. https://doi.org/10.1093/comjnl/bxs074

Barr

Stephenson

(2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community? ACM Inroads, 2(1), 48–54. https://doi.org/10.1145/1929887.1929905

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238

Bentler

P. M.

Bonett

D. G.

(1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606. https://doi.org/10.1037//0033-2909.88.3.588

Bocconi

Chioccariello

Dettori

Ferrari

Engelhardt

(2016). Developing computational thinking in compulsory education – Implications for policy and practice. Publications Office of the European Union. https://doi.org/10.2791/792158

Bollen

K. A.

(1989). Structural equations with latent variables: Wiley. https://doi.org/10.1002/9781118619179

Brennan

Resnick

(2012). New frameworks for studying and assessing the development of computational thinking. Proceedings of the 2012 Annual Meeting of the American Educational Research Association. Vancouver, Canada, 13-17 April.

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.

Corradini

Lodi

Nardelli

(2017). Conceptions and misconceptions about computational thinking among Italian primary school teachers. Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, USA, 18-20 August, 136–144. https://doi.org/10.1145/3105726.3106194

10.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/bf02310555

11.

CSTA & ISTE . (2011). Operational definition of computational thinking for K-12 education. https://cdn.iste.org/www-root/Computational_Thinking_Operational_Definition_ISTE.pdf

12.

Doleck

Bazelais

Lemay

D. J.

Saxena

Basnet

R. B.

(2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: Exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4(4), 355–369. https://doi.org/10.1007/s40692-017-0090-9

13.

El-Hamamsy

Zapata-Cáceres

Barroso

E. M.

Mondada

Zufferey

J. D.

Bruno

(2022). The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school. Journal of Educational Computing Research, 60(7), 1818–1866. https://doi.org/10.1177/07356331221081753

14.

Ertugrul-Akyol

(2019). Development of computational thinking scale: Validity and reliability study. International Journal of Educational Methodology, 5(3), 421–432. https://doi.org/10.12973/ijem.5.3.421

15.

Fang

Sun

Zeng

Liu

(2021). Development of assessment scale for computational thinking of pre-service teachers based on teaching competency: An exploration on sternberg’s theory of successful intelligence and teaching theory of thinking. E-Education Research, 2, 112–120. https://doi.org/10.13811/j.cnki.eer.2021.02.016

16.

Fessakis

Prantsoudi

(2019). Computer science teachers’ perceptions, beliefs and attitudes on computational thinking in Greece. Informatics in Education, 18(2), 227–258. https://doi.org/10.15388/infedu.2019.11

17.

Field

(2009). Discovering statistics using SPSS (3rd ed.). Sage Publications.

18.

Finch

J. F.

West

S. G.

(1997). The investigation of personality structure: Statistical models. Journal of Research in Personality, 31(4), 439–485. https://doi.org/10.1006/jrpe.1997.2194

19.

Fraillon

Ainley

Schulz

Duckworth

Friedman

(2018). IEA international computer and information literacy study 2018 assessment framework. Springer Open. https://doi.org/10.1007/978-3-030-19389-8

20.

Garvin

Killen

Plane

Weintrop

(2019). Primary school teachers’ conceptions of computational thinking. Proceedings of the 50th ACM Technical Symposium on Computer Science Education, Minneapolis, USA, 27 February-2 March, 899–905. https://doi.org/10.1145/3287324.3287376.

21.

Grover

Pea

(2013). Computational thinking in K – 12 : A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051

22.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2010). Multivariate data analysis (7th ed.). Pearson.

23.

Halpern

D. F.

(2014). Thought and knowledge: An introduction to critical thinking (5th ed.). Psychology Press. https://doi.org/10.4324/9781315885278

24.

Huang

Looi

C.-K.

Yeter

I. H.

(2022). Comparison of STEM, non-STEM, and mixed-disciplines pre-service teachers’ early conceptions about computational thinking. Proceedings of Sixth APSCE International Conference on Computational Thinking Education and STEM Education 2022, Delft, Netherlands, 15-17 June, 38–43. https://doi.org/10.34641/ctestem.2022.462

25.

Hutcheson

Sofroniou

(1999). The multivariate social scientist. Sage.

26.

Israel

Liu

Yan

Sherwood

Martin

Fancseli

Rivera-Cash

Adair

(2022). Understanding barriers to school-wide computational thinking integration at the elementary grades: Lessons from three schools. In Ottenbreit-Leftwich

Yadav

(Eds.), Computational Thinking in PreK-5: Empirical Evidence for Integration and Future Directions (pp. 64–71). ACM and the Robin Hood Learning + Technology Fund.

27.

Jacob

Nguyen

Tofel-Grehl

Richardson

Warschauer

(2018). Teaching computational thinking to English learners. NYS TESOL Journal, 5(2), 12–24.

28.

Jocius

Joshi

Dong

Robinson

Catete

Barnes

Albert

Andrews

Lytle

(2020). Code, connect, create: The 3C professional development model to support computational thinking infusion. Proceedings of the 51st ACM Technical Symposium on Computer Science Education, Portland, USA, 11-14 March, 971–977. https://doi.org/10.1145/3328778.3366797

29.

Jong

M. S. Y.

Nguyen

Geng

Chai

C. S.

Lin

P. Y.

(2020). Development and predictive validity of the computational thinking disposition questionnaire. Sustainability, 12(11), 4459.

30.

Kafai

Y. B.

Burke

(2013). The social turn in K-12 programming: Moving from computational thinking to computational participation. Proceeding of the 44th ACM Technical Symposium on Computer Science Education, Denver, USA, 6-9 March, 603–608. https://doi.org/10.1145/2445196.2445373

31.

Kale

Akcaoglu

Cullen

Goh

Devine

Calvert

Grise

(2018). Computational what? Relating computational thinking to teaching. TechTrends, 62(6), 574–584. https://doi.org/10.1007/s11528-018-0290-9

32.

Ketelhut

D. J.

Mills

Hestness

Cabrera

Plane

McGinnis

J. R.

(2020). Teacher change following a professional development experience in integrating computational thinking into elementary science. Journal of Science Education and Technology, 29(1), 174–188. https://doi.org/10.1007/s10956-019-09798-4

33.

Kim

Yuan

Hill

R. B.

Doshi

Thai

C. N.

(2015). Robotics to promote elementary education pre-service teachers’ STEM engagement, learning, and teaching. Computers & Education, 91, 14–31. https://doi.org/10.1016/j.compedu.2015.08.005

34.

Kline

R. B.

(2016). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.

35.

Kong

S. C.

Lai

Sun

(2020). Teacher development in computational thinking: Design and learning outcomes of programming concepts, practices and pedagogy. Computers & Education, 151, Article 103872. https://doi.org/10.1016/j.compedu.2020.103872

36.

Korkmaz

Ö.

Çakir

Özden

M. Y.

(2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005

37.

Lai

R. P. Y.

(2022). Teachers’ ontological perspectives of computational thinking and assessment: A text mining approach. Journal of Educational Computing Research, 60(3), 661–695. https://doi.org/10.1177/07356331211043547

38.

Mannila

Dagiene

Demo

Grgurina

Mirolo

Rolandsson

Settle

(2014). Computational thinking in K-9 education. Proceedings of the Working Group Reports of the 2014 on Innovation and Technology in Computer Science Education Conference (pp. 1–29). ACM. https://doi.org/10.1145/2713609.2713610

39.

Marzano

R. J.

Kendall

J. S.

(2007). The new taxonomy of educational objectives. Corwin Press.

40.

Mason

S. L.

Rich

P. J.

(2019). Preparing elementary school teachers to teach computing, coding, and computational thinking. Contemporary Issues in Technology and Teacher Education, 19(4), 790–824.

41.

McDonald

R. P.

M. H. R.

(2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. https://doi.org/10.1037/1082-989X.7.1.64

42.

Ministry of Education of the People’s Republic of China . (2022). English curriculum standards for compulsory education (2022 version). Beijing Normal University Publishing Group.

43.

Papert

(1996). An exploration in the space of mathematics educations. International Journal of Computers for Mathematical Learning, 1(1), 95–123. https://doi.org/10.1007/bf00191473

44.

Rich

K. M.

Yadav

Schwarz

C. V.

(2019). Computational thinking, mathematics, and science: Elementary teachers’ perspectives on integration. Journal of Technology and Teacher Education, 27(2), 165–205. https://par.nsf.gov/servlets/purl/10183080

45.

Saito-Stehberger

Garcia

Warschauer

(2021). Modifying curriculum for novice computational thinking elementary teachers and English language learners. Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education, Virtual Event, Germany, June 26-July 1, 136–142. https://doi.org/10.1145/3430665.3456355.

46.

Sands

Yadav

Good

(2018). Computational Thinking in K-12: In-service teacher perceptions of computational thinking. In Khine

M. S.

(Ed.), Computational Thinking in the STEM Disciplines (pp. 151–164). Springer.

47.

Shute

V. J.

Sun

Asbell-Clarke

(2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003

48.

Sun

Zhou

Yang

(2022). Evaluation and developmental suggestions on undergraduates’ computational thinking: A theoretical framework guided by Marzano’s new taxonomy. Interactive Learning Environments, 31(10), 6588–6610. https://doi.org/10.1080/10494820.2022.2042311

49.

Sun

Wang

(2021). Construction of evaluation framework of computational thinking based on Mazano’s new taxonomy of educational goals. E-Education Research, 42(6), 20–26. https://doi.org/10.13811/j.cnki.eer.2021.06.003

50.

Tang

Yin

Lin

Hadad

Zhai

(2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, Article 103798. https://doi.org/10.1016/j.compedu.2019.103798

51.

Tsarava

Moeller

Román-González

Golle

Leifheit

Butz

M. V.

Ninaus

(2022). A cognitive definition of computational thinking in primary education. Computers & Education, 179, Article 104425. https://doi.org/10.1016/j.compedu.2021.104425

52.

Weintrop

Beheshti

Horn

Orton

Jona

Trouille

Wilensky

(2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5

53.

Wing

J. M.

(2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215

54.

Yadav

Gretter

Good

McLean

(2017). Computational thinking in teacher education. In Rich

Hodges

(Eds.), Emerging research, practice, and policy on computational thinking (pp. 205–218). Springer. https://doi.org/10.1007/978-3-319-52691-1_13

55.

Yadav

Hong

Stephenson

(2016). Computational thinking for all: Pedagogical approaches to embedding 21st century problem solving in K-12 classrooms. TechTrends, 60(6), 565–568. https://doi.org/10.1007/s11528-016-0087-7

56.

Yadav

Krist

Good

Caeli

E. N.

(2018). Computational thinking in elementary classrooms: Measuring teacher understanding of computational ideas for teaching science. Computer Science Education, 28(4), 371–400. https://doi.org/10.1080/08993408.2018.1560550

57.

Yadav

Mayfield

Zhou

Hambrusch

Korb

J. T.

(2014). Computational thinking in elementary and secondary teacher education. ACM Transactions on Computing Education, 14(1), 1–16. https://doi.org/10.1145/2576872