Development and Validation of the Computational Thinking Test for Elementary School Students (CTT-ES): Correlate CT Competency With CT Disposition

Abstract

This study aimed to develop the Computational Thinking Test for Elementary School Students (CTT-ES) to assess young children’s CT competencies in non-programming contexts and also examine the relationship between CT competencies and CT dispositions. A survey including a pool of CTT-ES candidate items and the Computational Thinking Scale (CTS) was administered to 631 elementary school students. Rasch model of the Item Response Theory and the discrimination analysis of the Classical Testing Theory were conducted for item analyses. Pearson’s correlation analyses and hierarchical multiple regression analyses were used to examine the relationships between CTT-ES and CTS scores. The results showed that the final CTT-ES including 16 items had a good fitness, discrimination, and reliability to evaluate elementary students’ domain-general CT competencies. The convergent validity of CTT-ES was confirmed by its significant correlations with the CTS scores. The significant regression model not only showed students’ CT competencies can be predicted by their CT dispositions but also supported The Developmental Model of CT. This study provided a valid and reliable tool for assessing young children’s CT abilities. It also furthered our understanding about the developmental orders of CT abilities and contributed to the theoretical construction of CT.

Keywords

Computational thinking assessment competency test elementary school non-programming domain-general CT

Background

In the rapid changing society of the 21st century, researchers, educators, and educational policy makers in many countries have been urging educational reform by integrating computational thinking (CT) into the school curricula to develop all students’ complex problem-solving skills (Román-González et al., 2019). According to Wing (2006, 2011), CT involves thought processes as well as problem-solving skills for systematically dealing with issues that occur every day for everyone. From a computer literacy perspective, Tsai et al. (2021) divided the definitions of CT into domain-specific and domain-general categories. Domain-specific CT refers to solving problems using specific approaches in a specific learning domain, such as computer scientists usually define CT as to solve problems systematically using computer programming (Brennan & Resnick, 2012). Domain-general CT refers to one’s abilities to solve complex problems in daily-life contexts, such as the five abilities (abstraction, decomposition, algorithmic thinking, evaluation, and generalization) suggested by Selby and Woollard (2013).

Although not every child is expected to become a computer programmer in his/her future career development, every 21^st century citizen needs to be prepared with competencies to solve complex problems in daily lives. Especially for young children, the domain-general CT should be more important and have more impacts than the domain-specific CT, because they have more experiences or involved longer in daily-life problem-solving. Measuring young children’s CT skills only through computer programming contexts may not be enough for a comprehensive understanding of their CT skill developments. Understanding and promoting the development of young children’s computational thinking abilities from a domain-general angle or a literacy perspective should be more emphasized in contemporary computer education research. Meanwhile, assessments play an influential role in determining the success of integrating computer programming or CT into K-12 education. Without appropriate measurement instruments, it is difficult to validly and reliably assess the effectiveness of any intervention, nor to mention comparing differences across studies. Due to there is a need to develop appropriate CT assessment tools for young children, this study focuses mainly on the discussions of domain-general CT and its related assessment tools in non-programming contexts.

Domain-general CT Assessment Tool

Domain-general CT has been defined as one’s abilities to solve complex problems in daily-life contexts, such as the five abilities (abstraction, decomposition, algorithmic thinking, evaluation, and generalization) suggested by Selby and Woollard (2013). This domain-general framework has been adopted by Computing At School in the United Kingdom in developing guidance on computational thinking for teachers (Csizmadia et al., 2015). Nevertheless, based on prior review studies (Cutumisu et al., 2019; Hsu et al., 2018; Tang et al., 2020), most existing CT assessment tools focus on assessing students’ programming outcomes or attitudes about programming learning. That is, most existing CT assessments are involved in domain-specific contexts, especially computer programming contexts. So far, few domain-general assessment tools have been developed in terms of self-reported questionnaires or performance tests.

Recently, some researchers have attempted to develop self-reported questionnaires for assessing domain-general CT. Tsai et al. (2021) developed a self-reported Computational Thinking Scale (CTS) based on the five CT elements suggested by Selby and Woollard (2013). With an overall reliability of 0.91, the CTS with 19 items assessed students’ CT dispositions in five subscales: Abstraction subscale assessed the ability to decide what details should be retained or ignored; Decomposition subscale assessed the ability to break down problems into smaller, understandable, and manageable parts; Algorithmic Thinking subscale assessed the ability to think procedurally as a step-by-step set of solutions to the problems; Evaluation subscale assessed the ability to decide which solution appropriately fits the purpose; and Generalization subscale assessed the ability to adapt and transfer solutions from one problem to another. With an adaptable stem question, the CTS can be used in referring to both domain-general and domain-specific contexts (such as programming contexts). The CTS then further contributed to the construction and validation of the Developmental Model of Computational Thinking (DMCT) in a follow-up study (Tsai et al., in press). Another prior study (Korkmaz et al., 2017), using a three-dimensional definition (knowledge, skill, and attitude) of CT, claimed to develop a CT questionnaire with 29 items to assess college students’ self-perceived computational thinking skills (also abbreviated as CTS). This scale was assessed under a framework including the four factors: creativity, cooperativity, algorithmic-critical thinking, and problem-solving, which in turn may have assessed students’ computer programming self-efficacy similar to the CPSES developed by Tsai et al. (2019).

Not many performance tests are currently available for teachers or educational researchers to assess students’ domain-general CT in teaching practices or educational studies. A common tool to assess K-12 students’ domain-general CT through solving “real-life” problems is the performance tests used in the Bebras Challenge (Dagienė & Futschek, 2008). The Challenge is a competition hosted annually and internationally since early 2010s. In the year of 2020, students from over 40 countries competed in their national contests. Its ultimate goal is to motivate students to be interested in learning informatics concepts and practicing CT skills through solving a series of small tasks in non-programming contexts (Dagienė et al., 2017). Thus, the Bebras tests can be administered to students without any prior knowledge of programming languages such that it can be regarded as a domain-general CT assessment tool. On the other hand, since it requires students to transfer their CT skills to diverse problems, contexts, and situations, it can be also regarded as a CT skills transfer tool (Román-González et al., 2019).

Association between CT Questionnaire and CT Test

No matter domain-specific CT or domain-general CT is defined in studies, CT can be assessed from a process-oriented or an outcome-oriented angle (Tsai et al., 2021). The process-oriented assessment aims to understand one’s thinking processes such as dispositions or attitudes for solving complex problems, while the outcome-oriented assessment aims to evaluate one’s performances or achievements in solving complex problems. Self-reported CT questionnaire is an example of process-oriented CT assessment because questionnaires can reflect one’s affective variables such as attitudes, anxiety, interests, or motivations for learning. For example, the Computational Thinking Scale (Tsai et al., 2021) was designed to assess one’s computational thinking dispositions in the five cores of CT. On the other hand, CT performance test is belonged to the category of outcome-oriented CT assessment. Computer programming tasks may be the most commonly used CT performance tests so far in the literature (Cutumisu et al., 2019; Hsu et al., 2018; Tang et al., 2020). Most of the assessment tests may be designed by computer programming instructors and require prior knowledge and skills of particular programming languages. For instance, Moreno-León and Robles (2015) developed Dr. Scratch that allows one to conduct automatic analysis of Scratch projects uploaded as a way to assess their development of CT skills. The relationship between CT processes and CT performances so far is still not quite clear in the literature. This may be due to that the inconsistent CT definitions have been adopted in prior literature, and not many valid and reliable instruments are available for conducting the examination.

Prior studies have suggested that, although many assessment tools are developed to examine students’ CT, how the different measurements relate to each other is a critical issue worthy of further investigation (Román-González et al., 2019). Additionally, probing convergent validity of CT instruments may demonstrate the robustness of a study’s findings (Carlson & Herdman, 2012). For instance, among the limited literature examining convergent validity of CT assessments, Román-González et al. (2019) probed the relations among their CT, spatial ability, reasoning ability, and problem-solving ability. They confirmed the convergent validity of the Computational Thinking test (CTt; Román-González, 2015) with the Bebras test based on the statistically significant, positive, and moderate correlation found between the two assessments. They further suggested the necessity to explore the convergent validity of CT assessment tests with other types of assessment tools, such as the tools assessing the attitudes, perceptions or dispositions of CT (e.g., the Computational Thinking Scale, CTS, developed by Tsai et al., 2021). Therefore, examining the relationship between CT performance tests and CT thinking processes is important for both theoretical understanding and instrumental development.

In sum, before we merge CT into a national curricula for young children, assessment tools are important and required for researchers and policy makers to understand and to draw conclusions about the effectiveness of the educational curriculum reform. Given that there is few valid and reliable domain-general CT test available for school teachers or educational researchers to use in teaching practices or educational studies, this study aimed to develop a domain-general CT test for elementary school students and to validate the CT tests with a valid self-reported CT scale (Tsai et al., 2021). Meanwhile, due to the relationship between the CT performance and the self-reported CT disposition is worth of exploring, this study also aimed to examine the correlations between students’ CT performance and CT disposition, especially examining whether young students’ CT dispositions can predict their CT performance.

Purpose

Based on the aforementioned literature, there were two research purposes of this study: One was to design a CT assessment tool, the Computational Thinking Test for Elementary School Students (CTT-ES), to measure elementary school students’ computational thinking competencies (or skills) in a non-programming context. The convergent validity of the CTT-ES was also examined with a self-reported assessment tool, the CTS (Tsai et al., 2021). The other purpose of this study was to explore the relationship between young students’ self-reported dispositions of CT and their CT competencies. Specifically, this study also examined whether young students’ CT disposition scores obtained from the CTS questionnaire can predict their CT performance scores assessed by the CTT-ES test.

Method

In order to develop the Computational Thinking Test for Elementary School Students (CTT-ES) to assess primary school students’ computational thinking competencies in a non-programming context, a pool of candidate items was developed based on the items of the Bebras Challenges (Dagienė & Futschek, 2008). A survey including the CTT-ES and the CTS (Tsai et al., 2021) was then conducted to analyze and validate the items of CTT-ES. The Rasch model of Item Response Theory (IRT) and the discrimination of the Classical Testing Theory were conducted for item analysis. To investigate the convergent validity of different CT tools, Pearson’s correlation analyses between dimensional scores of the CTT-ES and of the CTS were conducted, followed by hierarchical regression analysis. Thus, the sequential exploratory mixed method (Creswell, 2013) was used in this study.

Item Development

The CT test items were developed based on the items in the Bebras Challenges (Dagienė & Futschek, 2008), an international challenge on informatics and computational thinking. The original Bebras items were designed for students from Grade 1^st to 12^th and were categroized into six age groups, two grades a group. Each item in the booklets was marked by the level of difficulty, ranging from low, medium, to high levels, and how it responded to Selby and Woollard’s (2013) CT framework including algorithm, abstraction, generalization, decomposition, and evaluation. In the process of revising the items, the following steps were taken in this study. First, the published item booklets from 2014-2016 (downloaded from https://www.computingatschool.org.uk/) were used as the original item pool for selecting items. Among these items, the researchers selected 16 originally designed for fifth and sixth graders. The researchers’ selection criteria were to 1) include a balanced selection among the five themes of computational thinking, and 2) include diverse selection among the three difficulty levels. Second, the items were translated into Chinese and then the context was revised to become more adaptive to the social-cultural environment in Taiwan. For example, the main character Bebras was changed to a goose which is a more common animal for children to see in Taiwan, and the context was redesigned as a farm called Jack’s farm. The translated and revised items were then reviewed by an expert team consisting of elementary and middle school teachers and university professors to examine whether the item difficulty was appropriate, whether the wording of each item was suitable for the elementary school students, and to which subscale (dimension) each item belonged. Finally, the 16 items were piloted on several elementary students (fifth and sixth graders) to test the time for completion and to further identify any potential problems with the wording and content (including the graphs). Recommendations were analyzed and final revisions were made where necessary. The final version including 16 items was then used for this study. Table 1 shows the items corresponding to each of the five CT factors, which were discussed and verified via an expert group meeting. Scoring of the items was based on the CT factors it responds to. For instance, Item Q01 was related to Decomposition and Evaluation factors. Therefore, its score is two and the total scores for the 16 items were 33. Figure 1 illustrates two sample items, one difficult and the other easy, for the CTT-ES.

Table 1.

Items of the CTT-ES responding to Selby and Woollard’s (2013) CT framework.

Item	Decomposition	Abstraction	Algorithm	Evaluation	Generalization
Q01. Searching for a goose	V			V
Q02. Swimming teams			V
Q03. Feeding time	V		V
Q04. Flowers codes	V		V		V
Q05. Soccer games		V		V
Q06. Storage for food pellets		V		V
Q07. Fruit robot			V	V
Q08. Boring men		V		V	V
Q09. Purchase order		V		V
Q10. Harvesting vegetables			V		V
Q11. Filling food pellets					V
Q12. Doggy chain		V			V
Q13. Watering system		V	V
Q14. Stone abacus	V	V			V
Q15. Fruit shipment 1	V			V
Q16. Fruit shipment 2	V			V
Frequency of total items	6	7	6	8	6

Figure 1.

Sample items of the CTT-ES. Searching for a goose (Q1) is an easy item and Feeding geese (Q3) is a difficult one.

Participants

Since the CTT-ES was designed for elementary school upper-division students, a total of 631 elementary school students, 294 fifth graders and 337 sixth graders, in Taiwan were selected as the sample of this study. They were drawn conveniently from 28 intact classes in five elementary schools in southern Taiwan. According to the National Education Act in Taiwan (Ministry of Education, 2009), primary and secondary schools should enforce mixed-ability grouping in class. Thus, the sample was considered to be normally distributed regarding their academic achievements.

Tools

This study utilized the Test Analysis Modules (TAM) and the Wright Map packages in the R software to estimate item difficulties and students’ abilities on the same logit scale (Robitzsch et al., 2020; Irribarra & Freund, 2014). Additionally, the Computational Thinking Scale (CTS) (Tsai et al., 2021) with a non-programming context was used to validate the CTT-ES developed in this current study. The CTS (Tsai et al., 2021) included 19 items categorized into five subscales: decomposition, abstraction, algorithmic thinking, evaluation, and generalization. Each item was evaluated by a 5-point Likert scale ranging from 1 (not at all like me) to 5 (very much like me). Sample items for this CTS included “I usually think if it is possible to decompose a problem” (decomposition subscale), “I usually think of a problem from a whole point of view, rather than looking at the details” (abstraction subscale), “I am used to figuring out the procedures step-by-step for a solution” (algorithmic thinking subscale), “I tend to find a correct solution for a problem” (evaluation subscale), and “I tend to solve a new problem according to my experience” (generalization subscale). When originally reported with a programming context, the reliability (Cronbach’s alpha) of the overall CTS was .91 and ranged from .74 to .83 for the subscales (Tsai et al., 2021). In current study, the CTS with a non-programming context was used, and the reliability of Cronhach’s alpha was .95 for the overall scale and ranged from .79 to .86 for the subscales.

Data Collection

An online survey including the CTT-ES test (16 items) and the CTS scale with non-programming context (19 items) was administered to all of the study participants. The data were all collected in school computer classrooms. There was no time limitation for answering the CTT-ES and the CTS items. The participants on average spent approximately 25 minutes completing the CTT-ES. Demographic variables such as grade levels and genders were also collected in the survey.

Data Analysis

Item Analysis

To understand whether this computational thinking test is an appropriate measurement for elementary school students, we applied the Rasch model, a one-parameter logistic Item Response Theory model for dichotomous items (Andrich & Marais, 2019; Mayer et al., 2014; Rasch, 1960). The data were analyzed using the Test Analysis Modules and the Wright Map packages in R software (Robitzsch et al., 2020; Irribarra & Freund, 2014).

Correlation Analysis

To validate the CTT-ES assessment test, Pearson’s correlation coefficient analyses were conducted between the participants’ CTT-ES assessment test scores and their self-reported CTS disposition scores (Tsai et al., 2021). The examinations focused on not only the correlation between the two total scores but also on the correlations between each pair of corresponding subscale scores. For example, the correlation between the Abstraction sub-score of CTT-ES and the Abstraction sub-score of CTS was examined.

Hierarchical Multiple Regression Analysis

To further verify the relationships between the CTS self-reported scores and the CTT-ES performance scores, a hierarchical multiple regression analysis based on the structural model of the CTS (Tsai et al., in press) conducted by using the five factors of CTS as the predictors and the CTT-ES performance scores as the predicted variable. In the model, the five factors of CTS had a four-level hierarchical relationship. Therefore, the five factors of CTS were put into the hierarchical multiple regression model in four steps in the following order: Abstraction and Decomposition in the first round, followed by Algorithm, Evaluation, and Generalization.

Results

IRT Analysis and Classical Discrimination

The Wright Map obtained from the IRT analysis is shown in Figure 2, in which the distribution of the persons’ proficiencies (i.e., students’ abilities, on the left side) and item difficulties (on the right side) were both illustrated. The item difficulty is defined as the respondents have 50% chance of answering this item correctly on the logit scale of Rasch model and the item difficulties in the CTT-ES ranged from 5.27 logits (most difficulty, See Table 2) to −1.19 logits (least difficult). In addition, persons’ proficiencies refer to the level of latent trait of respondent who has 50% chance of answering items correctly on the logit scale of Rasch model and the persons’ proficiencies ranged from 3.61 logits (most able) to −3.57 logits (least able) in the CTT-ES. This shows that CTT-ES includes varied difficulties of items and is able to measure a range of persons’ proficiency. As shown in Figure 1, Q10, Q6, and Q16 had the highest item difficulties, while Q4 as well as Q13 had the lowest item difficulties. Most of the testing items were scattered over the scope of difficulties, and the complete item difficulties of the CTT-ES are listed in Table 2.

Figure 2.

Wright Map of Rasch analysis of the CT Test scores of the elementary school students.

Table 2.

Item properties of Rasch analysis and classical discrimination on Elementary school students’ computational thinking test scores.

Item	Percent correct (%)	Difficulty	Discrimination	Infit MNSQ
Q10	0.79	5.27	0.09	1.00
Q6	3.96	3.59	0.30	0.95
Q16	8.40	2.74	0.30	1.00
Q3	17.74	1.79	0.48	0.94
Q15	23.45	1.39	0.47	0.96
Q14	29.32	1.04	0.55	0.92
Q2	30.90	0.95	0.33	1.10
Q5	40.73	0.44	0.40	1.06
Q7	41.84	0.39	0.37	1.08
Q8	48.65	0.06	0.46	1.01
Q9	54.52	−0.22	0.34	1.12
Q11	55.31	−0.26	0.49	0.98
Q12	62.44	−0.61	0.52	0.96
Q1	67.19	−0.85	0.41	1.03
Q4	73.06	−1.18	0.54	0.92
Q13	73.22	−1.19	0.50	0.94

Note: EAP/PV reliability = 0.69, WLE reliability =0.66.

The psychometric properties of all items in the CTT-ES are illustrated in Table 2, listing from the lowest to the highest percentage of correct responses. The item difficulties ranged from 5.27 (most difficult) to −1.19 (least difficult). The average persons’ proficiency was 0.008 logits (SD = 1.13). This study also applied point biserial correlations for the correct answers to obtain the classical discrimination values which ranged from 0.09 to 0.55. The MNSQ for each item ranged from 0.92 to 1.12 (Mean = 1.00, SD = 0.06), which was between 0.70 and 1.30, thus indicating a good fit to the Rasch model at the item level (Bond & Fox, 2015). Although Q10 has the lowest discrimination (.09) in classical discrimination, it provides higher difficulty (5.27 logits) to measure persons’ proficiency (the student’s ability) that reaches 3.61 logits in CTT-ES. Also, the infit MNSQ of Q10 is 1.00 which indicates a good fit to the Rasch model. This study decided to keep Q10 after the expert consultation meeting. Finally, the Weighted Likelihood Estimate (WLE) person-separation reliability was 0.66, the Expected A Posteriori estimate based on Plausible Values (EAP/PV) reliability was 0.69, and the Cronbach’s alpha was 0.69, suggesting good reliabilities for the CTT-ES test.

Correlation Analyses

A correlation analysis was conducted to examine the convergent validity of both the CTT-ES and CTS instruments. Table 3 showed the CTT-ES test total score was significantly and positively correlated (r = .274, p < .001) with the CTS self-reported total score. Meanwhile, the correlation coefficients located in the diagonal of the table were also positively significant (r = .154 to .291, p < .001). This indicated that the total scores of two instruments as well as all paired sub-scores with the same name from the two instruments (e.g., CTT-ES_AB and CTS_AB) were correlated significantly and positively. The results suggested the factors of the two instruments are convergent. Finally, it is reasonable to observe that the CTT-ES sub-scores were interrelated with the sub-scores of CTS, because each CTT-ES task was associated with more than one CT factor in CTS.

Table 3.

Pearson’s correlations between CTT-ES scores and CTS total and subscale scores. (N=530).

	CTT-ES_AB	CTT-ES_DE	CTT-ES_AL	CTT-ES_EV	CTT-ES_GE	CTT-ES_Total
CTS_AB	.225 ^***	.263^***	.172^***	.231^***	.238^***	.268^***
CTS_DE	.135^***	.180 ^***	.060	.118^***	.174^***	.155^***
CTS_AL	.170^***	.220^***	.154 ^***	.174^***	.205^***	.214^***
CTS_EV	.207^***	.257^***	.214^***	.224 ^***	.243^***	.275^***
CTS_GE	.228^***	.230^***	.197^***	.230^***	.291 ^***	.281^***
CTS_Total	.221^***	.262^***	.185^***	.225^***	.263^***	.274 ^***

Note: *** p < .001; AB = Abstraction, DE = Decomposition, AL=Algorithmic Thinking, EV = Evaluation, GE = Generalization, CTS = Computational Thinking Scale, CTT = Computational Thinking Test.

Hierarchical Multiple Regression Analyses

Based on the above correlation analysis result and the hierarchical structure suggested in the Developmental Model of Computational Thinking (DMCT) (Tsai et al., in press), this study further examined the relationships between the factors of CT and the CTT-ES performance using a hierarchical multiple regression analyses. That is, factors of CTS were put into the multiple regression models according to the level order, from the fundamental to the higher levels, in the DMCT.

The hierarchical multiple regression analysis results of students’ CTT-ES performance score are presented in Table 4. To predict students’ CTT-ES performance scores, the students’ responses to five factors in the CTS questionnaire (Abstraction, Decomposition, Algorithm, Evaluation, and Generalization) were recruited as the predictors in this regression model. Based on Pearson’s correlation results and the structural model of the CTS (Tsai et al., in press), students’ self-reported scores of Abstraction and Decomposition in CTS were the input to step 1 in model 1. The results of model 1 (F value = 20.88, p < .001, R² adjusted = 0.07) revealed that only students’ self-reported scores of Abstraction had a significant relationship with their CTT-ES performance scores (t = 5.30, p < .001). In step 2, students’ self-reported scores of Algorithm were recruited. The results of model 2 (R² adjusted = 0.07, R² change = 0.00, F change = 0.98, p > .05) indicated that students’ self-reported scores of Algorithm did not have any significant relation with their CTT-ES performance scores (t = 0.99, p > .05). In step 3, students’ self-reported scores of Evaluation were recruited. The results of model 3 (R² adjusted = 0.09, R² change = 0.02, F change = 10.16, p < .01) showed that students’ self-reported scores of Abstraction and Evaluation played significant roles in their CTT-ES performance scores (t = 5.59 and 3.19, p < .01). In step 4, students’ self-reported scores of Generalization were recruited in the final model 4. The results of model 4 (R² adjusted = 0.10, R² change = 0.01, F change = 7.70, p < .01) indicated that students’ self-reported scores of Abstraction (t = 2.56, p < .05), Decomposition (t = −2.32, p < .05), and Evaluation (t = 2.12, p < .05) were significantly associated with their CTT-ES performance scores. Although students’ self-reported scores of Decomposition were positively correlated with their CTT-ES performance scores (r = 0.155, p < .001) in the Pearson’s correlation coefficient analyses, Decomposition turned into a negative predictor in hierarchical regression model 4. This may be due to the suppression effect which usually happens when the independent variables positively and highly correlate to each other and positively correlate to the dependent variable (Conger, 1974; Pandey & Elliott, 2010). In sum, the significant predictors of students’ CTT-ES performance score were found to be Abstraction, Decomposition, Evaluation, and Generalization.

Table 4.

Hierarchical regression results of using students’ CTS sub-scores to predict CTT-ES test score.

Steps and variables		Model 1			Model 2			Model 3			Model 4
		B	β ^a	T	B	β	T	B	β	t	B	β	t
1	Constant	3.47		6.87^***	3.30		6.19^***	3.00		5.59^***	2.78		5.16^***
	CTS-AB	1.01	0.31	5.30^***	0.90	0.27	4.07^***	0.69	0.21	3.00^**	0.59	0.18	2.56^*
	CTS-DE	−0.18	−0.06	−1.00	−0.25	−0.08	−1.28	−0.34	−0.11	−1.76	−0.45	−0.15	−2.32^*
2	CTS-AL				0.22	0.07	0.99	−0.18	−0.05	−0.71	−0.37	−0.11	−1.40
3	CTS-EV							0.77	0.24	3.19^**	0.54	0.17	2.12^*
4	CTS-GE										0.69	0.21	2.78^**
	R² adjusted	0.07			0.07			0.09			0.10
	R² change	0.07			0.00			0.02			0.01
	F Change	20.88^***			0.98			10.16**			7.70^**

Note: ^a standard coefficients; ^* p < 0.05 ^**p < 0.01 ^***p < 0.001; AB = Abstraction, DE = Decomposition, AL = Algorithmic Thinking, EV = Evaluation, GE = Generalization, CTS = Computational Thinking Scale, CTT = Computational Thinking Test.

Discussion

According to the above results, the current study has two major findings: (1) The designed assessment tool, the Computational Thinking Test (CTT-ES), is fitted and reliable for evaluating young children’s computational thinking competencies (skills or performances) in a non-programming context. Also, it is valid via an examination along with the Computational Thinking Scale (CTS) (Tsai et al., 2021) which is another computational thinking assessment tool concerning self-reported perceptions or dispositions. (2) A significant relationship was found between students’ computational thinking dispositions and computational thinking competencies. The results of the hierarchical regression analysis further supported the construct of the Developmental Model of Computational Thinking (DMCT) (Tsai et al., in press). The discussion is therefore focused on the two folds as the following.

The Current CTT-ES Assessment Test

This study developed the CTT-ES based on the five major themes of computational thinking, including algorithm, abstraction, generalization, decomposition, and evaluation (Selby & Woollard 2013). Through both the item analyses based on the Item Response Theory (IRT) and the Classical Testing Theory, the CTT-ES including 16 items (tasks) is well-fitted and well-discriminated for elementary school students to assess computational thinking skills (or competencies). While the Bebras Challenge tasks have been used worldwide for assessing computational thinking competencies, only a few studies have examined the psychometric properties of the tasks. Hubwieser and Mühling (2015) used the IRT method to examine a more dated version (i.e., the German Bebras Contest of 2009) than the versions referenced in the current study. Other studies have used qualitative evaluation (e.g., content analysis or rubrics) or quantitative measures, such as questionnaires or success rate for identifying item difficulty (Izu et al., 2017; van der Vegt, 2018). Past studies have pointed out that the chosen items from the Bebras item pool represent a joint construct; nevertheless, there exist problems with the quality of items (Hubwieser & Mühling, 2015). Results in the current study have shown that the revised Bebras items, with attention to the wording, the representation, and the context, is a reliable set of tasks for assessing computational thinking competencies.

This study further examined and confirmed the convergent validity of the CTT-ES (outcome-oriented CT assessment) with a self-reported assessment tool, the CTS (process-oriented CT assessment). Prior literature (Román-González et al., 2019) merely confirmed the convergent validities among any two outcome-oriented CT assessments (e.g., Scratch tasks and Bebras tests). Results of this study suggested that the convergent validity can be conformed between CT thinking process and CT performance outcomes. Additionally, since the CTT-ES and the CTS were developed according to the same framework of Selby and Woollard (2013), this study moved a step further by investigating how each pair of corresponding dimensional scores of the two instruments was correlated. The results showed that they were positively and weakly correlated to each other (with low-level correlations). Compared to the stronger (middle-level) correlations reported in prior literature for two CT outcome assessments, it is reasonable to have lower correlations between two assessments from two different categories than those from the same category.

Relationship between CT Disposition and CT Competency

Several significant findings have been obtained in this study regarding examining the relationships between CT disposition (via CTS) and CT competency (via CTT-ES). First, mapped correlations were found between each pair of CTS and CTT-ES sub-scores, suggesting that students’ CT disposition and CT competency are associated correspondingly in each dimension. Similarly, students’ CT disposition factors are almost interrelated with CT competency factors, for example, the Abstraction in CTS was also correlated with the Generalization in CTT-ES. Such a finding may support the structural relationship suggested in DMCT (Tsai et al., in press) that the disposition of abstraction thinking is also useful for inducing the generalization performance. Therefore, a hierarchical multiple regression analysis was further performed based on the suggested model of DMCT.

Results of the hierarchical multiple regression analyses can be explained by the DMCT in two aspects. First, in current study, the abstraction and the decomposition were firstly drawn together as the two significant predictors in the final regression model. This can be explained by the first claim of the DMCT that abstraction and decomposition are the two fundamental elements of CT (Tsai et al., in press). Second, in the final regression model, following the drawn of abstraction and decomposition, significant predictors were drawn in an order from evaluation to generalization. The second claim of the DMCT suggests that there is a liner-order prediction relationship from algorithmic thinking, evaluation, to generalization (Tsai et al., in press). Thus, the finding is in parallel with the second claim, except the algorithmic thinking is not drawn in current study, which is discussed later in this paper. In sum, except for the missing of algorithmic thinking, all CT disposition factors were drawn into the final regression model in an order consistent to the structural model suggested in the DMCT. Therefore, theoretically, the two findings basically supported the DMCT regarding the developmental order of young students’ CT abilities. This study furthered our understanding about elementary school students’ CT development and the results can contribute to the build of CT theory.

One thing worth noting was that the algorithmic thinking disposition factor did not significantly predict the elementary school students’ CT competencies. It might be possible that elementary students may not have much experience relating to algorithm designs in school curricula. Future studies may replicate the current study for elementary school students and reexamine the model especially for the arithmetic thinking factor. Another interesting finding was the negative relationship shown in the regression model between the two dispositions: Abstraction thinking and Decomposition thinking. This finding could suggest that the cognitive processes of the two thinking are contradicted to each other. For example, the information processing of the two thinking could be processed in opposite directions. Abstraction may be processed in a bottom-up direction like induction, while Decomposition may be processed in a top-down direction like deduction. Future studies can further examine this hypothesis as well.

Moreover, results of the regression model may have some teaching implications and provide some feedbacks for education practices. The model suggested that abstraction and decomposition abilities may be the most important factors influencing the development of young children’s CT abilities. The developmental order of CT abilities should be considered in national curriculum reforms, especially the fundamental abilities (abstraction and decomposition) should be emphasized more in elementary school levels. In the daily-live problem-solving or non-programming contexts, enhancing elementary school students’ CT dispositions may benefit their CT performances. Regarding the designs of CT assessment tests, the findings could have provided some feedbacks for future CT assessment item designs. For example, it has been reported that more than 70% of the items in Bebras Challenge were associated with algorithm and procedures and less than 10% were classified as abstraction (Izu et al., 2017). In designing future assessment items, perhaps more attention should be paid to abstraction and decomposition.

Finally, the mapped relationships between CTS and CTT-ES provide another implication for practices and research. That is, CTS and CTT-ES can be used for different purposes at different stages of teaching. For example, when time is limited, teachers can use CTS as a proxy of the CTT-ES for a quick and pre-interventional evaluation to guide the design of learning activities. Then, CTT-ES can be used for more formal formative or summative evaluation for diagnosis. Future studies can further replicate the mapped relationship between the process-oriented CT (or formative assessment of CT) and the outcome-oriented CT (or summative assessment of CT) for learners in different school levels.

Conclusion

The present study designed and validated the Computational Thinking Test for Elementary School Students (CTT-ES) to measure students’ CT competencies (or skills) in a non-programming context. Item analysis results by Item Response Theory (IRT) and Classical Testing Theory showed that the CTT-ES had plausible difficulties and good reliability to measure young children’s CT performance. In addition, this study examined the convergent validity of the CTT-ES (CT skills transfer tools) with the CTS (CT perceptions–attitudes scales). The findings identified positive and weak correlations among all dimensions of the two instruments. Furthermore, hierarchical multiple regression analyses were conducted to investigate how the factors of the CTS predicted the performance scores of the CTT-ES. The results showed that students who usually tried to find the critical aspects of the problem (Abstraction), to search for effective solutions to the problem (Evaluation), and to apply a solution to different problems (Generalization) could perform better on the CTT-ES. This in turn suggested that students’ CT perceptions-attitudes (or dispositions) can predict their CT performances (or competencies). In sum, the CTT-ES can be applied to examine elementary school students’ computational thinking skills in a non-programming context. For computer educators or researchers, it can be used as a pretest or a posttest instrument to collect data regarding students’ computational thinking performance or competency in an instructional experiment. For elementary school teachers, it may be used to understand or diagnose students’ prior skills in computational thinking and learning progress in a computer literacy course or computer programming learning activities. Future studies are suggested to utilize the CTT-ES to track the development of young students’ CT skills in a non-programming context, to assess the relationships between young students’ CT performance and their academic performance, and to evaluate the impacts of various CT embedded curricula on young students’ CT performance. Finally, this current study only examines the relationship between young students’ CT disposition and their CT performance in a non-programming context. Future studies may also reexamine the relationships in a programming context or for students of differing ages.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was financially supported by the Ministry of Science and Technology in Taiwan under the following project numbers: MOST 106-2511-S-003-065-MY3, 108-2511-H-003-004-MY3, 109-2628-H-020-001-MY3, 109-2511-H-003-019-MY3, and 109-2511-H-003-052-MY3. It was also supported by the ‘Institute for Research Excellence in Learning Sciences’ of National Taiwan Normal University (NTNU) from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan.

ORCID iDs

Meng-Jung Tsai

Francis Pingfan Chien

Silvia Wen-Yu Lee

Chung-Yuan Hsu

Jyh-Chong Liang

Author Biographies

Meng-Jung Tsai is a Professor at the Program of Learning Sciences in National Taiwan Normal University. Her research interests center at computer literacy education and technology-enhanced science learning. Recently, she focused on topics including information searching, computational thinking, Robotics STEAM curricula and 21^stcentury skills, socio-scientific issues, and learning analytics with eye-tracking.

Francis Pingfan Chien is currently a research assistant at the Program of Learning Sciences in National Taiwan Normal University. He is interested in utilizing multiple approaches to understand human behavior and decipher the brain function of learning and information seeking. Recently, he is studying computational models to the mechanism of curiosity and the description of learning behaviors.

Silvia Wen-Yu Lee is a Professor at the Graduate Institute of Information and Computer Education in National Taiwan Normal University. Her research interests include epistemic beliefs in science, scientific modeling, and computer-supported learning.

Chung-Yuan Hsu holds a Ph. D. from Ohio University. He is a professor at Department of Child Care, National Pingtung University of Science and Technology, Taiwan. His research interests focus on digital game-based learning, TPACK, and computer-assisted science learning.

Jyh-Chong Liang is a Professor at the Program of Learning Sciences in National Taiwan Normal University. His research interests deal with scientific epistemological beliefs, conceptions of and approaches to learning science, TPACK, and Web information assessment.

References

Andrich

Marais

(2019). A course in Rasch measurement theory. Singapore: Springer.

Bond

T. G.

Fox

C. M.

(2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed). New York, NY: Routledge.

Brennan

Resnick

(2012). New frameworks for studying and assessing the development of computational thinking. Proceedings of the 2012 annual meeting of the American educational research association, Vancouver, Canada.

Carlson

K. D.

Herdman

A. O.

(2012). Understanding the impact of convergent validity on research results. Organizational Research Methods, 15(1), 17–32. https://doi.org/10.1177/1094428110392383.

Conger

A. J.

(1974). A revised definition for suppressor variables: A guide to their identification and interpretation. Educational and Psychological Measurement, 34(1), 35–46. https://doi.org/10.1177/001316447403400105.

Creswell

J. W.

(2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications.

Csizmadia

Curzon

Dorling

Humphreys

Selby

Woollard

(2015). Computational thinking: A guide for teachers. http://community.Computingatschool.org.uk/resources/2324.

Cutumisu

Adams

(2019). A Scoping Review of Empirical Research on Recent Computational Thinking Assessments. Journal of Science Education and Technology, 28(6), 651–676. https://doi.org/10.1007/s10956-019-09799-3.

Dagienė

Futschek

(2008). Bebras International Contest on Informatics and Computer Literacy: Criteria for Good Tasks. In Mittermeir

R.T.

Sysło

M.M.

(Eds), Informatics Education - Supporting Computational Thinking. ISSEP 2008 Lecture Notes in Computer Science (vol 5090, pp. 19–30). Berlin, Heidelberg: Springer.

10.

Dagienė

Sentance

Stupurienė

(2017). Developing a Two-Dimensional Categorization System for Educational Tasks in Informatics. Informatica, 28(1), 23–44. https://doi.org/10.15388/Informatica.2017.119.

11.

Hsu

T. C.

Chang

S. C.

Hung

Y. T.

(2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature [Review]. Computers & Education, 126, 296–310. https://doi.org/10.1016/j.compedu.2018.07.004.

12.

Hubwieser

Mühling

(2015). Investigating the psychometric structure of Bebras contest: Towards mesuring computational thinking skills [paper presentation]. 2015 international conference on learning and teaching in computing and engineering. Taipei: Taiwan. https://doi.org/10.1109/LaTiCE.2015.19.

13.

Irribarra

D.T.

Freund

(2014). Wright Map: IRT item-person map with ConQuest integration. http://github.com/david-ti/wrightmap.

14.

Izu

Mirolo

Settle

Mannila

Stupurienė

(2017). Exploring Bebras Tasks Content and Performance: A Multinational Study. Informatics in Education, 16(1), 39–59. https://dx-doi-org.web.bisu.edu.cn/10.15388/infedu.2017.03.

15.

Korkmaz

Ö.

Çakir

Özden

M. Y.

(2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72(C), 558–569. https://doi.org/10.1016/j.chb.2017.01.005.

16.

Mayer

Sodian

Koerber

Schwippert

(2014). Scientific reasoning in elementary school children: Assessment and relations with cognitive abilities. Learning and Instruction, 29, 43–55. https://doi.org/10.1016/j.learninstruc.2013.07.005.

17.

Ministry of Education (2009). Guidelines to mixed-ability grouping in primary and secondary school. https://law.moj.gov.tw/LawClass/LawAll.aspx?pcode=H0070026.

18.

Moreno-León

Robles

(2015). Dr. Scratch: A web tool to automatically evaluate Scratch projects. Proceedings of the workshop in primary and secondary computing education. London, United Kingdom. https://doi.org/10.1145/2818314.2818338.

19.

Pandey

Elliott

(2010). Suppressor variables in social work research: Ways to identify in multiple regression models. Journal of the Society for Social Work and Research, 1(1), 28–40. https://doi.org/10.5243/jsswr.2010.2.

20.

Rasch

(1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.

21.

Robitzsch

Kiefer

(2020). TAM: Test analysis modules. R package version 2.6-2. http://CRAN.R-project.org/package=TAM.

22.

Román-González

(2015). Computational thinking test: Design guidelines and content validation. Proceedings of the 7th annual international conference on education and new learning technologies. EDULEARN 2015 (pp. 2436–2444). https://doi.org/10.13140/RG.2.1.4203.4329.

23.

Román-González

Moreno-León

Robles

(2019). Combining Assessment Tools for a Comprehensive Evaluation of Computational Thinking Interventions. In Kong

S.-C.

Abelson

(Eds), Computational Thinking Education (pp. 79–98). Singapore: Springer.

24.

Selby

C. C.

Woollard

(2013). Computational thinking: The developing definition. Presented at the 18th annual conference on innovation and technology in computer science education. Canterbury.

25.

Tang

Yin

Lin

Hadad

Zhai

(2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798.

26.

Tsai

M.-J.

Wang

C.-Y.

Hsu

P.-F.

(2019). Developing the Computer Programming Self-Efficacy Scale (CPSES) for computer literacy education. Journal of Educational Computing Research, 56(8), 1345–1360. https://doi.org/10.1177/0735633117746747.

27.

Tsai

M.-J.

Liang

J.-C.

Hsu

C-Y.

(2021). The Computational Thinking Scale (CTS) for computer literacy education. Journal of Educational Computing Research, 59(4), 579–602. https://doi.org/10.1177/0735633120972356.

28.

Tsai

M.-J.

Liang

J.-C.

Lee

S. W.-Y.

Hsu

C.-Y.

(in press). Structural validation for the Developmental Model of Computational Thinking. Journal of Educational Computing Research. https://doi.org/10.1177/07356331211017794.

29.

van der Vegt

(2018). How hard will this task be? Developments in analyzing and predicting question difficulty in the Bebras Challenge. Olympiads in Informatics, 12, 119–132. https://dx-doi-org.web.bisu.edu.cn/10.15388/ioi.2018.10.

30.

Wing

(2006). Computational thinking. Communications of the ACM, 49(3), 33–36. https://dx-doi-org.web.bisu.edu.cn/10.1145/1118178.1118215.

31.

Wing

(2011). Research notebook: Computational thinking – what and why? Carneige Mellon.