A Log-Based Analysis of the Associations Between Creativity and Computational Thinking

Abstract

Creativity and Computational Thinking (CT) have been both extensively researched in recent years. However, the associations between them are still not fully understood despite their recognition as essential competencies for the digital age. This study looks to bridge this gap by examining the association between CT and two types of creativity, i.e., Creative Thinking and Computational Creativity. The research was conducted among 124 middle school students from Spain, who were divided into control and experimental groups; the intervention included an explicit encouragement to be as creative as possible (i.e., to submit multiple correct solutions) in a given learning task. Data were analyzed from a standardized creativity test (Torrance's TTCT) and cross-referenced with log files that documented the students' activities in the Kodetu game-based learning environment. Our research findings indicate some interesting associations between CT and Creativity. First, we found that creativity contributes to CT. Second, we found that CT is transferable across different domains. Finally, we found that Computational Creativity can develop and improve over time.

Keywords

creativity computational thinking computational creativity learning analytics log analysis

Technology and science have seen major developments in recent years alongside the constant growth in data and information. These trends compel students to acquire appropriate skills to help them meet the challenges of the future. Given these circumstances, the educational system is undergoing a change to support the acquisition of skills over content memorization (Scott, 2015). 21st-century skills include competencies such as problem-solving, creativity, and computational thinking (CT), which are considered essential to thrive in our technology-driven world (Bocconi et al., 2016; Deschryver & Yadav, 2015; Nouri et al., 2020). These skills are essential for all disciplines, and their integration is necessary to develop students' analytical and technological capabilities, and prepare them to be innovative problem solvers and critical thinkers (Deschryver & Yadav, 2015; Gretter & Yadav, 2016). The importance of these skills is a catalyst to the increasing implementation of different initiatives for imparting CT. Educational systems around the world recognize the need to nurture these skills from an early age (World Bank, 2019). One of the strategies to foster these skills is through programming and coding environments, as they enable high-order abilities such as logical thinking, critical thinking, and CT (Combéfis et al., 2016; Kafai & Burke, 2013). Strengthening coding skills is also associated with the notion that coding can enhance creativity (Eu Commission, 2019). The growing interest in promoting these skills and the environments that impart them have also accelerated the research in the field (Zhang & Nouri, 2019). However, there is a lack of sufficient research on the interrelationships between creativity and CT, and most analysis are based on qualitative methods and limited data volumes (Brennan & Resnick, 2012; Tang et al., 2020). Learning analytics methods make it possible to deeply explore and evaluate the learning environments and the learning processes involved (Krumm et al., 2018). Various studies in recent years have adopted these methods, and have shown their effectiveness in researching the learning processes and the acquisition of CT skills (Berland et al., 2013; Blikstein, 2011; Eguíluz et al., 2017; Hershkovitz et al., 2019). This study continues this line of research.

The purpose of this study is to examine the relationship between CT and creativity, and analyze how these skills are reflected in a game-based learning environment. Such an understanding may benefit various stakeholders who can better design the learning experiences to promote CT and creativity.

Computational Thinking

Computational Thinking (CT) is the conceptual basis required to define and solve real-world problems using algorithmic methods to reach transferable solutions; it is considered as a necessary skill in diverse contexts and multiple disciplines (Shute et al., 2017). CT comprises a large collection of mental strategies, such as recursive and procedural thinking, modeling, abstraction, decomposition, heuristic reasoning and parallelism (Wing, 2006, 2010). The term was first coined by Seymour Papert, who predicted that computational ideas can change the way children think in any field (Papert, 1980). Papert's vision was indeed realized, and it is now clear that CT is a universal skill that every child should acquire (Barr & Stephenson, 2011; Voogt et al., 2015). In the past, CT was considered to be primarily related to STEM fields, but today it is recognized as relevant to a broad spectrum of fields, including sciences, humanities and the arts (Gretter & Yadav, 2016; Kalelioğlu et al., 2016).

Large organizations, including the OECD and UNESCO, consider CT as a necessary literacy for tomorrow's citizens (Organisation for Economic Co-operation and Development, 2018; Scott, 2015; World Economic Forum, 2015). In 2012, the National Research Council (NRC) defined CT as one of the eight practices to be integrated into science studies (NRC, 2012). In 2016, a report was published by the European Commission, emphasizing that CT should be incorporated into compulsory education in order for students to fully participate in the digital world (Bocconi et al., 2016). Since then, many opinion papers, reports and initiatives dealing with the implementation of CT in the curricula have been published (International Society for Technology in Education & Computer Science Teachers Association [ISTE & CSTA], 2011; Kafai & Burke, 2013; Seow et al., 2019; World Bank, 2019).

Despite its vast popularity, there is little consensus regarding CT's operative definition and its evaluation strategies (Tang et al., 2020). CT is not narrowly focused on teaching and learning in a specific topic, but rather emphasizes the acquisition of extensive skills and knowledge that can be applied in a variety of contexts (Günbatar, 2019).

From a cognitive psychological point of view, CT's implementation in the curriculum can be seen from two perspectives: domain-general and domain-specific. A domain-general perspective sees CT as a universal skill that is not identified with a particular field of knowledge, while a domain-specific perspective sees CT as a skill related to a specific field, particularly computer science and programming (Lai, 2019). Referring to the latter point of view, the association between CT and programming is somewhat complex. On the one hand, coding is considered a fertile ground for learning and practicing CT (Buitrago Flórez et al., 2017; Lye & Koh, 2014; Román-González et al., 2017; Tsarava et al., 2017). On the other hand, some argue that CT represents thought processes used to solve problems that are not necessarily translated into codes (Tedre & Denning, 2016). Therefore it is important to separate the common association between CT and programming (Hu, 2011). Of course, the domain-specific perspective may be important in a broader context, considering that a skill that is acquired in a particular context may be transferrable, hence may affect other contexts as well (Perkins & Salomon, 1992).

Over the past decade, many online learning environments have been developed to facilitate and foster CT concepts acquisition. Among these environments are many user-friendly, game-based environments designed for children and adolescents (Kim & Ko, 2017). Many such environments are taking a constructivist approach, that is, promoting learning via an active process of the learner, and constructing knowledge by experimenting and linking existing knowledge with new knowledge (Fosnot & Perry, 1996; Jonassen & Ronrer-Murphy, 1999; Piaget, 1964). The constructionist approach, which extends the constructivist approach, also underlies some of the environments for acquiring CT (Csizmadia et al., 2019; Kesler et al., 2019). According to this approach, learning is a product of knowledge construction by using existing knowledge and hands-on experience, while creating meaningful, concrete artifacts. Some of the learning environments have embraced game design principles, creating highly interactive learning experiences that increase student motivation and improve learning outcomes (Ibáñez et al., 2014; Kazimoglu et al., 2012; Vu & Feinstein, 2017). These environments are usually built around linearly progressing challenges, that facilitate knowledge construction by breaking down the concepts into incremental learning components. They also support trial and error behavior for improving knowledge acquisition (Wang & Chen, 2010). Students can learn through practical experience, and gradually link the theoretical computing concepts to their ability to solve problems (Webb et al., 2018). The log files documenting the actions of the participants in these environments enable the collection and analysis of the processes involved in acquiring CT concepts. Previously, we have used the log files drawn from this platform to analyze students' persistence throughout the CT acquisition process, and to explore associations between creativity and CT (Israel-Fishelson et al., 2020; Israel-Fishelson & Hershkovitz, 2020).

Creativity

Creativity is a thinking ability that enables people to solve problems innovatively and to produce original and valuable products (Torrance, 1974). In the past, creativity was primarily thought of in the contexts of design and art. Still, today it is recognized for its importance for human inventive potential in all disciplines, and its impact is evident in various spheres of life (Donovan et al., 2014; Navarrete, 2013). In recent years creativity has been recognized as an essential skill for the 21st century (Said-Metwaly et al., 2017) that can be nurtured and should be included in curricula from an early age (Beghetto, 2010; Vygotsky, 2004).

There are a great many conceptualizations of the term “creativity”, but there is an overall agreement that this is a multidimensional construct comprising four dimensions, as suggested by Elis Paul Torrance, which are fluency, flexibility, originality, and elaboration (Torrance, 1965). Broadly speaking, these dimensions refer, respectively, to the production of many ideas of different types, which are new and can be articulated in detail. Creativity has been studied extensively over the years from various perspectives (Kaufman & Beghetto, 2009; Runco & Jaeger, 2012), among which are the thinking processes involved in this process; qualities of creative people; and characteristics of creative products. For a long time, a common belief among various scholars has held that creativity is a personal trait, linked to cognitive skills and characteristics such as IQ, motivation, and divergent thinking (Batey & Furnham, 2006). According to this perception, people have different levels of creativity, and one person may be more creative than another (Amabile & Pillemer, 2012). Today, an alternative theory suggests that creativity is influenced by different forces, some of which are related to personality and others to context (Amabile & Pillemer, 2012; Reiter-Palmon et al., 2009). Furthermore, some believe that creativity is not a permanent trait or a congenital phenomenon, but a skill that can be taught, practiced and improved (Hsiao et al., 2014).

The question of whether creativity is transferable, i.e., whether it is domain-general or domain-specific, has been the topic of an extensive discussion (Plucker & Beghetto, 2002). The perception of creativity as domain-general assumes that a person who demonstrates creativity in one area is more likely to show creativity in other areas. In contrast, the perception of creativity as domain-specific assumes that creativity in one domain is not necessarily related to creativity in another domain (Plucker & Beghetto, 2002; Said-Metwaly et al., 2017). The answer to this question is still open and unresolved, with some scholars suggesting that creativity is both domain-general and domain-specific (Baer, 2010; Hong & Milgram, 2010). This issue, therefore, has a direct impact on both the measurement of creativity and on its promotion. This research assumes that creativity may be, to some extent, transferable, and that exercising creativity in one domain may improve creativity in another (Stolaki & Economides, 2018; K. Wang & Nickerson, 2017), and therefore it is of great importance to cultivate and promote creativity. Indeed, our previous study supports this notion, as it had shown positive associations between different measures of creativity, outside and inside the CT acquisition process (Israel-Fishelson et al., 2020). Our current research design, as detailed below, aims at further testing this assumption and learn how creativity is expressed in the acquisition of CT.

Creativity and CT

Four decades ago, Papert (1980) emphasized the potential for developing creativity using computers. However, only in recent years has creativity been accepted as directly related to computer science, and its central role in fostering motivation and interest in this field of study has been acknowledged (Romeike, 2007). Different studies have demonstrated the mutual contribution of creativity and computer science, and CT in particular, on each other (Kong, 2019; Miller et al., 2013; Pérez Poch et al., 2016; Resnick, 2006; Seo & Kim, 2016). Creativity and CT are not to be considered as mutually exclusive, as both refer to knowledge-construction processes, and creativity involves a set of thinking tools that overlap with the fundamentals of CT, specifically, observation, imagination and visualization, abstraction, and creation and identification of patterns (Yadav & Cooper, 2017). Indeed, creativity is part of the International Society for Technology in Education and the Computer Science Teachers Association's definition of CT, as they refer to it as a reflection of logical and algorithmic thinking, problem-solving skill, and creative thinking (ISTE & CSTA, 2011). The role of creativity in CT acquisition is not limited to the construction of creative products, but is also manifested in finding new ways of thinking when facing problems that need to be solved (Dagiene et al., 2019).

In recent years, various educational initiatives worldwide have begun to establish national K-12 curricula, academic standards, and instructional activities that entwine these two skills together. Educational institutions have started to use digital learning environments that promote CT while allowing the expression and development of creative thinking (Kong, 2019; Roque et al., 2016; World Bank, 2019). Often, this requires adapting learning environments so that they will instill the main CT concepts in a logical progression, while simultaneously help in developing creative thinking (Dagiene et al., 2019).

Studies have examined the relationship between creativity and CT from different perspectives, including mostly examining creativity within the realm of CT and exploring the impact of these structures on each other (Miller et al., 2013; Seo & Kim, 2016). However, only a few studies focused on the association between these two perspectives. Our previous, seminal study had shown that creativity may contribute to CT acquisition (Israel-Fishelson et al., 2020); however, further research is needed to shed light on the associations between these two constructs. Therefore, the main goal of the current study is to explore the associations between creativity and CT among sixth-grade students in the context of a game-based learning environment for CT acquisition. The study refers to two types of creativity: creative thinking, defined as creativity as reflected by traditional measures of creativity, and computational creativity, a measure of how creativity is displayed in solutions within the learning environment.

Research Questions

We formulated the following research question to help meet our research goals:

How do the personal characteristics of students who were explicitly encouraged to think creatively and those who were not, differ when comparing their Creative Thinking, Computational Thinking, and Computational Creativity?

What are the associations between Creative Thinking, Computational Thinking, and Computational Creativity, among students who were explicitly encouraged to think creatively and those who were not?

What are the differences in students’ personal characteristics between those who submitted multiple solutions, and those who only managed to submit a single solution, when comparing their Creative Thinking, Computational Creativity, and Computational Thinking?

How do the multiple solutions submitted differ from each other? How are these differences associated with students' personal characteristics?

Methods

Learning Analytics

Learning analytics (LA) is “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Ferguson, 2012, p. 2). LA usually uses visualization, statistical, and machine learning techniques to assess and improve the understanding of learning processes and learning platforms involved in these processes (Krumm et al., 2018). Such methods are useful for predicting students' success (Emerson et al., 2019), detecting difficulties while acquiring CT concepts (Román-González et al., 2019), and evaluating the acquisition of CT concepts by aggregating students' achievements in various learning tasks (Kong, 2019).

In the context of computer science education, LA approaches have been previously used to study students' programming activities (Berland et al., 2013; Blikstein, 2011; Boutnaru & Hershkovitz, 2015; Eguíluz et al., 2017; Gal et al., 2017; Grover et al., 2017; Hershkovitz et al., 2019; Lu et al., 2017; Nutbrown & Higgins, 2016). These studies are often based on data that is automatically collected via an online learning platform, and objectively reflects the learning process; this data is stored in the learning systems' log files, which document each action taken in the system. The current study follows this line of research by applying LA methods to measure CT and computational creativity.

The Learning Environment: Kodetu

The Kodetu online game-based environment is used to implement basic CT concepts. It is targeted at elementary and middle-school students, and is based on the block-based mechanism of Google Blockly (Eguíluz et al., 2017). The platform has three official games, as well as the ability for users to create their own games. Each game consists of several levels. At each level, the user is required to solve a maze-based challenge and lead an astronaut to a marked target point. The user determines the astronaut's displacement using visual coding blocks. Progression to the next level is enabled when the current level is completed successfully. The user can reset the level and solve it again. Kodetu is available in three languages: English, Spanish, and Basque. During use, the system logs any action taken by its users, i.e., the user ID, the coding blocks being used, the solution correctness, and the action timestamp.

The Kodetu platform has already been used in several CT-related studies (Eguíluz et al., 2017, 2018; Saavedra-Sánchez et al., 2019). Below are the reasons for choosing this environment for the benefit of our research. First, Kodetu is based on block-based programming, and is therefore suitable for younger students with no prior experience in programming. Moreover, since the platform leads learners on a path along which they are gradually introduced to new concepts, it serves as a good platform for assessing CT while learning it. Second, it can be easily adjusted to fit different research objectives and questions. For example, we can control the number of challenges given to students, the nature of the challenges, and their order, as well as the feedback messages they receive. Indeed, we did so, as described below. Finally, this platform offers easy access to the system log files, as the software was developed for research purposes by some of the authors (third, fourth, and fifth). Platforms such as Kodetu allow for a portfolio-driven approach to assess CT, that allows for measuring its acquisition at multiple points in time throughout the learning process (Tang et al., 2020).

For the current study, a dedicated game was created in the Kodetu platform. The game, which has two versions, which we will refer to as “Kodetu Single” and “Kodetu Multi”, includes ten levels with increasing difficulty. These levels cover some CT concepts such as sequences, loops, and conditionals, as detailed below. The first four levels are designed to enable users to practice the concept of sequences. Level 1 is a trivial level to show how the platform works. Levels 2 and 3 involve turns and perspective. Level 4 presents a large maze in which a long sequence of actions, including more than one rotation, is needed to reach the goal. Level 5 limits the number of blocks that can be used (i.e., code length) to prevent participants from using long sequences and to encourage them to take advantage of new code structures of loops. Level 6 introduces loops in a trivial challenge. Level 7 also works on sequences and loops with a limitation on block usage. Level 8 limits the number of blocks that can be used (i.e., code length) to prevent participants from using long sequences, and to encourage them to take advantage of new code structures for conditionals. Level 9 introduces if-else conditionals, and requires nested structures and a limited number of blocks. Level 10 (shown in Figure 1) works on loops and conditional statements, while presenting a general maze with some distracting elements.

Figure 1.

An Example Level of the Kodetu Game Used in This Study (Level 10).

Both versions of the game include the same set of levels, with only one difference in the system response for correctly solving Level 10. In “Kodetu Multi”, once a student is submitting a correct solution to Level 10, they have the option to submit another solution by clicking a “Repeat Level” button.

Participants and Procedure

124 sixth-grade students (ages 11–12 y/o) from two different schools in northern Spain participated in this study, that took place during June 2019. The participating students went to an outreach workshop about technology, programming, and robotics, in which they participated. The willingness to participate in the workshop as well as in the research was the criterion for choosing these schools. The workshop was organized by the Faculty of Engineering at the University of Deusto. During that workshop, the students played a designated Kodetu game for about 60 minutes. Included in the current analysis are only those students who progressed in the game until at least Level 9.

Students were divided into experimental and control groups. The division was based on the order of their group’s arrival at the workshop. Prior to their Kodetu session, all participants took a pen-and-paper creativity test (Torrance's TTCT – Figural Test) and filled-up a short online questionnaire that collected background information. Following that, students played Kodetu, either in the “Single” mode (control group) or in the “Multi” mode (experimental group).

The control group included 43 students (23 girls and 20 boys); the experimental group included 81 students (36 girls and 44 boys). For the vast majority of the students, this was their first encounter with programming experience (79% of the control group and 80% of the experimental group). Additionally, 44% of the students in the control group and 60% of the experimental group reported having a high affinity for technology. No statistically significant differences were found between the groups with respect to their personal characteristics. The results are summarized in Table 1.

Table 1.

Demographics and Personal Characteristics of the Participants.

Group	N	Gender	Previous coding knowledge	Affinity for technology
Control	43	F: 20, M: 23	No: 34, Yes: 9	Low: 24, High: 19
Experimental	81	F: 36, M: 44	No: 65, Yes: 16	Low: 32, High: 49
Chi-square Value		0.03, p = 0.87	0.03, p = 0.88	3.02, p = 0.08

Data and Analysis

Our dataset, drawn from the learning environment's log files, contained 21,340 rows, each representing an action taken by a user, including the action's timestamp, the level at which it was taken, its result [Success, Failure, Timeout, Error], and the code associated with this action. All statistical analyses were conducted using IBM SPSS version 26.

Different research questions required defining different sets of relevant participants. For the first and second research questions, which are focused on analyzing students' behavior in the two research groups while playing levels 1–9 — which were identical for both groups — we compared the distribution of the dependent variables between the control group (N = 43) and the experimental group (N = 81), taking into consideration activity in levels 1-9 only.

For answering the third research question, which is focused on identifying the differences between those students who submitted multiple solutions (N = 11) in Level 10, to those who only submitted a single solution (N = 47), we partitioned the experimental group into two respective sub-groups, including in the former, students who at least tried to submit another solution after submitting a correct one.

For answering the fourth research question, which is focused on comparing the multiple solutions submitted, we only considered those students from the experimental group who submitted multiple correct solutions in Level 10 (N = 9).

Data from the Kodetu log files were cross-referenced with the data obtained via the creativity task, by using a unique ID assigned to each participant. The participants wrote down their Kodetu-generated ID on the creativity test form.

Instruments: Creative Thinking Test

The Torrance Test for Creative Thinking (TTCT) was used for assessing creativity. TTCT is considered one of the popular tests for such assessments. The TTCT is based on Gilford's theoretical model of divergent thinking, and examines Creative Thinking along four dimensions: fluency, flexibility, originality, and elaboration (Torrance, 1974). Alongside the figural test, the TTCT also offers a verbal test. The validity and reliability of both tests have been repeatedly proven (Cramond et al., 2005; K. H. Kim, 2011). However, we found the figural test more appropriate for this study. First, the tasks involved in the studied platform were mostly visual, both in terms of the puzzle presented to students and in terms of the blocks they used to build their code. Second, conceptual problem-solving of this type involves more graphic thinking than literal thinking (Liu & Lu, 2002). Furthermore, a recent analysis of both figural and verbal versions of the TTCT showed that, while the scores on the two versions are highly associated, the figural version is a more comprehensive, reliable, and valid measure of creativity (K. H. Kim, 2017). The TTCT – Figural Test has been previously used successfully for studying associations with creativity in the context of programming or CT (Liu & Lu, 2002; Seo & Kim, 2016).

To perform the TTCT we used a pen-and-paper test, where students received a sheet of paper with 12 identical empty circles. They were asked to draw as many drawings as possible, with the circles being an integral part of the drawing. An eligible drawing used the circle as part of the drawing. See examples in Figure 2.

Figure 2.

Example of Eligible (top row) and Non-eligible (bottom row) Drawings from TTCT—Figural Test.

Research Variables

Demographics and Personal Characteristics

The participants self-reported their personal characteristics at the beginning of the session, using a short online questionnaire that includes the following variables:

Gender [M/F]

Previous Programming Background [Yes/No]

Affinity for Technology [1-10 Likert scale, with 1 being “Low”, and 10 being “High”]

Computational Thinking

Three variables were used to measure student-level acquisition of CT, each computed first for each level separately, and then averaged across levels:

Total Solution Attempts – counting all solutions attempts, including correct and incorrect ones.

Correct Solution Attempts – counting only correct solution attempts; note that despite being able to progress to the next level upon submitting a correct solution, users can — and sometimes indeed do — re-try to solve the current challenge.

Completion Time [min] – calculated as the difference between the time of loading the level and the time of moving on to the next level.

Creative Thinking

To score the pen-and-paper TTCT figural test, we used eligible drawings only, that is, only drawings in which a circle was considered an integral part of the drawing. To ensure the reliability of eligibility determination, each of the first two authors separately coded 20 sheets for eligibility. We then ran an inter-rater reliability assessment using Cohen’s Kappa, which yielded a satisfactory coefficient of 0.81. The authors then discussed borderline cases and agreed on guidelines for the rest of the coding, which was done by the first author.

Originality and flexibility were based on drawing categories. To this end, we needed to classify the full pool of drawings into categories. The coding of the first 20 sheets was done independently by each of the first two authors. Following the coding, the authors discussed their results until they reached unanimous agreements on the results. The remaining pages were coded by the first author, with continuous interaction with the second author on the category definitions, splitting and mergers. This iterative process ended with a final list consisting of 59 categories (e.g., Emoji, Sun, Flower, Signpost, Animal). Examples of the various analyzed drawings can be seen in Figure 3 with such categories as Face (top), Ball (middle), and Animal (bottom).

Figure 3.

Examples of Drawings in Categories of Face (top), Ball (middle), and Animal (bottom).

Using a process similar to the one used in determining the categories, the first two authors rated each of the eligible drawings for its elaboration on a scale of 1 (Low Elaboration) – 6 (High Elaboration), dependent on the drawing’s detail level. Figure 4 exemplifies the coding process of six drawings in the “Pizza” category, with the drawing on the left rated as 1, i.e., low elaboration, and the drawing on the right is rated as 6, i.e., high elaboration.

Figure 4.

Examples of Six Levels of Elaboration as Were Evident in the: ‘Pizza’ Category, From Low (Left) to High (Right).

Four variables were computed for each student:

Fluency: number of eligible drawings;

Flexibility: number of different drawing categories;

Originality: frequency of drawing categories, averaged across all the student's drawings;

Elaboration: number of ideas/details used in each eligible drawing, averaged across all the student's drawings.

Computational Creativity

In our analysis of creative solutions, we analyzed 1591 rows representing correct solutions. We did not refer to other logged solution attempts. We used originality as a single proxy for creativity, as fluency, flexibility, and elaboration were not applicable; this is because the Kodetu platform, like many other platforms, does not promote participants to submit multiple solutions. Once a level has been completed, participants are directed to the next level.

The frequency of a solution among all correct solutions was used to represent Originality on a 0–1 scale. The average frequency of correct solutions was used in cases when a participant submitted several correct solutions.

This process is demonstrated for level 9 (see Figure 5). The astronaut’s expected path, according to the participant’s code, involves walking straight, turning left, and several subsequent straight walks. Denoting “D” for a do-while loop, “I” for if condition, “E” for the else part of the condition, “F” for going forward, “L” for turning left. The common solution was: DIFFEL (Do: if the path forward, then forward, else turn left). This solution was submitted in 71% of the cases. However, there was another correct solution, DILLEF (submitted in 29% of the cases).

Figure 5.

Initial Setting in Level 9.

Results

Comparing Between the Experimental and the Control Groups

In order to better understand whether and how Computational Thinking, Creative Thinking, and Computational Creativity differ between the experimental group and the control group, we first report on descriptive statistics of each of the variables.

Computational Thinking, Creative Thinking, and Computational Creativity

When comparing the performance related to Computational Thinking, we found that the Completion Time was higher for the control group than for the experimental group, with t(122) = 2.41, at p < 0.001. This difference has a small-medium effect size of d = 0.45. That is, the students in the control group had progressed in the game slower than the students in the experimental group. No differences were found between the groups in Solution Attempts or Correct Solution Attempts.

When comparing the performance related to Creative Thinking and Computational Creativity, we found no statistically significant differences between the groups. The results are summarized in Table 2.

Table 2.

Descriptive Statistics for Computational Thinking, Creative Thinking and Computational Creativity.

	Control		Experimental		t(122)	p	Cohen’s d
	M	SD	M	SD	t(122)	p	Cohen’s d
Solution attempts	5.66	2.33	5.67	2.06	−0.03	0.98	0.00
Correct solution attempt	1.01	0.07	0.99	0.07	1.62	0.11	0.29
Completion time	2.94	0.78	2.6	0.73	2.41	<0.05	0.45
Fluency	7.40	3.35	7.67	3.77	−0.4	0.69	0.08
Flexibility	4.91	3.12	3.99	3.00	1.6	0.11	0.30
Originality	0.89	0.16	0.90	0.13	−0.45	0.65	0.07
Elaboration	2.85	0.68	3.07	0.92	−1.41	0.16	0.27
Computational creativity	0.26	0.07	0.25	0.07	0.39	0.7	0.14

Associations Between Computational Thinking and Creative Thinking

We tested for correlations between Computational Thinking and Creative Thinking. We found that for the control group, Fluency was moderately positively correlated with Correct Solution Attempts, with Spearman's ρ = 0.36, at p < 0.05. No further statistically significant correlations were found in the control group between Fluency and Computational Thinking, and no statistically significant correlations at all were found in the experimental group.

As for Flexibility and Originality, we found that in the control group they were statistically significantly negatively correlated with Solution Attempts, with ρ = −0.37 and ρ = −0.34, respectively, at p < 0.05. Additionally, these variables were also negatively correlated with Completion Time, with ρ = −0.31 and ρ = -0.35, respectively, at p < 0.05. These findings suggest that students in the control group who provided more varied and original drawings took less time and attempts to solve the game challenges.

Interestingly, we found that in the experimental group, Flexibility and Originality were statistically significantly negatively correlated with Correct Solution Attempts, with ρ = −0.36 and ρ = −0.35, respectively, at p < 0.01, and with average completion time, with ρ = −0.27 and ρ = −0.22, respectively, at p < 0.05. These findings imply that the more flexible and original the students in the experimental group were, the less time they needed to complete the game challenges and the fewer attempts they made to submit multiple correct solutions.

No statistically significant correlations were found between Elaboration and Computational Thinking variables in neither of the groups. The results are summarized in Table 3.

Table 3.

Correlations between Computational Thinking and Creative Thinking by Condition.

	Control			Experiment
	Solution attempts	Correct solution attempts	Average completion time	Solution attempts	Correct solution attempts	Average completion time
Fluency	ρ=−0.15p = 0.35	ρ = 0.36*	ρ=−0.01p = 0.95	ρ = 0.22p = 0.05	ρ−0.09p = 0.44	ρ = 0.14p = 0.2
Flexibility	ρ=−0.37*	ρ = 0.1p = 0.52	ρ=−0.31*	ρ = 0.12p = 0.31	ρ=−0.36^**	ρ=−0.35^**
Originality	ρ=−0.34*	ρ=−0.11p = 0.5	ρ=−0.33*	ρ = 0.02p = 0.87	ρ=−0.27*	ρ=−0.22*
Elaboration	ρ = 0.07p = 0.66	ρ=−0.27p = 0.07	ρ = 0.13p = 0.42	ρ=−0.18p = 0.11	ρ = 0.06p = 0.62	ρ=−0.06p = 0.62

*p < 0.05, **p < 0.01.

Associations Between Computational Thinking and Computational Creativity

Next, we examined the correlations between Computational Thinking and Computational Creativity. No statistically significant correlations were found between the variables for the control group. In contrast, we found that in the experimental group, Computational Creativity was statistically significantly correlated with all measures of computational thinking. Computational Creativity was negatively correlated with Solution Attempts and Completion Time, with ρ = -0.26 at p < 0.05 and ρ = −0.41, respectively, both at p < 0.01. A positive correlation was found between Computational Creativity and Correct Solution Attempts, with ρ = −0.26, at p < 0.05. These results indicate that the more creative the students from the experimental group were in producing a solution, the less effort it took them to complete the game challenges. Findings are summarized in Table 4.

Table 4.

Correlations Between Computational Creativity and the Three Computational Thinking Variables, by Condition.

	Solution attempts	Correct solution attempts	Average completion time
Control	ρ = 0.03p = 0.87	ρ = 0.03p = 0.86	ρ=−0.00p = 0.98
Experimental	ρ=−0.26*	ρ = 0.26*	ρ=−0.41^**

*p < 0.05, **p < 0.01.

Association Between Creative Thinking and Computational Creativity

Finally, we studied the correlations between the two creativity-related measures, that is, Computational Creativity and Creative Thinking. We found that in the experimental group, a statistically significant positive correlation existed between Originality and Computational Creativity, with ρ = 0.24, at p < 0.05 (see Table 5). This result indicates that the more original the students were in TTCT, the more creative they were in solving levels in the game. No correlations were found in the control group.

Table 5.

Correlations Between Creative Thinking and Computational Creativity by Condition.

	Fluency	Flexibility	Originality	Elaboration
Control	ρ = 0.15p = 0.34	ρ = 0.25p = 0.11	ρ = 0.19p = 0.23	ρ=−0.06p = 0.7
Experimental	ρ=−0.14p = 0.2	ρ=−0.00p = 0.99	ρ = 0.24*	ρ = 0.02p = 0.84

*p < 0.05.

Comparing Within the Experimental Group (Multiple vs. Single Participants)

Next, we examined the characteristics of those students from the Experimental group who submitted — or, at least, tried to submit — multiple correct solutions in level 10 (N = 11). We refer to this group as “Multi”. We compared their performance in the pre-test and in levels 1-9 with those who submitted only one correct solution (N = 47). We refer to the latter group as “Single”.

Due to the ordinal nature of the data and the small sample size, we used the nonparametric Mann-Whitney U-test (an alternative to the t-test for parametric data) to compare the characteristics of the two groups regarding the research variables. This test is suitable for examining small samples of subjects (Nachar, 2008). Based on the results of the Mann-Whitney U-test (all having p > 0.05), we assert that there are no statistically significant differences between the “Single” group and the “Multi” group. The results are summarized in Table 6.

Table 6.

Demographics, Personal Characteristics and Computational Thinking, Creative Thinking and Computational Creativity.

	Single		Multi		Mann–Whitney U	Z	p
	M	SD	M	SD	Mann–Whitney U	Z	p
Gender	1.59	0.5	1.64	0.5	240.5	−0.3	0.77
Previous coding knowledge	0.23	0.43	0.18	0.4	245	−0.37	0.71
Affinity for technology	0.57	0.5	0.64	0.5	242.5	−0.37	0.71
Solution attempts	5.21	1.81	6.52	2.23	163.5	−1.89	0.06
Correct solution attempts	1.02	0.04	1.00	0.00	220	−1.35	0.18
Average time	2.45	0.71	2.67	0.75	221	−0.74	0.46
Fluency	7.45	3.8	8.0	4.17	240.5	−0.36	0.72
Flexibility	3.47	2.71	4.27	3.85	257.5	−0.02	0.98
Originality	0.88	0.15	0.91	0.1	199.5	−1.18	0.24
Elaboration	2.98	0.98	3.49	0.9	183.5	−1.49	0.14
Computational creativity	0.27	0.07	0.24	0.04	212.5	−0.92	0.36

Table 7.

Correlations Between Computational Thinking and Creative Thinking by Condition.

	Single			Multi
	Solution attempts	Correct solution attempts	Average completion time	Solution attempts	Correct solution attempts	Average completion time
Fluency	ρ = 0.4^**	ρ=−0.14p = 0.35	ρ = 0.28p = 0.06	ρ=−0.28p = 0.41	−	ρ = 0.07p = 0.84
Flexibility	ρ=−0.19p = 0.2	ρ=−0.33*	ρ=−0.41^**	ρ=−0.63*	−	ρ=−0.87^**
Originality	ρ=−0.14p = 0.34	ρ=−0.28p = 0.06	ρ=−0.35*	ρ = 0.22p = 0.51	−	ρ=−0.06p = 0.87
Elaboration	ρ=−0.38^**	ρ = 0.05p = 0.74	ρ=−0.22p = 0.14	ρ = 0.1p = 0.77	−	ρ = 0.29p = 0.39

*p < 0.05, **p < 0.01.

Associations Between Computational Thinking and Creative Thinking

When examining the correlation between computational thinking and creative thinking, we found some notable findings. We found that in the “Single” group, a positive correlation was observed between Fluency and Solution Attempts, with ρ = 0.4 at p < 0.01. That is, the more fluent the students were in the standardized creativity test, the more solutions they provided in the game. Additionally, in this group, flexibility was negatively correlated with Correct Solution Attempts and Completion Time, with ρ = −0.33 at p < 0.05, and ρ = −0.41 at p < 0.01, respectively. These findings imply that the more flexible the students were, the less time they needed and the fewer correct solutions they submitted.

In the “Multi” group, we observed strong negative correlations between Flexibility, Solution Attempts, and Completion Time, with ρ = −0.63 at p < 0.05, and ρ = −0.87 at p < 0.01, respectively. That is, the more flexible students in the “Multi” group were in their drawings, the less effort it took them to solve the game challenges.

Moreover, a negative correlation was observed between Originality and Completion Time, with ρ = −0.35, at p < 0.05. For this group, we also found a negative correlation between Elaboration and Solution Attempts, with ρ = −0.38, at p < 0.01. That is, the more elaborated and original students were, the less attempts and less time it took them to complete the game challenges. The results are summarized in Table 7.

Associations Between Computational Thinking and Computational Creativity

Interestingly, we found that in the “Single” group, computational creativity was negatively correlated with all measures of computational thinking. That is, the more original the solutions were, fewer attempts were submitted by the students, they required less time, and their solutions were less correct. No statistically significant correlations were found for the “Multi” group. The results are summarized in Table 8.

Table 8.

Correlations Between Computational Thinking and Computational Creativity by Condition.

	Solution attempts	Correct solution attempts	Average completion time
Single	ρ=−0.32*	ρ=−0.4^**	ρ=−0.4^**
Multi	ρ = 0.03p = 0.94	–	ρ=−0.03p = 0.37

*p < 0.05, **p < 0.01.

Association Between Creative Thinking and Computational Creativity

We found that in the “Single” group, Fluency was negatively correlated with Computational Thinking, with ρ = −0.29, at p < 0.05. No statistically significant correlations were found for the “Multi” group. The results are summarized in Table 9.

Table 9.

Correlations Between Creative Thinking and Computational Creativity by Condition.

	Fluency	Flexibility	Originality	Elaboration
Single	ρ=−0.29*	ρ = 0.16p = 0.29	ρ = 0.27p = 0.07	ρ=−0.00p = 0.99
Multi	ρ = 0.12p = 0.72	ρ = 0.35p = 0.29	ρ = 0.36p = 0.28	ρ=−0.28p = 0.41

*p < 0.05, **p < 0.01.

Comparing Multiple Correct Solutions and Their Submitters (Experimental Group, Level 10)

Lastly, we focused only on the students from the experimental group who provided more than one correct solution in Level 10 (N = 9, of whom 4 are girls and 5 are boys). We tried to identify the differences between the submitted multiple solutions, and highlight some common personal characteristics of those who submitted them. Only one boy had previous coding knowledge, while, in contrast, only one girl had a low affinity for technology. Except for one girl who submitted three correct solutions, all the students submitted two correct solutions. Remarkably, we found that except for two cases, all these students were able to improve their Computational Creativity score, meaning that their second/third solution was more original than the previous one they have submitted. The average Computational Creativity score for the first correct solution was 0.68 (SD = 0.17), while the average for the second correct attempt was 0.92 (SD = 0.14). A Wilcoxon Signed-Ranks Test indicated that the second correct attempt rank was statistically higher than the first correct attempt rank, with p < 0.05. In total, 10 different correct solutions were found. The most common solution (found in 35% of the cases) was the shortest one. However, this solution only used sequencing commands, i.e., forward, turn-right, and turn-left: FFFRFFFRFFTRF. Most solutions only used these commands in different variations and sequences. Only two solutions used the combination of sequencing command with loop and conditions. An example of such a solution is illustrated in Figure 6. It should be noted that this solution is not the most effective in terms of code length. The most effective solution consists of only four blocks: DoWhile, If-Else, Forward, and TurnRight. No participant has submitted such code.

Figure 6.

An Example of a Solution Which Used Sequencing Commands, Loop, and Conditions.

It is also important to highlight that students submitted a relatively high number of attempts, ranging between 3 and 30. The average number of Total Attempts was 12.67 (SD = 8.89). Examining the sequence of solutions submitted indicates that, with the exception of one student who succeeded in the first attempt, all students failed before and between their correct solutions. The results are summarized in Table 10.

Table 10.

Characteristics of Those Who Submitted Multiple Solutions in Level 10.

User	Gender	Previous coding knowledge	Affinity for technology	Total attempts	Success attempts	Solution attempts	Total time	Computational creativity	Solutions’ trend
3932	Boy	Yes	High	3	2	S-F-S	4.90	0.98	→
9130	Girl	No	High	5	2	2F-S-S-F	2.95	0.72	↑
1z4m	Girl	No	Low	30	2	6F-S-22F-S	10.95	0.93	↑
nwm8	Boy	No	High	12	2	7F-S-3F-S	3.50	0.57	→
o4n8	Boy	No	High	18	2	7F-S-9F-S	7.48	0.99	↑
q79j	Boy	No	High	8	2	2F-S-4F-S	6.72	0.99	↑
wwk4	Boy	No	High	5	2	2F-S-F-S	4.52	0.78	↑
x2lq	Girl	No	High	21	3	5F-S-7F-S-6F-S	6.80	0.95	↑
ywmx	Girl	No	High	12	3	2F-S-2F-S-5F-S	7.55	0.81	↑

S=Success, F=Fail.

Discussion

In this study, we conducted a controlled experiment to examine middle-school students' personal characteristics and the association between CT, creative thinking, and computational creativity. These metrics were tested using a standard creativity test and student solutions in a dedicated game-based platform aimed at acquiring different CT concepts. Our intervention included an explicit request from the experimental group students to try to solve the last challenge of the game in as many different ways as possible; this request was made at the beginning of the session, that is, before starting using the game.

Our findings indicate that despite no statistically significant differences were found between the experimental and the control groups with respect to demographic and personal characteristics, creative thinking, and computational creativity, some differences were found with respect to CT acquirement. In particular, students in the experimental group needed less time to solve the game's challenges than students in the control group. It is possible that the students in the experimental group rushed through the challenges in order to reach level 10 where they were asked to submit multiple solutions. It was indeed found that giving students a task in a computer-mediated platform with a particular goal in mind accelerated students' performance (Schoeller, 2005)

We also found that there were similarities and differences between the experimental group and the control group, in the associations between creative thinking and CT. In both groups, we found statistically significant negative correlations between originality and flexibility and completion time. As students were more creative in the standardized creativity test, they required less time to solve the levels in the game. This is in line with previous studies which found associations between standardized creativity test and academic achievements (Anwar et al., 2012; Whalley & Ogier, 2020). Moreover, these findings provide the notion that creativity may contribute to computer science and CT in particular (Kong, 2019; Miller et al., 2013). Notably, we found that originality and flexibility were negatively correlated with solution attempts in the control group, while in the experimental group, these metrics were negatively correlated with correct solution attempts. We also found that in the control group, the higher the fluency score was, the more correct solutions were provided. These findings also reinforce the notion that creativity may be effective in acquiring CT, but this effect is most noticeable when there is no explicit pressure for being creative.

Our findings indicate some intriguing associations between all metrics of CT and computational creativity in the experimental group. The more creative the students were in producing a solution, the less effort it took them to solve the levels in the game. That is, creativity within the platform and CT acquirement are interconnected. This finding is consistent with previous studies that indicated the mutual contribution of creativity and CT, and that platforms for CT acquisition can develop and promote these skills simultaneously (Figueiredo & García-Peñalvo, 2017; Hershkovitz et al., 2019; Mishra & Yadav, 2013; Resnick, 2006). Additionally, we found that the more original the students in the experimental group were — as measured by a traditional pen-and-paper creativity test — the more creative they were in solving the levels of the game; this supports the notion that creative tasks can affect the expression of creative thinking in various domains, and may imply a “transfer of creativity” from one domain to another (Hong & Milgram, 2010; Liu & Lu, 2002).

When we compared those who submitted — or at least tried to submit — multiple correct solutions in level 10 (“Multi”) with those who submitted only one correct solution (“Single”), we found some remarkable findings. In the Single group, we found that creative thinking was negatively correlated with CT measures, excluding a positive correlation between fluency and solution attempts. As students were more flexible, they had fewer correct solutions, and as students were more elaborate, they had fewer attempts before achieving a correct solution. It is possible that elaboration hinders multiple solutions as students are more rigorous and immersed in perfecting their solutions. This assumption is consistent with the conclusions drawn from a study of 460 subjects who performed the Alternative Uses Test, whereby in time-limited situations, elaboration may hinder the production of original ideas (Dippo & Kudrowitz, 2015). Furthermore, we found that computational creativity was also negatively correlated with CT measures. It is apparent that these students found it difficult to provide creative solutions despite the amount of time they invested and the number of attempts they submitted. Their difficulty may be related to their lack of understanding of the concepts presented to them in the game. Indeed, it was found that creativity depends on expertise and familiarity with the content (Reilly, 2008).

As for the Multi group, we found that students who were more flexible required less time and fewer attempts in the game. Such association was found in the first phase, as detailed above. Moreover, since no statistically significant differences in group characteristics were found, it is possible that the students in the Multi group were able to provide multiple correct solutions because they understood the material better than the students in the Single group. This direction should be further investigated, as we plan to do in our future work.

An examination of the students who submitted more than one correct solution in level 10, revealed that the majority of these participants had a high affinity for technology, no previous coding experience, and performed more attempts in the game. As noted, there were no statistically significant differences in the characteristics of these students compared to the rest of the students in the experimental group. Indeed, previous studies have found that experience with technology usage does not necessarily predict CT acquisition (Durak & Saritepeci, 2018) or creativity (Jackson et al., 2012). Still, students were able to submit multiple correct solutions and even improve their creativity scores among the solutions. Therefore, it is possible that it was not their prior experience with technology that impacted their performance, but rather their tinkering — as is evident by the relatively high number of attempts — that has contributed to enhancing their creativity (Siemon et al., 2016). Moreover, tinkering was found as an effective strategy for learning programming, especially for novices. Importantly, the design of the learning environment and the immediate feedback that it suggests encourage this strategy (Berland et al., 2013). Therefore, the impact of the learning environment attributes on student performance should be further examined. Our findings show that most solutions that were submitted in that stage only incorporated basic commands, and although they were creative, they were not necessarily programmatically effective. It is possible that using more complex commands requires experience, which, as mentioned above, helps to develop creativity. Therefore, the relationship between code efficiency and creativity should be further examined.

As with the majority of studies, the design of the current study is subject to limitations. First, because we analyzed data from a single learning platform (Kodetu), it is possible that our findings were a result of some unique attributes of this platform (Saito et al., 2017). Furthermore, the analysis is based on students from a single country (Spain). Personal and cultural characteristics may affect how creativity is exhibited (Deng et al., 2016; Runco & Johnson, 2002; Zhou et al., 2013). It is possible that students from other origins and with different backgrounds will perform differently on such a platform. Therefore, we recommend replicating this study in other countries to offer a more international and multicultural view. Indeed, this is our plan.

Conclusions

This study advances the understanding of creativity and CT and enriches the limited knowledge base on computational creativity. Our findings suggest that creativity contributes to CT and is transferable across domains. Therefore, there is great importance of cultivating creativity while promoting CT. The findings also show that computational creativity can develop and improve over time, but at the same time, limitations can interfere with the expression of creativity. Many learning environments focus on code efficiency, sometimes at the expense of fostering creativity; original solutions which are longer than the desired solution will be perceived as less successful. However, longer solutions are not necessarily less effective than shorter ones, and might bring the user to the same successful outcome (Chao et al., 2014). Therefore, instructional designers need to inherently assimilate mechanisms to encourage creativity within learning platforms. Additionally, educators who want to better cultivate CT skills should encourage student creativity, whether by using dedicated learning platforms or by providing unplugged creative exercises.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rotem Israel-Fishelson

Arnon Hershkovitz

Author Biographies

Rotem Israel-Fishelson is a PhD candidate in the Department of Mathematics, Science, and Technology Education at Tel Aviv University's School of Education. Her main research interest is the study of computational thinking and creativity using learning analytics methods. She earned her MSc in Media Technology (2016) from Linnaeus University (Sweden) and her BA in Instructional Design (2012) from Holon Institute of Technology (Israel). In addition, Rotem is a lecturer in the Faculty of Instructional Technologies at the Holon Institute of Technology.

Arnon Hershkovitz is a senior lecturer at Tel Aviv University’s School of Education (Israel). He earned his PhD in Science Education from Tel Aviv University, Israel, in 2011, after completing his MA in Applied Mathematics (2003) and BA in Mathematics and Computer Science (1996) at the Technion – Israel Institute of Technology. He was a post-doctoral research associate in the Department of Human Development at Teachers College Columbia University (NYC), and had spent a visiting professor fellowship at Northwestern University’s School of Education and Social Policy (Evanston, IL). He has published about 60 scientific papers in peer-reviewed journals, conferences, and books. He had served as a co-editor of Technology, Instruction, Cognition and Learning (TICL) and as a co-editor of the Journal of Learning Analytics (JLA), and has served as chair and reviewer for many peer-reviewed conferences. He is an alumni of the European Commission's Marie Curie Career Integration Grant, and is currently a co-PI in two projects funded by the Israeli government. His main research interests are the application of learning analytics methods to studying the skills that learners and instructors require in today's digital age.

Andoni Eguíluz is a researcher and lecturer in the Computer Engineering Department, University of Deusto (Spain). He is a fellow of the Advanced Study Program of the Massachusetts Institute of Technology (USA), and a member of the Deusto Learning Lab research group. His main research interests are education, games, multimedia, audiovisual and web accessibility, technology and innovation. Prior to his academic career, he was a technological entrepreneur, and was director and researcher on a number of software projects involving games, serious games, accessibility, multimedia, web, 3D, user interaction, graphic interfaces, compiling environments, compiler generators, and more.

Pablo Garaizar is a professor in the Faculty of Engineering, at the University of Deusto (Spain). He is a member of the Deusto Learning Lab research group. He holds a BSc in Computer Engineering (University of Deusto), a BA in Psychology (UNED), an MEng in Networks and Systems Administration (University of Deusto), and a PhD in Computer Engineering (University of Deusto). He is interested in finding new ways of using information and communication technologies to improve learning, and is also interested in web programming, computer security, telematic networks and systems administration.

Mariluz Guenaga is a professor in the Faculty of Engineering at the University of Deusto (Spain), and serves as the director of the Deusto Learning Lab research group. She is the Coordinator of the Spanish Network of Learning Analytics (SNOLA) and a senior member of the IEEE. She obtained a PhD in Computer Engineering from the University of Deusto (Spain). Her research interests focus on the innovative use of technology in STEM education, learning analytics, and game-based learning.

References

Amabile

T. M.

Pillemer

(2012). Perspectives on the social psychology of creativity. The Journal of Creative Behavior, 46(1), 3–15. https://doi.org/10.1002/jocb.001

Anwar

M. N.

Aness

Khizar

Naseer

Muhammad

(2012). Relationship of creative thinking with the academic achievements of secondary school students. International Interdisciplinary Journal of Education, 1(3), 1–4.

Baer

(2010). Is creativity domain specific? In Kaufman

J. C.

Sternberg

R. J.

(Eds.), The Cambridge handbook of creativity (pp. 321–341). Cambridge University Press.

Barr

Stephenson

(2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community? ACM Inroads, 2(1), 48–54.

Batey

Furnham

(2006). Creativity, intelligence, and personality: A critical review of the scattered literature. Genetic, Social, and General Psychology Monographs, 132(4), 355–429. https://doi.org/10.3200/MONO.132.4.355-430

Beghetto

R. A.

(2010). Creativity in the classroom. In Kaufman

J. C.

Sternberg

R. J.

(Eds.), The Cambridge handbook of creativity (pp. 447–463). Cambridge University Press.

Berland

Martin

Benton

Petrick Smith

Davis

(2013). Using learning analytics to understand the learning pathways of novice programmers. Journal of the Learning Sciences, 22(4), 564–599.

Blikstein

(2011). Using learning analytics to assess students’ behavior in open-ended programming tasks. In Long

Siemens

Conole

Gasevic

(Eds.), Proceedings of the 1st International Conference on Learning Analytics and Knowledge—LAK ’11 (pp. 110–116). ACM Press.

Bocconi

Chioccariello

Dettori

Ferrari

Engelhardt

Kampylis

Punie

(2016). Developing computational thinking in compulsory education. European Commission, JRC Science for Policy Report.

10.

Boutnaru

Hershkovitz

(2015). Software quality and security in teachers’ and students’ codes when learning a new programming language. Interdisciplinary Journal of E-Skills and Lifelong Learning, 11, 123–148.

11.

Brennan

Resnick

(2012). New frameworks for studying and assessing the development of computational thinking. In Tyson

C. A.

Ball

A. F.

(Eds.), Proceedings of the 2012 annual meeting of the American educational research association (pp. 1–25). American Educational Research Association.

12.

Buitrago Flórez

Casallas

Hernández

Reyes

Restrepo

Danies

(2017). Changing a generation’s way of thinking: Teaching computational thinking through programming. Review of Educational Research, 87(4), 834–860.

13.

Chao

J.-Y.

Liu

C.-H.

Chen

J.-Y.

(2014). The influence of courses integrating atayal culture and LEGO Dacta on the programming ability and creativity of aboriginal children. Global Journal of Computers & Technology, 1(2), 34–43.

14.

Combéfis

Beresnevičius

Dagienė

, & Electronics and IT Unit, École Centrale des Arts et Métiers. (2016). Learning programming through games and contests: Overview, characterisation and discussion. Olympiads in Informatics, 10(1), 39–60. https://doi.org/10.15388/ioi.2016.03

15.

Cramond

Matthews-Morgan

Bandalos

Zuo

(2005). A report on the 40-year follow-up of the Torrance tests of creative thinking: Alive and well in the new millennium. Gifted Child Quarterly, 49(4), 283–291. https://doi.org/10.1177/001698620504900402

16.

Csizmadia

Standl

Waite

(2019). Integrating the constructionist learning theory with computational thinking classroom activities. Informatics in Education, 18(1), 41–67. https://doi.org/10.15388/infedu.2019.03

17.

Dagiene

Futschek

Stupuriene

(2019). Creativity in solving short tasks for learning computational thinking. Constructivist Foundations, 14(3), 382–396.

18.

Deng

Wang

Zhao

(2016). How creativity was affected by environmental factors and individual characteristics: A cross-cultural comparison perspective. Creativity Research Journal, 28(3), 357–366. https://doi.org/10.1080/10400419.2016.1195615

19.

Deschryver

M. D.

Yadav

(2015). Creative and computational thinking in the context of new literacies: Working with teachers to scaffold complex technology-mediated approaches to teaching and learning. Journal of Technology and Teacher Education, 23(3), 411–431.

20.

Dippo

Kudrowitz

(2015). The effects of elaboration in creativity tests as it pertains to overall scores and how it might prevent a person from thinking of creative ideas during the early stages of brainstorming and idea generation [Paper presentation]. The 27th International Conference on Design Theory and Methodology. American Society of Mechanical Engineers. https://doi.org/10.1115/DETC2015-46789

21.

Donovan

Green

T. D.

Mason

(2014). Examining the 21st century classroom: Developing an innovation configuration map. Journal of Educational Computing Research, 50(2), 161–178. https://doi.org/10.2190/EC.50.2.a

22.

Durak

H. Y.

Saritepeci

(2018). Analysis of the relation between computational thinking skills and various variables with the structural equation model. Computers and Education, 116, 191–202. https://doi.org/10.1016/j.compedu.2017.09.004

23.

Eguíluz

Garaizar

Guenaga

(2018). An evaluation of open digital gaming platforms for developing computational thinking skills. In Cvetković

(Ed.), Simulation and gaming (pp. 143–168). InTech. https://doi.org/10.5772/intechopen.71339

24.

Eguíluz

Guenaga

Garaizar

Olivares-Rodríguez

(2017). Exploring the progression of early programmers in a set of computational thinking challenges via clickstream analysis. IEEE Transactions on Emerging Topics in Computing, 8(1), 256–261. https://doi.org/10.1109/TETC.2017.2768550

25.

Emerson

Smith

Rodríguez

Wiebe

Mott

Boyer

Lester

(2019). Predicting early and often: Predictive student modeling for block-based programming environments. In Lynch

C. F.

Merceron

Desmarais

Nkambou

(Eds.), Proceedings of the 12th international conference on educational data mining (pp. 39–48). International Educational Data Mining Society.

26.

Eu Commission

D. S. M.

(2019). Coding—The 21st century skill. https://ec.europa.eu/digital-single-market/en/coding-21st-century-skill

27.

Ferguson

(2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. https://doi.org/10.1504/IJTEL.2012.051816

28.

Figueiredo

García-Peñalvo

F. J.

(2017). Improving computational thinking using follow and give instructions [Paper presentation]. 5th International Conference on Technological Ecosystems for Enhancing Multiculturality—TEEM 2017, Part F1322 (pp. 1–7). ACM. https://doi.org/10.1145/3144826.3145351

29.

Fosnot

C. T.

Perry

R. S.

(1996). Constructivism: A psychological theory of learning. Constructivism: Theory, Perspectives, and Practice, 2, 8–33.

30.

Gal

Hershkovitz

Morán

A. E.

Guenaga

Garaizar

(2017). Suggesting a log-based creativity measurement for online programming learning environment [Paper presentation]. Fourth (2017) ACM Conference on Learning @ Scale—L@S ’17 (pp. 273–277). ACM Press. https://doi.org/10.1145/3051457.3054003

31.

Gretter

Yadav

(2016). Computational thinking and media and information literacy: An integrated approach to teaching twenty-first century skills. TechTrends, 60(5), 510–516. https://doi.org/10.1007/s11528-016-0098-4

32.

Grover

Basu

Bienkowski

Eagle

Diana

Stamper

(2017). A framework for using hypothesis-driven approaches to support data-driven learning analytics in measuring computational thinking in block-based programming environments. ACM Transactions on Computing Education, 17(3), 1–25.

33.

Günbatar

M. S.

(2019). Computational thinking within the context of professional life: Change in CT skill from the viewpoint of teachers. Education and Information Technologies, 24(5), 2629–2652. https://doi.org/10.1007/s10639-019-09919-x

34.

Hershkovitz

Sitman

Israel-Fishelson

Eguíluz

Garaizar

Guenaga

(2019). Creativity in the acquisition of computational thinking. Interactive Learning Environments, 27(5-6), 628–644. https://doi.org/10.1080/10494820.2019.1610451

35.

Hong

Milgram

R. M.

(2010). Creative thinking ability: Domain generality and specificity. Creativity Research Journal, 22(3), 272–287. https://doi.org/10.1080/10400419.2010.503535

36.

Hsiao

H. S.

Chang

C. S.

Lin

C. Y.

P. M.

(2014). Development of children’s creativity and manual skills within digital game-based learning environment. Journal of Computer Assisted Learning, 30(4), 377–395. https://doi.org/10.1111/jcal.12057

37.

(2011). Computational thinking—What it might mean and what we might do about it [Paper presentation]. ITiCSE’11—16th Annual Conference on Innovation and Technology in Computer Science (pp. 223–227). ACM Press. https://doi.org/10.1145/1999747.1999811

38.

Ibáñez

M.-B.

Di-Serio

Delgado-Kloos

(2014). Gamification for engaging computer science students in learning activities: A case study. IEEE Transactions on Learning Technologies, 7(3), 291–301.

39.

International Society for Technology in Education & Computer Science Teachers Association. (2011). Operational definition of computational thinking. Report, 1030054.

40.

Israel-Fishelson

Hershkovitz

(2020). Persistence in a game-based learning environment: The case of elementary school students learning computational thinking. Journal of Educational Computing Research, 58(5), 891–918. https://doi.org/10.1177/0735633119887187

41.

Israel-Fishelson

Hershkovitz

Eguíluz

Garaizar

Guenaga

(2020). The associations between computational thinking and creativity: The role of personal characteristics. Journal of Educational Computing Research. Advance online publication. https://doi.org/10.1177/0735633120940954

42.

Jackson

L. A.

Witt

E. A.

Games

A. I.

Fitzgerald

H. E.

Von Eye

Zhao

(2012). Information technology use and creativity: Findings from the children and technology project. Computers in Human Behavior, 28(2), 370–376. https://doi.org/10.1016/j.chb.2011.10.006

43.

Jonassen

D. H.

Rohrer-Murphy

(1999). Activity theory as a framework for designing constructivist learning environments. Educational Technology Research and Development, 47(1), 61–79.

44.

Kafai

Y. B.

Burke

(2013). Computer programming goes back to school. Phi Delta Kappan, 95(1), 61–65.

45.

Kalelioğlu

Gülbahar

Kukul

(2016). A framework for computational thinking based on a systematic research review. Baltic Journal of Modern Computing, 4(3), 583–596.

46.

Kaufman

J. C.

Beghetto

R. A.

(2009). Beyond big and little: The four C model of creativity. Review of General Psychology, 13(1), 1–12. https://doi.org/10.1037/a0013688

47.

Kazimoglu

Kiernan

Bacon

MacKinnon

(2012). Learning programming at the computational thinking level via digital game-play. Procedia Computer Science, 9(0), 522–531.

48.

Kesler

Shamir-Inbal

Blau

(2019). To compute or to create? This is the question! examining creative programming artifacts on the scratch platform in relation to pedagogical perspectives of code and robotics’ teachers. In Eshet-Alkalai

Blau

Caspi

Geri

Kalman

Etgar

(Eds.), Proceedings of the 14th chais conference for the study of innovation and learning technologies: Learning in the technological era (pp. 92–100). The Open University of Israel.

49.

Kim

A. S.

A. J.

(2017). A pedagogical analysis of online coding tutorials. In Caspersen

M. E.

Edwards

S. H.

Barnes

Garcia

D. D.

(Eds.), Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education—SIGCSE ’17 (pp. 321–326). ACM.

50.

Kim

K. H.

(2011). Proven reliability and validity of the Torrance tests of creative thinking (TTCT). Psychology of Aesthetics, Creativity, and the Arts, 5(4), 314–315. https://doi.org/10.1037/a0021916

51.

Kim

K. H.

(2017). The Torrance tests of creative Thinking—Figural or verbal: Which one should we use? Creativity. Theories—Research—Applications, 4(2), 302–321. https://doi.org/10.1515/ctra-2017-0015

52.

Kong

(2019). Components and methods of evaluating computational thinking for fostering creative problem-solvers in senior primary school education. In Kong

Abelson

(Eds.), Computational thinking education (pp. 119–142). Springer.

53.

Krumm

Means

Bienkowski

(2018). Learning analytics goes to school: A collaborative approach to improving education. In learning analytics goes to school: A collaborative approach to improving education. Routledge.

54.

Lai

R. P. Y.

(2019). What underlies computational thinking: Exploring its cognitive mechanism and educational implications. Proceedings of International Conference on Computational Thinking Education, 2019, 204–208.

55.

Liu

M.-C.

H.-F.

(2002). A study on the creative problem-solving process in computer programming. Proceeding of the International Conference on Engineering Education (ICEE 2002), August 18–21, 2002, Manchester, UK.

56.

O. H. T.

Huang

J. C. H.

Huang

A. Y. Q.

Yang

S. J. H.

(2017). Applying learning analytics for improving students engagement and learning outcomes in an MOOCs enabled collaborative programming course. Interactive Learning Environments, 25(2), 220–234.

57.

Lye

S. Y.

Koh

J. H. L.

(2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012

58.

Miller

L. D.

Soh

L. K.

Chiriacescu

Ingraham

Shell

D. F.

Ramsay

Hazley

M. P.

(2013). Improving learning of computational thinking using creative thinking exercises in CS-1 computer science courses [Paper presentation]. Frontiers in Education Conference, FIE (pp. 1426–1432). https://doi.org/10.1109/FIE.2013.6685067

59.

Mishra

Yadav

(2013). Of art and algorithm: Rethinking technology & creativity in the 21st century. TechTrends, 57(3), 10–14.

60.

Nachar

(2008). The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution. Tutorials in Quantitative Methods for Psychology, 4(1), 13–20. https://doi.org/10.20982/tqmp.04.1.p013

61.

National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press.

62.

Navarrete

C. C.

(2013). Creative thinking in digital game design and development: A case study. Computers & Education, 69, 320–331. https://doi.org/10.1016/j.compedu.2013.07.025

63.

Nouri

Zhang

Mannila

Norén

(2020). Development of computational thinking, digital competence and 21st century skills when learning programming in K-9. Education Inquiry, 11(1), 1–17. https://doi.org/10.1080/20004508.2019.1627844

64.

Nutbrown

Higgins

(2016). Static analysis of programming exercises: Fairness, usefulness and a method for application. Computer Science Education, 26(2–3), 104–128.

65.

Organisation for Economic Co-operation and Development. (2018). The future of education and skills: Education 2030. OECD Education Working Papers 23. https://doi.org/10.1111/j.1440-1827.2012.02814.x

66.

Papert

(1980). Mindstorms: Children, computers, and powerful ideas. Basic Books.

67.

Pérez Poch

Olmedo Torre

Sánchez Carracedo

Salán Ballesteros

M. N.

López Álvarez

(2016). On the influence of creativity in basic programming learning at a first-year engineering course. International Journal of Engineering Education, 32(5(B), 2302–2309.

68.

Perkins

D. N.

Salomon

(1992). Transfer of learning. International Encyclopedia of Education, 6452–6457. https://doi.org/10.1016/j.dr.2010.03.001

69.

Piaget

(1964). Part I: Cognitive development in children: Piaget development and learning. Journal of Research in Science Teaching, 2(3), 176–186. https://doi.org/10.1002/tea.3660020306

70.

Plucker

J. A.

Beghetto

R. A.

(2002). Why creativity is domain general, why it looks domain specific, and why the distinction does not matter. In Sternberg

R. J.

Grigorenko

E. L.

Singer

J. L.

(Eds.), Creativity: From potential to realization (pp. 153–167). American Psychological Association. https://doi.org/10.1037/10692-009

71.

Reilly

R. C.

(2008). Is expertise a necessary precondition for creativity? A case of four novice learning group facilitators. Thinking Skills and Creativity, 3(1), 59–76. https://doi.org/10.1016/j.tsc.2008.02.002

72.

Reiter-Palmon

Illies

M. Y.

Kobe Cross

Buboltz

Nimps

(2009). Creativity and domain specificity: The effect of task type on multiple indexes of creative problem-solving. Psychology of Aesthetics, Creativity, and the Arts, 3(2), 73–80. https://doi.org/10.1037/a0013410

73.

Resnick

(2006). Computer as paintbrush: Technology, play, and the creative society. In Singer

Golikoff

Hirsh-Pasek

(Eds.), Play = learning: How play motivates and enhances childrens cognitive and socialemotional growth (pp. 1–16). Oxford University Press.

74.

Román-González

Moreno-León

Robles

(2019). Combining assessment tools for a comprehensive evaluation of computational thinking interventions. In Kong

Abelson

(Eds.), Computational thinking education (pp. 79–98). Springer.

75.

Román-González

Pérez-González

J. C.

Jiménez-Fernández

(2017). Which cognitive abilities underlie computational thinking? Criterion validity of the computational chinking test. Computers in Human Behavior, 72, 678–691.

76.

Romeike

(2007). Three drivers for creativity in computer science education [Paper presentation]. IFIP-Conference on Informatics, Mathematics and ICT, Boston, MA, United States, 2007.

77.

Roque

Rusk

Resnick

(2016). Supporting diverse and creative collaboration in the scratch online community. In Cress

Moskaliuk

Jeong

(Eds.), Mass collaboration and education (pp. 241–256). Springer International Publishing. https://doi.org/10.1007/978-3-319-13536-6_12

78.

Runco

M. A.

Jaeger

G. J.

(2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92–96. https://doi.org/10.1080/10400419.2012.650092

79.

Runco

M. A.

Johnson

D. J.

(2002). Parents’ and teachers’ implicit theories of children’s creativity: A cross-cultural perspective. Creativity Research Journal, 14(3–4), 427–438. https://doi.org/10.1207/S15326934CRJ1434_12

80.

Saavedra-Sánchez

Vidal-Sepúlveda

Olivares-Rodríguez

(2019). Effort analysis of computational thinking process over a gamified and non-gamified environments. In Scheihing

Guerra

Henríquez

Olivares

Muñoz-Merino

P. J.

(Eds.), The 2nd Latin American conference on learning analytics (Vol. 2425, pp. 89–99).

81.

Said-Metwaly

Van den Noortgate

Kyndt

(2017). Methodological issues in measuring creativity: A systematic literature review. Creativity. Theories—Research—Applications, 4(2), 276–301. https://doi.org/10.1515/ctra-2017-0014

82.

Saito

Sasaki

Washizaki

Fukazawa

Muto

(2017). Quantitative learning effect evaluation of programming learning tools. In Wong

K. W. G.

C. F. H.

(Eds.), 2017 IEEE 6th International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 209–216). IEEE. https://doi.org/10.1109/TALE.2017.8252335

83.

Schoeller

B. A.

(2005). The effects of reading goals on learning in a computer mediated environment. Issues in Informing Science and Information Technology, 2, 405–422. https://doi.org/10.28945/838

84.

Scott

C. L.

(2015). The futures of learning 2: What kind of learning for the 21st century? Education Research and Foresight Working Papers 3.

85.

Seo

Y.-H.

Kim

J.-H.

(2016). Analyzing the effects of coding education through pair programming for the computational thinking and creativity of elementary school students. Indian Journal of Science and Technology, 9(46), 1–5. https://doi.org/10.17485/ijst/2016/v9i46/107837

86.

Seow

Looi

How

Wadhwa

(2019). Educational policy and implementation of computational thinking and programming: Case study of Singapore. In Kong

Abelson

(Eds.), Computational thinking education (pp. 345–361). Springer.

87.

Shute

V. J.

Sun

Asbell-Clarke

(2017). Demystifying computational thinking. Educational Research Review, 22, 142–158.

88.

Siemon

Plaumann

R. M.

Regenberg

Yuan

Liu

Robra-Bissantz

(2016, August 11–14). “Tinkering for creativity”: An experiment to utilize MaKey MaKey invention kit as group priming to enhance collaborative creativity [Paper presentation]. Americas Conference on Information Systems, AMCIS 2016, San Diego, CA, USA.

89.

Stolaki

Economides

A. A.

(2018). The creativity challenge game: An educational intervention for creativity enhancement with the integration of information and communication technologies (ICTs). Computers and Education, 123, 195–211. https://doi.org/10.1016/j.compedu.2018.05.009

90.

Tang

Yin

Lin

Hadad

Zhai

(2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019

91.

Tedre

Denning

P. J.

(2016). The long quest for computational thinking [Paper presentation]. Koli Calling Conference on Computing Education Research (pp. 120–129). ACM Press. https://doi.org/10.1145/2999541.2999542

92.

Torrance

E. P.

(1965). Scientific views of creativity and factors affecting its growth. Daedalus, 94(3), 663–681. https://doi.org/10.2307/20026936

93.

Torrance

E. P.

(1974). Torrance tests of creative thinking. Scholastic Testing Service.

94.

Tsarava

Moeller

Pinkwart

Ninaus

(2017, October 5–6). Training computational thinking: Game-based unplugged and plugged-in activities in primary school [Paper presentation]. 11th European Conference on Game-Based Learning (pp. 687–695). Academic Conferences International Limited.

95.

Voogt

Fisser

Good

Mishra

Yadav

(2015). Computational thinking in compulsory education: Towards an agenda for research and practice. Education and Information Technologies, 20(4), 715–728.

96.

Feinstein

(2017). An exploratory multiple case study about using game-based learning in STEM classrooms. International Journal of Research in Education and Science, 3(2), 582–582.

97.

Vygotsky

L. S.

(2004). Imagination and creativity in childhood. Journal of Russian and East European Psychology, 42(1), 7–97.

98.

Wang

Nickerson

J. V.

(2017). A literature review on individual creativity support systems. Computers in Human Behavior, 74, 139–151. https://doi.org/10.1016/j.chb.2017.04.035

99.

Wang

L. C.

Chen

M. P.

(2010). The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning. Innovations in Education and Teaching International, 47(1), 39–52.

100.

Webb

M. E.

Bell

Davis

Katz

Y. J.

Fluck

Sysło

M. M.

Kalaš

Cox

Angeli

Malyn-Smith

Brinda

Micheuz

Brodnik

(2018). Tensions in specifying computing curricula for K-12: Towards a principled approach for objectives. IT—Information Technology, 60(2), 59–68. https://doi.org/10.1515/itit-2017-0017

101.

Whalley

Ogier

(2020). Paperclips, circles, and six-legged spiders. An exploration of self-perceived and measured creativity among CS students [Paper presentation]. Twenty-Second Australasian Computing Education Conference (pp. 11–20). ACM. https://doi.org/10.1145/3373165.3373167

102.

Wing

J. M.

(2006). Computational thinking. Communications of the ACM, 49(3), 33–35.

103.

Wing

J. M.

(2010). Computational thinking: What and why? The Link Magazine. Carnegie Mellon University.

104.

World Bank. (2019). Children learning to code: Essential for 21st century human capital.

105.

World Economic Forum. (2015). New vision for education unlocking the potential of technology.

106.

Yadav

Cooper

(2017). Fostering creativity through computing. Communications of the ACM, 60(2), 31–33. https://doi.org/10.1145/3029595

107.

Zhang

L. C.

Nouri

(2019). A systematic review of learning computational thinking through scratch in K-9. Computers and Education, 141, 103607. https://doi.org/10.1016/j.compedu.2019.103607

108.

Zhou

Shen

Wang

Neber

Johji

(2013). A cross-cultural comparison: Teachers’ conceptualizations of creativity. Creativity Research Journal, 25(3), 239–247. https://doi.org/10.1080/10400419.2012.730006