Enhancing and Modeling Teachers’ Design Beliefs and Efficacy of Technological Pedagogical Content Knowledge for 21st Century Quality Learning

Abstract

This study proposed a new conceptualization of technological pedagogical content knowledge (TPACK) that focuses exclusively on the intersection of technology, pedagogy, and content specifically for selected dimensions of 21st century learning. In addition, teachers’ design beliefs were investigated with the teachers’ TPACK. Given the conceptualization, a new instrument was designed and validated. An associated intervention program to enhance the preservice teachers' TPACK was designed and the pre- and post-course surveys were conducted. To unpack the relationships between teachers’ design beliefs and their TPACK, structural equation models were constructed and validated. The findings indicate that the instrument possesses good construct, discriminant and convergence validity, and reliabilities. The intervention enhanced the teachers’ TPACK efficacies and their design beliefs significantly, and the structural equation models indicate that the teachers’ design beliefs are significant predictors of the teachers’ TPACK. The implications of this study suggest that TPACK may be conceived differently and this may promote new intervention programs to foster preservice teachers’ TPACK and design beliefs.

Keywords

technological pedagogical content knowledge design thinking teachers’ beliefs structural equation modeling quality learning preservice teachers scale development

Introduction

As technology continues to enhance the affordances of the 21st century classroom, teaching and learning practices that are founded on the epistemological assumptions and pedagogical research of the last century are continuously being disrupted (Collins & Halverson, 2010; Geisinger, 2016). The status of teachers and textbooks, which once were the authoritative sources of knowledge are now being challenged. Multiple sources of high-quality knowledge, and even pedagogically strong delivery of subject matter, are readily available in the Internet. Many YouTube teaching videos are created by well-respected educators. In other words, the advantages of traditional schooling can be and is increasingly being replicated and propagated through information and communication technologies (ICTs). Consequently, classroom pedagogical practices and teachers’ pedagogical competencies need to be transformed beyond excellent content delivery. Emerging literature on teacher education is pointing toward teachers’ ability to create technological, pedagogical, and content knowledge (TPACK; Koh, Chai, Wong & Hong, 2015; Mishra & Koehler, 2006; Valtonen et al., 2017) for 21st century classrooms.

As with pedagogical content knowledge (PCK), TPACK has emerged to be a powerful notion to unpack the knowledge and skills teachers need to design lessons for 21st century classrooms (Harris, Phillips, Koehler & Rosenberg, 2017). Nonetheless, current research indicates that the emergence of TPACK has not transformed the state of technology integration in classrooms (Heitink, Voogt, Fisser, Verplanken, & van Braak, 2017; Pringle, Dawson, & Ritzhaupt, 2015; Tondeur, Aesaert, et al., 2017). More research on developing teachers’ TPACK specifically to promote 21st century competencies through various pathways is needed (Koehler, Mishra, Kereluik, Shin, & Graham, 2014; Valtonen et al., 2017). On the other hand, quantitative TPACK research has focused more on validating the seven-factor model (see Mishra & Koehler, 2006) and using the validated models to assess teachers’ growth in terms of their efficacy before and after ICT courses (e.g., see Chai & Koh, 2017). While several studies have been able to validate the seven-factor model quantitatively (Chai, Koh, & Tsai, 2016), recent research has challenged the model with different ways with which TPACK could be conceived of (e.g., see Jang & Tsai, 2013; Lee & Tsai, 2010; Yeh, Hsu, Wu, Hwang, & Lin, 2014).

Given the aforementioned background, this study attempts to conceptualize a new representation of TPACK created specifically for 21st century learning, which is labeled as TPACK-21st century quality learning (TPACK-21CQL). TPACK-21CQL deals directly with the intersection among technology, pedagogy, and content of the seven-factor model of TPACK without considering the intermediate factors such as technological pedagogical knowledge (TPK), PCK, or technological content knowledge (TCK) or the elementary factors such as technological knowledge, pedagogical knowledge, or content knowledge. In other words, instead of conceptualizing TPACK as constituted by various subfactors, it contributes to current TPACK research by exploring directly the intersection of TPACK and hypothesizes that the central TPACK factor can be multidimensional when it addresses selected dimensions of 21st century learning. TPACK-21CQL may open up new perspectives on how TPACK should be conceptualized, measured, and fostered. In addition, teachers’ beliefs have been recognized as an area that needs to be researched in conjunction with teachers’ TPACK as these constructs are closely intertwined in influencing teachers’ instructional decision-making (Tondeur, van Braak, Ertmer, & Ottenbreit-Leftwich, 2017). In particular, teachers’ design beliefs have been identified as a multidimensional construct that is associated with teachers’ TPACK, and teachers’ design beliefs may predict the teachers’ TPACK (Chai & Koh, 2017). Given the new conceptualization of TPACK, this study aims to (a) create and validate an instrument that represents TPACK-21CQL, (b) investigate how the preservice teachers’ beliefs and TPACK-21QL efficacies change throughout an intervention program designed around TPACK-21QL, and (c) examine the structural equation model (SEM) of design beliefs and TPACK-21QL before and after the intervention to further understand the effects of the intervention.

Literature Review

The following section first identifies the significant dimensions of 21st century learning and its relation to TPACK. It then argues for a new representation of TPACK-21CQL. Subsequently, professional devlopment models for TPACK are reviewed along with research about relevant aspects of teachers’ beliefs. The purpose is to identify important design heruistics that could promote TPACK-21CQL and foster teachers’ beliefs toward 21st century learning.

Dimensions of 21st Century Learning for TPACK-21CQL

Driven by the advances in technologies, multiple frameworks of 21st century learning has been formulated (see Dede, 2010; Geisinger, 2016; Voogt & Roblin, 2012). Examples of well-known international frameworks include the Partnership for 21st century skills (P21CS, 2009), Organisation for Economic Co-operation and Development (OECD) 21st century skills and competences for new millennium learners (Ananiadou & Claro, 2009), and the Intel, Microsoft, and Cisco sponsored assessment and teaching of 21st century skills (Binkley et al., 2012; see also http://www.atc21s.org/). These international frameworks are from three different continents, which are America, Europe, and Australia, respectively. While variation exists among the frameworks, the common underlying competencies are learners’ ability to leverage on ICT to communicate and collaborate with others. The ACT21S classified ICT as tools for working in the 21st century workplace while the P21CS framework further divides the ICT competencies as information, media, and technological skills. Communication and collaboration are the desired ways of working for the knowledge society, which is intertwined with critical and creative thinking (P21CS, 2009). The OECD framework, however, took the approach of categories of skills and organized the 21st century competencies as cogntive skills, intrapersonal skills, interpersonal skills, and technical skills (Ananiadou & Claro, 2009). Cognitive skills involve solving emerging real-world problems which are complex in nature through different modes of thinking. Intrapersonal skills involve self-management such that one could adapt and perform self-regulated learning in a dynamically changing world.

Drawing from the OECD framework and also the meaningful learning framework which has been portrayed as being compatible with the 21st century learning framework (Howland, Jonassen, & Marra, 2014), this study identifies authentic learning (AUL), collaborative learning (COL), reflective learning (RL), and active and cosntructive learning (ACTL) as the four key dimensions of 21st century quality learning that preservice teachers need to be able to foster in the classroom, supported by technologies to facilitate content mastery. AUL emphasizes getting students to resolve real-world complex problems. COL involves students co-construction of content knowledge through in-depth negotiation. RL engages students in planning, monitoring, reviewing, and reflecting their learning processes while ACTL requires students to construct digital artifacts with appropriate ICT tools as mediators of learning. The framework is contextualized to the current development of Singapore education and its emphasis on 21st century learning and it is named as 21st century quality learning (Chee & Chai, 2017).

While teachers are called to transform pedagogical practices to create 21st century learning in their classroom, it is obvious that specific research into how to equip teachers for such transformations is needed. Generally, the notion of 21st century learning is discussed conceptually and methods of assessing 21st century learning has just begun to emerge (Care, Scoular, & Griffin, 2016; Geisinger, 2016). Voogt, Erstad, Dede, and Mishra’s (2013) review of international studies indicates that while there is consensus about the major constituents of 21st century skills, the enactment of 21st century learning in classrooms is lacking, owing to the absence of integration between the 21st century skills with curriculum and assessment. Kereluik, Mishra, Fahnoe, and Terry (2013) highlighted the need for teachers to develop TPACK for 21st century learning. The TPACK framework articluates the complex forms of knowledge and knowing that teachers need to create lessons that promote technology-based learning. While many TPACK studies are based on the premise of engendering 21st century learning (e.g., Angeli & Valanides, 2009; Mishra & Koehler, 2006), specific efforts that are directed toward promoting specific 21st century skills may need further research (Valtonen et al., 2017), especially given that the current state of technology integration generally supports traditional learning (Pringle et al., 2015; Tondeur, Aesaert, et al., 2017).

Current Conceptualizations of TPACK and Associated Quantitaive Surveys

Since the inception of TPACK around 2005, its representations have been contested (see Angeli & Valanides, 2009; Koehler et al., 2014). Nonetheless, the most widely cited framework is the seven-factor model articulated by Mishra and Koehler (2006). Schmidt et al. (2009) created the first instrument to measure TPACK based on the seven-factor representation. Subsequently, many researchers have built on their efforts to test, refine, or create and further contextualize the TPACK efficacy questionnaire (see Chai et al., 2016; Koehler et al., 2014). For example, Lee and Tsai (2010) attempted to contextualize TPACK for web-based learning and they were able to identify some relevant factors. Earlier research has also questioned whether it is possible to identify the seven-factor model (e.g., see Archambault & Barnett, 2010). Recent reviews have provided substantial evidences that it is possible to identify the seven factors with adequate validity and reliability (Chai et al., 2016; Drummond & Sweeney, 2017; Harris et al., 2017). The seven-factor model has also guided professional development activities to build teachers’ TPACK, mainly drawing on the teachers’ TK, PCK, and TPK to be synthesized and transformed into ICT-based lesson design (Chai & Koh, 2017; Koehler, Mishra & Yahya, 2007).

It is obvious that the short academic development history of the seven-factor model illustrates that conceptual models guide measurement, research, the design of intervention, and subsequently the development of teachers’ competencies. Researchers who are sensitive to the implications of the conceptual model have thus challenged the model from both the theoretical and empirical fronts (e.g., Archambault & Barnett, 2010; Brantley-Dias & Ertmer, 2013). The seven-factor model has also been criticized as adopting a more integrative view rather than a transformative view of TPACK (Angeli & Valanides, 2009). In other words, there are other possible representations of TPACK that could guide teachers’ development.

Through Delphi technique, Yeh et al. (2014) presented an eight-factor model based on experts’ inputs about TPACK from the perspective of educators’ practical experiences with integrating technology. Examples of the eight factors include using ICT to assess students and using ICT-integrated strategies for teaching. Closer examination of the factors reveals that they can be considered as expanded forms of TPK. Yeh et al.’s (2014) study implies that TPK can be further broken down to subfactors. However, as the authors argue that their representation of TPK is solely drawn from practicing science educators, the eight factors may be taken as representing technological pedagogical science knowledge.

Recent reviews have also suggested that future TPACK research may move into measuring specific combinations of technology, pedagogy, and content areas (e.g., Chai et al., 2016; Jang & Tsai, 2013; Koehler et al., 2014; Lee & Tsai, 2010). To focus teachers’ development of TPACK specifically for selected dimensions of 21st century learning, this study proposed and validated a survey that focused on TPACK-21CLQ as explained earlier.

Models to Facilitate Preservice Teachers’ TPACK Development

Several models based on different contexts of teacher education have been researched and shown to be effective for raising preservice teachers’ competencies for creating TPACK in the form of ICT-based lesson designs. Most notable among them is Angeli and Valanides’s (2009) work on technology mapping. Technology mapping begins with the identification of difficult topics that ICT could better represent, followed by the creation of technological representations of the content that could facilitate student-centric learning. Appropriate teaching and learning strategies are then chosen to engage students in learning the content. The effectiveness of technology mapping is verified by assessing preservice teachers’ lesson design. The study indicates that it is important to support preservice teachers with an overall instructional model.

Building on the technology mapping processes, Kramarski and Michalsky (2010) articulated a set of 16 metacognitive prompts to help preservice teachers in different phases of designing ICT-integrated lessons. The underlying reason for supporting the preservice teachers with the metacognitive prompts was based on the complexities of design tasks where multiple sources of knowledge have to be activated and transformed to design a coherent and effective lesson. Kramarski and Michalsky (2010) tested the effectiveness of their model with TPACK comprehension tests and evaluated the preservice teachers’ lesson design. Apparently, their study indicates that preservice teachers who were supported by the prompts outperformed those who did not have the prompts. An important lesson that can be derived from their study is that the complex and knowledge intensive design processes need to be scaffolded especially for preservice teachers. One limitation of this model could be that it was implemented with a 56-hour timeframe which many teacher education institutes may not be able to afford. There may be a need to find ways to support the preservice teachers in a just-in-time manner.

To date, there are criticisms about the TPACK framework as it is complex and may not help teachers in their day-to-day practice (Brantley-Dias & Ertmer, 2013; Dobozy & Campbell, 2016). In addition, Angeli, Valanides and Christodoulou (2016) have advocated that meaningful TPACK could not be achieved through the integrative approach. These authors have argued for the transformative approach that requires the teachers to synthesize isolated knowledge of technology, pedagogy, and content in a manner that result in new teaching and learning practices (see Angeli & Valnides, 2009). Thus, to be well versed in utilizing the framework, extensive help to challenge teachers’ assumption about what constitutes good technology integration and to help them transform isolated knowledge is needed (Koh, Chai, & Lim, 2017). For preservice teachers who may lack experience, the preservice ICT course design needs to help them to interweave multiple source of knowledge and tutors facilitation is essential.

Integrating the aforementioned models, Chai and Koh (2017) created the Scaffolded TPACK Lesson Design Model (STLDM) as an overall design process for ICT integrated lesson. Based on the theory of epistemic framing (Koh, Chai, Wong, et al., 2015; see also Rosenberg, Hammer, & Phelan, 2006) that emphasizes helping learners to view the task at hand as knowledge construction, the STLDM synthesizes important metacognitive prompts from past research as a means to activate the relevant knowledge of the various subfactors of TPACK (see Figure 2) to facilitate the preservice teachers’ design. While previous research indicated that the STLDM (Chai & Koh, 2017) improves the preservice teachers’ self-efficacy for designing lessons and fosters relevant design beliefs, more instructional support can be further integrated. These supports could be instructor modeling of design thinking to avoid early fixation (Razzouk & Shute, 2012), experiential learning among the preservice teachers to foster deep understanding, and reflection and collaboration among the preservice teachers (Heitink et al., 2017; Tondeur et al., 2012).

Figure 1.

Structural equation model of teachers’ design beliefs predicting 21CQL.

Past effort in promoting teachers’ TPACK has surfaced some important lessons. First, learning design through authentic design experience, that is, learning by design, is a tried-and-tested approach. Second, framing the design with relevant knowledge and considerations helps preservice teachers to draw upon relevant knowledge needed for good lesson design. Third, multiple forms of cognitive prompts can support the preservice teachers to consider the myriad contextual and pedagogical knowledge. The actualization of these learning from past research is elaborated in our revised STLDM (see Methods section). The revised STLDM may contribute to promote educators’ effort in enhancing preservice teachers TPACK.

Teachers’ Design Beliefs

The importance of teachers’ beliefs in shaping teachers’ instructional decision has been emphasized in the context of engaging teachers to design lesson (Heitink et al., 2017; Tondeur, van Braak, et al., 2017). Teachers possess many different forms of beliefs (e.g., beliefs about technology, students, subject matter, etc.) and these beliefs could be highly individualized and implicit. Nonetheless, these beliefs shape teachers’ design instruction. In relation to TPACK, empirical research on teachers’ beliefs and how it is associated with their TPACK has drawn attention but with relatively few studies being published. Boschman, McKenny, and Voogt’s (2015) qualitative content analysis indicates that early literacy teachers’ existing beliefs and attitudes form the basis of their technology use in classrooms. Quantitative research, however, was largely confined to teachers’ design beliefs (Chai & Koh, 2017; Koh, Chai, Hong, & Tsai, 2015; Koh et al., 2017). The teachers’ design beliefs reported in these studies include their beliefs about new culture of learning, beliefs about themselves as designers, their design disposition (i.e., their comfort level with the inherently ambiguous design situation), and their self-efficacy for design thinking. The beliefs toward new culture of learning assess if teachers are agreeable with emerging participatory culture of learning that employs ICT as a collaborative, curative, constructive, and inquiry tool (Thomas & Brown, 2011).

Past research (Chai & Koh, 2017; Koh, Chai, Hong, et al., 2015) indicates that the teachers’ design beliefs predict their TPACK when TPACK is treated as a single factor where all other TPACK subfactors intersect. In addition, teachers’ design thinking efficacy may mediate other form of teacher design beliefs such as design disposition. This study hypothesizes that the predictive relationship should be generally supported as a second-order SEM was supported in the previous study (see Chai & Koh, 2017). As the conceptualization of TPACK evolves, it seems necessary for research in teachers’ beliefs to coevolve given the importance of teachers’ beliefs in influencing instructional decisions. In particular, research on how the SEM may change before and after the preservice teachers have been engaged in creating TPACK is apparently still lacking.

Methods

Based on the foregoing arguments that a new conceptualization of TPACK-21CQL could open up new ways to measure and foster preservice teachers’ competencies and beliefs about 21st century learning, and it may shape the relationships between the teachers’ design beliefs and their TPACK-21CQL, the following research questions were formulated to guide this study.

Research Questions

Does the TPACK-21CQL with teachers’ design beliefs questionnaire possess adequate psychometric properties to be regarded as a reliable and valid instrument?

Does the revised STLDM enhance Singapore preservice teachers’ TPACK-21CQL efficacies and their design beliefs significantly?

Can the hypotheses of teachers’ design beliefs predicting TPACK-21CQL efficacies be supported in the SEMs for both the pre- and postcourse data?

Participants

A total of 564 preservice teachers who enrolled for a 1-year teacher certification program participated in this study, accounting for 73.2% of the overall preservice teachers enrolled for the year of 2016. These preservice teachers were training for a wide range of subjects including languages, science, mathematics, humanities, and social studies. Of them, 295 (52.3%) are from primary school teacher training while the rest are from secondary or high school teacher training. There are 403 female teachers (71.5%). The mean age for the teachers is 26.2 years (SD = 5.71). Participation in the survey is voluntary with informed consent as per the procedures approved by the university’s institutional review board.

As a result of the emphasis placed on ICT in education in Singapore that begins in elementary school, the participants are generally well versed with basic technological skills which include the abilities to use Microsoft Office packages, surfing the Internet, and using Web 2.0 applications to collaborate. These software were used during the course for a variety of learning activities and their assignments. In addition, it should be noted that all participants passed the course which required them to design technology-enhanced lessons that meets the course requirements.

Instrument

The TPACK-21CQL survey is a newly constructed TPACK questionnaire with five items each for the four dimensions of 21st century (AUL, COL, RL, and ACTL) mentioned earlier. The initial pool of items was generated by the authors based on our understanding of TPACK and 21st century learning. The items were subjected to three university professors’ review to establish face validity. The items for teachers’ design beliefs were adopted from previous research (Chai & Koh, 2017), which has been established as valid and reliable. The factors include beliefs about new culture of learning (BNCL, seven items), teachers as designer (TAD; five items), design disposition (DD, six items), and efficacy in design thinking (DT, five items). All items are scored on a 7-point Likert scale (1 = strongly disagree to 7 = strongly agree).

Data Collection

There were two points of data collection. The precourse survey was conducted during the first lesson before any teaching commenced. The postcourse survey was conducted 12 weeks later during the last session of the course. All data were collected through an online survey and the preservice teachers spent around 15 minutes completing the survey.

Data Analysis

After the responses were matched for the pre and postsurveys, the precourse data were analyzed to establish construct validity through exploratory factor analysis using principal component analysis and the alpha reliabilities were computed. This was followed by further analysis of average variance extracted (AVE) and construct reliability to support convergent validity. The correlations between the factors were also computed along discriminant validity obtained through the square root of AVEs. Subsequently, confirmatory factor analysis was performed on the postcourse data to provide further evidence about the construct validity. To examine the effects of the course on the preservice teachers’ efficacy of designing 21CQL and the changes in their design beliefs, paired sample t-tests and analysis of Cohen’s d were conducted. The SEM with hypotheses based on previous research (Chai & Koh, 2017; Koh, Chai, Hong, et al., 2015) were constructed (see Figure 1) and tested.

Description of the Intervention

The mandatory course, “ICT for meaningful learning,” is a 24-hour course conducted over 12 weeks for all preservice teachers in Singapore. Several design principles undergird the course. First, learning by design through actual designing. To equip the preservice teachers with the necessary TPACK design skills for TPACK-21CQL, there were two design assignments, supported by a lesson design template. The first design assignment requires the preservice teachers to choose a topic pertaining to a cyberwellness issue (e.g., cyber bullying) and design a lesson to facilitate students in learning how to be safe and responsible users of ICT after they have been taught the basic skills of designing lessons with the support of the revised STLDM (see the following section). The skills taught include how to formulate lesson objectives, design ICT-based lesson activities, prepare learning resources, and evaluate students’ understanding. The preservice teachers worked in groups of four to five, drawing on collaborative design to reduce the preservice teachers’ cognitive load of learning the design process. The second assigment is an individual assignment for a topic based on the subject they were training to teach. This assignment was intended to further strengthen the preservice teachers’ design skills. In both assignments, the preservice teachers were supported by coaching from the tutors through consultation within and outside class sessions.

Second, framing lesson design with relevant technological, pedagogical, and content knowledge through cognitive prompts. The revised STLDM as depicted in Figure 2 was provided to guide the preservice teachers in creating ICT-integrated lessons for 21CQL. As shown in Figure 2, multiple cognitive prompts denoting relevant TPACK domains were formulated and spread across the two stages of design. In Stage 1, the cognitive prompts help the preservice teachers to activate relevant knowledge resources and gather information as the basis of diagnosing and deciding appropriate learning objectives for the lesson. In Stage 2, the cognitive prompts push the preservice teachers to consider good practices for using specifics of technologies and to create lesson activities that incorporate 21CQL. The revised STLDM has incorporated Kirschner’s (2015) suggestions on building the teaching professionals which focuses on teachers as designer.

Figure 2.

The revised scaffolded TPACK lesson design model (R-STLDM).

Third, building preservice teachers’ TPACK through reflection of learning experiences. To build the preservice teachers’ TPACK, multiple forms of technology-supported lessons were introduced weekly. For instance, to build the preservice teachers’ knowledge about 21CQL, the preservice teachers were provided with website resources and a set of Powerpoint or Google slides with guiding questions. Preservice teachers were tasked to co-construct knowledge for one dimension of 21CQL and subsequently perform peer teaching. After the peer teaching, the tutors invited the preservice teachers to consider the strengths and weaknesses of peer teaching as they have experienced, the possible technologies, resources, grouping instruction, and so forth, needed to support peer teaching; possible problems and solutions with the pedagogical approach and what is needed to improve it. In other words, reflecting on the learning experiences created by the teacher, educators’ design of TPACK with the aims of improving the lesson was one of the means to help the preservice teachers to unpack lesson design.

Fourth, tutor modeling of TPACK co-creation. In many sessions, the preservice teachers were tasked to tinker with ICT tools for the content areas, learn the technical skills, and build knowledge on how to use the tool meaningfully for relevant content-based topics. The tutors modeled how they investigate the affordances and pedagogical considerations needed to use the ICT tools in the classroom in the first few lessons. The tutors’ way of pedagogical sense making was gradually transferred to the preservice teachers to build their ability to make sense of emerging ICT tools. For instance, the tutors may task the preservice teachers to build a ICT-based concept map using Cmap (see https://cmap.ihmc.us/) for a given topic relevant to their teaching needs. This is followed by a knowledge construction question “How can an ICT-based concept map be used to facilitate 21st century quality learning in your subject matter?” The preservice teachers contribute ideas about suitable topics and tasks that the tool can be used for, the pedagogical provision such as providing guiding questions and linking words between the concepts to help students, video resources showing advanced features of the tool, possible problems and solutions when using the tool, and how collaboration and reflection should be structured.

Findings

Research Question 1 assesses the psychometric properties of the survey to establish its validity and reliability. The exploratory factor analysis yielded eight factors with Eigen value above 1 as it was designed to measure. The total variance explained was 73.0%. Nine items with factor loading lower than .50 were removed. The overall alpha reliability of the remaining items was .94. Table 1 provides further information about means, standard deviations, factor loadings, alpha reliabilities, AVE, and CR.

Table 1.

Exploratory Factor Analysis With Descriptive Statistics, Factor Loadings, Alpha Reliabilities (α), Average Variance Extracted (AVE), and Composite Reliabilities (CR), N = 564.

No	Items	Factor loadings
Reflective Learning with ICT (RL); Mean = 4.73, SD = 1.16, α = .93, AVE = 0.57, CR = 0.87
RL2	I can use technology-based tools (e.g., online KWL, concept maps) to prompt students in monitoring their progress in understanding the content	.81
RL4	I can create self-directed learning activities of the content knowledge with appropriate ICT tools (e.g., Webquest, Flipped lessons)	.78
RL3	I can guide students in diagnosing their knowledge gaps with various forms of online feedback systems (quizzes, analytics etc.).	.75
RL5	I can help students to review the strength and weaknesses of their ICT-supported learning for the subject I teach.	.75
RL1	I can use technology (e.g., blogs. e-portfolio) to support students’ self-reflection about their learning strategies for the subject matter.	.69
Authentic Learning with ICT (AUL), Mean = 4.61, SD = 1.05, α = .88, AVE = 0.49, CR = 0.82
AUL2	I can engage students in learning the subject matter using the ICT tools that subject matter experts use	.73
AUL4	I can use technologies to scaffold students’ in solving complex problems arising from the topics that I teach.	.73
AUL5	I can arouse students’ interest in solving real-world problems using subject related software.	.73
AUL1	I am able to create real-world problem scenarios for my teaching subject using online web site creators.	.66
AUL3	I am competent in searching for online video resources to initiate real-world problem solving related to the subject matter.	.63
Collaborative Learning with ICT (COL) Mean = 4.49, SD = 1.02, α = .91, AVE = 0.53, CR = 0.82
COL4	I am competent in prompting students to talk deeply about the content knowledge in online platforms.	.73
COL1	I can formulate in-depth discussion topics about the content knowledge for students’ online discussion.	.73
COL3	I can engage students in substantial peer critiquing work through collaborative software.	.73
COL2	I can facilitate students’ co-construction of subject matter representations when they are working in small groups around a computer.	.72
Active Constructive Learning with ICT (ACTL) Mean = 4.92, SD = 0.99, α = .84, AVE = 0.56, CR = 0.79
ACTL1	I am able to use technology to stimulate my students’ higher order thinking about the subject matter.	.81
ACTL2	I can engage students in constructing deep understanding about the subject matter with various forms of technology (e.g., Google site, concept maps etc.)	.80
ACTL3	I am competent in helping my students to critically synthesize information from various web-based resources for content learning.	.63
Beliefs of New Culture of Learning (BNCL) Mean = 5.80, SD = 0.76, α = .88, AVE = 0.59, CR = 0.88
BNCL5	Today’s learners should be able to remix relevant resources to publish their ideas.	.81
BNCL3	Remeshing digital resources responsibly is a good way to learn.	.79
BNCL1	It is important nowadays for students to be able to participate in online learning communities.	.76
BNCL7	Managing personal online learning resources is a desirable skill.	.74
BNCL4	Producing creative digital works is a meaningful task.	.74
Design Disposition (DD), Mean = 5.35, SD = 0.85, α = .84, AVE = 0.54, CR = 0.86
DD4	I am comfortable to deviate from established practices.	.780
DD5	I am comfortable with occasional failures from trying out new approaches for teaching.	.78
DD3	I am comfortable to explore conflicting ideas	.76
DD6	I am constantly seeking to turn constraints into opportunities.	.70
DD1	I am comfortable with the presence of uncertainty.	.66
Design Thinking Efficacy (DT), Mean = 4.27, SD = 1.12, α = .91, AVE = 0.57, CR = 0.84
DT3	I am competent in detecting possible implementation difficulties in the ICT-based lesson plan.	.78
DT2	I am able to choose the most feasible ICT-based lesson design for my students’ learning.	.78
DT1	I can easily generate a few ICT-based lesson ideas.	.74
DT4	I am confident that my design will optimize my students’ learning.	.73
Teachers as designer (TAD), Mean = 6.08, SD = 0.72, α = .91, AVE = 0.64, CR = 0.87
TAD3	Teachers should be empowered to design lessons	.85
TAD2	My students’ learning experience is an outcome of my lesson design.	.83
TAD1	It is my responsibility to master the skills of designing lessons.	.80
TAD4	Working like designer is part of the teacher’s duty.	.70

In addition to the psychometric information provided in Table 1, Table 2 provides evidence for the discriminant validity. In addition, confirmatory factor analysis (CFA) of the postcourse survey indicates good model fit with a range of indices (chi-square = 1192.26, degree of freedom = 532, χ²/df = 2.24, Root Mean Square Error of Approximation (RMSEA) = 0.047, Goodness of Fit Index (GFI) = 0.89, Confirmatory Fit Index (CFI) = 0.96; see Hair, Black, Babin, & Anderson, 2010). Other fit indexes include Normed Fit Index (NFI) = 0.93; Non-Normed Fit Index (NNFI) = 0.96; Incremental fit index (IFI) = 0.96, Root Mean Square Residual (RMSR) = 0.025, and Adjusted Goodness of Fit Index (AGFI) = 0.87. Overall, there are sufficient evidences to support the construct, convergent, and divergent validity and the reliability of the TPACK-21CQL with teachers’ design beliefs survey.

Table 2.

Correlations and Discriminant Validity of the Factors Based on Precourse Survey.

Components	RL	AUL	COL	ACTL	BNCL	DD	DT	TAD
RL	(0.75)
AUL	.65	(.70)
COL	.68	.64	(.73)
ACTL	.56	.58	.56	(.75)
BNCL	.30	.32	.30	.31	(.77)
DD	.32	.31	.38	.34	.48	(.73)
DT	.66	.63	.63	.51	.28	.32	(.75)
TAD	.21	.18	.19	.23	.49	.44	.20	(.80)

Note. All correlations are significant at .01, figures in parenthesis are the square root of AVE.

Research Question 2 is concerned with the effects of the course design supported by the revised STLDM on the preservice teachers’ TPACK21CQL efficacies and their design beliefs. Table 3 shows that other than the factor teachers as designer (TAD), all factors measured were significantly enhanced pre- and postcourse. Analyses of the effect sizes indicate that, in general, the course effects on TPACK-21CQL dimensions (RL, AUL, and COL) and the teachers’ DT were large (d > 0.8; see Table 3). The effects on ACTL are medium while the effects on two of the teachers’ design beliefs (DD and BNCL) were small. Overall, the course effects should be regarded as positive though some improvement may be needed.

Table 3.

Paired Sample t-Test and Effect Sizes (N = 564).

Measured factors	Pre-study survey		Post-study survey		t-test	Cohen’s d
Measured factors	M	SD	M	SD	t-test	Cohen’s d
BNCL	5.80	0.76	5.89	0.74	2.42*	0.10
DT	4.27	1.12	5.61	0.75	28.61***	1.20
DD	5.35	0.85	5.42	0.80	5.37 ***	0.23
TAD	6.08	0.72	6.12	0.71	1.22	0.05
ACTL	4.92	0.99	5.61	0.73	15.02 ***	0.63
COL	4.49	1.02	5.54	0.75	24.02 ***	1.01
AUL	4.61	1.05	5.66	0.74	23.40 ***	0.98
RL	4.73	1.16	5.74	0.75	20.44 ***	0.86

*p < 0.05, ***p < .001.

The SEM tested 19 hypotheses with regard to whether the teachers’ design beliefs could predict directly or indirectly their TPACK-21CQL. Table 4 reports the findings for both the pre- and postcourse surveys. Figure 3 depicts the supported hypotheses graphically with standardized estimates. The SEM fit indices show that both models based on pre- and postcourse data can be accepted (precourse: chi-square = 1447.80, degree of freedom = 538, χ²/df = 2.69, RMSEA = 0.055, TLI = 0.92, CFI = 0.93; postcourse: chi-square = 1335.72, degree of freedom = 538, χ²/df = 2.48, RMSEA = 0.051, TLI = 0.95, CFI = 0.95; see Hair et al., 2010). For the precourse survey, 9 out of 19 hypotheses were supported. For the postcourse survey, 15 out of 19 hypotheses were supported. The comparison reveals that before the intervention, only BNCL and DT were significant predictors of the factors of TPACK-21CQL. DD only has indirect effect through DT on the TPACK-21CQL factors while TAD has no effect. After the intervention, the teachers’ design beliefs were more able to predict the preservice teachers’ TPACK-21CQL either directly or indirectly through DT. The notable changes in predictive relationship are for the factor of DD follow by TAD. This indicates that the course may have increased the strength of the teachers’ design beliefs as positive predictors. Overall, BNCL and DT have emerged as important positive predictors of the teachers’ TPACK-21CQL for both pre- and postcourse data.

Table 4.

Supported or Unsupported Hypotheses of Structural Equation Models.

Hypotheses		Standardized estimate		Standard error		Critical ratio		Supported Yes/No
Hypotheses		Pre	Post	Pre	Post	Pre	Post	Pre	Post
H1	BNCL→DT	0.29	0.19	0.09	0.05	3.12**	3.50***	Yes	Yes
H2	BNCL→RL	0.13	0.14	0.06	0.04	2.23*	3.41***	Yes	Yes
H3	BNCL→AUL	0.24	0.33	0.07	0.04	3.49***	7.58***	Yes	Yes
H4	BNCL→COL	0.10	0.16	0.06	0.04	1.83	3.89***	No	Yes
H5	BNCL→ACTL	0.21	0.30	0.07	0.05	3.05**	6.52***	Yes	Yes
H6	DD→DT	0.43	0.37	0.08	0.04	5.43***	8.35***	Yes	Yes
H7	DD→RL	0.05	0.10	0.05	0.03	1.04	2.95**	No	Yes
H8	DD→AUL	0.09	0.09	0.06	0.03	1.46	2.67**	No	Yes
H9	DD→COL	0.20	0.22	0.05	0.04	4.02	6.11***	No	Yes
H10	DD→ACTL	0.10	0.00	0.06	0.04	1.66	0.05	No	No
H11	TAD→DT	−0.10	0.29	0.10	0.05	−1.04	5.62***	No	Yes
H12	TAD→RL	−0.03	0.21	0.06	0.04	−0.45	5.44***	No	Yes
H13	TAD→AUL	−0.11	0.03	0.07	0.04	−1.55	0.77	No	No
H14	TAD→COL	−0.12	0.06	0.06	0.04	−2.05*	1.45	No	No
H15	TAD→ACTL	−0.01	0.04	0.07	0.04	0.08	1.01	No	No
H16	DT→RL	0.63	0.57	0.04	0.04	15.44***	13.31***	Yes	Yes
H17	DT→AUL	0.69	0.53	0.05	0.05	14.34***	11.74***	Yes	Yes
H18	DT→COL	0.52	0.57	0.04	0.04	14.31***	13.03***	Yes	Yes
H19	DT→ACTL	0.43	0.50	0.04	0.05	11.16***	10.88***	Yes	Yes

*p < .05. **p < .01. ***p < .001.

Figure 3.

SEM of teachers’ design beliefs predicting 21CQL for pre- and post-course surveys.

Discussion

This section is organized according to the findings. First, this study proposed a new conceptualization of TPACK-21CQL, created a new survey to operationalize the concept, and provided adequate evidences that the new survey is valid and reliable. It may contribute to research in 21st century learning, specifically in the area of teacher education for the 21st century. Measurement for 21st century learning is in its nascent stage (Care et al., 2016; Geisinger, 2016). The survey can enrich the field with more instruments to assess teachers’ TPACK, especially for 21st century learning. Valtonen et al. (2017) attempted to create an instrument to measure 21st century TPACK. However, their instrument adopted the seven-factor model and it did not deal directly with the intersection of technology, pedagogy, and content, that is, they did not have a factor that represents the intersection of technology, pedagogy, and content. They have also suggested that TPACK for 21st century learning needs to consider reflective learning. Tondeur, Aesaert, et al. (2017) also recognize the needs to create new instruments for 21st century learning with ICT but their effort was focused on ICT competencies. Building on past research that propose alternatives of the seven-factor TPACK model (e.g., Jang & Tsai, 2013; Yeh et al., 2014), this study factorized the intersection of technology, pedagogy, and content with dimensions of 21st century. The essence of TPACK lies in teachers’ pedagogical reasoning of the intersection among TK, PK, and CK (Mishra & Koehler, 2006). This instrument focuses on the intersection and thus is a potentially useful instrument for teacher educators who are interested in promoting teachers’ 21st century TPACK. We believe that the four factors could be used independently for teacher education programs which may focus on one or more dimensions of 21st century learning. Nonetheless, our current instrument did not include other important 21st century skills such as critical and creative thinking. Future research may consider expanding the instrument to include other subscales.

The second research question dealt with the effects of the course design based on TPACK-21CQL that we have implemented. Building on the lessons learnt from past research (Angeli & Valanides, 2009; Kramarski & Michalsky, 2010; Mishra & Koehler, 2006), the course design was undergirded by learning by design and supported through cognitive prompts, reflective experiential learning, tutor coaching and modeling, and collaborative design. The paired-sample t-test indicates that the preservice teachers’ gain in self-efficacies for design thinking and TPACK-21CQL are significant with good effect sizes. This study thus provides evidence that fostering preservice teachers’ TPACK may include the design principles mentioned. The intervention may contribute to Kirschner’s (2015) vision of professional teachers who are adept at designing instruction. Kirschner has pointed out that studies about teacher as designer tends to adopt the case study methodology which involves small samples. This research is a large-scale study that promotes teachers as designer with possible design principles for teacher educators to consider. A limitation of this study is that the evidence is confined to teachers’ self-report. Future study needs to include more evidence such as scoring the lesson plans with rubrics (see e.g., Harris, Grandgenett, & Hofer, 2010). In addition, the effect sizes of the change in the teachers’ design beliefs are mostly small. Further effort in fostering the teachers’ design beliefs is needed.

The findings with regard to teachers’ design beliefs in this study indicate that preservice teachers’ beliefs toward TAD and BNCL were high even before the course commenced (TAD = 6.08, BNCL = 5.80). Preservice teachers in this study were generally in agreement that teachers should be designers and teachers should design lesson that promote new culture of learning. Nonetheless, these teachers did not possess strong beliefs in dealing with design as indicated by the precourse mean scores for DT (M = 4.27) and DD (M = 5.35). In general, these teachers have positive beliefs about design that could facilitate their learning in the course. The precourse findings also show that all factors of the teachers’ design beliefs are significantly related to their TPACK-21CQL (see Table 2).

The pre- and postcourse SEMs indicate that before the intervention, only BNCL and DT were significant predictors for the TPACK-21CQL. After the course, other factors of teachers’ design beliefs (DD and TAD) also gained importance as predictors. In particular, closer direct relationships between design dispositions and TPACK-21CQL were established. The teachers’ design disposition indicates their comfort level to deal with ambiguous design situations. The results seem to imply that while the changes in design disposition were significant though small, the intervention has improved their capacity to deal with design situations sufficiently such that they have become more adept and comfortable with the demands of creating TPACK-21CQL. We would argue that it is important to raise teachers’ comfort level in dealing with the emerging volatile and unpredictable technological pedagogical contexts of today’s classroom and equip them with the disposition to rise above the challenges through design thinking.

There are a number of hypotheses that were not supported during the precourse survey. The unsupported hypotheses were mainly from DD and TAD (see Figure 3). While factors can be established and are positively correlated to all other factors surveyed, they are not predictors of the teachers DT or the factors of TPACK-21CQL when the preservice teachers are untrained. This may indicate that the SEM may not be representative of how DD and TAD are structurally associated to the mediating factor DT and the subsequent factors. Both factors are relatively new in research and hypothesized in accordance to past research (Koh, Chai, Hong, et al., 2015). An alternative model could be that they predict BNCL, which in turn predict DT and subsequent factors. This is however beyond the scope of this study and could be tested in future research.

The SEM based on the postcourse data also did not support several hypotheses, namely DD as predictor for ACTL and TAD as predictor to AUL, COL, and ACTL. Further correlation analysis indicates stronger correlations for the factors concerned after the course (generally > 0.4, see Table 2). Nonetheless, it seems clear that TAD effects are mediated mostly through DT. Again, this may indicate that there may be better models beyond the one examined in this study.

In summary, this study provides further evidences that support past research about the importance of teachers’ beliefs in influencing teachers’ lesson designs (Heitink et al., 2017; Tondeur, van Braak, et al., 2017). The SEM we obtained for the postcourse survey is congruent with past research (Chai & Koh, 2017; Koh, Chai, Hong, et al., 2015). These findings imply that teacher educators may need to attend to teachers’ design beliefs. Together with the paired sample t-tests on the teachers’ design beliefs, it seems that more effort in fostering the teachers’ design beliefs could be important. Previous research by Tee and Lee (2011) indicates that the experience of designing TPACK shifted the teachers’ attribution of problems with ICT integration from external environment factor to personal design ability. Thus, one possible way forward could be to structure reflective questions for the preservice teachers to explicitly discuss their beliefs about design and how their beliefs are connected to their design acts.

One limitation to our study with regard to teachers’ design beliefs and their TPACK is that our sample comprised only preservice teachers. Practicing teachers may have very different views about the beliefs we have measured due to their embodied understanding of the teaching profession. It would be interesting and valuable to replicate the study with practicing teachers and to compare how preservice and practicing teachers differ in terms of their beliefs and their responses toward the intervention. Practicing teachers may be less positive about the conception of “teachers as designer” and their beliefs toward new culture of learning. In addition, the revised STLDM may not have large effects on their TPACK-21CQL as it may be oversimplified to capture the nuances in teachers’ actual design considerations. Another limitation is that we did not test the effects of the intervention for the different subject areas that the preservice teachers were training to teach. As content specializations may pose different challenges to the implementation of 21st century learning, further research is definitely needed. Finally, as this research is quantitative in nature, future research may consider interviewing and observing the preservice teachers to provide richer and deeper understanding of the preservice teachers’ TPACK.

Implications and Conclusion

There are several rise-above implications that this study points to. First, to promote 21st century learning which is most likely undergirded by technology-enhanced learning, educators may need to create and validate specific instruments to investigate the relations among the psychologically and pedagogically relevant factors and the effectiveness of instruction. Second, designing comprehensive courses to address the challenges of 21st century technology-enhanced learning requires teacher educators to address teachers’ design beliefs. This is likely to require collective and coordinated efforts among teacher educators, especially for those who are specializing in different subject areas. It seems that sustained effort in learning by design and reflecting the assumptions of design is equally important for teacher educators and teachers. As pointed out by Angeli et al. (2016), there are still many gaps that need attention for teacher and teacher educators to create transformative TPACK. To build the knowledge base needed to transform current educational practices and associated beliefs, it is unlikely that the process can happen in isolation. It is more likely that beliefs and practices are dialectically intertwined and thus need to be addressed concurrently.

Raising preservice teachers’ competencies for the 21st century classroom is an important mission that teacher education institutes need to address. This study documents our effort in furthering the collective endeavor of teacher educators in transforming today’s classroom. We have attempted to change how TPACK can be theorized to influence the design of our ICT course into one that fosters preservice teachers’ design thinking. Nonetheless, design is open-ended, contextually bounded, and subjected to dynamic challenges. More research on how TPACK can be conceptualized and enacted is needed to enrich this nascent field of research.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Ching Sing Chai is a professor from the Chinese University of Hong Kong. His research interest is in teacher education, specifically technological pedagogical content knowledge, teacher’s beliefs, and design thinking.

Joyce Hwee Ling Koh is an associate professor with the Learning Sciences and Technologies Academic Group at the National Institute of Education, Nanyang Technological University, Singapore. She teaches preservice and graduate courses in the areas of ICT integration and instructional design, respectively. She has published numerous peer-reviewed articles in the area of TPACK and design thinking in SSCI-listed journals and has served as the Principal Investigator for two research projects focusing on design thinking and ICT integration in the context of 21st century learning.

Yiong Hwee Teo is a lecturer with the Learning Sciences and Technologies Academic Group at the National Institute of Education, Nanyang Technological University, Singapore. He teaches preservice and graduate courses in the areas of ICT integration, instructional design, and classroom management. He was part of the team in the Ministry of Education that planned and implemented Singapore’s first three ICT Masterplans for schools. His research focus is in asynchronous online discussion and blended learning.

References

Ananiadou, K., & Claro, M. (2009). 21st century skills and competences for new millennium learners in OECD countries (OECD Education Working Papers, No. 41). OECD Publishing. Retrieved from http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=edu/wkp(2009)20&doclanguage=en.

Angeli

Valanides

(2009) Epistemological and methodological issues for the conceptualization, development, and assessment of ICT-TPCK: Advances in technological pedagogical content knowledge (TPCK). Computers & Education 52(1): 154–168.

Angeli

Valanides

Christodoulou

(2016) Theoretical considerations of technological, pedagogical content knowledge. In: Herring

Koehler

Mishra

(eds) TPACK handbook V2.0: TPACK research and approaches, New York, NY: Routledge, pp. 11–30.

Archambault

L. M.

Barnett

J. H.

(2010) Revisiting technological pedagogical content knowledge: Exploring the TPACK framework. Computers & Education 55(4): 1656–1662. doi:10.1016/j.compedu.2010.07.009.

Binkley

Erstad

Herman

Raizen

Ripley

Miller-Ricci

Rumble

(2012) Defining twenty-first century skills. In: Griffin

McGaw

Care

(eds) Assessment and teaching of 21st century skills, London, NY: Springer.

Boschman

McKenney

Voogt

(2015) Exploring teachers’ use of TPACK in design talk: The collaborative design of technology-rich early literacy activities. Computers & education 82: 250–262.

Brantley-Dias

Ertmer

P. A.

(2013) Goldilocks and TPACK: Is the construct ‘Just right?’. Journal of Research on Technology in Education 46(2): 103–128.

Care

Scoular

Griffin

(2016) Assessment of collaborative problem solving in education environments. Applied Measurement in Education 29(4): 250–264.

Chai

C. S.

Koh

J. H. L.

Tsai

C.-C.

(2016) A review of quantitative measures of technological pedagogical content knowledge (TPACK). In: Mary

Matthew

Punya

(eds) TPACK handbook V2.0: TPACK research and approaches, New York, NY: Routledge, pp. 87–106.

10.

Chai

C. S.

Koh

J. H. L.

(2017) Changing teachers’ TPACK and design beliefs through the Scaffolded TPACK lesson design model (STLDM). Learning: Research and Practice 3(2): 114–129.

11.

Chee

Y. S.

Chai

C. S.

(2017) Quality 21st century learning: A framework for 21st century education. In: Chai

C. S.

Koh

H. L. J.

Teo

Y. H.

(eds) Technology-enhanced 21st century learning, Singapore: Pearson, pp. 1–14.

12.

Collins

Halverson

(2010) The second educational revolution: Rethinking education in the age of technology. Journal of Computer Assisted Learning 26(1): 18–27.

13.

Dede

(2010) Comparing frameworks for 21st century skills. In: Bellanca

Brandt

(eds) 21st century skills: Rethinking how students learn, Bloomington, IN: Solution Tree Press, pp. 51–75.

14.

Dobozy

Campbell

(2016) The complementary nature of learning design and TPACK. In: Dalziel

(ed.) Learning design: Conceptualizing a framework for teaching and learning online, New York, NY: Routledge, pp. 96–116.

15.

Drummond

Sweeney

(2017) Can an objective measure of technological pedagogical content knowledge (TPACK) supplement existing TPACK measures? British Journal of Educational Technology 48(4): 928–939.

16.

Geisinger

K. F.

(2016) 21st century skills: What are they and how do we assess them? Applied Measurement in Education 29(4): 245–249.

17.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2010) Multivariate data analysis, 7th ed. New York, NY: Prentice Hall.

18.

Harris, J., Grandgenett, N., & Hofer, M. (2010). Testing a TPACK-based technology integration assessment rubric. In C. Crawford, D. A. Willis, R. Carlsen, I. Gibson, K. McFerrin, J. Price, & R. Weber (Eds.), Proceedings of the society for information technology & teacher education international conference 2010 (pp. 3833–3840). Chesapeake, VA: AACE.

19.

Harris

Phillips

Koehler

Rosenberg

(2017) TPCK/TPACK research and development: Past, present, and future directions. Australasian Journal of Educational Technology 33(3): i–viii. doi:10.14742/ajet.3907.

20.

Heitink

Voogt

Fisser

Verplanken

van Braak

(2017) Eliciting teachers’ technological pedagogical knowledge. Australasian Journal of Educational Technology 33(3): 96–109. doi:10.14742/ajet.3505.

21.

Howland

J. L.

Jonassen

D. H.

Marra

R. M.

(2014) Meaningful learning with technology, 4th ed. Essex, England: Pearson.

22.

Jang

S. J.

Tsai

M. F.

(2013) Exploring the TPACK of Taiwanese secondary school science teachers using a new contextualized TPACK model. Australasian Journal of Educational Technology 29(4): 566–580.

23.

Kereluik

Mishra

Fahnoe

Terry

(2013) What knowledge is of most worth: Teacher knowledge for 21st century learning. Journal of Digital Learning in Teacher Education 29(4): 127–140.

24.

Kirschner

P. A.

(2015) Do we need teachers as designers of technology enhanced learning? Instructional Science 43(2): 309–322.

25.

Koehler

M. J.

Mishra

Yahya

(2007) Tracing the development of teacher knowledge in a design seminar: Integrating content, pedagogy and technology. Computers & Education 49(3): 740–762.

26.

Koehler

M. J.

Mishra

Kereluik

Shin

T. S.

Graham

C. R.

(2014) The technological pedagogical content knowledge framework. In: Spector

J. M.

Merill

M. D.

Elen

Bishop

M. J.

(eds) Handbook of research on educational communications and technology, New York, NY: Springer, pp. 101–111.

27.

Koh

J. H. L.

Chai

C. S.

Wong

Hong

H. Y.

(2015) Technological pedagogical content knowledge (TPACK) and design thinking: A framework to support ICT lesson design for 21st century learning. The Asia Pacific Education Researcher 24(3): 535–543.

28.

Koh

J. H. L.

Chai

C. S.

Hong

H. Y.

Tsai

C. C.

(2015) A survey to examine teachers’ perceptions of design dispositions, lesson design practices and their relationships with technological pedagogical content knowledge (TPACK). Asia Pacific Journal of Teacher Education 43(5): 378–391.

29.

Koh

J. H. L.

Chai

C. S.

Lim

W. Y.

(2017) Teacher professional development for TPACK-21CL: Effects on teacher ICT integration and student outcomes. Journal of Educational Computing Research 55(2): 172–196.

30.

Kramarski

Michalsky

(2010) Preparing preservice teachers for self-regulated learning in the context of technological pedagogical content knowledge. Learning and Instruction 20(5): 434–447.

31.

Lee

M. H.

Tsai

C. C.

(2010) Exploring teachers’ perceived self efficacy and technological pedagogical content knowledge with respect to educational use of the World Wide Web. Instructional Science 38(1): 1–21.

32.

Mishra

Koehler

M. J.

(2006) Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record 108(6): 1017–1054.

33.

Partnership for 21st Century Skills. (2009). Framework for 21st century learning. Retrieved from http://www.p21.org/documents/P21_Framework.pdf.

34.

Pringle

R. M.

Dawson

Ritzhaupt

A. D.

(2015) Integrating science and technology: Using technological pedagogical content knowledge as a framework to study the practices of science teachers. Journal of Science Education and Technology 24(5): 648–662.

35.

Razzouk

Shute

(2012) What is design thinking and why is it important? Review of Educational Research 82(3): 330–348.

36.

Rosenberg

Hammer

Phelan

(2006) Multiple epistemological coherences in an eighth-grade discussion of the rock cycle. The Journal of the Learning Sciences 15(2): 261–292.

37.

Schmidt

D. A.

Baran

Thompson

A. D.

Mishra

Koehler

M. J.

Shin

T. S.

(2009) Technological pedagogical content knowledge (TPACK) the development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education 42(2): 123–149.

38.

Tee

M. Y.

Lee

S. S.

(2011) From socialisation to internalisation: Cultivating technological pedagogical content knowledge through problem-based learning. Australasian Journal of Educational Technology 27(1): 89–104.

39.

Thomas

Brown

J. S.

(2011) A new culture of learning: Cultivating the imagination for a world of constant change, Lexington, KY: CreateSpace.

40.

Tondeur

van Braak

Sang

Voogt

Fisser

Ottenbreit-Leftwich

(2012) Preparing pre-service teachers to integrate technology in education: A synthesis of qualitative evidence. Computers & Education 59: 134–144. doi:10.1016/j.compedu.2011.10.009.

41.

Tondeur

Aesaert

Pynoo

Braak

Fraeyman

Erstad

(2017) Developing a validated instrument to measure preservice teachers’ ICT competencies: Meeting the demands of the 21st century. British Journal of Educational Technology 48(2): 462–472.

42.

Tondeur

van Braak

Ertmer

P. A.

Ottenbreit-Leftwich

(2017) Understanding the relationship between teachers’ pedagogical beliefs and technology use in education: a systematic review of qualitative evidence. Educational Technology Research and Development 65(3): 555–575.

43.

Valtonen

Sointu

Kukkonen

Kontkanen

Lambert

Mäkitalo-Siegl

(2017) TPACK updated to measure pre-service teachers’ twenty-first century skills. Australasian Journal of Educational Technology 33(3): 15–31. doi:10.14742/ajet.3518.

44.

Voogt

Roblin

N. P.

(2012) A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. Journal of Curriculum Studies 44: 299–321. doi:10.1080/00220272.2012.668938.

45.

Voogt

Erstad

Dede

Mishra

(2013) Challenges to learning and schooling in the digital networked world of the 21st century. Journal of Computer Assisted Learning 29(5): 403–413.

46.

Yeh

Y. F.

Hsu

Y. S.

H. K.

Hwang

F. K.

Lin

T. C.

(2014) Developing and validating technological pedagogical content knowledge‐practical (TPACK‐practical) through the Delphi survey technique. British Journal of Educational Technology 45(4): 707–722.