Teaching higher order thinking skills to gifted students: A meta-analysis

Abstract

The current study examined the effects of higher order thinking skills (HOTS) interventions with gifted students in Taiwan. A total of 25 studies published between 1997 and 2017 were included. Twenty-nine effect sizes were extracted for the 25 studies. The small number of existing studies indicates a lack of scholarly attention to HOTS in gifted education in Taiwan in the past two decades. On the other hand, the effect sizes, ranged from 0.26 to 2.01, with a mean of 0.78 and standard deviation of 0.39, showed moderately large effect sizes for these interventions, which can be interpreted as evidence for general effectiveness. Subgroup analyses indicated that intervention effects did not vary significantly by grade level, type of program, intervention dosage, and type of dissemination. However, a statistically significant difference was found between the effect sizes in different types of instructional design (i.e. stand-alone HOTS unit vs. integrated HOTS unit). Implications are discussed.

Keywords

Gifted education higher order thinking skills meta-analysis gifted students evidence-based practices creative thinking problem-solving critical thinking

In today’s fast-changing world, students are required to not only acquire knowledge but also learn skills that help them synthesize and generate knowledge (Ananiadou and Claro, 2009). Thus, higher order thinking skills (HOTS) are highly valued. According to the World Economic Forum (2016), large global firms forecasted that by 2020, creativity will be one of the top three most desired job skills, along with critical thinking and complex problem-solving skills. In recent years, many education systems, including that of Taiwan, have been responding to these emergent demands (Ananiadou and Claro, 2009). While many studies in North America have shown that HOTS can be enhanced via instructional interventions (e.g. Abrami et al., 2008; Yasin and Yunus, 2014), there is no study that demonstrates the effectiveness of HOTS interventions in Taiwan, where HOTS may be viewed differently (Chan, 2013). The main purpose of this study is to explore the nature and the effectiveness of HOTS instructional interventions conducted in Taiwanese K-12 settings.

An overview of gifted education in Taiwan

Gifted education in the Taiwanese public K-12 system started in 1973 when the Taiwanese Ministry of Education launched a pilot program for intellectually gifted elementary students (Wu, 2013). Today, students identified as gifted by local school districts in Taiwan are served through one of the following two schemes. First, there are school-based pullout resource classrooms—where gifted students remain mostly in regular classrooms while receiving 6–10 hours of enriched curriculum weekly in a resource classroom. Second, there are district-based gifted programs where students are enrolled in part-time programs coordinated by several schools via resource sharing. These part-time programs are typically held during weekends, winter vacation, or summer vacation and may last for several weeks. The content of the programs is flexible and may involve independent studies of natural sciences, social sciences and humanities, leadership, and creativity. Generally speaking, enrichment is the primary form of gifted education in Taiwan. Only a few students each year are qualified for grade skipping or other forms of acceleration.

Embracing evidence as a foundation for practice in Taiwan

Gifted education policies in Taiwan have drawn inspiration from the North American system since their onset (Wu, 2013). Recently, evidence-based practices (Collinson et al., 2009) have emerged as critical for decision-making in Taiwan.

In 2002, the United States launched the No Child Left Behind Act (NCLB), which altered the discourse of evidence-based practices in education (Edyburn, 2010). NCLB stressed efforts to encourage empirically supported pedagogical techniques that could meet the growing demands for increased accountability and improved outcomes through rigorous scientific research (Robinson et al., 2007). Evidence-based practices have since influenced many professions, including gifted education (Adelson and Matthews, 2019; Callahan and Moon, 2007; Cook and Odom, 2013; Groccia and Buskist, 2011; McBee et al., 2018; Reis et al., 2007). Although evidence-based practices are considered optimal for teachers and policy makers in Taiwan (Hung et al., 2015; Tseng et al., 2011), there is a lack of evidence-based inquiries in gifted education.

The importance of meta-analysis on gifted education

Steenbergen-Hu and Olszewski-Kubilius (2016) argued that meta-analyses can make unique contributions to the field of gifted education. First, the findings are reliable since the results derive from systematic and replicable steps (see also Fidler, 2010). Second, by summarizing the current status of evidence, meta-analyses provide opportunities for researchers to situate their insights within the big picture. Third, meta-analyses allow researchers to examine the effects of a large number of independent variables and potential moderating influences. Similarly, Vaughn et al. (1991) have argued that meta-analyses are a more comprehensive method for conducting program evaluations in gifted education given that one could examine a wide range of independent and moderating variables simultaneously.

In the past few decades, a large number of studies on gifted education have been conducted in Taiwan (Wu, 2000; Wu and Cho, 1993; Yu et al., 2017), most of which have shown statistically significant results. However, as Warner (2008) indicated, statistical significance alone is not a warrant of practical usefulness, unlike effect size, which allows researchers and practitioners to look at the magnitude of the obtained difference between the sample mean and the hypothesized population mean. In addition, existing meta-analyses on the effects of intervention in the Taiwanese context are scarce. A review of the literature only identified one meta-analysis published in 2002, which investigated the overall effect sizes of selected instructional implementation on creative thinking performance (Peng, 2002).

Current study

While the current study focuses on the effects of empirical studies conducted in Taiwan, it nevertheless sheds light on programmatic features that can apply to the competency-focused, process-based 21st-century pedagogical approaches more generally. Through a meta-analysis, this study seeks to answer the following questions:

What is the state of HOTS studies in the field of gifted education in Taiwan over the past two decades (1997–2017)?

How effective were the HOTS interventions for gifted students in Taiwan?

What study characteristics produced better outcomes?

Conceptual and operational definitions

Gifted students

The identification of giftedness in Taiwan is based on multiple criteria and follows a systematic procedure (Kao, 2012). According to The Implementation Regulations of Disability and Gifted Students Identification Process in Taiwan (2013), students identified as gifted must meet the following criteria: (a) scored at least 2 standard deviations above the mean or above the 97th percentile on an approved standardized individual intelligence test and (b) the existence of evidence regarding an applicant’s advanced learning needs (e.g. teachers’ responses on observational scales and recommendation letters). In the present study, gifted students refer to those who had been identified during their compulsory education (grades 1–12).

Higher order thinking skills

The term “higher order thinking” has been referred to variously as reflective thinking, higher level reasoning, synthesis, evaluation, creative thinking, critical thinking, and problem-solving (Bloom et al., 1956; Glaser, 1984; Lewis and Smith, 1993; Newmann, 1988; Norris and Ennis, 1989). Despite critiques of the overlapping terminologies and unclear definitions of higher order thinking, most researchers have agreed that it involves complicated cognitive activities such as reasoning and problem-solving, integrating and synthesizing information, and thinking creatively and yielding multiple solutions (Barak and Dori, 2009; Grossen, 1992; Lewis and Smith, 1993; Philp, 1985; Sternberg and Lubart, 1996; Zohar, 2006). In the current study, HOTS is summarized and partitioned into four abilities: creative thinking, critical thinking, problem-solving ability, and reflective thinking.

Method

Shokraneh (2019) recommended the documentation of strategies and steps adopted in a meta-analysis to facilitate reproducibility or new updates of a meta-analysis. In this study, the analytical strategies and steps adopted were as the following.

Search for eligible studies

To ensure a full coverage of empirical studies on the topic, we searched peer-reviewed journal articles, master’s theses, and doctoral dissertations through the following electronic databases: Airiti Library Collection, Index to Taiwan Periodical Literature System, ERIC Database, National Digital Library of Theses and Dissertations in Taiwan, and Google Scholar. The time span of the study search was set from 1997 to 2017, a period of 20 years.

The literature search was primarily conducted using the following keywords: gifted students, higher order thinking, thinking strategies, thinking skills, gifted programs, creative thinking, critical thinking, problem-solving skills, reflective thinking, and all possible combinations and permutations of these terms. Studies containing these keywords in their titles or abstracts were initially selected and individually reviewed to find additional references. Moreover, we conducted more specific searches focusing on research published in the following Taiwanese gifted education and special education journals: Bulletin of Special Education (ISSN: 1026-4485), Journal of Gifted Education (ISSN: 1561-3801), Journal of Special Education (ISSN: 1561-3798), and Special Education Forum (ISSN: 1994-1935). Lastly, we examined the references listed in the included studies for additional studies that could also be included. The first author held a professorship in gifted education, whereas the second author was a doctoral candidate in gifted education. Both the first and second authors were well trained to conduct a systematic literature search.

Study inclusion and exclusion criteria

Once the initial search of studies was completed, these studies were further screened in the following three stages.

Screening I

An initial screening was conducted by examining the following criteria. (a) Subject of the study: The subject of a selected study needed to speak to HOTS interventions or programming for gifted students in a K-12 setting. (B) Outcomes of the study: The outcomes of a selected study had to present changes in HOTS performance among the participants. Students’ performance was used as the dependent variable in this meta-analysis (Henfield et al., 2017). (c) Completeness and nonredundancy: To avoid a single study exerting a disproportionate influence on the overall results, the content of a selected study was reviewed to ensure that there was no overlap with other studies that derived from the same source of research findings. If the same study was reported in more than one article, only the most complete version would be included for this analysis.

Screening II

After the initial screening, we further selected studies that consisted of appropriate research methods and the establishment of comparison groups. First, an experimental design or a quasi-experimental design was required. Studies selected for this meta-analysis had to include pre- and posttest measurements. Second, studies selected had to include appropriate comparison groups. The treatment groups and control groups had to be matched in terms of subjects’ major aptitudes. Appropriate comparisons included, for example, specific teaching methods for gifted students versus general teaching methods for gifted students (Kim, 2016; Steenbergen-Hu and Moon, 2011).

Screening III

Finally, measurement instruments and statistical results were examined before the final inclusion of the study. To be included, the study must (a) use standardized tests to measure HOTS and (b) contain sufficient statistical information for effect size extraction (e.g. means and standard deviations, p values, analysis of variance [ANOVA] tables) or other essential statistics (e.g. t values, F values).

The initial search resulted in 245 studies. Twenty-five studies met all of our inclusive criteria (see Figure 1), including 17 (68%) studies on creative thinking, 6 (24%) studies on critical thinking, and 6 (24%) studies on problem-solving. Among the included studies, three studies provided results for both creative thinking and problem-solving and one study provided results for both creative thinking and critical thinking.

Figure 1.

Flow diagram of study selection process.

Study coding

Pilot coding was first performed with three studies that led to some revisions to the initial coding sheet. The final code set contained the following two dimensions: (a) general features of the study and (b) format of the data in the study.

The first dimension included the following information: (a) contributor(s) of the article, (b) publication year, (c) type of article (i.e. journal article, thesis, or dissertation), (d) research design (i.e. experimental, quasi-experimental), (e) sample size, (f) demographics of the research participants, (g) program type (e.g. pullout gifted resource class, district-based gifted program), (h) treatment design and implementation, and (i) outcome measures (e.g. type of instrument, content areas to be measured).

The second dimension included the following information: (a) mean, standard deviation, and sample size of each study, (b) test statistics and associated degrees of freedom, and (c) effect size. Since most studies contained multiple outcome variables, only outcomes that directly assessed HOTS were coded. When difficulties in coding arose, further discussion was employed to resolve these difficulties. Interrater reliabilities were good to excellent (Cohen’s κ = 0.80), and differences were settled through discussion.

Effective size calculation

Effect sizes were used for this meta-analysis to summarize the findings of the selected studies. An effect size speaks to the strength of the relationship between two variables and reflects the magnitude of a treatment effect. When calculating the effect sizes, Lipsey and Wilson (2001) argued that the effect sizes are biased if they are based on small sample sizes, particularly for samples of fewer than 20 participants. Since the samples in many studies of gifted students are quite small in this meta-analysis, Hedges’ g was chosen as the effect size index to eliminate the influence of sample size (Borenstein, 2009). Generally, effect sizes of the research outcomes included in the studies were calculated based on raw means, standard deviations, and sample size in the studies. We subtracted the mean of one group from the other (M1 − M2) and divided by pooled standard deviation. If the comparison involved a single sample tested pre- and postmeasurements, the d-index was calculated first. The d-index would then be converted to Hedges’ g. Cohen’s (1988) criteria were used for interpretation of standardized mean differences and summarized effect sizes as small (≤0.20), medium (0.5), and large (≥0.80). Positive effect sizes were interpreted as treatment groups having stronger results than control groups.

Borenstein et al. (2009) indicated that since interventions and outcomes measures across different studies are not exactly the same, a random-effects model is preferred over fixed-effects model in a meta-analysis. Therefore, we decided to use the random-effects model to estimate the effect size. Comprehensive Meta-Analysis 3.0 software package (CMA) was used for effect size synthesis and moderator analyses. Moreover, since most of the studies contained more than one effect size, the average of these results was calculated to represent each outcome measure, as recommended by Cooper (2010).

Heterogeneity analysis

A random-effects model allows true effect sizes to vary from study to study. In this case, the observed effects varied from one to another for two reasons. One was the real heterogeneity in effect size (i.e. treatment characteristics), and the other was the within-study error (i.e. sample error). Heterogeneity analysis is used to examine whether sampling error alone might be responsible for the variance among the effect sizes (Borenstein and Higgins, 2013; Cooper, 2010; Higgins and Green, 2008; Steenbergen-Hu and Olszewski-Kubilius, 2016). A set of heterogeneity statistics were computed through the CMA software.

We examined both Q _T and I ² statistics to decide whether there was significant variability across studies. The Q _T statistic is a heterogeneity statistic commonly used in assessing the collection of effect sizes (Borenstein et al., 2009; Steenbergen-Hu and Olszewski-Kubilius, 2016). A significant Q _T value indicates large variability across the effect sizes that is greater than what is likely to have resulted from subject-level sampling error alone. Statistically significant heterogeneity statistics indicate that moderator variables may account for variability in effect size (Lipsey and Wilson, 2001). However, Q _T is known for being sensitive to sample sizes (Abrami et al., 2008). Higgins and Thompson (2002) recommend the use of I ² as a complement to determine how much of the variance between studies is due to true variance rather than sampling error. I ² is the ratio of between-study variance and total variance, which represents the true heterogeneity of total variance across the observed effect estimates. It ranges from 0% to 100%. Based on Higgins and Green’s (2008) guide for I ² interpretation, an I ² of 0–40% indicates “likely not important,” 30–60% is associated with “possible moderate heterogeneity,” 50–90% is associated with “possible substantial heterogeneity,” and 75–100% is associated with “considerable heterogeneity.” Heterogeneity was considered significant for p values of a Q _T statistic less than 0.05 and I ² greater than 50%.

Subgroup analysis

To identify variables that may affect the interventions, we categorized the effect sizes into several groups and conducted subgroup analyses with random-effects ANOVA-like procedures for meta-analysis (Henfield et al., 2017). As Lipsey and Wilson (2001) suggested, there are three types of independent variables commonly used for meta-analysis: substantive variables, method variables, and extrinsic variables. Based on our interests and research purpose, these variables were categorized within the subgroup analysis. The substantive variables included the grade level, intervention dosage, implementation approach, and type of program. The extrinsic variable was the type of dissemination (i.e. published/unpublished) (Hedges and Olkin, 1985; Henfield et al., 2017). The computations produced from a subgroup analysis compartmentalized the total variance (Q _T) into between-group effect (Q _B) and within-group effect (Q _W). Statistically significant differences between the group means would indicate that the effects of HOTS interventions were influenced by the effectiveness of interventions by defined subgroups (Ellis, 2010; Pertti et al., 2008).

Assessing publishing bias

Publication bias refers to the fact that significant results are more likely to be published than insignificant ones, which may lead to an overestimation of the effects found in meta-analyses (Rothstein et al., 2005). To identify possible publication bias, visual inspection of a funnel plot and the trim-and-fill procedure were used in the present study (Borenstein et al., 2009). The funnel plot provides a roughly funnel-shaped symmetric distribution, which represents the magnitude of the effect of each trial compared with a measure of its size, including the standard error. Symmetrical plots can be interpreted as having an absence of publication bias. Asymmetrical data can be adjusted by using the trim-and-fill procedure on a precision plot (Duval and Tweedie, 2000). In the trim-and-fill procedure, extreme effect sizes from the skewed side of the funnel plot are removed and projected missing effect sizes are imputed. The procedure is an iterative process that adjusts the overall effect size by identifying the number of missing studies that would balance the plot to provide an unbiased estimate of effect size. The degree of difference between the adjusted and the observed mean effect size indicates the magnitude of the impact of the publication bias on the mean effect size (Steenbergen-Hu and Olszewski-Kubilius, 2016).

Results

Among the 25 studies, 14 studies were unpublished master’s theses and 11 studies were journal articles. The majority (22 studies) were conducted in elementary school settings (grades 3–6) and the rest (3 studies) were conducted in junior high school settings (grades 7 and 8). Among the 22 studies conducted in elementary school settings, 16 included creative thinking skills, 6 included problem-solving skills, and 4 included critical thinking skills. Three studies examined both creative thinking and problem-solving skills. Among the three studies conducted in junior high school settings, two focused on critical thinking skills, whereas the remaining one focused on creative thinking skills.

In terms of the research designs employed by the studies, 22 used a quasi-experimental, nonequivalent pre–post-test design, while the remaining 3 used a single group pre–post-test design. The participants in both experimental and control groups in these studies were all enrolled in gifted programs. The range of the sample sizes was between 11 and 76, while the pooled sample size was 995 (experimental group = 511, control group = 484). Table 1 provides an overview of the descriptive information of the included studies. The following further describes the characteristics of the intervention programs and the characteristics of the studies.

Table 1.

Summary of inclusive studies.

	Author(s)	Year	Intervention program	Intervention dosage (sessions)	Type of programs	Type	Grade level	Comparison group	N	Outcome measure/instrument	Content
1	Chang	2011	Future study in environment issue	34/high	PRC	P	5-6	Gifted/one-group pre–post-test	11	CT/CTT1	SU
2	Chao	2004	Scientific creative games training program	24/moderate	PRC	UP	3	Gifted/quasi	57	CRT/CAP	SU
3	Cheng	2015	CoRT active thinking program	20/low	PRC	UP	5–6	Gifted/quasi	33	PS/TPS	SU
4	Chiang	2005	Creative problem-solving training (CPS)	38/high	DBGP	P	3–6	Gifted/quasi	68	CRT/TTCT	SU
4	Chiang	2005	Creative problem-solving training (CPS)	38/high	DBGP	P	3–6	Gifted/quasi	68	PS/TPS	SU
5	Chiu	2008	Creative writing program	20/low	PRC	UP	4	Gifted/quasi	48	CRT/NCTT	IU
6	Chou	2010	Critical thinking program	16/low	PRC	UP	7	Gifted/quasi	66	CT/CTT1	IU
7	Chou	2004	CPS	32/high	PRC	UP	7	Gifted/quasi	58	CRT/NCTT	SU
8	Lee, C. J.	2010	LEGO computer theme	24/moderate	PRC	UP	5–6	Gifted/quasi	25	PS/TPS	SU
9	Lee and Chang	2005	Metaphor mode instruction	20/low	PRC	P	8	Gifted/quasi	30	CT/QDTCT	IU
10	Li and Cheng	2007	Drama instruction	26/moderate	PRC	P	4	Gifted/quasi	58	CRT/NCTT	SU
11	Liang	2003	Verbal logical thinking training	16/low	PRC	UP	4, 6	Gifted/quasi	55	CT/CTT1	SU
12	Liao	2009	Mind mapping program	12/low	PRC	UP	5	Gifted/one-group pre–post-test	23	CRT/TTCT	IU
13	Lin	2001	Critical thinking program	20/moderate	DBGP	UP	4-5	Gifted/quasi	76	CT/CCTTX	SU
14	Tsai	2010	DISCOVERY program	26/moderate	PRC	P	5	Gifted/quasi	51	PS/TPS	SU
	Tsai	2010	DISCOVERY program	26/moderate	PRC	P	5	Gifted/quasi	51	CRT/TTCT	SU
15	Tseng	2008	Creative thinking teaching with picture books	10/low	PRC	UP	3	Gifted/quasi	55	CRT/NCTT	SU
16	Wang	2010	Mathematic creative thinking program	16/low	PRC	UP	3	Gifted/one-group pre–post-test	13	CRT/TTCT	IU
17	Wang	2005	Philosophical thinking program	32/high	PRC	P	5	Gifted/quasi	37	CT/CTT1	SU
	Wang	2005	Philosophical thinking program	32/high	PRC	P	5	Gifted/quasi	37	CRT/CAP	SU
18	Wang and Chang	2014	Game design program	16/low	PRC	P	3–4	Gifted/quasi	35	CRT/TTCT	SU
19	Wang	2003	“Six Thinking Hat” training program	22/moderate	PRC	UP	5	Gifted/quasi	28	PS/TPS	SU
20	Wu	2012	Lateral thinking program	20/low	PRC	UP	5–6	Gifted/quasi	24	CRT/TTCT	SU
21	Xie and Chang	2016	CPS	28/moderate	PRC	P	5–6	Gifted/quasi	26	CRT/TTCT	SU
	Xie and Chang	2016	CPS	28/moderate	PRC	P	5–6	Gifted/quasi	26	PS/CS-CPS	SU
22	Yang and Chang	2011	Linear and intuitive thinking program	20/low	PRC	P	6	Gifted/quasi	34	CRT/TTCT	SU
23	Yu	2003	CoRT active thinking program	20/low	PRC	UP	5	Gifted/quasi	30	CRT/NCTT	SU
24	Yu and Chang	2016	Program of future imagination for school	24/moderate	PRC	P	5	Gifted/quasi	24	CRT/TTCT	SU
25	Zeng and Chang	2013	Intel K-12 Educator’ Thinking Tools	30/moderate	PRC	P	5–6	Gifted/quasi	30	CRT/TTCT	SU

Note: Type of programs: PRC: pull-out resource classroom; DBGP: district-based gifted program. Type of studies: P: published; UP: unpublished. Outcome measure: CRT: creative thinking; CT: critical thinking; PS: problem-solving. Measurement: TTCT: Torrance Tests of Creative Thinking; CAP: Creativity Assessment Packet; NCTT: New Creative Thinking Test; CTT1: Critical Thinking Test–Level 1; QDTCT: The Questionnaire of Dispositions Toward Critical Thinking; CCTTX: Cornell Critical Thinking Test–Level X; TPS: Test of Problem Solving; CS-CPS: Children Scientific Creative Problem-Solving. Content: SU: stand-alone unit; IU: integrated unit.

Characteristics of intervention

In terms of the intervention programs, diverse gifted education programs were used to improve different thinking skills. For creative thinking skills, the most common teaching methods in the primary studies were the Creative Problem Solving (CPS) program (three studies) (Chiang, 2005; Chou, 2004; Xie and Chang, 2016). Similarly, the CPS program was the most commonly used for improving problem-solving abilities (two studies) (Chiang, 2005; Xie and Chang, 2016). For critical thinking skills, the most widely used teaching methods were critical thinking skills programs (two studies; Chou, 2010; Lin, 2001) that involved five stages: identifying problems, deduction, induction, exploration, and evaluation.

Concerning the intervention dosage, the intervention period varied from 10 sessions to 38 sessions. Studies with the intervention lasting from 10 sessions to 20 sessions were coded as low (12 studies), from 21 sessions to 30 sessions as moderate (9 studies), and more than 31 sessions as high (4 studies). Overall, intervention sessions were implemented one to two times per week for 40–80 min.

With regard to outcome variables, diverse standardized instruments were used to measure the HOTS. For creative thinking performance, the most common outcome variables in the primary studies were standardized aptitude test results, such as the Torrance Tests of Creative Thinking, the Creativity Assessment Packet, and the New Creative Thinking Test. For critical thinking performance, the most widely studied variables were produced from the Critical Thinking Test–Level 1, the Questionnaire of Dispositions Toward Critical Thinking, and the Cornell Critical Thinking Test–Level X. Finally, the Test of Problem Solving and the Children Scientific Creative Problem Solving were used for estimating problem-solving ability performance.

Random-effects model results

In total, 29 effect sizes were extracted from the 25 studies. Table 2 presents the effect sizes in individual studies analyzed and relevant statistical information, including Hedges’ g, standard error, variance, the lower limit and upper limit of the 95% confidence interval (CI) for the effect size, Z value, and the p value. The random-effects model demonstrated that the effect sizes ranged from 0.26 to 2.01. A medium effect was found on the mean effect size (g = 0.76, 95% CI [0.62, 0.90]), which implied a general effectiveness of the HOTS interventions. The grand effect size between the treatment and control groups was significantly different from zero (Z = 10.48, p < 0.001). These results revealed that students receiving teaching interventions performed better than those in the control group on HOTS measures.

Table 2.

Summary of combined effect sizes for HOTS.^a

Study		Year	Effect sizes	SE	Lower limit	Upper limit	Z	p
1	Chang	2011	0.61	0.32	0.01	1.24	1.92	0.06
2	Chao	2004	1.47	0.31	0.87	2.08	5.77	0.00
3	Cheng	2015	0.69	0.33	0.05	1.33	2.10	0.04
4	Chiang	2005	0.82	0.25	0.33	1.31	3.25	0.00
4	Chiang	2005	0.44	0.24	0.04	0.92	1.81	0.07
5	Chiu	2008	0.58	0.29	0.00	1.15	1.96	0.05
6	Chou	2010	0.53	0.25	0.03	1.01	2.08	0.04
7	Chou	2004	2.01	0.33	1.38	2.65	6.20	0.00
8	Lee	2010	0.36	0.39	0.41	1.12	0.10	0.37
9	Lee and Chang	2005	1.07	0.39	0.31	1.82	2.76	0.01
10	Li and Cheng	2007	0.55	0.27	0.01	1.08	2.01	0.05
11	Liang	2003	0.66	0.28	0.13	1.20	2.41	0.02
12	Liao	2009	1.07	0.26	0.56	1.57	4.13	0.00
13	Lin	2001	0.37	0.23	0.80	0.83	1.61	0.11
14	Tsai	2010	0.26	0.26	0.25	0.78	1.00	0.32
14	Tsai	2010	0.61	0.24	0.13	1.08	2.50	0.01
15	Tseng	2008	0.76	0.28	0.22	1.31	2.75	0.01
16	Wang	2010	0.90	0.32	0.28	1.53	2.83	0.01
17	Wang	2005	1.36	0.36	0.66	2.06	3.79	0.00
17	Wang	2005	0.62	0.33	0.03	1.28	1.88	0.06
18	Wang and Chang	2014	1.46	0.39	0.69	2.22	3.72	0.00
19	Wang	2003	0.81	0.39	0.05	1.57	2.08	0.04
20	Wu	2012	1.21	0.47	0.28	2.13	2.56	0.01
21	Xie and Chang	2016	0.77	0.16	0.02	1.55	1.92	0.06
21	Xie and Chang	2016	0.39	0.15	0.37	1.14	1.00	0.32
22	Yang and Chang	2011	0.69	0.35	0.00	1.37	1.96	0.05
23	Yu	2003	0.69	0.34	0.03	1.36	2.04	0.04
24	Yu and Chang	2016	0.53	0.41	0.26	1.33	1.32	0.19
25	Zeng and Chang	2013	0.47	0.36	0.23	1.19	1.31	0.19
	Total		0.76	0.07	0.62	0.90	10.48	0.00

HOTS = higher order thinking skills; SE = standard error.

^a Heterogeneity: Q value = 43.18, p = 0.03; I ² = 35%.

The test for heterogeneity yielded a Q statistic of 43.18 (p < 0.05), indicating some variability among the effect sizes. A further investigation into the I ² statistic estimation resulted in 25%. However, according to Higgins and Green’s guideline, an I ² of 0–40% indicates that it is likely not important. Hence, the heterogeneous analysis did not meet the criteria that we predetermined.

Outcome analysis

To examine the effect of selected instructional implementation on different HOTS, effect sizes were grouped by different dimensions of HOTS (see Table 3). Three types of HOTS were treated here as dependent variables, including creative thinking skills (k = 17), critical thinking skills (k = 6), and problem-solving abilities (k = 6). Under a random-effects model, the effects of HOTS interventions were significantly different from zero (p < 0.01) for creative thinking, critical thinking, and problem-solving. In addition, large effect sizes were found on creative thinking skills (g = 0.90, 95% CI [0.71, 1.10]), medium effects were found on critical thinking skills (g = 0.71, 95% CI [0.43, 0.99]), and small effects were found on problem-solving abilities (g = 0.47, 95% CI [0.21, 0.73]). The effect sizes were significantly higher for creative thinking skills than for critical thinking skills and problem-solving abilities (Q _B = 6.98, p < 0.05). The results supported the conclusion that the instructional interventions examined in this meta-analysis had a significantly positive effect on different types of HOTS.

Table 3.

Mean effect sizes on the three categories of HOTS.^a

Outcome	k	Hedges’ g	SE	Lower limit	Upper limit	Z	Q _B
Measure category							6.98^b
Creative thinking	17	0.90	0.10	0.71	1.10	8.90^c
Critical thinking	6	0.71	0.14	0.43	0.99	4.92^c
Problem-solving	6	0.47	0.13	0.21	0.73	3.58^c

HOTS = higher order thinking skills; SE = standard error.

^a k = Number of effect sizes included in the analysis.

^b p < 0.05.

^c p < 0.01.

Subgroup analyses

Several coding characteristics (i.e. grade level, type of program, intervention dosage, instructional design, and type of dissemination) were selected as independent variables for a subgroup analysis. These variables are subsequently discussed. For a summary of all subgroup analysis results, see Table 4.

Table 4.

Results of subgroup analysis.^a

Moderator	k	Hedges’ g	SE	Lower limit	Upper limit	Z	Q _B
Grade level							3.40
Grades 3–4	8	0.84	0.13	0.58	1.10	6.28^b
Grades 5–6	16	0.70	0.09	0.53	0.87	8.16^b
Grades 7–8	3	1.21	0.23	0.27	2.15	2.53^b
Type of program							2.94
Pullout gifted resource class	26	0.82	0.08	0.66	0.98	10.10^b
District-based gifted program	3	0.54	1.14	0.26	0.82	3.81^b
Intervention dosage							3.41
Low	12	0.76	0.09	0.57	0.94	8.20^b
Moderate	11	0.61	0.10	0.41	0.81	5.95^b
High	6	1.04	0.22	0.61	1.48	4.73^b
Instructional design							7.01^b
Stand-alone unit	24	0.70	0.07	0.56	0.83	10.12^b
Integrated unit	5	1.12	0.25	0.63	1.60	4.53^b
Study’s dissemination							2.67
Published	15	0.69	0.08	0.53	0.85	8.47^b
Unpublished	14	0.90	0.13	0.64	1.16	6.75^b

SE = standard error.

^a k = Number of effect sizes included in the analysis.

^b p < 0.01.

Grade level

For grade level, effect sizes were grouped by studies including grades 3–4 (k = 8), grades 5–6 (k = 16), and grades 7–8 (k = 3). With the random-effects model, the effect sizes for students who are gifted in grades 3–4 and 5–6 in elementary school and grades 7–8 in junior high school were significantly greater than zero. Among the studies, the largest effect size was observed for gifted students in junior high school (g = 1.21), followed by the effect size for grades 3–4 (g = 0.84) and grades 5–6 (g = 0.70). However, the mean effect size of studies with different grade level students did not differ from each other (Q _B = 3.40, p > 0.05). The results suggested HOTS improvement across grade levels in this meta-analysis.

Type of program

For the type of program, effect sizes were grouped by studies including pullout gifted resource classes (k = 26) versus district-based gifted programs (k = 3). In examining the type of program as a moderator, the effect sizes of both pullout resource class and district-based gifted program were significantly different from zero (p < 0.01), with average effect sizes of 0.82 and 0.54, respectively. There was no significant difference between the effect sizes in different types of programs (Q _B = 2.94, p > 0.05). The results suggested that the type of program did not influence the effect sizes of the intervention.

Intervention dosage

For intervention dosage, effect sizes were grouped into low (k = 12), moderate (k = 10), and high (k = 7) dosage of intervention. With the random-effects model, all the effect sizes of low, moderate, and high dosage of intervention were significantly greater than zero (M = 0.76, 0.61, and 1.04, respectively; for all, p < 0.05). There was no statistically significant group difference in intervention dosage (Q _B = 3.41, p > 0.05). The results suggested that the effects of selected instructional implementation did not vary significantly by the length of instruction.

Instructional design

Regarding instructional design, we compared the effect sizes of two models: stand-alone HOTS units developed by researchers versus integrated HOTS units derived from a regular curricular subject (e.g. social sciences). Units derived from a regular curricular subject produced a significantly greater intervention effect (g = 1.12), compared with independent units developed by researchers (g = 0.70).

Type of dissemination

With regard to types of dissemination, effect sizes were grouped by peer-reviewed journal articles (k = 15) and unpublished master’s theses (k = 14). Under the random-effects model, the effect sizes of both published and unpublished studies were significantly greater than zero (M = 0.69, 0.90, respectively; for both subgroups, p < 0.01). There were no statistically significant group differences between types of dissemination (Q _B = 2.67, p > 0.05). The results suggested that types of dissemination in these studies did not correlate with the effectiveness found in the interventions.

Publication bias

The assessment of publication bias for the meta-analysis of HOTS was performed on the 29 effect sizes extracted from the selected studies, of which 15 (51.7%) were extracted from published journal articles and 14 (48.3%) were extracted from unpublished master’s theses. The funnel plot in Figure 2 showed the effect size of observed studies and four imputed studies. Through visual inspection, asymmetry was observed in the funnel plot, which indicates the probable presence of publication bias. The trim-and-fill method was conducted to impute the studies and adjust the effect size. After replacing the trimmed studies, the mean effect size for the combined studies moved from 0.76 (with a 95% CI [0.62, 0.90]) to 0.84 (with a 95% CI [0.69, 0.97]). Therefore, the effect of publication bias on the results of this meta-analysis was small. This finding implied that the original effect size (g = 0.76) may have been biased downward by publication bias.

Figure 2.

Funnel plot of standard error by Hedges’ g.

Discussion and implications

The present meta-analysis aimed to answer three research questions. The first question was concerned with the overall effect of HOTS instructional interventions pertaining to gifted students in the existing literature in Taiwan. According to Cohen’s criteria, the overall effect size of 0.76 is considered between medium and large. The results suggest that HOTS could be enhanced through effective teaching methods (e.g. The Cognitive Research Trust [CoRT] program (de Bono, 1987), Discovering Intellectual Strengths and Capabilities while Observing Varied Ethnic Responses [DISCOVERY] (Maker, 2001)).

The second research question focused on the effect sizes of specific types of HOTS. To answer this question, we categorized the HOTS into three types of thinking skills by measured outcomes and treated these thinking skills as dependent variables. Our analysis revealed an overall positive effect consistent with findings from previous systematic reviews conducted outside of Taiwan (e.g. Abrami et al., 2008; Huber and Kuncel, 2016; Lee et al., 2016; Niu et al., 2013; Peltier and Vannest, 2017; Yasin and Yunus, 2014). In addition, the interventions on creative thinking skills demonstrated the largest effects among all HOTS skills. This may be because creative thinking skills were most commonly used in teaching gifted students in Taiwan and students had had more opportunities to practice them.

The third research question focused on which study characteristics produced the largest intervention effect. To answer this question, we evaluated group differences among the following variables: (a) grade level, (b) type of program, (c) intervention dosage, (d) instructional design, and (e) study’s dissemination. For grade level, the intervention effects were statistically significant at all grade levels. However, the differences in the effect sizes for the grade-level comparisons were not statistically significant at the 0.05 level. The results indicate that HOTS interventions are appropriate for all of the grade levels included in this meta-analysis. However, there should be caution in interpreting this result since the number of studies conducted in junior high school settings may be too small to provide reliable insights.

Similarly, the intervention effect was statistically significant for both types of programs (pullout gifted resource classrooms or districted-based gifted programs) with no significant difference in the effect sizes found between the two types. This finding suggests that instructions carried out through the two schemes could both improve gifted students’ HOTS. However, the number of studies for the district-based gifted program might be too small to detect any differences in effects and the finding should be interpreted with caution.

The intervention dosage did not significantly affect the HOTS effect sizes in this study, suggesting that the effects of HOTS instruction did not vary whether the length of intervention was brief or not. More importantly, the average effect size for HOTS instruction was significantly greater than zero for each category of intervention dosage. The finding suggested that there was no reason to expect exponential effects in intervention effectiveness based on the length of instruction. When thinking about adding HOTS to their regular teaching repertoire, teachers could have the confidence that it is a matter of focus (i.e. reminding students of the importance of HOTS) rather than a matter of length.

In terms of the comparison between the two instructional design methods, the effect was statistically significant for both methods. However, the difference between the two effect sizes was statistically significant. This suggested that an integrated unit derived from a regular curriculum subject was more effective than a stand-alone unit designed specifically to enhance HOTS. This finding is promising for educators since additional instructional hours for students to acquire HOTS may not be necessary. This finding could also suggest that teachers be equipped with some theoretical foundation regarding HOTS and have the capacity to embed HOTS within their daily instruction.

Conclusion and study limitation

This study has a few limitations. First, the number of usable empirical studies for this meta-analysis was much smaller than expected, given the voluminous body of literature related to the effects of HOTS instruction. Many studies were excluded because they were not clear about the definitions of HOTS or their practices. Hence, there should be some caution in interpreting the results from this meta-analysis. Second, this meta-analysis did not include qualitative studies since they do not contain the quantitative information needed for a meta-analysis. In the future, synthesized studies, including both qualitative and quantitative information, would be valuable in providing a more comprehensive understanding related to the effects of HOTS instruction. Third, the findings from this meta-analysis may not generalize to other settings than those in which targeted instruction methods were implemented.

Despite the limitations of this meta-analysis, this study makes contributions to the field of gifted education in several ways. First, the findings from this meta-analysis support the positive effects of HOTS instructions on gifted students in terms of creative thinking, critical thinking, and problem-solving. Second, this meta-analysis has updated previous synthesized research on the effects of HOTS instructions for gifted students. There has been no current meta-analytic studies focusing on HOTS instruction to gifted students in Taiwan, so this study provides research-based information on HOTS instruction for practitioners in different grade levels. Third, this study found that HOTS instruction can promote the development of students’ thinking. However, the effect size of the integrated HOTS units is greater than that of the stand-alone units designed by researchers. It means that it would be beneficial to integrate HOTS into the instruction of regular school subjects and make them a part of students’ daily learning experience. In summary, although previous literature has explored the effects of HOTS instruction, this study provided new information and supported previous literature that can benefit researchers and practitioners.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

C Owen Lo

References

Abrami

Bernard

Borokhovski

, et al. (2008) Instructional interventions affecting critical thinking skills and dispositions: a stage 1 meta-analysis. Review of Educational Research 78: 1102–1134.

Adelson

Matthews

(2019) Gifted Child Quarterly’s commitment to transparency, openness, and research improvement. Gifted Child Quarterly 62: 83–85.

Ananiadou

Claro

(2009) 21st Century Skills and Competences for New Millennium Learners in OECD Countries. Education Working Papers, No. 41. Paris: OECD.

Barak

Dori

(2009) Enhancing higher order thinking skills among inservice science teachers via embedded assessment. Journal of Science Teacher Education 20: 459–474.

Bloom

Engelhart

Furst

, et al. (1956) Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay.

Borenstein

(2009) Effect sizes for continuous data. In: Cooper

Hedges

Valentine

(eds), The Handbook of Research Synthesis and Meta-analysis, 2nd ed. New York: Russell Sage Foundation.

Borenstein

Hedges

Higgins

JPT

, et al. (2009) Introduction to Meta-analysis. Hoboken: Wiley.

Borenstein

Higgins

(2013) Meta-analysis and subgroups. Prevention Science 14: 134–143.

Callahan

Moon

(2007) Sorting the wheat from the chaff: what makes good evidence of effectiveness in the literature in gifted education? Gifted Child Quarterly 51: 305–319.

10.

Chan

ZCY

(2013) A systematic review of critical thinking in nursing education. Nurse Education Today 33: 236–240.

11.

Chang

(2011) The effect of future study in environmental issue on gifted elementary school students’ critical thinking. Journal of Gifted Education 10(2): 1–32.

12.

Chao

(2004) The study of the teaching effectiveness on scientific creative games training program for elementary school gifted students. Unpublished master’s thesis , National Chiayi University, Chiayi.

13.

Cheng

(2015) Impact of blending course for humor and CoRT 6 active thinking tools on sense of humor and problem solving ability of elementary gifted students. Unpublished master’s thesis , National Kaohsiung Normal University, Kaohsiung.

14.

Chiang

(2005) A study on the effects of the creative problem solving training courses on improving creativity and problem solving ability of elementary gifted students. Journal of Gifted Education 5(2): 83–106.

15.

Chiu

(2008) A study on the effectiveness of the writing program of creative thinking. Unpublished master’s thesis , National Taiwan Normal University, Taipei.

16.

Chou

(2010) The study of promoting critical thinking ability of secondary gifted students by using critical thinking programs. Unpublished master’s thesis , National Changhua University of Education, Changhua.

17.

Chou

(2004) A study on application of the creative problem solving teaching method for the gifted junior high school students. Unpublished master’s thesis , National Taiwan Normal University, Taipei.

18.

Cohen

(1988) Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

19.

Collinson

Kozina

Lin

, et al. (2009) Professional development for teachers: a world of change. European Journal of Teacher Education 32: 3–19.

20.

Cook

Odom

(2013) Evidence-based practices and implementation science in special education. Exceptional Children 79: 135–144.

21.

Cooper

(2010) Research Synthesis and Meta-analysis: A Step by Step Approach, 5th ed. Thousand Oaks, CA: Sage.

22.

de Bono

(1987) CoRT Thinking Lessons: Breadth. Clive: de Bono Thinking Systems.

23.

Duval

Tweedie

(2000) Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56: 455–463.

24.

Edyburn

(2010) Understanding the quality of the science supporting the special education technology evidence base. Journal of Special Education Technology 25: 63–68.

25.

Ellis

(2010) The Essential Guide to Effect Sizes: Statistical Power, Meta-analysis and the Interpretation of Research Results. Cambridge: Cambridge University Press.

26.

Fidler

(2010) Statistical significance, result worthiness and evidence: What lessons are there for giftedness education in other disciplines? In Thompson

Subotnik

(Eds), Methodologies for conducting research on giftedness (pp. 71–88). Washington, DC: American Psychological Association.

27.

Glaser

(1984) Education and thinking: the role of knowledge. American Psychologist 19: 93–104.

28.

Groccia

Buskist

(2011) Need for evidence-based teaching. New Directions for Teaching and Learning 128(Winter): 5–11.

29.

Grossen

(1992) The fundamental skills of higher order thinking. In: Carnine

Kameenui

(eds), High Order Thinking. Austin: Pro-ed, pp. 157–182.

30.

Hedges

Olkin

(1985) Statistical Methods for Meta-analysis. Orlando: Academic Press.

31.

Henfield

Woo

Bang

(2017) Gifted ethnic minority students and academic achievement: a meta-analysis. Gifted Child Quarterly 61: 3–19.

32.

Higgins

Green

(eds) (2008) Cochrane Handbook for Systematic Reviews of Interventions, Vol. 5. Chichester: Wiley-Blackwell.

33.

Higgins

Thompson

(2002) Quantifying heterogeneity in a meta-analysis. Statistics in Medicine 21: 1539–1558.

34.

Huber

Kuncel

(2016) Does college teach critical thinking? A meta-analysis. Review of Educational Research 86: 431–468.

35.

Hung

Huang

Tsai

, et al. (2015) Current state of evidence-based practice education for undergraduate nursing students in Taiwan: a questionnaire study. Nurse Education Today 35: 1262–1267.

36.

Kao

(2012) The educational predicament confronting Taiwan’s gifted programs: an evaluation of current practices and future challenges. Roper Review 34: 234–243.

37.

Kim

(2016) A meta-analysis of the effects of enrichment programs on gifted students. Gifted Child Quarterly 60: 102–116.

38.

Lee

(2010) The effects of using different teaching modes in the LEGO computer theme courses on gifted students’ problem solving ability and learning motivation. Unpublished master’s thesis , National Pingtung University, Pingtung.

39.

Lee

Gong

, et al. (2016) A meta-analysis of the effects of non-traditional teaching methods on the critical thinking abilities of nursing students. BMC Medical Education 16: 240–248.

40.

Lee

Chang

(2005) The effects of critical thinking instruction for the gifted in junior high school via metaphor model. Journal of Gifted Education 5(2): 25–62.

41.

Lewis

Smith

(1993) Defining higher order thinking. Theory Into Practice 32: 131–137.

42.

Cheng

(2007) The effect of creative drama instruction to creativity and interpersonal communication of elementary gifted students. Journal of Gifted Education 7(2): 19–45.

43.

Liang

(2003) The study of the teaching effectiveness on verbal logical thinking training program for elementary school gifted students. Unpublished master’s thesis , National Changhua University of Education, Changhua.

44.

Liao

(2009) The study on the effects of mind mapping instruction on creativity to the elementary gifted students. Unpublished master’s thesis , National Changhua University of Education, Changhua.

45.

Lin

(2001) The study on promoting critical thinking ability of elementary gifted students by using critical thinking training program. Unpublished master’s thesis , National Taiwan Normal University, Taipei.

46.

Lipsey

Wilson

(2001) Practical Meta-analysis. Thousand Oaks, CA: Sage.

47.

Maker

(2001) DISCOVER: assessing and developing problem solving. Gifted Education International 15: 232–251.

48.

McBee

Makel

Peters

, et al. (2018) A call for open science in giftedness research. Gifted Child Quarterly 62: 374–388.

49.

Newmann

(1988) Higher order thinking in the high school curriculum. NASSP Bulletin 72: 58–64.

50.

Niu

Horenstein

LSB

Garvan

(2013) Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educational Research Review 9: 114–128.

51.

Norris

Ennis

(1989) Evaluating Critical Thinking. Pacific Grove, CA: Midwest Publications.

52.

Peltier

Vannest

(2017) A meta-analysis of schema instruction on the problem-solving performance of elementary school students. Review of Educational Research 87: 899–920.

53.

Peng

(2002) The content-analysis and meta-analysis of the researches in gifted education of Taiwan, R. O. C. Unpublished master’s thesis , National Taiwan Normal University, Taipei.

54.

Pertti

Leonard

Julia

(2008) The SAGE Handbook of Social Research Methods. Thousand Oaks, CA: Sage.

55.

Philp

(1985) Assessing Higher Order Thinking Skills. Washington, DC: American Institutes for Research.

56.

Reis

McCoach

Coyne

, et al. (2007) Using planned enrichment strategies with direct instruction to improve reading fluency, comprehension, and attitude toward reading: an evidence based study. The Elementary School Journal 108: 3–24.

57.

Robinson

Shore

Enersen

(eds) (2007) Best Practices in Gifted Education. Waco: Prufrock.

58.

Rothstein

Sutton

Borenstein

(2005) Publication Bias in Meta-analysis: Prevention, Assessment and Adjustments. Chichester: Wiley.

59.

Shokraneh

(2019) Reproducibility and replicability of systematic reviews. World Journal of Meta-Analysis 7: 66–71.

60.

Steenbergen-Hu

Moon

(2011) The effects of acceleration on high-ability learners: a meta-analysis. Gifted Child Quarterly 55: 39–53.

61.

Steenbergen-Hu

Olszewski-Kubilius

(2016) How to conduct a good meta-analysis in gifted education. Gifted Children Quarterly 60: 134–154.

62.

Sternberg

Lubart

(1996) Investing in creativity. American Psychologist 51: 677–688.

63.

Tsai

(2010) The effect of discover curriculum on problem solving ability and creativity of elementary gifted students. Journal of Gifted Education 10(1): 63–93.

64.

Tseng

Chou

Wang

, et al. (2011) The effectiveness of problem-based learning and concept mapping among Taiwanese registered nursing students. Nurse Education Today 31: e41–e46.

65.

Tseng

(2008) The effects of the picture book with creative thinking teaching program on the creativity of the gifted students. Unpublished master’s thesis, University of Taipei, Taipei.

66.

Udall

Daniels

(1991) Creating the Thoughtful Classroom. Tucson: Zephyr Press.

67.

Vaughn

Feldhusen

Asher

(1991) Meta-analyses and review of research on pull-out programs in gifted education. Gifted Child Quarterly 35: 92–98.

68.

Wang

(2010) The study of the creativity performance on mathematic creative thinking teaching for the third-graders’ gifted students. Unpublished master’s thesis , National University of Tainan, Tainan.

69.

Wang

(2005) The effect of philosophical thinking program in the creativity and critical thinking processes on the gifted elementary school students. Bulletin of Special Education and Rehabilitation 13: 1–25.

70.

Wang

Chang

(2014) Creative thinking skills integrated into game design program on creativity for the elementary gifted students. Special Education Forum 16: 62–78.

71.

Wang

(2003) The effect of “six thinking hats” training course on improving problem solving ability of elementary gifted students. Unpublished master’s thesis , National Taiwan Normal University, Taipei.

72.

Warner

(2008) Applied Statistics: From Bivariate Through Multivariate Techniques. Thousand Oaks, CA: Sage.

73.

World Economic Forum (2016) The future of jobs-employment, skills and workforce strategy for the fourth industrial revolution. Available at: https://www.voced.edu.au/content/ngv:71706.

74.

(2012) The lateral thinking of teaching programs to improve the level of gifted students of creativity. Master’s thesis . Available at: https://hdl.handle.net/11296/vdnfvt.

75.

(2000) Talent identification and development in Taiwan. Roeper Review 22: 131–134.

76.

(2013) 40th anniversary of gifted education in Taiwan: a retrospection. Gifted Education Quarterly 126: 1–11.

77.

Cho

(1993) Programs and practices for identifying and nurturing giftedness and talent in Asian (outside the Mainland of China). In: Heller

Mönks

Sternberg

Subotnik

(eds), International Handbook of Giftedness and Talent, 2nd ed. Oxford: Pergamon, pp. 797–807.

78.

Xie

Chang

(2016) Creative problem solving integrated into scientific toys making instruction on creativity and scientific creative problem solving for elementary gifted students. Special Education Forum 20: 20–35.

79.

Yang

Chang

(2011) A study of linear and intuitive thinking teaching program on creativity for elementary gifted students. Special Education Forum 10: 1–18.

80.

Yasin

Yunus

(2014) A meta-analysis study on the effectiveness of creativity approaches in technology and engineering education. Asian Social Science 10: 242–252.

81.

(2003) A study of the effects of the de Bono’s CoRT program on primary school gifted students’ linguistic creativity. Unpublished master’s thesis , National Taiwan Normal University, Taipei.

82.

Chang

Jen

(2017) Policy and practice in science education for the gifted in Taiwan. In: Sumida

Taber

(eds) Policy and Practice in Science Education for the Gifted: Approaches from Diverse National Contexts. London: Routledge.

83.

Chang

(2016) The future school teaching program on creativity and imagination for gifted students. Special Education Forum 20: 1–19.

84.

Zeng

Chang

(2013) Effect of intel K-12 educator’ thinking tools integrated into teaching program on reasoning ability and creativity by gifted students in the elementary school. Special Education Forum 14: 1–19.

85.

Zohar

(2006) The nature and development of teachers’ metastrategic knowledge in the context of teaching higher-order thinking. Journal of the Learning Sciences 15: 331–377.