Measuring and Predicting Divergent Thinking With a Self-Report Questionnaire,Teacher Rating Scale,and Self-Regulated Learning Microanalysis

Abstract

This study examined the relationships among three measurement methodologies that are used to assess characteristics and processes associated with creativity (i.e., a self-report questionnaire, teacher ratings, and a structured interview). In addition, we examined the predictive contributions of these three measurement methodologies for a divergent thinking test (Torrance Test of Creative Thinking-Figural; TTCT-F). Participants were 89 adolescents in the Midwestern United States. Results revealed that the self-report questionnaire and structured interview measure of self-efficacy correlated significantly (r = .34), but no other significant relationships among measurement methodologies were observed. Neither the self-report ratings nor teacher ratings significantly predicted performance on the TTCT-F, but the structured interview measures did.

Keywords

self-regulated learning personality/individual differences motivation metacognition divergent thinking self-regulated learning microanalysis

Creativity promotes societal innovation, academic achievement, and personal motivation (Gajda et al., 2017; Liu et al., 2016), inspiring researchers and leaders to advocate for the intentional development of creativity within school settings (OECD, 2018). Yet, the development of creativity presents a complex, multifaceted challenge that requires both a clear definition and aligned assessment methods to determine the efficacy of efforts to develop creative skills. One systematic approach is to dissect creativity into more discrete components and then define, develop, and assess those components. The creative process has been dissected into two major processes: the generation of ideas (i.e., divergent thinking) and the selection of ideas (i.e., convergent thinking; Jaarsveld et al., 2012). Of these two processes, divergent thinking is more consistently researched as a measure of creative potential (Reiter-Palmon et al., 2019). Examining divergent thinking provides two key benefits. First, it can be assessed before significant creative accomplishments occur, thus making them more appropriate measures for developing students. Second, divergent thinking predicts creative accomplishments (Kim, 2008a, 2008b; Plucker, 1999; Runco et al., 2010).

Given that divergent thinking is a component of creativity, understanding factors that contribute to and improve students’ divergent thinking becomes a worthwhile endeavor. Specifically, educators, school psychologists, and researchers need measures that (a) identify students who need extra support or added rigor, (b) highlight processes that can be developed, (c) determine the effectiveness of interventions designed to develop creativity, and (d) understand environmental variables that support or inhibit students’ creative growth.

An incredible array of assessment techniques has been developed (e.g., Kaufman et al., 2008; Thys et al., 2014). Self-report questionnaires (SRQs) and teacher ratings are often used to examine characteristics and processes at a more global level; however, other recently developed measures assess regulatory, metacognitive, and motivational processes at a task-specific level. While this proliferation of measurements brings depth to the field, many questions remain regarding the conceptual overlap and relationships of these measures and which measures best predict important creative processes such as divergent thinking.

This article addresses this gap by examining the relations among three measurement formats (i.e., SRQ, teacher ratings, and a structured interview). Next, we examine the extent to which these measurement formats predict a commonly used measure of divergent thinking (i.e., Torrance Test of Creative Thinking-Figural [TTCT-F]).

Defining and Assessing Divergent Thinking

Creativity has been broadly conceptualized as the interaction among personal characteristics, the creative process, and environmental factors to produce a novel and useful product or idea (Plucker et al., 2004). In 1950, Guilford seminally proposed examining a series of interrelated “creative abilities” that facilitate creative outcomes, including the ability (a) to think fluently and flexibly and (b) to generate novel and elaborate ideas. Several of these “abilities” are synthesized within the broader cognitive process of divergent thinking. Specifically, divergent thinking entails the generation of many (i.e., fluency), different (i.e., flexibility), novel (i.e., originality) ideas (Finke et al., 1992; Runco, 2008). Further, divergent thinking significantly predicts future creative accomplishments (Kim, 2008a, 2008b; Plucker, 1999; Runco et al., 2010).

When Guilford (1950) proposed these components of creativity, he simultaneously recognized the challenges of assessing them due, in part, to the rarity of truly excellent creative accomplishments and the difficulty of designing valid measures. This recognition led to the proliferation of divergent thinking tests to assess creative potential (e.g., TTCT, Guilford Divergent Thinking Tasks, Wallach and Kogan Divergent Thinking Tasks). Of these assessments, Kim (2008a, 2008b) reported in a meta-analysis that the TTCT was most commonly used (n = 142) and demonstrated the second highest correlation with creative achievements (r = .33). The Group Inventory for Finding Creative Talent was the most highly correlated (r = .33), but only five studies used that assessment in the meta-analysis.

Assessing Factors Contributing to Divergent Thinking

Multiple measurement formats have been used to examine characteristics and processes that support divergent thinking (and broader creative skills as well) including SRQs, teacher ratings, and task-specific interviews; however, they have not been examined concurrently.

SRQs

First, SRQs emphasize Likert-scaled items addressing multiple personal characteristics, beliefs, and behaviors that support creative accomplishments (Batey, 2012; Kaufman, 2019). Moreover, SRQs are important because they capture individuals’ perceptions of themselves, which educational professionals can then use to (a) design instruction to help students view their creativity more accurately, (b) identify potential processes and characteristics that could be enhanced, and (c) determine the extent to which students value and recognize their creativity. Self-report questionnaires are used often when researchers are interested in creative processes. For example, approximately 40% of empirical studies of creativity include an SRQ (Forgeard & Kaufman, 2016). This frequency may be due, in part, to strong psychometric properties and efficient administration/scoring (Kaufman et al., 2008; Runco et al., 2001).

Teacher ratings

Researchers and educators also use teacher ratings, which share many content and format similarities with SRQs. Furthermore, they tend to demonstrate strong psychometric properties and are efficient to administer and score. Teacher ratings are critical because SRQs may be prone to children’s inaccurate reporting, and teacher ratings provide an additional, external data source to corroborate evidence (Miller et al., 2014; Winne & Jamieson-Noel, 2002). Further, teacher ratings systematically guide teachers to consider specific aspects when assessing students’ creativity, which is helpful because teachers’ beliefs of creativity may not match researchers’ definitions (Rubenstein et al., 2018; Westby & Dawson, 1995).

Task-specific measurements

In contrast to SRQs and teacher ratings, which measure one’s general characteristics and general processes across a variety of settings, task-specific measures have also been used to examine participants’ thinking, processes, or behaviors during one specific task. For example, researchers have used observation techniques (Jang & Ko, 2017), think-aloud protocols (Gilhooly et al., 2007), and structured interviews called self-regulated learning (SRL) microanalysis (i.e., SRL microanalysis; Callan et al., 2019; Rubenstein et al., 2019). Collectively, task-specific measurements provide distinct benefits in educational settings. Specifically, they contribute fine-grained information regarding how students engage with a specific task, which is important because students’ complex cognitive processes often vary across tasks (Lodewyk et al., 2009). Further, these measurement techniques detect small shifts in students’ processes that are not captured by SRQs (Cleary et al., 2017). Being able to monitor these shifts could help teachers differentiate and adapt learning experiences. Moreover, some research indicates that task-specific measures are stronger predictors of achievement compared to more global measures (Cleary, 2015; Callan & Cleary, 2018; Young & Worrell, 2018); however, this topic has not been explored within the field of creativity.

Although multiple task-specific measurements exist, the current study uses SRL microanalysis, which is a structured interview designed to capture participants’ SRL while they are engaged in a specific task of interest (Cleary, et al., 2012). Self-regulated learning includes a variety of processes, such as setting goals, planning, holding adaptive motivational beliefs, using strategies, self-monitoring, and reflecting (Zimmerman, 2000), which facilitate achievement in a variety of domains (Schunk & Greene, 2017), including creativity (Callan et al., 2019; Rubenstein et al., 2019). Self-regulated learning skills are particularly important because they can be taught through intervention programs (i.e., Cleary et al., 2017; Graham et al., 2012) and by embedding SRL supportive practices within regular classroom instruction (Perry et al., 2002).

Self-regulated learning microanalysis interviews use brief questions that are administered before, during, and after an individual engages with a task. The administration timing of each question is matched to the target task and SRL theory (Zimmerman, 2000) so that constructs are measured at the point in time that they are most salient. For example, goal setting, planning, and self-efficacy measures are administered just before an individual engages in the target task because these processes are most important while an individual prepares for task engagement (Zimmerman, 2000). Likewise, strategy use and self-monitoring measures are administered during or just after task engagement, whereas self-reflective processes are measured after task engagement.

Self-regulated learning microanalysis can be advantageous because it enables the collection of fine-grained and real-time data about participants’ thoughts and behaviors. Given the real-time nature of data, SRL microanalysis questions are carefully designed to limit interruptions to the natural flow to the task and to prevent cueing participant engagement in SRL. Prior research using SRL microanalysis suggests several key processes emerge as strong predictors of achievement within academic and creativity domains. Specifically, SRL microanalysis measures of self-efficacy, planning, and strategy use have been shown to be particularly powerful predictors of achievement in a variety of domains (Cleary & Callan, 2017), including creative problem solving (CPS) (Callan et al., 2019; Rubenstein et al., 2019).

Comparing Measurement Methods

Different types of measures (i.e., SRQs, teacher ratings, and SRL microanalysis) each have advantages and drawbacks. Moreover, while conceptual overlaps exist, they each emphasize different aspects or perspectives that may relate to divergent thinking (i.e., personal characteristics vs. facilitative processes). Thus, understanding the relationships among these different measurement methodologies may help practitioners interpret data from multiple measurement formats. Prior research has examined the relationship between SRQs and teacher ratings of creativity, finding either no significant correlations or very weak correlations (rs ranged from .06–.23; Mann, 2009; Ridgley et al., 2020). One recent study did examine the relations among SRQs, teacher ratings, and SRL microanalysis within the context of mathematics by administering these measures to 100 eighth graders (Callan & Cleary, 2018). The SRQ and teacher ratings correlated significantly (r = .30), but SRL microanalysis did not correlate with the SRQ (r = .03) nor the teacher ratings (r = .06). However, we are unaware of existing research examining the relationships among these measures within the context of divergent thinking.

In addition to comparing measures of divergent thinking characteristics and processes, researchers have also used those measures in isolation to predict divergent thinking. Self-report questionnaires have correlated with students’ performance on divergent thinking tasks (Mann, 2009). Moreover, the teacher form of the Scales for Identifying Gifted Students-Creativity Subscale (T-SIGS-C) has shown large correlations (r = .62) with divergent thinking tests (Ryser & McConnell, 2004). Although SRL microanalysis predicted performance on a CPS task (Callan et al., 2019), additional research is needed to examine the contributions of SRL microanalysis beyond the predictive contributions of SRQs and teacher ratings.

To our knowledge, researchers have not examined the predictive contributions of these measures concurrently. Thus, a question still remains as to which measurement tools best predict divergent thinking. Understanding how SRQs, teacher ratings, and SRL microanalysis predict students’ performance on the TTCT-F is important and can guide educators’ and curriculum designers’ measurement procedures, which in turn should inform their instructional practices.

This study addresses two broad objectives. First, this study examines the relationships among measures of creative characteristics and processes (i.e., SRQs, teacher rating, and SRL microanalysis). “What relationships exist among measurement tools (i.e., SRQs, teacher ratings, and SRL microanalysis)?” Next, we examine how these measures predict performance on the TTCT-F. “To what extent do different measurement tools (i.e., SRQ, teacher ratings, and SRL microanalysis) predict performance on a test of divergent thinking (i.e., TTCT-F)?”

Methods

Participants

Participants were 89 adolescents (Mean age = 13.90; SD = 0.65) enrolled in two schools in the Midwestern United States. The majority of students were enrolled in eighth–ninth grades with ages ranging from 11 to 15 years. A small number of sixth–seventh graders were included because they were enrolled in classes with eight–ninth grade students. All participants were enrolled in general education. The populations of both schools were similar regarding socioeconomic status, ethnicity, and performance on state assessments. No statistically significant school differences existed regarding outcome variables. Forty-two participants were female and seventy-nine were Caucasian (five students identified as multiracial, two as Asian, two as Black, one as Latinx, and one as American Indian). Data were missing for three participants due to an audio recorder malfunction during their response to interview questions about planning. Regarding descriptive statistics for the TTCT-F, our sample mean score on the TTCT-F was 107 with a SD of 16. Compared to the normative mean and SD (i.e., 100; 20), our scores were slightly higher and less variable.

General Procedures

Participants met individually with a trained research assistant to complete several CPS tasks and respond to SRL microanalysis interview questions (see Table 1 for a summary of procedures). These meetings were audio recorded and transcribed to ensure accuracy and to facilitate coding of open-ended responses. Meetings began with a CPS-Introduction-Task, which required participants to generate solutions and select their best solution to help a stakeholder solve a problem. An example problem is, “The school nurse accidentally ordered 1000 boxes of band-aids, rather than 1000 individual band-aids. Can you help the school nurse come up with different ways to use extra boxes of band-aids?” These CPS tasks were developed and published in prior research (Callan et al., 2019; Rubenstein et al., 2019, 2020).

Table 1.

SRL Microanalysis Measures, Sample Wording, and Coding.

Measures	Measure Description and Samplesa	Coding or Scaling of Response
CPS-Introduction-Task	Two CPS tasks. (E.g., “The school nurse accidently ordered 1000 boxes of bandaids rather than 1000 individual bandaids. Can you help the nurse come up with different ways to use the extra boxes of bandaids?”)	Fluency: Number of ideas
		Flexibility: Number of types of ideas
		Originality: Idea uniqueness
		Usefulness: Idea quality
Micro self-efficacy	“How sure are you that you can create many ideas to solve the problem?”	Likert rating: 1 (not at all sure) to 7 (very sure)
Micro planning	“What can you do if you get stuck or have trouble thinking of ideas?”	Number of CPS strategies reported
CPS-Final-Task	One CPS task parallel to previous CPS-Introduction-Task	Fluency: Number of ideas
		Flexibility: Number of types of ideas
		Originality: Idea uniqueness
		Usefulness: Idea quality
Micro strategy use	“Tell me all the things you did to help you solve this problem”	Identical to strategic planning items

Note. SRL = Self-regulated learning; CPS = creative problem solving.

Prompts are paraphrased and abbreviated. See Methods section for further details. “Micro” indicates that this measure was SRL microanalysis.

The CPS-Introduction-Task familiarized participants with the format of CPS tasks, before they responded to SRL microanalysis questions embedded around a second, parallel CPS task, called the CPS-Final-Task. One SRL microanalysis measure targeted self-efficacy (i.e., expectations for success) and two measures addressed strategic thinking and action (i.e., planning and strategy use). The self-efficacy and planning items were administered just before participants attempted to solve the CPS-Final-Task and the strategy use measure was administered just after participants attempted to solve the CPS-Final-Task. Participants completed the TTCT-F and SRQ within 2 weeks of this meeting. Teacher ratings were completed within 2 weeks of these measures as well.

Measures

Teacher version—Scales for Identifying Gifted Students—Creativity Subscale

The SIGS was originally designed as a teacher rating scale to identify K-12 students for gifted services (Ryser & McConnell, 2004). The full scale includes seven subscales; however, this study only used the creativity subscale. The T-SIGS-C items were crafted from the cognitive, process, and personality characteristics discussed in the creativity literature. The T-SIGS-C includes 12 Likert scale items examining teachers’ perceptions of how much items describe a specific student (e.g., “Does not mind uncertainty”). Prior research has shown strong internal consistency (α = .93) and test–retest reliability (ranging from 0.70 to 0.91; Ryser & McConnell, 2004). Furthermore, differential item functioning analyses suggests that SIG items are invariant across gender or ethnic identities (Ryser & McConnell, 2004). Cronbach’s alpha with the current sample was .87. Some states, including the state in which this study was conducted, recommend the SIGS as the primary scale for gifted identification (Marschand, 2013).

Student version–Scales for Identifying Gifted Students—Creativity Subscale (S-SIGS-C)

Several SRQs exist within the field of creativity; however, the alignment of content and items between most SRQs and teacher ratings is inconsistent and fragmented. Thus, we opted to have students and teachers both rate the items from the SIGS-C for consistency and parsimony. Minor adjustments were made to item wording to reflect the rating of one’s own behaviors as opposed to the original format of the SIGS-C that referred to others’ behavior. Although the SIGS was originally designed for teacher ratings of students, some prior research used the SIGS as an SRQ (Ridgley et al., 2020). This work showed that the S-SIGS-C was strongly correlated with a single-item self-rating of overall creativity (r = .64; Ridgley et al., 2020). Within the current sample, Cronbach’s alpha was .80. Taken together, these relationships provide evidence for the reliability and validity of the S-SIGS-C.

SRL microanalysis

Three microanalysis measures examined participants’ self-efficacy, planning, and strategy use. The measure of self-efficacy utilized numeric responses on a Likert scale, whereas planning and strategy use were measured using free-response items. The coding of planning and strategy use is described below. The SRL microanalysis interview questions were developed and validated in prior research (Callan et al., 2019; Rubenstein et al., 2019, 2020).

Self-Efficacy

After the CPS-Introduction-Task, but before participants were informed of the details of the CPS-Final-Task, the interviewer said, “I am just about to tell you the new story, but before I do…,” and then the interviewer administered four self-efficacy items to measure students’ perceptions of their ability to (a) generate many solutions, (b) generate many types of solutions, (c) generate a solution no one else would think of, and (d) generate a solution that would be helpful. As an example, “Using this scale where one means that you are ‘Not At All Sure’ and seven means that you are ‘Very Sure,’ how sure are you that you can create many solutions to the new problem?” The ratings across all four items were aggregated and displayed acceptable reliability (α = .73). Prior use of a similar item has differentiated between experts, nonexperts, and novices (Kitsantas & Zimmerman, 2002).

Planning

This one-item measure targeted participants’ plans to use strategies prior to attempting a CPS task. The interviewer asked, “What can you do if you get stuck or have trouble thinking of ideas?” If the participant provided a response other than “no” or “I don’t know,” the interviewer prompted, “Is there anything else that you can do?” This prompt was provided a maximum of two times. Coding responses to the planning item included counting the number of strategies identified. This measure was used in prior research and was shown to predict achievement on a CPS task (Callan et al., 2019; Rubenstein et al., 2019). Coding was completed by the second and third authors with substantial inter-rater agreement (κ = 0.66). Inter-rater agreement was determined using Kappa, which is interpreted differently than other reliability coefficients such as alpha (see McHugh (2012) for further details regarding the interpretation of Kappa statistics).

Strategy Use

The single-item strategy use measure was administered immediately after task performance. The measure wording was, “Tell me all the things that you did to help you solve this problem.” Prompting and coding procedures were identical to the planning measure. This measure was used in prior microanalytic research and predicted achievement on CPS tasks (Callan et al., 2019; Rubenstein et al., 2019). Coding was completed by the second and third authors with substantial inter-rater agreement (κ = 0.67).

Divergent Thinking Test.

The TTCT-F assesses students’ ability to transform shapes or lines into pictures. The TTCT-F consists of three open-ended subtests that assess fluency (number of ideas), originality (uniqueness of ideas), elaboration (amount of detail), abstractness of titles, and resistance to premature closure (delayed or purposeful closure of an open figure). Standard scores are calculated for each of the skills. Additionally, the responses from the subtests are also scored for creativity strengths (e.g. emotional expressiveness, unusual visualization, vivid imagery, etc.), and these strengths are combined with the scores on the individual creativity skills to yield an overall creativity index. The TTCT-F has demonstrated high reliability (KR21 = 0.83 – 0.93) and high inter-rater reliability (0.96–0.99; Torrance, 2008). The TTCT-F has been shown to be a stronger predictor of creative achievements (r = .33) than other divergent thinking measures (Kim, 2008a, 2008b); however, the relationship is still modest. For the purpose of this study, we used the TTCT-F Grade Standard Score for the Creativity Index.

Results

Prior to completing analyses for our key research questions, the authors examined descriptive statistics (see Table 2) and assumptions for the planned correlation and regression analyses. Assumptions for both analyses were met. Thus, the authors addressed the planned research questions.

RQ1: What relationships exist among measurement tools (i.e., SRQs, teacher ratings, and SRL microanalysis)?

Table 2.

Descriptive Statistics.

Variable	Mean(SD)	Skewness (SE)	Kurtosis (SE)	Min	Max
SRQ	31.66 (7.6)	−1.1 (.25)	3.0 (.49)	0	46
Teacher rating	32.96 (7.1)	−0.80 (.25)	1.03 (.50)	9	48
Micro self-efficacy	4.41 (1.0)	−0.44 (.25)	0.40 (.50)	1.33	6.67
Micro planning	2.27 (1.2)	0.81 (.26)	0.89 (.50)	0	6
Micro strategy use	2.14 (1.2)	1.17 (.25)	1.3 (.50)	0	6
TTCT-F	107.43 (15.8)	−0.22 (.25)	−0.19 (.49)	61	141

Note. TTCT-F = Torrance Test of Creative Thinking-Figural; SRQ = Self-report questionnaire; SE = standard error.

“Micro” indicates that this measure was SRL microanalysis.

Pearson correlations were computed to examine relationships among the SRQ, teacher ratings, and SRL microanalysis. The SRQ and the teacher ratings did not correlate significantly (r = .13, p = .25). The SRQ did not correlate significantly with SRL microanalysis measures of planning and strategy use but did correlate significantly with the SRL microanalysis measure of self-efficacy (r = .34, p = .001). The teacher ratings and SRL microanalysis measures did not correlate significantly (see Table 3).

RQ2: To what extent do different measurement tools (i.e., SRQ, teacher ratings, and SRL microanalysis) predict performance on a test of divergent thinking (i.e., TTCT-F)?

Table 3.

Correlations Among SRQ, Teacher Ratings, SRL Microanalysis, and TTCT-F.

Measure	1	2	3	4	5
1. SRQ	—
2. Teacher ratings	.13	—
3. Micro self-efficacy	.34**	−.15	—
4. Micro planning	.15	.21	.33**	—
5. Micro strategy usage	.12	.16	.21*	.49***	—
6. TTCT-F	.16	.12	.36***	.23*	.32**

Note. “Micro” indicates that this measure was SRL microanalysis. TTCT-F indicates Torrance Test of Creative Thinking-Figural; SRL: Self-report questionnaire.

*p < .05.

** p < .01.

***p < .001.

The second objective was to examine the extent to which an SRQ, teacher rating, and SRL microanalysis predicted students’ TTCT-F performance. Neither the SRQ nor teacher ratings correlated significantly with the TTCT-F. However, the SRL microanalysis measures of self-efficacy, planning, and strategy use each correlated significantly with the TTCT-F (see Table 3). A subsequent regression analysis was completed including all three SRL microanalysis measures as predictors (see Table 4). The overall model was significant, F(3, 83) = 6.8, p < .001, R² = .20, explaining a medium amount of variance in TTCT-F scores. Strategy use and self-efficacy each uniquely explained a significant but small amount of unique variance in TTCT-F performance (5% and 8%, respectively).

Table 4.

Prediction of Performance on a Divergent Thinking Test (i.e., TTCT-F).

Variable	Semi-partial Correlation (sr²)	β	T	R ²
Micro self-efficacy	.29 (8%)	.31	2.93**	.20***
Micro planning	<.01 (0%)	<.01	0.03
Micro strategy use	.29 (8%)	.26	2.31*

Note. “Micro” indicates that this measure was SRL microanalysis. TTCT-F indicates Torrance Test of Creative Thinking-Figural. Total/adjusted R² = .20/.17; sr² = semi-partial squared represents the proportion of unique variance in mathematics test scores accounted for a specific predictor after controlling for all other variables.

*p < .05.

** p < .01.

***p < .001.

Discussion

The primary objectives of this study were to (a) examine relationships among an SRQ, teacher ratings, and a task-specific interview measuring SRL processes and then (b) explore the extent to which these three measurement techniques predicted divergent thinking. Collectively, these findings add to the literature by empirically examining the relations among multiple measurement formats that are used within the creativity literature and how these measures predict divergent thinking outcomes.

Relationships Among Measurement Types

Regarding the first objective, SRL microanalysis of self-efficacy correlated significantly with the SRQ, but no other significant relationships were found across measurement types. This may suggest these measurement methods target different facilitators of divergent thinking (i.e., personal characteristics and processes).

SRQs and teacher ratings

The SRQ and teacher ratings exhibited small, nonsignificant relationships (r = .13). Although one might suspect larger correlations when raters respond to the same items, the size of correlations that we found are similar to prior research using both a teacher and student version of the SIGS-C. Moreover, research examining multiple informant ratings of more objective and observable behaviors has often found only modest relationships (Miller et al., 2014). Thus, we expect minimal rater agreement when students and teachers respond about internal, cognitive processes.

Another possible reason for the lack of correlation may be teacher lack of awareness of students’ divergent thinking skills as prior research indicates that students and teachers rarely share their creative endeavors with each other (Ridgley et al., 2020). Furthermore, teachers reported being uncomfortable assessing students’ creative thinking (Myhill & Wilson, 2013). Overall, the lack of a significant correlation suggests teachers and students either view students’ divergent thinking characteristics differently or teachers do not have the knowledge/comfort level to assess it accurately. It is also possible that the poor correlations resulted from using the SIGS-C as a self-report when it was not originally designed for that purpose. Additional research is needed to examine these topics further.

SRQs and SRL microanalysis

Interestingly, the SRQ correlated significantly with the SRL microanalysis measure of self-efficacy but did not correlate significantly with the planning or strategy use measures. That is, the SRQ better related to how well participants thought they would do than how adaptively they approached the task. There are multiple potential interpretations of these findings. First, given that one’s prior successes and failures are the best predictor of self-efficacy (Bandura, 1997), students who have succeeded in previous CPS tasks should report higher self-efficacy and may also be expected to self-report more characteristics of creative people. Another interpretation is that SRQs better reflect one’s perceptions of ability rather than their specific characteristics or behaviors. For example, students may perceive themselves to be a creative person and thus endorse greater alignment with characteristics that are stereotypical of creative persons.

Teacher ratings and SRL microanalysis

Teacher ratings did not correlate significantly with any of the SRL microanalysis measures. These findings are consistent with prior research comparing teacher ratings and SRL microanalysis situated within mathematics context (Callan & Cleary, 2018). However, the relationship between planning and teacher ratings approached significance. With a larger sample size, this relationship may be significant. The current study was a first examination of these relationships within the context of creativity. One potential explanation for the lack of significant correlations is that teacher ratings of creativity may not be informed by students’ task approach but instead other factors such as creative behaviors in class, academic performance, or people pleasing behaviors (Gralewski & Karwowski, 2013; Westby & Dawson, 1995).

An additional explanation for the lack of correspondence between these measures is the difference in “measurement granularity.” That is, teacher ratings addressed creativity across multiple contexts, while SRL microanalysis measures targeted SRL within one specific creativity task. The lack of convergence might indicate differences between one’s general creative characteristics and how one addresses a specific task. This interpretation resonates with research suggesting creativity and SRL processes may be task-specific rather than global (Plucker & Zabelina, 2009; Silvia, et al., 2009). Future research could administer task-specific measures, such as SRL microanalysis, in relation to both applied CPS tasks and divergent thinking tests to address the consistency across task types.

Prediction of Divergent Thinking

Our second objective was to examine the prediction of students’ divergent thinking per the TTCT-F. Although the SRQ and teacher ratings did not relate to TTCT-F, the SRL microanalysis measures of self-efficacy, planning, and strategy use did. Surprisingly, the SRQ and teacher rating did not predict the TTCT-F, given that prior research showing large correlations between the T-SIGS-C and the TTCT (r = .62). Our findings may have differed from prior research for several reasons. First, the sample size within the T-SIGS norming data was relatively modest (n = 30; Ryser & McConnell, 2004), whereas the sample size for our study was much larger (n = 89). The utility of the SIGS-C could decline when teachers are asked to complete ratings of many students as opposed to the original study that included a smaller sample size per teacher rater. Last, in our study, teachers had known students for a minimum of 3 months, but it is unclear how long teachers had known students in the normalization studies. Future research should utilize another SRQ that was specifically designed and validated for this purpose to examine these results further.

Although it was surprising the SRQ and teacher ratings did not significantly predict divergent thinking skills, it was encouraging that SRL microanalysis did because SRL microanalysis measured specific processes that can be taught through interventions (Cleary, 2015) and classroom practices (Perry et al., 2002). This provides concrete guidance for teachers as they develop divergent thinking skills among their students.

Our findings also parallel some prior research. Young and Worrell (2018) demonstrated a similar finding that measures of what one does during a task are more predictive of achievement than measures that are removed from the task. Further Callan and Cleary (2018) found SRL microanalysis to be more predictive of some mathematics achievements when compared to a self-report questionnaire and a teacher rating scale. While it is important that students and teachers are aware of students’ characteristics, our findings indicate that measurement processes that examine what students actually think and do during a task is a stronger predictor.

Interestingly, SRL microanalysis has typically been used to measure performance on the immediate task around which the microanalytic questions were administered. The current study and one other study (Callan & Cleary, 2018) have examined outcomes outside of the interview procedures. The results of these two studies indicate that SRL microanalysis predicts performance on similar tasks that are provided outside of the immediate interview context. These findings are encouraging because they may indicate that SRL microanalysis measures can help to predict a variety of important, related outcomes. However, these findings are somewhat surprising, given that SRL does vary across tasks, and microanalysis is designed to measure SRL within one single task. Thus, it might have been expected that SRL microanalysis would need to be administered in relation to TTCT-F in order to predict achievement on the TTCT-F.

Limitations and Future Directions

There are several limitations of the present study. First, we conducted SRL microanalysis interviews within the context of the applied CPS problems, but we did not administer microanalysis in relation to the TTCT-F. Doing so would have enabled the authors to also examine the consistency of SRL processes across two different types of creative tasks. In addition, when examining the relations among SRQ, teacher ratings, and SRL microanalysis, it is important to note that these different assessment procedures varied by methodology (rating scales vs. interviews), source (teachers vs. self-report), and the targeted constructs (i.e., creative characteristics and SRL processes). Given that the method, source, and topic of these measures differed, it is difficult to definitively determine why these measures differed. Future research should examine multiple types of assessments which are aligned in content.

Summary

Our results examined the interrelations of three measurement formats and examined the unique predictive contributions of these measures. Although replication and further research is needed, our initial results have important implications for educational practitioners who intend to (a) interpret multiple measurement methodologies designed to measure characteristics and processes that support divergent thinking and (b) develop students’ divergent thinking skills.

Regarding measurement interpretation, the predictor measures in this study did not correlate significantly. This is likely because creative processes, such as divergent thinking, are multifaceted, and different measures tap into unique aspects of divergent thinking. To capture these multiple facets, educators should collect data with multiple measurement formats and from multiple sources. However, given that SRL microanalysis best related to divergent thinking outcomes, educators interested in identifying effective divergent thinkers, or those who need support with these skills, may be better suited to use microanalysis. Further research is needed to examine additional divergent thinking outcomes before making high stakes decisions.

Regarding attempts to improve divergent thinking skills, our data provide support for using SRL microanalysis to identify processes that are related to divergent thinking. Not only did SRL microanalysis better predict divergent thinking but using microanalysis is conceptually advantageous as well. For example, microanalytic interviews target processes, such as strategy use or self-efficacy, that can be taught or developed through interventions. Educators can use microanalysis interviews in several ways to teach SRL skills. First, microanalysis can be administered to identify SRL skill deficits, and interventions can be designed to meet individual student needs. Moreover, microanalysis can be administered posttest to document changes in targeted skills. However, potentially of greater application for educators is to administer microanalysis throughout the course of intervention to monitor progress toward intervention goals and prompt intervention adaptations if necessary. The authors note that replication of our findings and additional research is needed prior to implementation of these findings.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article was financially supported by Ball State University, through Academic Excellence Grant.

ORCID iD

Gregory L. Callan

References

Bandura

(1997). Self-efficacy: The exercise of control. Macmillan.

Batey

(2012). The measurement of creativity: From definitional consensus to the introduction of a new heuristic framework. Creativity Research Journal, 24(1), 55-65. doi:10.1080/10400419.2012.649181

Callan

G. L.

Cleary

T. J.

(2018). Multidimensional assessment of self-regulated learning with middle school math students. School Psychology Quarterly, 33(1), 103. doi:10.1037/spq0000198

Callan

G. L.

Rubenstein

L. D.

Ridgley

L. M.

McCall

J. R.

(2019). Measuring self-regulated learning during creative problem-solving with SRL microanalysis. Psychology of Aesthetics, Creativity, and the Arts, 5, 136-148. doi:10.1037/aca0000238

Cleary

T. J.

(2015). Self-regulated learning interventions with at-risk youth: Enhancing adaptability, performance, and well-being. American Psychological Association Press Publications.

Cleary

T. J.

Callan

G. L.

(2017). Assessing self-regulated learning using microanalytic methods. In Schunk

D. H.

Greene

(Eds.), Handbook of Self-Regulation of Learning and Performance (2nd ed., pp. 338-351). Routledge.

Cleary

T. J.

Callan

G. L.

Zimmerman

B. J.

(2012). Assessing self-regulation as a cyclical, context specific phenomenon: Overview and analysis of SRL microanalytic protocols. Education Research International, 2012, 1-19. doi:10.1155/2012/428639

Cleary

T. J.

Velardi

Schnaidman

(2017). Effects of the self-regulation empowerment program on middle school students’ strategic skills, self-efficacy, and mathematics achievement. Journal of School Psychology, 64, 28-42. doi:10.1016/j.jsp.2017.04.004

Finke

R. A.

Ward

T. B.

Smith

S. M.

(1992). Creative cognition: Theory, research, and applications. The MIT Press.

10.

Forgeard

M. J. C.

Kaufman

J. C.

(2016). Who cares about imagination, creativity, and innovation, and why? A review. Psychology of Aesthetics, Creativity, and the Arts, 10, 250-269. doi:10.1037/aca0000042

11.

Gajda

Karwowski

Beghetto

R. A.

(2017). Creativity and academic achievement: A meta-analysis. Journal of Educational Psychology, 109(2), 269-299. doi:10.1037/edu0000133

12.

Gilhooly

K. J.

Fioratou

Anthony

S. H.

Wynn

(2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects. British Journal of Psychology, 98(4), 611-625. doi:10.1111/j.2044-8295.2007.tb00467

13.

Graham

McKeown

Kiuhara

Harris

K. R.

(2012). A meta-analysis of writing instruction for students in the elementary grades. Journal of Educational Psychology, 104(4), 879-896. doi:10.1037/a0029185

14.

Gralewski

Karwowski

(2013). Polite girls and creative boys? Students' gender moderates accuracy of teachers' ratings of creativity. Journal of Creative Behavior, 47, 290-304. doi:10.1002/jocb.36

15.

Guilford

J. P.

(1950). Creativity. American Psychologist, 5(9), 444-454. doi:10.1037/h0063487

16.

Jaarsveld

Lachmann

Van Leeuwen

(2012). Creative reasoning across developmental levels: Convergence and divergence in problem creation. Intelligence, 40(2), 172-188. doi:10.1016/j.intell.2012.01.002

17.

Jang

(2017). Sources of scientific creativity: Participant observation of a public research institute in Korea. Journal of Open Innovation: Technology, Market, and Complexity, 3(1), 1-16. doi:10.1186/s40852-017-0052-5

18.

Kaufman

J. C.

(2019). Self-assessments of creativity: Not ideal, but better than you think. Psychology of Aesthetics, Creativity, and the Arts, 13, 187-192. doi:10.1037/aca0000217

19.

Kaufman

J. C.

Plucker

J. A.

Baer

(2008). Essentials of creativity assessment. John Wiley & Sons.

20.

Kim

K. H.

(2008a). Commentary: The torrance tests of creative thinking already overcome many of the perceived weaknesses that Silvia et al.'s (2008) methods are intended to correct. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 97-99. doi:10.1037/1931-3896.2.2.97

21.

Kim

K. H.

(2008b). Meta-analyses of the relationship of creative achievement to both IQ and divergent thinking test scores. Journal of Creative Behavior, 42, 106-130.

22.

Kitsantas

Zimmerman

B. J.

(2002). Comparing self-regulatory processes among novice, non-expert, and expert volleyball players: A microanalytic study. Journal of applied sport psychology, 14(2), 91-105. doi:10.1080/10413200252907761

23.

Liu

Jiang

Shalley

C. E.

Keem

Zhou

(2016). Motivational mechanisms of employee creativity: A meta-analytic examination and theoretical extension of the creativity literature. Organizational Behavior and Human Decision Processes, 137, 236-263. doi:10.1016/j.obhdp.2016.08.001

24.

Lodewyk

K. R.

Winne

P. H.

Jamieson-Noel

D. L.

(2009). Implications of task structure on self-regulated learning and achievement. Educational Psychology, 29(1), 1-25. doi:10.1080/01443410802447023

25.

Mann

E. L.

(2009). The search for mathematical creativity: Identifying creative potential in middle school students. Creativity Research Journal, 21, 338-348. doi:10.1080/10400410903297402

26.

Marschand

(2013). Handbook for coordinators of high ability programs. Indiana Department of Education.

27.

McHugh

M. I.

(2012). Interrater reliability: The kappa statistic. Biochemia Medica - Journal of the Croatian Society of Medical Biochemistry, 22(3), 276-282. doi:10.11613/BM.2012.031

28.

Miller

L. D.

Martinez

Y. J.

Shumka

Baker

(2014). Multiple informant agreement of child, parent, and teacher ratings of child anxiety within community samples. Canadian Journal of Psychiatry, 59, 34-39. doi:10.1177/0829573514566377

29.

Myhill

Wilson

(2013). Playing it safe: Teachers’ views of creativity in poetry writing. Thinking Skills and Creativity, 10, 101–111. doi:10.1016/j.tsc.2013.07.002

30.

OECD (2018). The future of education and skills: Education 2030. Retrieved from https://www.oecd.org/education/2030/E2030%20Position%20Paper%20(05.04.2018).pdf

31.

Perry

VandeKamp

K. O.

Mercer

Nordby

(2002). Investigating teacher-student interactions that foster self-regulated learning. Educational Psychologist, 37(1), 5-15. doi:10.1207/00461520252828519

32.

Plucker

J. A.

(1999). Is the proof in the pudding? Reanalyses of Torrance’s (1958 to present) longitudinal data. Creativity Research Journal, 12, 103-114. doi:10.1207/s15326934crj1202_3

33.

Plucker

J. A.

Beghetto

R. A.

Dow

G. T.

(2004). Why isn't creativity more important to educational psychologists? Potentials, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83-96.

34.

Reiter-Palmon

Forthmann

Barbot

(2019). Scoring divergent thinking tests: A review and systematic framework. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 144-152. doi:10.1037/aca0000227

35.

Ridgley

L. M.

Rubenstein

L. D.

Finch

W. H.

(2020). Issues and opportunities when using rating scales to identify creatively gifted students: Applying an IRT approach. Gifted and Talented International, 34, 6-18.doi:10.1080/15332276.2020.1722041

36.

Rubenstein

L. D.

Callan

G. L.

Ridgley

L. M.

Henderson

(2019). Students’ strategic planning and strategy use during creative problem solving: The importance of perspective-taking. Thinking Skills and Creativity, 34. doi:10.1016/j.tsc.2019.02.004

37.

Rubenstein

L. D.

Callan

G. L.

Speirs Neumeister

K. L.

Ridgley

L. M.

Hernández Finch

M. E.

(2020). How problem identification strategies influence creative outcomes. Contemporary Educational Psychology, 60. doi:10.1016/j.cedpsych.2020.101840

38.

Rubenstein

L. D.

Ridgley

L. M.

Callan

G. L.

Karami

Ehlinger

(2018). How teachers perceive factors that influence creativity development: Applying a social cognitive theory perspective. Teaching and Teacher Education, 70, 100-110.

39.

Runco

M. A.

(2008). Commentary: Divergent thinking is not synonymous with creativity. Psychology of Aesthetics, Creativity, and the Arts, 2, 93-96.

40.

Runco

M. A.

Millar

Acar

Cramond

(2010). Torrance tests of creative thinking as predictors of personal and public achievement: A fifty-year follow-up. Creativity Research Journal, 22, 361-368. doi:10.1080/10400419.2010.523393

41.

Runco

M. A.

Plucker

J. A.

Lim

(2001). Development and psychometric integrity of a measure of ideational behavior. Creativity Research Journal, 13(3), 393-400. doi:10.1037/a0020499

42.

Ryser

G. R.

McConnell

(2004). Scales for identifying gifted students. Prufrock Press.

43.

Schunk

Greene

(2017). Handbook of self-regulation of learning and performance. Routledge. doi:10.4324/9781315697048

44.

Silvia

P. J.

Kaufman

J. C.

Pretz

J. E.

(2009). Is creativity domain-specific? Latent class models of creative accomplishments and creative self-descriptions. Psychology of Aesthetics, Creativity, and the Arts, 3(3), 139. doi:10.1037/a0014940

45.

Thys

Sabbe

De Hert

(2014). The assessment of creativity in creativity/psychopathology research—a systematic review. Cognitive Neuropsychiatry, 19(4), 359-377. doi:10.1080/13546805.2013.877384

46.

Torrance

E. P.

(2008). Torrance Tests of creative thinking: Norms-technical manual: Figural forms A & B. Scholastic Testing Service Inc.

47.

Westby

E. L.

Dawson

V. L.

(1995). Creativity: Asset or burden in the classroom?. Creativity Research Journal, 8(1), 1-10. doi:10.1207/s15326934crj0801_1

48.

Winne

P. H.

Jamieson-Noel

(2002). Exploring students’ calibration of self-reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551-572. doi:10.1016/S0361-476X(02)00006-1

49.

Young

A. E.

Worrell

F. C.

(2018). Comparing metacognition assessments of mathematics in academically talented students. Gifted Child Quarterly, 62(3), 259-275. doi:10.1177/0016986218755915

50.

Zimmerman

B. J

(2000). Attaining self-regulation: A social-cognitive perspective. In Boekaerts

Pintrich

P. R.

Zeidner

(Eds.), Handbook of self-regulation (pp. 13–39). Academic Press. doi:10.1016/B978-012109890-2/50031-7