Students’ General Knowledge of the Learning Process: A Mixed Methods Study Illustrating Integrated Data Collection and Data Consolidation

Abstract

There were two purposes for this mixed methods study: to investigate (a) the realistic meaning of awareness and understanding as the underlying constructs of general knowledge of the learning process and (b) a procedure for data consolidation. The participants were 11th-grade high school and first-year university students. Integrated data collection and data transformation provided for positive but small correlations between awareness and understanding. A comparison of the created combined and integrated new data sets showed that the integrated data set provided for an expected statistically significant outcome, which was in line with the participants’ developmental difference. This study can contribute to the mixed methods research because it proposes a procedure for data consolidation and a new research design.

Keywords

integrated data combined data data transformation data consolidation educational research

Educational researchers employing different theoretical approaches to learning and development agree that metacognitive knowledge involves an awareness and understanding, or beliefs and knowledge, about one’s own cognitive processes (Hacker, Dunlosky, & Graesser, 2009; Pintrich, 2002). However, they also agree that we still understand too little about the phenomenon of metacognitive knowledge (cf. Efklides & Misailidi, 2010; Veenman, Van Hout-Wolters, & Afflerbach, 2006). Specifically, in the context of learning, there are no empirical studies on the theoretical distinction between awareness and understanding; thus, the realistic meaning of awareness and understanding seems to be an overlooked topic. In other words, the measurements of awareness and understanding are typically distinct from one another, as they involve quantitative measurements and qualitative measurements, respectively. To investigate their realistic meaning—or their comparative function from the perspective of education (Maxwell & Mittapalli, 2010)—a mixed methods study is required that genuinely integrates the quantitative and qualitative data.

Mixed methods research is particularly useful for investigating complex and multifaceted phenomena, such as metacognitive knowledge because the quantitative and qualitative data can complement one another and lead to a better understanding of such phenomena (Johnson, Onwuegbuzie, & Turner, 2007). With respect to mixing quantitative and qualitative data, a distinction can be made between combining or connecting, where the types of data are largely independent of one another, and integration or merging, where the types of data are either highly or largely dependent of one another (Creswell & Plano Clark, 2011; Woolley, 2009; Yin, 2006). However, Bryman (2007) concluded that it is often difficult to genuinely integrate findings in mixed methods research studies due to several barriers (i.e., losing sight of the rationale for conducting mixed methods research, uncertainty about how to connect the quantitative and qualitative data, and writing down the results).

The mixing of quantitative and qualitative methods can take place at data collection, data analysis, and data interpretation (Creswell & Plano Clark, 2011; Greene, 2007; Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009). Mixing in mixed methods studies mostly occurs at the level of interpretation but rarely at the levels of data collection and data analysis (Newman, Onwuegbuzie, & Hitchcock, 2015; Niglas, 2004; Onwuegbuzie, Slate, Leech, & Collins, 2009). This finding seems to refer in particular to data consolidation or the creation of data sets as a new variable for data analysis, which can be useful to uncover fresh insights or new perspectives (Caracelli & Greene, 1993). Then, the creation of new data sets to enable data analysis is not without its challenges where genuine integration of the quantitative and qualitative data is concerned.

Recently, mixed methods researchers have focused on enabling data integration for data analysis, and have proposed mixed data analysis approaches, such as computer software (Bazeley, 2006), Bayes statistics (Newman, Onwuegbuzie, & Hitchcock, 2014), and advanced statistical techniques (Onwuegbuzie & Hitchcock, 2015). An important point regarding these approaches is the question of whether one could say confidently that different types of data are integrated. This study aimed to contribute to this line of research by including a second mixed methods research question to investigate whether the newly created data sets included genuinely integrated data.

Therefore, there were two purposes for this study. The first was to obtain a better understanding of the students’ awareness in relation to their understanding of general knowledge of the learning process (GKLP). For this purpose, a mixed methods study was required because of the collection of two kinds of data. Next, although researchers may not want to mix these kinds of data, they could be mixed because of their theoretical and practical (see Data Integration and Consolidation section) relationships. Specifically, this offered the possibility of obtaining a better understanding of the data creation procedure. Therefore, the second purpose of this study was to integrate, with a reasonable amount of confidence, these two kinds of data into new data sets for data analysis. This article begins with an overview of the main steps in this study through a discussion of the mixed methods research literature and the phenomenon of GKLP.

Data Integration and Consolidation

In mixed methods research, data integration relates to the thoughtful integration of methods and paradigmatic assumptions to obtain a meaningful understanding of the subject under investigation (Freshwater & Cahill, 2013; Greene & Hall, 2010). This follows a realistic philosophy in which different valid perspectives of the world lie at the basis of research studies (Maxwell & Mittapalli, 2010). Nevertheless, genuine data integration requires specific attention to a study’s design. First, whether data integration and data consolidation were appropriate had to be determined (Tashakkori & Creswell, 2007). In this study, the purpose or conceptual rationale was to investigate awareness and understanding as the underlying constructs of the complex phenomenon of GKLP, which would require the integration of different data types. In other words, the theoretical relationship between awareness and understanding might enable the attempt to create a genuinely integrated data set.

Second, the rationale for using a mixed methods design had to be established (Bryman, 2006). For example, Greene, Caracelli, and Graham (1989) provided a framework for the rationale in mixed methods studies that included triangulation, complementarity, development, initiation, and expansion. The rationale of this study was complementarity (i.e., completeness; Bryman, 2007) in terms of obtaining a better understanding of awareness and understanding as the underlying constructs of GKLP.

Third, appropriate data collection methods (e.g., interviews, observations, questions, tests, and checklists) had to be determined for this study. For example, regarding data collection characterized by integration, Yin (2006) argued that it is important in mixed studies to tighten the use of quantitative and qualitative data collection methods. In other words, the more the “items” overlap or complement one another, the more they can be part of a single study. This study established an integrated question design for data collection.

Fourth, to integrate different types of data to enable data consolidation, quantitizing and qualitizing were options (Collingridge, 2013; Sandelowski, Voils, & Knafl, 2009; Seltzer-Kelly, Westwood, & Peňa-Guzman, 2012). Quantitizing and qualitizing refer to the converting of one type of data into the other to allow for statistical or thematic analyses (Caracelli & Greene, 1993; Tashakkori & Teddlie, 2010). Quantitizing means assigning numeric values to nonnumeric data (Sandelowski, 2001; Sandelowski et al., 2009), and qualitizing means converting quantitative data into narrative representations (Jang, McDougall, Pollon, Herbert, & Russell, 2008; Teddlie & Tashakkori, 2009). In this study, quantitizing was appropriate because the qualitative data provided for hierarchical categories.

Quantitizing based on the characteristics of the qualitative data means following a pragmatic stance in that a workable solution is sought for a problem (Freshwater & Cahill, 2013) in terms of what would be practical (Johnson et al., 2007). Generally, the decision regarding quantitizing and qualitizing, as well as the selection of data collection methods, data analysis techniques, and the place of data integration in the design, can be influenced by the different methodologies and philosophies that are favored by communities of scholars in a research field (Freshwater & Cahill, 2013; Mertens, 2012; Morgan, 2007; Onwuegbuzie & Combs, 2011). That is, the mixing of methods can refer to a dialectic stance (Greene, 2007), a design-related stance (Creswell & Plano Clark, 2011), and a pragmatic stance (Johnson et al., 2007; Johnson & Onwuegbuzie, 2004; Teddlie & Tashakkori, 2009).

Finally, with regard to data consolidation, Caracelli and Greene (1993) provided a review of program evaluation studies that described data consolidation as the creation of new variables, expressed either in a quantitative or qualitative form, to enable further data analysis. In this study, the creation of two new data sets from the collected data types that measured awareness and understanding of GKLP enabled a comparison that investigated their appropriateness in terms of genuine integration.

General Knowledge of the Learning Process: A Complex Phenomenon

Metacognition, which is generally defined as thinking about cognitive processes, is divided into metacognitive knowledge and executive processes (Brown, 1987; Flavell, 1979). In recent decades, research on metacognition in education has focused primarily on the executive processes responsible for monitoring and control (e.g., Hacker et al., 2009; National Research Council, 2000; Zimmerman & Schunk, 2011) and to a much lesser degree on metacognitive knowledge (Dinsmore, Alexander, & Loughlin, 2008; Efklides & Misailidi, 2010). Consequently, educational researchers must develop a better understanding of the phenomenon of metacognitive knowledge.

Metacognitive knowledge is defined as an awareness and understanding of cognitive processes (Hacker et al., 2009; Pintrich, 2002). Schraw (1998) argued that metacognitive knowledge appears to span a wide variety of subject areas and domains. In other words, as students acquire more metacognitive knowledge in a number of domains, they may develop general metacognitive knowledge that cuts across all academic domains (Neuenhaus, Artelt, Lingel, & Schneider, 2011). Although researchers from a variety of theoretical perspectives agree that students become more aware and knowledgeable about cognitive processes with development, research has shown that many students fail to develop accessible metacognitive knowledge (De Backer, Van Keer, & Valcke, 2012; Pintrich, 2002). Therefore, a better understanding of metacognitive knowledge might lead to insights and help students develop this knowledge (van Velzen, 2012; van Velzen, 2016).

Although awareness and understanding theoretically make up the concept of metacognitive knowledge, they are mostly studied separately. Thus, studies on awareness primarily use closed-ended rating-scale questions that enable participants to state how frequently they think about their metacognitive knowledge (e.g., Kleitman & Stankov, 2007; Schraw & Dennison, 1994). Such studies are quantitative studies. Conversely, studies on an understanding of metacognitive knowledge have used open-ended questions (Annevirta & Vauras, 2001; Swanson, 1990), interviews (Kipnis & Hofstein, 2008; McCrindle & Christensen, 1995), and observations (cf. Hurme & Järvelä, 2005; Schofield, 2012). These latter studies are qualitative studies.

In this study, the focus was on the realistic meaning of the underlying constructs of awareness and understanding as part of the phenomenon of metacognitive knowledge. That is, the conceptual rationale was to obtain a better understanding of the function of awareness and understanding in relation to each other with regard to GKLP.

Conceptual Framework of General Knowledge of the Learning Process

The theories developed by Flavell (1979) and Brown (1987) underpin most studies on metacognitive knowledge (e.g., Kipnis & Hofstein, 2008). In this study, synthesizing Flavell and Brown’s theories to define GKLP resulted in two starting points. First, for students to become self-directed learners in that they can think through their learning before beginning to execute their learning activities, GKLP is essential (Brown, 1987). That is, students need GKLP to set up an effective learning plan to direct learning as distinct from setting up an efficient action plan to manage the executive processes. Second, GKLP refers to the interaction between the learning person and the learning environment (Brown, 1987), which includes the learning person, learning strategies, and learning tasks (Flavell, 1979). In other words, by emphasizing the similarities more than differences of opinion, GKLP will include three components: (a) general knowledge of developing cognitive knowledge (GKDCK), (b) general knowledge of the learning-task demands (GKLTD), and (c) general knowledge of oneself as a learner (GKOL; see Figure 1).

Figure 1.

Theoretical framework of general knowledge of the learning process (van Velzen, 2016).

GKDCK included Flavell’s concept of strategy characteristics and Brown’s subcomponents of declarative, procedural, and conditional metacognitive knowledge from a cognitive perspective. That is, it included knowledge of general cognitive factors that influence the learning of subject matter (i.e., knowing about developing cognitive knowledge). Consequently, the subcomponent of declarative GKLP did not include GKOL (Brown, 1987; Schraw & Moshman, 1995) because this was a separate component in this study (cf. Pekrun, Goetz, Titz, & Perry, 2002).

GKLTD referred to Flavell’s concept of task requirements and Brown’s concept of task conditions. Research showed that effective performance on any task depends on the students’ awareness of the task demands, as well as their understanding about meeting these demands (Armbruster, Echols, & Brown, 1982). Therefore, this component included an understanding of the general differences in learning tasks and the consequences of effective learning (cf. Stillman, 2004).

GKOL referred to Flavell’s concept of person characteristics and Brown’s concept of strengths and weaknesses of oneself as a learner. Because GKOL, or the students’ self-schemas that include generalizations of self-perceptions about who they are in varying learning contexts, does not refer to situation-specific knowledge. This can include a person’s general knowledge and beliefs regarding memory, learning ability, and self-concept (Pintrich & Schunk, 2001).

To conclude, GKLP is a complex and multifaceted phenomenon in that it is late developing, refers to awareness and understanding, and includes multiple components and subcomponents. The following research question was thus posited:

Research Question 1: What is the realistic meaning of awareness and understanding with respect to GKLP?

Accordingly, because the theoretical relatedness and the difference in the types of data of awareness and understanding of GKLP enabled data consolidation, the following mixed methods research question was stated:

Research Question 2: How can an integrated data set appropriate for data analysis be obtained so that it provides an adequate level of confidence for capturing GKLP?

Method

Research Design

Different mixed methods research designs exist in educational research (Creswell, 2011; Leech & Onwuegbuzie, 2009; Teddlie & Tashakkori, 2009). The design of this study included the mixing of quantitative and qualitative approaches at many stages (Onwuegbuzie & Johnson, 2006) of the investigation (see Figure 2). That is, integrated data collection and data transformation preceded data consolidation that consisted of the creation of two types of data sets that enabled comparisons. Therefore, the design of this study resembled a convergent (Creswell & Plano Clark, 2011) and conversion (Teddlie & Tashakkori, 2009) design, though a conversion consolidation–investigation design might be a more appropriate description because the consolidation phase was used to investigate whether genuinely integrated data sets could be created.

Figure 2.

Mixed methods design for this study.

Participants

Adolescent students were the targeted group in this study because they already have developed some metacognitive knowledge (Schneider, 2008) but they are still further developing this knowledge (Weil et al., 2013). In addition, 11th-grade students, because they form the middle grade of senior high school students (i.e., Grades 10-12, preparing for university entrance), and first-year university students were selected. That is, during discussions with senior high school students and their teachers prior to this study, it became evident that 12th-grade high school students, in particular, learn how to become self-directed learners in their final school year. Therefore, 11th-grade high school students are likely to differ developmentally from first-year university students with respect to GKLP. This is also in line with the research literature (Weil et al., 2013).

The high school students came from six schools that had subscribed to a call for research participants. The university students came from the Pedagogy and Educational Psychology Department of a University in the Netherlands. Random assignment of the high school classes and the university’s mentor groups to the three GKLP groups resulted in six subsamples (see Table 1). Next, although sex is not a variable in this study, examination (Mann–Whitney U test) showed that the high school women provided for significantly higher level responses regarding GKLTD (n = 96; skewness = −.42; p = .008) and GKOL (n = 111; skewness = −.42; p = .00).

Table 1.

Summary of the Samples per Study Variable.

	High school					University
Component	n	Men	Women	Unknown	M _age	n	Men	Women	Unknown	M _age
GKDCK	109	41	66	2	17	30	1	29	0	19
GKLTD	96	38	57	1	17	55	3	48	4	19
GKOL	111	43	65	3	17	47	4	40	3	19
Total	316	122	188	6	17	132	8	117	7	19

Note. GKDCK = general knowledge of developing cognitive knowledge; GKLTD = general knowledge of learning-task demands; GKOL = general knowledge of oneself as a learner.

Data collection took place at the beginning of the school year. These participants had followed the obligatory school courses in reflection (i.e., reconsidering learning experiences to obtain improvements) and learning techniques (e.g., summarizing, note taking, and questioning) though they received no specific training in developing GKLP.

Instruments

By developing an integrated question design for data collection, a relatively large amount of students could participate. In other words, the pragmatic decision to use questions instead of, for instance, interviews resulted in obtaining a relatively large sample (Freshwater & Cahill, 2013; Johnson et al., 2007). The integrated question design consisted of related or integrated closed- and open-ended questions that needed to collect quantitative and qualitative data of an equal status in that both the collected numbers and words would have the same opportunity to provide information regarding GKLP.

The related closed- and open-ended self-report questions required responses in a written format regarding the three GKLP components (see Table 2). To this end, each question consisted of a statement that began with “I know . . . ” to refer to GKLP. That is, the statements consisted of general information that focused on the students’ stable GKLP across multiple school subjects and academic domains. Stated otherwise, the statements explained GKLP concepts per each component of GKLP (e.g., awareness and understanding of effective learning through certain learning techniques, such as summarizing, and regarding certain learning-task demands, such as availability) to enable construct refinement (cf. Pawson & Tilley, 2004).

Table 2.

Overview of the Theoretical Subcomponents.

Component	Subcomponent	Definition	Characteristic example
GKDCK^a	Declarative	Knowing that learning techniques are required to learn subject matter.	I know if I understand the essence of subject matter (RC), because I focus on . . .
	Procedural	Knowing how to perform learning techniques.	I know if I can summarize subject matter (RC), because I focus on . . .
	Conditional	Knowing when to apply learning techniques.	I know when a learning technique will work (RC), because I focus on . . .
GKLTD^b	Basic	Knowledge of the learning-task requirements of difficulty, availability, and comprehensiveness.	I know if a learning task is difficult or easy (RC), because difficult learning tasks include . . . and easy learning tasks include . . .
	Conditions	Knowledge of the memory-related, perceptual, engagement, language-related, representational, and organizational task conditions.	I know if a learning task requires engagement (RC), because a learning task requiring engagement include . . . and a learning task not requiring engagement . . .
GKOL^c	Metamemory	Knowledge of the personal strength and weaknesses regarding memorizing.	I know which subject matter I find easy or difficult to memorize (RC), because subject matter is memorized easily . . . and it is memorized with difficulty . . .
	Studying	Knowledge about oneself regarding one’s reasons for studying in varying school-learning situations.	I know my reasons for doing homework extensively or superficially (RC), because I study extensively . . . and I study superficially . . .

Note. GKDCK = general knowledge of developing cognitive knowledge; GKLTD = general knowledge of learning-task demands; GKOL = general knowledge of oneself as a learner; RC = response choice.

Adapted from Schraw and Dennison (1994) and Weinstein and Mayer (1986). ^bAdapted from Brown (1987), Flavell (1979), and Stillman (2004). ^cAdapted from Kleitman and Stankov (2007) and Schraw and Dennison (1994).

First, the response choice to a closed-ended question inquiring about the frequency that a student rated his or her awareness of a concept of GKLP, measured awareness. Next, an open-ended question followed the closed-ended question inquiring about that student’s understanding of the GKLP concept by enabling him or her to respond in his or her own words. For example, “I know if information relates to my prior knowledge: no, sometimes, neutral, often, always (i.e., response choice)—because I focus on . . . ” (i.e., constructed response).

In this way, mixed data were collected (Johnson & Christensen, 2014) via identical sampling (Leech & Onwuegbuzie, 2009). Additionally, the data collection questions were not survey questions because in this study, each question included a related quantitative and qualitative component. Because these questions would be new to the participants, the careful construction of the questions resulted in using natural and familiar language.

The pilot of the GKLP questions took place with an 11th-grade high school student and four first-year university students in the earth sciences. The time it took these students to respond to the GKLP questions and the comments they provided during the standardized open-ended interview afterward resulted in the selection of separate samples per each component of GKLP. In this way, the students would have the time and opportunity to respond thoughtfully and comprehensively to the questions. Including three separate samples also prevented the questions regarding GKDCK, which mentioned certain study techniques, from providing clues about the GKLTD and GKOL, and vice versa.

Next, determining whether the participants had understood the questions, that is, in whether they had understood the intended construct, required the establishment of related data collection instruments for the closed-ended questions per each component of GKLP to obtain convergent validity. For general knowledge about developing cognitive knowledge, the scale knowledge-of-cognition questions of Schraw and Dennison’s (1994) Metacognitive Awareness Instrument (coefficient alpha = .88 for 100-point rating scale) was used and included 5-point rating-scale frequency questions. For example, “I know what kind of information is most important to learn.”

For GKLTD, the developed questions referred to the students’ general strategic management of tasks, which is in line with the definitions of Entwistle, McCune, and Walker (2001). The strategic management of task questions assumed that comprehension of different learning-task demands is required when it is a student’s intention to achieve sufficient grades by managing his or her learning strategically. Six questions on a 5-point rating scale regarding frequency were established that inquired about (a) managing learning time efficiently, (b) finding the right learning materials and conditions, (c) being alert to learning assessment requirements, (d) monitoring the effectiveness of one’s learning, and (e) gearing learning to the perceived preferences of teachers. An example was “Generally, I can manage my learning time effectively.”

Regarding GKOL, the questions of the Academic Self-Concept test (Marsh & Yeung, 1997) were used (coefficient alpha = .93). That is, the self-concept of academic learning also includes knowledge of one’s strengths and weaknesses as a learner. Although the research of Marsh and Yeung indicated a relationship between subject-specific academic self-concepts and subject-specific achievements, research also showed that adolescents’ general academic self-concepts correlated with achievement, thereby suggesting the existence of a general academic self-concept for adolescents (Byrne & Worth Gavin, 1996). Therefore, the rephrasing of the Academic Self-Concept test questions in a general fashion resulted in 5-point rating scale questions regarding frequency. Examples were “Generally, I get good grades” and “Generally, I learn things quickly.”

Procedure

The participants responded to the questions individually and were specifically encouraged to respond to the open-ended questions as comprehensively as possible by elaborating and explaining their responses. Furthermore, the information presented to the participants mentioned that their responses would be used only for research purposes. Finally, either a familiar teacher (i.e., high school) or mentor (i.e., university) chaired the data collection sessions. They received instruction not to respond to the participants’ questions.

Data Analysis

General

Cronbach’s alpha was used to investigate the reliability of the item scales. Nonparametric and less-restrictive statistical techniques were used, such as Spearman rank-order correlations and principal component analysis, because the quantitizing provided for ordinal-level variables (Sandelowski et al., 2009) and the subscales did not always obtain a reliability >.70 (Nunnally, 1978). For convergent validity, Spearman rank-order correlations indicated the statistical significance of the completed closed-ended items and the related constructs, because the complex and multifaceted nature of GKLP was likely to result in small correlations (cf. Sperling, Howard, Staley, & DuBois, 2004).

Qualitative

The analysis of the qualitative data took place by interpreting the content of the open-ended responses (Miles, Huberman, & Saldana, 2014). First, the reading of the responses resulted in finding meaningful clusters. Next, the content analysis of the responses took place by copying them and placing them into different matrices. This resulted in the three inductively obtained categories of prelevel, simple-level, and complex-level GKLP. Next, the responses were coded per category and further searched for subcategories. Finally, the categories were organized in a classification system and intercoder reliability was obtained via another coder who was informed (i.e., not trained) about the categories. Examination (Clark-Carter, 2010) showed a Cohen’s kappa of .67, which can be regarded as good.

Quantitizing

The following considerations, including postpositivism and the interpretive researchers’ arguments, addressed the question of whether quantitizing or qualitizing would be appropriate in this study. First, the closed-ended questions could have collected the participants’ information that was considered inadequately or “untruthful” (e.g., socially desirable). Moreover, the open-ended questions could have identified indirect information. That is, a response needed to be constructed. However, the relationship between the two kinds of questions could compensate for the possible disadvantages of the closed- and open-ended questions. Second, a research bias can influence the analysis of the qualitative data. However, data collection took place through a written response to the questions, resulting in the consideration of selective observation, or the selective recording of information, so that it is not an issue. Additionally, intercoder reliability was established.

Therefore, because the qualitative data provided for hierarchical categories that were in line with Schraw and Moshman’s (1995) types of metacognitive theories (i.e., tacit, informal, and formal), the data were quantitized. That is, tacit metacognitive theories defined as including implicit metacognitive knowledge, appeared to be in line with the category of simple-level GKLP. Informal metacognitive theories defined as including some degree of explicit metacognitive knowledge, appeared to be in line with the complex-level GKLP subcategory of descriptive explicit responses. Finally, the definition of formal metacognitive theories, or explicit accounts of metacognitive knowledge, appeared to be in line with the complex-level GKLP subcategory of explanatory explicit responses. Because there were relatively few complex-level explanatory explicit GKLP responses (for a discussion of this kind of responses, see van Velzen, 2013), the qualitative data were quantitized into three levels: the lowest score for prelevel GKLP, the mediate score for simple-level GKLP, and the highest score for complex-level GKLP (i.e., including both descriptive and explanatory explicit responses).

Data Consolidation

Based on the theoretical relationship between awareness and understanding and their (now obtained) positive correlation, data consolidation could take place in this study, though only in search of a procedure for data consolidation. First, principal component analysis based on an extraction of eigenvalues used for each type of data enabled that only those questions that contributed to the investigation of GKLP were included. Additionally, the different types of data required separate principal component analyses. For example, because the readily scored closed-ended questions can have a more consistent scoring (cf. Kubiszyn & Borich, 1996), the quantitative data could provide for more satisfactory factor loadings than the quantitized data.

The practical significance of the factor loadings for the first factor was set to >.40 (see Table 3), for the quantitative data because these data preceded the quantitized data (i.e., indicating the participants’ awareness in terms of familiarity with the GKLP concept mentioned in the item), and .30, for the quantitized data because of the unfamiliarity of the participants with open-ended questions on GKLP. In this way, the obtained data reduction would not result in an insufficient number of items to interpret a complex subject such as GKLP (Hair, Black, Babin, & Anderson, 2010). The obtained factor loadings ranged from .11 to .63, and the data reduction resulted in the deletion of 13% of the items (see Table 3).

Table 3.

Factor Loadings for Exploratory Factor Analysis of the Quantitative and Quantitized Qualitative Data per Sample.

	GKDCK		GKLTD		GKOL
Item	Quanti	Qt-Quali	Quanti	Qt-Quali	Quanti	Qt-Quali
1	.51	.55	.61	.57	.48	.53
2	.46	.42	.29	.50	.58	.52
3	.45	.43	.46	.32	.56	.36
4	.39	.53	.50	.52	.54	.57
5	.42	.57	.48	.60	.50	.49
6	.36	.40	.27	.42	.51	.53
7	.44	.54	.33	.44	.52	.41
8	.53	.45	.56	.52	.53	.54
9	.44	.52	.29	.46	.48	.43
10	.39	.40	.44	.47
11	.63	.53	.11	.49
12	.41	.45	.56	.54
13	.29	.46
14	.46	.54
15	.37	.54
16	.50	.50
17	.42	.51
18	.47	.53

Note. GKLTD = general knowledge of learning-task demands; GKOL = general knowledge of oneself as a learner; GKDCK = general knowledge of developing cognitive knowledge. Extraction method: Principal component analysis based on eigenvalue. Factor loadings ≤.40, for quantitative data (Quanti), and ≤30, for quantitized qualitative data (Qt-Quali) are in boldface.

Next, the created data sets had to differ in terms of the degree of integration, resulting in a numerically combined (NC) and a meaningfully integrated (MI) data set. That is, NC had to result in a combination of data types, whereas MI had to result in genuinely integrated data types to enable a comparison of the data creation procedures. Although retaining the initial data results were essential, the characteristics of the quantitative and the quantitized data required investigation to understand their contribution to the creation of NC and MI. Table 4 shows the psychometric properties of the quantitative and quantitized data. First, the alpha coefficient was low for the university subsamples but not for the high school subsamples. Therefore, sample differences accounted for the differences in reliability but not for the differences in the data types. Second, the quantitative data had a more consistent scoring (α = 76.67, which is >.70) than the quantitized data (α = 67.83, which is <.70), though the quantitative data was skewed (ranging from −.98 to −.19). Only the quantitized data provided for statistically significant developmental differences, indicating that this was important information that needed to be included in the data sets that would be created. Therefore, the quantitized data needed a greater weight than the quantitative data in that the created data sets had to resemble a normal distribution.

Table 4.

Psychometric Properties, Significance of Spearman Correlations With the Related Constructs, and Mann–Whitney U Tests for Developmental Differences.

	High school				University				U test
Component	n	M (SD)	α	r _s	n	M (SD)	α	r _s	p
GKDCK	109				30
Quanti		3.01 (0.41)	.79	.44**		2.86 (0.66)	.84	.73**	.30^a
Qt-Quali		1.01 (0.41)	.85			1.02 (0.27)	.61		.89^a
MAI-k		2.51 (0.46)	.81			2.67 (0.40)	.79
GKLTD	96				55
Quanti		3.07 (0.68)	.75	.50**		2.94 (0.95)	.76	.47**	.19^a
Qt-Quali		0.76 (0.29)	.71			0.96 (0.24)	.54		.00
SMT		3.64 (0.67)	.65			3.92 (0.60)	.72
GKOL	111				47
Quanti		3.01 (0.58)	.76	.40**		2.95 (0.47)	.70	.22	.48^a
Qt-Quali		1.06 (0.23)	.71			1.23 (0.23)	.65		.01
ASC		3.47 (0.62)	.59			3.27 (0.59)	.72

Note. GKDCK = general knowledge of developing cognitive knowledge; GKLTD = general knowledge of learning-task demands; GKOL = general knowledge of oneself as a learner; Quanti = quantitative data; Qt-Quali = quantitized qualitative data; MAI-k = knowledge about cognition questions of the Metacognitive Awareness Instrument; SMT = strategic management of tasks questions; ASC = Academic Self-Concept test.

Equal variances assumed.

p < .05. **p < .01.

Table 5 shows the creation procedure for NC and MI. The NC data set involved the mere combination of data via a procedure based solely on arithmetic operations. That is, adding the highest quantitized score multiplied by two to the highest quantitative score resulted in eight (4 + 2 × 2 = 8). The dividing of this maximum score of eight by five, to obtain a 5-point score, resulted in a newly obtained scoring that enabled the recoding of the original scores of the participants. In this way, the combined data set consisted of numbers determining the creation procedure.

Table 5.

Description of the Creation Procedure for the Combined and Integrated Data Sets and the Distribution of the Scores.

Variable	Definition	Procedure	Distribution
NC	Adding the highest open-ended score multiplied by 2 to the highest quantitized closed-ended score resulted in 8 as the maximally possible score: 4 + (2 × 2). Because each closed-ended score could be accompanied by each of the three open-ended scores, the maximum score of 8 was divided by 5 to obtain 5-point scoring, resulting in the score of 1.6. This was rounded off.	(1.6 = 2) = 1	6
		(3.2 = 3) = 2	40
		(4.8 = 5) = 3	79
		(6.4 = 6) = 4	12
		(8 = 8) = 5	2
MI	Relating each open-ended score with the highest quantitized closed-ended score (i.e., 4) and with the remaining quantitized closed-ended scores (i.e., 0-3) separately. Except for the highest open-ended score that was connected once with the closed-ended score.	0 + (0-3) = 1	9
		0 + 4 = 2	15
		1 + (0-3) = 3	78
		1 + 4 = 4	35
		2 + (0-4) = 5	2

Note. NC = numerically combined data; MI = meaningfully integrated data. The scoring ranged from 0 to 4.

The MI data set involved the genuine integration of data by taking the levels of GKLP, which were presented in the quantitized data as the starting point to which the interpretation of the quantitative data results were added. That is, of the quantitative data, the score of four (i.e., “always” interpreted as occurring often) and the scores of zero up to and including three (i.e., “not” to “often” interpreted as occurring sometimes) were added to each score of the quantitized data. In this way, the quantitative data were interpreted, while at the same time, the extra weight of the quantitized data was realized in that the quantitative scores were divided into high (i.e., 4) or low (i.e., 0-3) scores. In addition, the highest score of the quantitized data was used only once in the creation of MI, because it included few explanations of GKLP. Therefore, the meaningfulness underlies the creation of MI because the meaning behind the numbers rather than the numbers themselves determined the creation procedure.

To compare NC and MI, three common statistical analyses for construct examination (Collingridge, 2013) were calculated: (a) reliability to determine the internal consistency of the items, (b) examination of the statistical significance regarding the relationship between the closed-ended items and the related constructs, and (c) Mann–Whitney U tests to examine the hypothesis of whether the NC and MI data sets would provide for statistically significant developmental differences in line with the research literature (Schneider, 2008; Weil et al., 2013).

Results

Quantitative Data

The reliability showed acceptable internal consistency for the three components of GKLP, though the subcomponents did not always obtain the required minimum of .70 (see Table 6). Statistically significant correlations between the closed-ended items and the related constructs were found (p < .01; the correlations ranged from .14 to .74), except for the university subsample of GKOL, indicating that these results could not have occurred by chance. The quantitative data showed that, on average, the participants indicated to be often aware of GKLP (M = 3.93; SD = 0.65).

Table 6.

Psychometric Properties and Significance of Spearman with Related Constructs for the (Sub)Components of the Quantitative Data.

	High school					University
Component	n	M	SD	α	r _s	n	M	SD	α	r _s
GKDCK	109	3.73	0.61	.84	.48**	30	3.89	0.58	.87	.74**
Declarative		3.57	0.68	.64	.48**		3.56	0.63	.64	.51**
Procedural		3.80	0.80	.72	.39**		3.93	0.64	.70	.50**
Conditional		3.85	0.66	.60	.40**		3.96	0.76	.81	.74**
GKLTD	96	4.09	0.56	.79	.52**	55	3.93	0.56	.81	.47**
Basic		3.82	0.73	.57	.32**		3.71	0.70	.52	.32**
Conditions		4.19	0.67	.64	.39**		4.13	0.67	.57	.47**
GKOL	111	4.01	0.58	.78	.40**	47	3.95	0.47	.71	.22
Metamemory		3.83	0.76	.78	.39**		3.60	0.52	.48	.24
Studying		4.21	0.69	.63	.24**		4.31	0.64	.82	.14

Note. GKDCK = general knowledge of developing cognitive knowledge; declarative = general knowledge of developing declarative knowledge; procedural = general knowledge of developing procedural knowledge; conditional = general knowledge of developing conditional knowledge; GKLTD = general knowledge of learning-task demands; basic = general knowledge of basic learning-task demands; conditions = general knowledge of conditional learning-task demands; GKOL = general knowledge of oneself as a learner; metamemory = general knowledge of oneself regarding memory; studying = general knowledge of oneself regarding studying.

p < .05. **p < .01.

Qualitative Data

The responses to the open-ended questions were diverse in that they ranged from short to comprehensive responses. Although open-ended questions may not always provide for the kind of rich and narrative data that interviews can generate, due to the number of questions, the mean amount for two responses to each open-ended question (see Table 7), and the sample size, the obtained degree of detail of the qualitative data was sufficiently comprehensive to enable interpretations (see Table 8).

Table 7.

Descriptive Statistics of the Amount of the Open-Ended Responses.

Component	High school			University
	n	M	SD	n	M	SD
GKDCK	109	1.05	0.52	30	1.17	0.44
Declarative		1.19	0.57		1.34	0.51
Procedural		1.05	0.52		1.20	0.45
Conditional		0.92	0.48		0.97	0.37
GKLTD	96	1.49	0.61	55	1.82	0.54
Basic		1.71	0.66		2.16	0.72
Conditions		1.26	0.56		1.47	0.37
GKOL	111	1.90	0.72	47	1.99	0.59
Metamemory		2.06	0.76		2.23	0.62
Studying		1.73	0.68		1.76	0.56

Note. GKDCK = general knowledge of developing cognitive knowledge; declarative = general knowledge of developing declarative knowledge; procedural = general knowledge of developing procedural knowledge; conditional = general knowledge of developing conditional knowledge; GKLTD = general knowledge of learning-task demands; basic = general knowledge of basic learning-task demands; conditions = general knowledge of conditional learning-task demands; GKOL = general knowledge of oneself as a learner; metamemory = general knowledge of oneself regarding memory; studying = general knowledge of oneself regarding studying.

Table 8.

Examples of Typical Participant Responses to the Open-Ended Questions.

		Response
Component	Question	High school	University
GKDCK	I know if I understand the essence of subject matter, because I focus on . . .	Because I focus on whether I can relate the information to my experiences and to already existing knowledge. I have my own frame of reference.	Because I focus on whether I already know a lot of a subject, and sometimes this may interfere with or promote what I know. When I have less knowledge about a subject I just take up the new information.
GKLTD	I know if a learning task is comprehensive or short, because comprehensive learning tasks . . . and short learning tasks . . .	Comprehensive learning tasks often include an introduction texts and questions requiring insight and understanding and short learning tasks do not require that one will think things through because the answer is already present in the material.	Comprehensive learning tasks include often require that one has to discuss a topic, show in-depth understanding, an opinion, similarities and differences, and applications and short learning tasks include usually knowledge questions such as “What does this concept mean?” and the answers are present in the material.
GKOL	I know in which ways I can memorize, because I can memorize well . . . and I cannot memorize well . . .	What works for me is rehearsing a lot, reading a lot, and finding background information. What does not work is starting too late, because then I am stressed and I start thinking that I cannot do it, and then I cannot concentrate anymore.	What works for me is writing things out, rehearsing in my mind, discussing the material with others, and developing an opinion. What does not work is reading the material while skipping over uninteresting things.

Note. GKDCK = general knowledge of developing cognitive knowledge; GKLTD = general knowledge of learning-task demands; GKOL = general knowledge of oneself as a learner.

The responses to the open-ended questions provided for the hierarchical categories of prelevel (19%), simple-level (61%), and complex-level (20%) GKLP, which was mainly based on explicitness (see Table 9). These kinds of levels appeared to be in line with Schraw and Moshman’s (1995) theory regarding metacognitive knowledge. That is, they argued that explicit metacognitive knowledge is necessary to develop systematic frameworks of mind that allow for making informed choices and for becoming the producer of one’s own development of metacognitive knowledge. In other words, the present results seemed to be in line with their theory stating that explicitness is required for conscious or reflective access to GKLP. Next, the obtained percentages for the GKLP components (Table 10) suggested that it must be difficult for adolescent students to be explicit about GKLP because the results showed a lesser degree of complex-level GKLP responses than simple-level GKLP responses.

Table 9.

Inductively Developed Categories and Subcategories.

Category	%	Definition	Characteristic description
Prelevel	19	Unrelated responses
Blank	9	No information provided
Irrelevant	2	Unclear bits of information or without obvious meaning	Incomprehensible due to irrelevant themes
Incoherent	8	Incorrect information or without related meaning	Illogically connected response to the question
Simple-level	61	Responses without explicit information
Rewriting	22	Descriptions derived from the question provided	No additional information
Examples	17	Descriptions of concrete examples	Mentioning of school disciplines and subjects
External	11	Descriptions of externally provided cues and prompts	Such teachers and textbooks
Feelings	11	Descriptions of information that cannot be recalled explicitly	It cannot be put into words
Complex-level	20	Responses with explicit information
Learning concepts	17	Explicit naming and descriptions of relevant learning concepts	Such as focusing attention, comprehending, and elaborating
Explanations	3	Explicit descriptions providing relevant, structured, and explanatory information	Reasons that amplify usefulness, necessity, and development

Note. Rewriting = rewriting of the question; examples = concrete examples; external = external sources; feelings = feelings of knowing.

Table 10.

Percentages for Pre-, Simple-, and Complex-Level GKLP for the (Sub)Components of the Qualitative Data.

	High school				University
Component	n	Pre	Simple	Complex	n	Pre	Simple	Complex
GKDCK	109	28.74	48.22	23.04	30	28.10	44.36	27.54
Declarative		26.82	51.02	22.16		25.17	56.95	17.88
Procedural		28.83	42.88	28.29		27.34	29.33	43.33
Conditional		28.49	57.72	13.79		24.00	58.67	17.33
GKLTD	96	32.90	57.47	9.63	55	20.63	62.65	16.72
Basic		6.40	70.70	22.90		3.03	66.67	30.30
Conditions		39.40	54.94	5.66		27.68	60.39	11.93
GKOL	111	7.01	79.58	13.41	47	3.75	69.72	26.53
Metamemory		7.92	76.76	15.32		3.83	65.96	30.21
Studying		5.85	83.11	11.04		3.66	74.35	21.99

Furthermore, although the framing of the questions regarding GKLTD and GKOL differed from the questions regarding GKDCK, this appeared to not have influenced the results. Additionally, the typical responses in the GKLP hardly (<1%) included activities concerning efficiency (i.e., how to do it: “I always first do . . . , then I do . . . ”), which in this study was considered distinct from effectiveness (i.e., what is needed to learn effectively). This finding suggested that the participants were able to respond in line with the first theoretical starting point of this study in that their responses for the most part referred to thinking through effective learning.

Data Interpretation

Spearman correlations between the quantitative and quantitized data showed a positive but small relationship (r _s = .17). This result suggested that, when measured in this way, awareness and understanding are hardly related underlying constructs in that both contributed to the phenomenon of GKLP, though somewhat differently. The following examples can illustrate this concept. When adolescent students are aware of GKLP, their understanding can refer to one of three levels. For example, a student who indicated an awareness of GKLP and who provided prelevel GKLP responses could have the following kind of reconsideration: “Do I understand this subject matter?—I think I do.” In this example, the question raised indicates awareness. However, the provided answer does not reveal an understanding of GKLP. In comparison, a student who indicated an awareness of GKLP and who provided complex-level GKLP responses could have the following kind of reconsideration: “Do I understand this subject matter?—Although the summary I made was useful in grasping the overall meaning of the subject matter, it was not specific enough because I could not answer certain practice test questions.” In this latter example, the answer to the question suggests conscious access to GKLP.

Data Interpretation of NC and MI

The results showed higher reliabilities for NC than for MI, except for GKDCK in the high school subsample and GKOL in the university subsample (see Table 11). Regarding the developmental differences, although the mean score of the university participants was higher than that of the high school participants for every NC and MI data set, a statistically significant result (p = .00) was found only for MI regarding GKLTD. This result suggested that the creation procedure of MI provided for a more appropriate or genuinely integrated data set than the creation procedure of NC because a statistically significant developmental difference was expected.

Table 11.

Psychometric Properties and Mann–Whitney U Tests for Developmental Differences.

	High school			University			U test
Variable	n	M (SD)	α	n	M (SD)	α	p
GKDCK	109			30
NC		4.75 (1.20)	.85		4.95 (0.94)	.77	.47^a
MI		3.26 (0.83)	.87		3.37 (0.57)	.66	.61^a
GKLTD	96			55
NC		4.38 (1.13)	.77		4.74 (0.84)	.70	.40^a
MI		2.83 (0.73)	.73		3.17 (0.56)	.61	.07
GKOL	111			47
NC		5.13 (0.83)	.75		5.25 (0.70)	.70	.94^a
MI		3.47 (0.49)	.72		3.59 (0.52)	.71	.00^a

Note. NC = numerically combined data; MI = meaningfully integrated data; GKDCK = general knowledge of developing cognitive knowledge; GKLTD = general knowledge of learning-task demands; GKOL = general knowledge of oneself as a learner.

Equal variances assumed.

p < .05. **p < .01.

Discussion

This mixed methods study resulted in two findings. First, awareness and understanding were constructs that contributed to the phenomenon of GKLP, although in a distinguishable manner. Second, the data consolidation involved an investigation. That is, an investigation of the newly created, NC and MI data sets took place in an attempt to obtain an adequate level of confidence (Onwuegbuzie & Collins, 2014; Onwuegbuzie & Johnson, 2006) about which the data creation procedure captured GKLP most appropriately in term of genuine data integration. Obtaining such confidence in mixed methods studies generally requires enhancing the rigor to reduce interpretive inconsistencies. In this study, interpretive consistency was strived for via pragmatically substantiated decisions, a focus on integration from the onset of the study (i.e., rationale and data collection), and an investigation of the procedure for creating the data set.

The basis for the data creation procedure was a data collection phase that was integrated by relating each closed-ended question with an open-ended question (Woolley, 2009; Yin, 2006). In this manner, diminishing incongruent information, which may occur because of the complex and multifaceted nature of the phenomenon of GKLP, occurred. Further diminishing of incongruent information included the deleting of those items that showed low factor loadings.

The MI data set provided for an expected statistically significant developmental outcome, indicating that a genuinely integrated creation procedure established the MI data sets. However, as the qualitized data provided for two significant developmental findings, and only one MI data set provided for this outcome, this finding suggested that either the creation procedure or the object under investigation contributed to this result.

In this respect, the university sample also may have been a limitation in this study. That is, although the comparison with university students can provide developmental results because they are older, and the results suggest that the courses the university participants followed did not influence their responses, the participants came from one university department, and their relatively new learning situation may have hindered them in responding to the open-ended questions.

Conclusion

This study proposes a procedure for creating and comparing new data sets by investigating the characteristics of these data sets. Consequently, this resulted in presenting a new research design, labeled a conversion consolidation–investigation design. I hope that this attempt to find a procedure for data consolidation can be useful for educational mixed methods researchers who want to apply mixed data analysis approaches. This and other procedures of data creation and comparisons may provide for a better understanding of the possibilities and consequences of genuine data integration for data analysis and interpretation. To this end, future research studies might focus on other study objects, methods, designs, qualitizing versus quantitizing, and paradigmatic stances to obtain a better understanding of the strengths of mixed methods research regarding data integration and consolidation.

Footnotes

Authors’ Note

I am very grateful for the constructive comments of the anonymous reviewers and editors on earlier versions of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Annevirta

Vauras

(2001). Metacognitive knowledge in primary grades: A longitudinal study. European Journal of Psychology of Education, 16, 257-282.

Armbruster

B. B.

Echols

C. H.

Brown

A. L.

(1982). The role of metacognition in reading to learn: A developmental perspective. Volta Review, 84(5), 45-56.

Bazeley

(2006). The contribution of computer software to integrate qualitative and quantitative data and analysis. Research in the Schools, 13(1), 64-74.

Brown

A. L.

(1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In Weinert

F. E.

Kluwe

R. H.

(Eds.), Metacognition, motivation, and understanding (pp. 65-116). Hillsdale, NJ: LEA.

Bryman

(2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6, 97-113.

Bryman

(2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8-22.

Byrne

B. M.

Worth Gavin

D. A.

(1996). The Shavelson Model revisited: Testing for structure of academic self-concept across pre-, early, and late adolescents. Journal of Educational Psychology, 88, 215-228.

Caracelli

V. J.

Greene

J. C.

(1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15, 195-207.

Clark-Carter

(2010). Quantitative psychological research: The complete student’s companion. East Sussex, England: Psychology Press.

10.

Collingridge

D. S.

(2013). A primer on quantitized data analysis and permutation testing. Journal of Mixed Methods Research, 7(1), 81-97.

11.

Creswell

J. W.

(2011). Controversies in mixed methods research. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), The Sage handbook of qualitative research (Vol. 4, pp. 269-284). Thousand Oaks, CA: Sage.

12.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

13.

De Backer

Van Keer

Valcke

. (2012). Exploring the potential impact of reciprocal peer tutoring on higher education students’ metacognitive knowledge and regulation. Instructional Science, 40, 559-588.

14.

Dinsmore

D. L.

Alexander

P. A.

Loughlin

S. M.

(2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychology Review, 20, 391-409.

15.

Efklides

Misailidi

(2010). Introduction: The present and the future of metacognition. In Efklides

Misailidi

(Eds.), Trends and prospects in metacognition research (pp. 1-19). New York, NY: Springer.

16.

Entwistle

McCune

Walker

(2001). Conceptions, styles, and approaches within higher education: Analytic abstractions and everyday experiences. In Sternberg

R. J.

Zhang

L. F.

(Eds.), Perspectives on thinking, learning, and cognitive styles (pp. 103-136). Mahwah, NJ: LEA.

17.

Flavell

J. H.

(1979). Metacognition and cognitive monitoring: A new area of cognitive developmental inquiry. American Psychologist, 34, 906-911.

18.

Freshwater

Cahill

(2013). Editorial: Paradigms lost and paradigms regained. Journal of Mixed Methods Research, 7(1), 3-5.

19.

Greene

J. C.

(2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.

20.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274.

21.

Greene

J. C.

Hall

J. N.

(2010). Dialectics and pragmatism: Being of consequence. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 119-144). Thousand Oaks, CA: Sage.

22.

Hacker

D. J.

Dunlosky

Graesser

A. C.

(Eds.). (2009). Handbook of metacognition in education. New York, NY: Routledge.

23.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2010). Multivariate data analysis (7th ed.). Upper Saddle River, NJ: Prentice Hall.

24.

Hurme

T. R.

Järvelä

(2005). Students’ activity in computer-supported collaborative problem solving in mathematics. International Journal of Computers for Mathematical Learning, 10, 49-73.

25.

Jang

E. E.

McDougall

D. E.

Pollon

Herbert

Russell

(2008). Integrative mixed methods data analytic strategies in research on school success in challenging circumstances. Journal of Mixed Methods Research, 2(3), 221-247.

26.

Johnson

R. B.

Christensen

(2014). Educational research: Quantitative, qualitative, and mixed approaches (5th ed.). Thousand Oaks, CA: Sage.

27.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.

28.

Johnson

R. B.

Onwuegbuzie

A. J.

Turner

L. A.

(2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112-133.

29.

Kipnis

Hofstein

A. V. I.

(2008). The inquiry laboratory as a resource for development of metacognitive skills. International Journal of Science and Mathematics Education, 6, 601-627.

30.

Kleitman

Stankov

(2007). Self-confidence and metacognitive processes. Learning and Individual Differences, 17, 161-173.

31.

Kubiszyn

Borich

(1996). Educational testing and measurement: Classroom application and practice (5th ed.). New York, NY: HarperCollins.

32.

Leech

N. L.

Onwuegbuzie

A. J.

(2009). A typology of mixed methods research designs. Quality & Quantity, 43, 265-275.

33.

Marsh

H. W.

Yeung

A. S.

(1997). Coursework selection: Relations to academic self-concept and achievement. American Educational Research Journal, 34, 691-720.

34.

Maxwell

J. A.

Mittapalli

(2010). Realism as a stance for mixed methods research. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 145-168). Thousand Oaks, CA: Sage.

35.

McCrindle

A. R.

Christensen

C. A.

(1995). The impact of learning journals on metacognitive and cognitive processes and learning performance. Learning and Instruction, 5, 167-185.

36.

Mertens

D. M.

(2012). What comes first? The paradigm or the approach. Journal of Mixed Methods Research, 6(4), 255-257.

37.

Miles

M. B.

Huberman

A. M.

Saldana

(2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage.

38.

Morgan

D. L.

(2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76.

39.

National Research Council. (2000). In Bransford

J. D.

Brown

A. L.

Cocking

R. R.

(Eds.), How people learn: Brain, mind, experience, and school (Expanded ed.). Washington, DC: National Academies Press.

40.

Neuenhaus

Artelt

Lingel

Schneider

(2011). Fifth graders metacognitive knowledge: General or domain-specific? European Journal of Psychology of Education, 26, 163-178.

41.

Newman

Onwuegbuzie

A. J.

Hitchcock

J. H.

(2014, June). Bayes methodology: Mixed methods in drag. Paper presented at the first annual meeting of the Mixed Methods International Research Association, Boston, MA.

42.

Newman

Onwuegbuzie

A. J.

Hitchcock

J. H.

(2015). Using the general linear model to facilitate the full integration of qualitative and quantitative analysis: The potential to improve prediction and theory building and testing qualitative & quantitative analysis. General Linear Model Journal, 41(1), 12-28.

43.

Niglas

(2004). The combined use of qualitative and quantitative methods in educational research (Unpublished dissertation on Social Sciences). Estonia: Tallinn Pedagogical University.

44.

Nunnally

J. C.

(1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.

45.

Onwuegbuzie

A. J.

Collins

K. M. T.

(2014). The role of Bronfenbrenner’s ecological systems theory in enhancing interpretive consistency in mixed research. International Journal of Research in Education Methodology, 5, 651-661.

46.

Onwuegbuzie

A. J.

Combs

J. P.

(2011). Data analysis in mixed research: A primer. International Journal of Education, 3(1), E13. Retrieved from http://www.macrothink.org/journal/index.php/ije/article/viewFile/618/550

47.

Onwuegbuzie

A. J.

Hitchcock

J. H.

(2015). Advanced mixed analysis approaches. In Hesse-Biber

Johnson

R. B.

(Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 275-295). New York, NY: Oxford University Press.

48.

Onwuegbuzie

A. J.

Johnson

R. B.

(2006). The validity issue in mixed research. Research in the Schools, 13(1), 48-63.

49.

Onwuegbuzie

A. J.

Slate

J. R.

Leech

N. L.

Collins

K. M. T.

(2009). Mixed data analysis: Advanced integration techniques. International Journal of Multiple Research Approaches, 3, 13-33.

50.

Pawson

Tilley

(2004). Realistic evaluations. London, England: Sage.

51.

Pekrun

Goetz

Titz

Perry

R. P.

(2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37, 91-106.

52.

Pintrich

P. R.

(2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory Into Practice, 41, 219-225.

53.

Pintrich

P. R.

Schunk

D. H.

(2001). Motivation in education: Theory, research, and applications. Englewood Cliffs, NJ: Prentice Hall.

54.

Sandelowski

(2001). Real qualitative researchers do not count: The use of numbers in qualitative research. Research in Nursing & Health, 24, 230-240.

55.

Sandelowski

Voils

C. I.

Knafl

(2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208-222.

56.

Schneider

(2008). The development of metacognitive knowledge in children and adolescents: Major trends and implications for education. Mind, Brain, and Education, 2, 113-121.

57.

Schofield

(2012). Why didn’t I think of that? Teachers’ influence on students’ metacognitive knowledge of how to help students acquire metacognitive abilities. Kairaranga, 13(1), 56-62.

58.

Schraw

(1998). Promoting general metacognitive awareness. Instructional Science, 26, 113-125.

59.

Schraw

Dennison

R. S.

(1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460-475.

60.

Schraw

Moshman

(1995). Metacognitive theories. Educational Psychology Review, 7, 351-371.

61.

Seltzer-Kelly

Westwood

S. J.

Peňa-Guzman

D. M.

(2012). A methodological self-study of quantitizing: Negotiating meaning and revealing multiplicity. Journal of Mixed Methods Research, 6(4), 258-274.

62.

Sperling

R. A.

Howard

B. C.

Staley

DuBois

(2004). Metacognition and self-regulated learning constructs. Educational Research and Evaluation, 10, 117-139.

63.

Stillman

(2004). Strategies employed by upper secondary students for overcoming or exploiting conditions affecting accessibility of applications tasks. Mathematical Education Research Journal, 16(1), 41-70.

64.

Swanson

H. L.

(1990). Influence of metacognitive knowledge and aptitude on problem solving. Journal of Educational Psychology, 82, 306-314.

65.

Tashakkori

Creswell

J. W.

(2007). Exploring the nature of research questions in mixed methods research. Journal of Mixed Methods Research, 1(3), 207-211.

66.

Tashakkori

Teddlie

(2010). Putting the human back in “human research methodology”: The researcher in mixed methods research. Journal of Mixed Methods Research, 4(4), 271-277.

67.

Teddlie

Tashakkori

(2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage.

68.

van Velzen

J. H

. (2012). Teaching metacognitive knowledge and developing expertise. Teachers and Teaching: Theory and Practice, 18(3), 365-380.

69.

van Velzen

J. H

. (2013). Students’ explanations of their knowledge of learning processes. Educational Studies, 39(1), 83-95.

70.

van Velzen

. (2016). Metacognitive learning: Advancing learning by developing general knowledge of the learning process. Cham, Switzerland: Springer.

71.

Veenman

M. V. J.

Van Hout-Wolters

B. H. A. M.

Afflerbach

(2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3-14.

72.

Weil

L. G.

Fleming

S. M.

Dumontheil

Kilford

E. J.

Weil

R. S.

Rees.

. . .Blakemore

S. J.

(2013). The development of metacognitive ability in adolescence. Conscious and Cognition, 22, 264-271.

73.

Weinstein

C. E.

Mayer

R. E.

(1986). The teaching of learning strategies. In Wittrock

M. C.

(Ed.), Handbook of research on teaching (pp. 315-327). New York, NY: Macmillan.

74.

Woolley

C. M.

(2009). Meeting the mixed methods challenge of integration in a sociological study of structure and agency. Journal of Mixed Methods Research, 3(1), 7-25.

75.

Yin

R. K.

(2006). Mixed methods research: Are the methods genuinely integrated or merely parallel? Research in the Schools, 13(1), 41-47.

76.

Zimmerman

B. J.

Schunk

D. H.

(Eds.). (2011). Handbook of self-regulation of learning and performance: Educational psychology handbook series. New York, NY: Routledge.