Examining the Level of Convergence Among Self-Regulated Learning Microanalytic Processes,Achievement,and a Self-Report Questionnaire

Conceptualization of SRL

Although several viable models of SRL exist in the literature (Boekaerts & Niemivirta, 2000; Pintrich, 2000; Winne & Hadwin, 1998), we adhered to a social-cognitive perspective and specifically Zimmerman’s (2000) three-phase model of SRL because it has served as the key framework for developing and refining SRL microanalytic assessments. From this theoretical paradigm, SRL is defined as a multi-dimensional process whereby individuals proactively and strategically generate, direct, and adapt their behaviors and thoughts to attain personally meaningful goals or objectives (Zimmerman, 2000). This process is characterized by a cyclical feedback loop consisting of three phases: forethought, performance, and self-reflection. Forethought processes (e.g., strategic planning, self-efficacy beliefs) precede students’ attempts to learn and collectively represent how students understand and approach a task. These “before learning” processes are hypothesized to affect how students will engage during task learning or performance, such as the extent to which they use strategies (e.g., to acquire information or to control emotions) along with their attempt to monitor learning progress. Self-generated or external feedback is then used to make “after learning” judgments, such as how well they performed the task (i.e., self-evaluation) and the factors that contributed to their performance (i.e., attributions). These judgments, in turn, affect other reflection processes, such as affective reactions (e.g., satisfaction) and inferences that individuals make regarding how best to improve future learning (i.e., adaptive inferences; Zimmerman, 2000).

As students transition to high school and ultimately college contexts, their capacity to effectively use SRL skills is critical because of an increase in demands and expectations for students to manage and self-direct their behaviors on independent, long-term learning activities, such as writing a laboratory report for high school science classes or preparing for mid-term and final exams in college (Grolnick & Raftery-Helmer, 2015; Weinstein & Acee, 2013). There is also a relatively strong literature showing that self-reflection phase processes (e.g., attributions, adaptive inferences, self-evaluation, and satisfaction) are particularly important in promoting adaptive motivation and achievement among college students (Cleary, Callan, & Zimmerman, 2012; Kitsantas & Zimmerman, 2006; Perry, Stupnisky, Daniels, & Haynes, 2008).

Measurement of SRL

Historically, self-report questionnaires have dominated the SRL assessment landscape, both in applied and research settings (Cleary, 2009; Dinsmore, Alexander, & Loughlin, 2008). These types of assessment tools, which include the Motivated Strategies and Learning Questionnaire (MSLQ; Pintrich et al., 1991), typically involve a series of statements depicting regulatory beliefs, attitudes, or behaviors to which students are asked to provide retrospective ratings. Student responses to individual items are then aggregated into a composite score, which is subsequently used to make inferences about the quality of students’ regulatory processes. From our perspective, questionnaires represent an important assessment tool because they are efficient and easy to use, convey how students perceive their own regulatory behaviors, and have been shown to relate to academic success across the developmental spectrum (Cleary & Chen, 2009; Pintrich & Smith, 1993; Weinstein & Acee, 2013).

The phrase self-report, however, does not automatically imply the use of a questionnaire format. Structured interviews can also be viewed as a type of SRL self-report measure because they elicit student responses to particular questions targeting strategic or regulated behaviors (Cleary et al., 2012; Zimmerman & Martinez-Pons, 1988). One relatively recent contextualized interview, called SRL microanalysis, was developed to examine students’ regulatory processes and motivation beliefs as they approach, perform, and reflect on specific situations or tasks (Cleary et al., 2012). Although this approach shares some qualities with questionnaires in that students are asked to respond to specific questions or statements and because students serve as the source of information, the two methods are quite distinct. SRL microanalytic protocols attempt to gather information about students’ regulatory processes as they occur during learning activities, which is distinct from the retrospective ratings elicited by questionnaires. Furthermore, unlike questionnaires which provide students with explicit statements about regulatory behaviors, most microanalytic measures use open-ended questions that necessitate students to provide information about how they approach, perform, and/or reflect during learning. Given the highly contextualized nature of this approach, single-item measures are often used to capture specific SRL processes.

An important advantage of SRL microanalysis is that it is theoretically grounded and aligned with Zimmerman’s three-phase model of SRL (Cleary et al., 2012; Zimmerman, 2000). Thus, SRL microanalytic protocols are structured so that the phase-specific microanalytic questions (e.g., attributions, strategic planning) are directly aligned with the temporal dimensions of learning tasks (i.e., before, during, and after), such that forethought questions are administered before learning occurs, performance phase questions are linked with the during-task dimension, and the self-reflection phase questions follow learning.

Purposes of the Study

SRL microanalytic interviews have been administered across several tasks including basketball free-throw shooting (Cleary & Zimmerman, 2001), clinical reasoning (Artino, Cleary, Dong, Hemmer, & Durning, 2014), and studying (DiBenedetto & Zimmerman, 2010); however, the number of studies examining this assessment approach in academic settings is generally lacking. Furthermore, although researchers have used microanalytic protocols to assess select SRL sub-processes within a given phase of the cyclical loop (Kitsantas & Zimmerman, 2002), very few studies have comprehensively examined a wide array of sub-processes within a single phase, such as self-reflection. Finally, few studies have attempted to examine the level of convergence between microanalytic measures and other SRL assessment tools or to investigate whether microanalytic measures more reliably predict academic outcomes than questionnaires or other assessment techniques.

Our first objective was to establish the convergent validity of SRL microanalytic reflection phase questions by examining the relations among these processes (i.e., self-evaluative standards, attributions, adaptive inferences, and satisfaction) and their convergence with post-exam self-efficacy and achievement. Given that SRL theory postulates that self-reflection phase processes are inter-related and because research has shown that adaptive reflection relates positively to self-efficacy and achievement (Perry et al., 2008), we expected to observe medium to large positive correlations among the four self-reflection phase processes and with post-exam self-efficacy beliefs. We also anticipated that high-achieving students would display more adaptive reflection phase processes than low achievers.

The second and third research objectives focused specifically on a microanalytic measure of strategic planning. For the second objective, we were interested in examining the extent to which the microanalytic strategy measure converged with a self-report questionnaire (MSLQ; Pintrich et al., 1991). Our intent was not to use the MSLQ as the benchmark against which to validate the microanalytic strategy measure, but rather to investigate the extent to which these two types of self-report measures converge. This objective is important and relevant given the recent evidence that highly contextualized and event forms of SRL measures, such as behavioral traces and think alouds, typically do not correlate well with SRL questionnaires (Veenman, Prins, & Verheij, 2003; Winne & Jamieson-Noel, 2002). Because SRL microanalytic protocols are considered a type of event measure, we speculated that the microanalytic strategy measure would exhibit only small to negligible correlations with the MSLQ subscales.

Finally, we examined whether the SRL microanalytic strategic planning measure and the MSLQ reliably predicted final course exam scores. Given that research has established a clear link between the SRL strategies and academic outcomes and because both the MSLQ and microanalytic strategy measure targeted these types of strategies, we expected both measures to correlate with final exam grades.

Method

Results

Before testing our research objectives, we conducted independent t tests to investigate gender differences across all metric variables, given that some research has shown that males and females differ in terms of their motivation and SRL processes (Bussey, 2011). No significant differences across any variables were observed and thus the entire sample was used for all subsequent analyses. Given that we established a priori, directional hypotheses regarding the relations among SRL measures and achievement, one-tailed tests were used for all correlation analyses with a significance level of p < .05. In addition, labels used to denote the magnitude of relations (i.e., small, medium, and large) were based on guidelines in Cohen (1988). Finally, there were no missing data for virtually all measures. However, one individual did not complete the three-item self-efficacy measure and final exam grades were not available for five individuals. These values were not replaced and thus list-wise deletion was used when reporting correlations (see Tables 1 and 2).

OPEN IN VIEWER

Table 1.

Relations Among Self-Reflection Phase Processes, Post-Exam Self-Efficacy, and Initial Exam Performance.

Measure	M (SD)	1	2	3	4	5	6	7
1. First course exam	77.87 (12.09)	—
2. Self-evaluative standards	91.82 (4.81)	.61*	—
3. Satisfaction	6.44 (2.58)	.74*	.60*	—
4. Post-exam self-efficacy	7.44 (1.60)	.68*	.64*	.61*	—
5. Attributions–Weighted	1.16 (1.46)	.49*	.27*	.51*	.38*	—
6. Attributions–Frequency	1.39 (0.79)	.59*	.38*	.68*	.46*	.89*	—
7. Adaptive inference–Weighted	2.82 (1.56)	.17	.10	.12	.00	.16	.08	—
8. Adaptive inference–Frequency	2.06 (0.85)	.17	.01	.10	−.05	.11	.06	.92*

Note. n = 49 for all correlation coefficients except those involving the first course exam (n = 44) and self-efficacy (n = 48).

OPEN IN VIEWER

Table 2.

Relations Among Microanalytic Strategy Measure, Self-Report Questionnaires, and Final Exam Grade.

Measure	M (SD)	1	2	3	4	5	6	7
1. Final exam score	77.59 (10.88)	—
2. MSLQ time/study environment	4.93 (1.00)	.02	—
3. MSLQ effort regulation	5.32 (1.02)	−.03	.55*	—
4. MSLQ help seeking	4.46 (1.35)	.08	.00	.14	—
5. MSLQ elaboration	5.30 (0.92)	−.01	.35*	.26*	−.25*	—
6. MSLQ organization	4.46 (1.31)	.16	.39*	.03	−.12	.39*	—
7. Micro strategy–Weighted	3.51 (1.24)	.29*	.20	−.02	.07	−.01	.06
8. Micro strategy–Frequency	4.31 (1.84)	.23	.30*	−.06	.11	.07	.07	.84*

Note. n = 49 for all correlation coefficients except for those involving the final exam score (n = 44). MSLQ = Motivated Strategies and Learning Questionnaire.

Given that students often provided multiple codable responses to the three free-response microanalytic questions (i.e., strategic planning, attributions, adaptive inferences), we transformed the qualitative codes into metric scores to facilitate interpretation. The two primary transformation methods used in prior microanalytic research have included frequency counts of adaptive responses (DiBenedetto & Zimmerman, 2010) and a weighted scoring scheme targeting adaptive and negative regulatory responses (Artino et al., 2014). In the current study, we wanted to examine if the scoring procedures resulted in consistent and similar relations among the target variables.

As a brief example of scoring schemes, consider the attribution question. The frequency count method entailed counting the number of adaptive attribution responses (i.e., SRL strategies, classroom engagement, test-taking skills, and effort), whereas the weighted scoring involved assigning positive points for each adaptive attribution and negative points for uncontrollable attributions (e.g., teacher difficulty, ability, and don’t know). High attribution scores across both methods represented students who believed that controllable factors were the primary reasons for their test performance. Similar procedures were followed for the strategic planning and adaptive inference questions.

Discussion

The current study was important for several reasons. First, we applied SRL microanalytic procedures to a self-reflection activity for an authentic, high-stakes outcome in a college context. We also observed large inter-correlations among several self-reflection microanalytic processes and between the reflection processes and achievement; findings that are consistent with SRL theoretical predictions and some prior research (Perry et al., 2008; Zimmerman & Kitsantas, 1999). To our knowledge, this was also the first study to compare SRL microanalysis procedures and an SRL questionnaire, a particularly important initiative given that self-report questionnaires tend to be the most common form of SRL measurement (Dinsmore et al., 2008) and because questionnaires and microanalysis are both types of self-reports. Moreover, this study was the first to examine whether different types of microanalytic scoring schemes yield similar correlation values among regulatory processes and achievement.

A few of our findings warrant further discussion. This study provides some evidence for the convergent and predictive validity of SRL microanalytic measures relative to test reflection and preparation. Of primary importance was that not only did the microanalytic reflection processes exhibit large inter-correlations as predicted by Zimmerman’s cyclical feedback model, but they were also linked with post-exam self-efficacy beliefs and achievement, a finding that replicates prior microanalytic research (Cleary & Zimmerman, 2001; Cleary et al., 2006; Kitsantas & Zimmerman, 2006). On a more practical level, individuals who did not perform well on the first course exam exhibited a less adaptive pattern of reflection phase processes than high achievers, characterized by lower levels of satisfaction and post-exam self-efficacy beliefs, as well as maladaptive or non-strategic attributions.

An unexpected finding was the very poor relations observed between students’ adaptive/defensive inferences and most other regulatory processes and achievement. In short, students who were successful on the first exam and who exhibited high levels of satisfaction and post-exam self-efficacy were no more likely than low achievers to identify strategic adaptations that they can use to improve or sustain future exam performance. This finding contradicts prior microanalytic research, which showed that high achievers tend to exhibit more strategic adaptive inferences than low achievers (Cleary et al., 2006; DiBenedetto & Zimmerman, 2010). Although speculative, it is possible that these conflicting results emerged because of differences in the specificity of the performance outcomes about which students were asked to reflect and the clarity of the connection between these outcomes and potential adaptations. In contrast to our study that required students to answer the adaptive inference question in response to their overall grade on a comprehensive course exam, prior microanalytic research has typically focused on more narrowly defined and specific outcomes (e.g., single test question targeting understanding of tornados; DiBenedetto & Zimmerman, 2010). Thus, it is possible that the adaptive inference measure used in our study did not elicit adequate variation in participant responses because the students were not sure how to interpret or use the global score on the unit exam to subsequently adapt or modify future test preparation.

Another critical finding was that the MSLQ subscales exhibited very weak relations with the microanalytic strategy planning measure. In our study, both the MSLQ and the microanalytic strategy measure were customized to studying and test preparation activities in the same course. Furthermore, the scheme used to code students’ strategic planning responses was conceptually linked with the strategies targeted by the MSLQ subscales. Despite this conceptual overlap, we predicted small to negligible correlations between these two types of self-report measures because of the emerging literature showing that SRL self-report questionnaires do not typically converge with contextualized, event measures (Veenman et al., 2003; Winne & Jamieson-Noel, 2002); our hypothesis was largely confirmed. We propose two explanations for this very low level of convergence. First, it is possible that the differences in the nature and response format of these two self-report measures resulted in the discrepant findings. That is, SRL microanalytic protocols, which typically use open-ended questions, do not provide any direction or prompts to students about desirable or targeted SRL behaviors. Thus, student responses appear to represent authentic responses about the strategies that they perceive to be relevant in a given context. In contrast, SRL self-report questionnaires simply require students to rate pre-established regulatory statements.

A second explanation, and one that is closely aligned with our hypothesis, was that the two types of self-report measures tap different dimensions of SRL. Because SRL questionnaires emphasize composites and aggregate scores whereas the microanalytic protocols examine student responses to highly contextualized questions with specific situation referents, it is likely that questionnaires address the question of how individuals regulate their thoughts, behaviors, or environment in general or on average, whereas microanalytic questions address how students approach, use strategies, and/or self-reflect about a particular event or activity. Thus, the observed divergence between these measures suggests that it may be unreasonable to expect that students’ general approach to regulation will consistently align with how they regulate during specific academic situations or contexts.

Finally, regarding the two microanalytic scoring schemes, our study provided preliminary evidence that the weighted and frequency count scoring schemes for free-response microanalytic questions yielded very similar results regarding the relations among microanalytic processes, post-exam self-efficacy, and exam performance; however, some differences emerged. Descriptive analysis showed that though the correlation between microanalytic attributions and most other variables (i.e., standards, satisfaction, initial course grade, and post-exam self-efficacy) was higher when using the frequency count rather than the weighted scoring procedure, the relation between microanalytic strategic planning and final exam grade was statistically significant only when the weighted approach was used. Given this lack of consistency and the fact that our examination of these two scoring methods was exploratory in nature, future research needs to more closely identify the particular circumstances and specific SRL processes when the use of each scoring method would be most appropriate.

Limitations and Areas of Future Research

One of the limitations of our study was the relatively restricted sample, in terms of the sample size and the ethnicity and gender of the participants. Moreover, because the participants were volunteers and thus may have represented a more motivated and engaged group of individuals, it is possible that the obtained results were biased. Thus, our findings should be viewed with caution. Furthermore, it is important for future research to attempt to replicate these findings and to examine whether they generalize to different student populations, such as middle and high school students as well as those from culturally and socioeconomically diverse backgrounds.

We also acknowledge that the nature of the task used in this study (i.e., exam reflection), although important and relevant to college contexts, was very narrow and thus did not permit us to examine a full array of SRL processes across forethought, performance, and self-reflection phases. Applying a more comprehensive protocol would allow one to more clearly examine the interplay among the sub-processes across the different phases of the cyclical feedback loop.

Finally, we did not directly measure SRL behavior. Including measures of overt SRL behaviors would allow researchers to not only examine how self-reflection phases processes affect subsequent patterns of strategic action, but can also provide an objective indicator against which to judge the validity of both the strategy microanalytic measures and self-report questionnaires.

Conclusion

Recent survey research has shown that most school-based practitioners and teachers do not readily target students’ self-regulation and motivation processes (Cleary, 2009; Wehmeyer, Agran, & Hughes, 2000). When these types of assessments do occur in schools or even in research contexts, self-report questionnaires have traditionally been the assessment tool of choice (Cleary, 2009; Dinsmore et al., 2008). The results of the current study suggest that SRL microanalytic data may provide unique information about students’ regulatory approaches to learning, relative to self-report questionnaires. Our results are also in line with the recent shifts observed in the SRL literature regarding the importance of considering contextualized and event-oriented measures as part of an SRL assessment battery (Butler, 2011).