Abstract
This study examined the predictive validity of a teacher rating scale called the Self-Regulation Strategy Inventory–Teacher Rating Scale (SRSI-TRS) and its level of convergence with several student self-report measures of self-regulated learning (SRL). Eighty-seven high school students enrolled in one of four sections of a mathematics course in an urban high school and one mathematics teacher participated in the study. Correlation analyses revealed moderate correlations between the SRSI-TRS and self-report questionnaires targeting students’ motivation beliefs (i.e., mathematics interest) and regulatory behaviors in mathematics. Students’ self-efficacy perceptions correlated with all SRL and achievement measures, but not the SRSI-TRS. Hierarchical regression analyses showed that the SRSI-TRS emerged as the primary SRL predictor of achievement although student reports of their maladaptive SRL behaviors was a significant predictor in the final model.
Developing reliable and valid self-regulated learning (SRL) assessment tools has been an area of interest among SRL researchers and practitioners over the past couple of decades, with a heightened focus in recent years (Winne & Perry, 2000; Zimmerman, 2008). In addition to self-report questionnaires, which have traditionally been the most widely used SRL measure (Cleary, 2009; Dinsmore, Alexander, & Loughlin, 2008), researchers have developed a variety of alternative assessment tools in recent years to expand the breadth of measures available in practice and clinical contexts (Cleary, 2011; Winne & Perry, 2000; Zimmerman, 2008). Despite these advances in SRL assessment methodologies, the development and use of SRL teacher and parent rating scales remains somewhat limited. The primary purpose of the current study was to address this gap by describing the development and psychometric properties of a teacher rating scale designed to capture urban students’ SRL behaviors, such as help-seeking and self-motivation. Examining SRL in urban populations is particularly important given that much of the SRL literature has not targeted this population and because many students who live within these environments experience challenges (e.g., insufficient academic instruction) that may adversely impact their skills to adaptively regulate and self-direct their lives (Cleary, 2006; Cleary, Platten, & Nelson, 2008).
Definition and Assessment of Self-Regulation
Over the past couple of decades, there has been extensive research linking SRL processes to various academic skills, such as mathematics (Pape, Bell, & Yetkin, 2003), writing (Graham & Harris, 2009), and content area subjects such as science (Cleary et al., 2008). In general, SRL is often conceptualized as a multi-dimensional process that involves purposeful efforts to behaviorally, cognitively, and metacognitively control and optimize learning in a particular context (Pintrich, 2000; Zimmerman, 2000). Two core components of most SRL models include students’ motivational beliefs (e.g., self-efficacy, interest) and their use of self-regulation strategies to enhance their achievement. From a social-cognitive perspective, motivation beliefs are hypothesized to be the primary sources underlying student efforts to reach personal goals (Pintrich, 2000). Thus, students who are most likely to engage and persist on school tasks typically possess strong beliefs in their personal capabilities (i.e., self-efficacy), exhibit interest and enjoyment in completing academic tasks (i.e., mathematics interest), and perceive academic tasks to be valuable or personally meaningful (Eccles & Wigfield, 2002; Usher & Pajares, 2008).
Although these motivational beliefs underlie the “will” to learn, it is the nature of students’ strategic skills that often determines their success in school. For example, research has shown that high-achieving students frequently use cognitive strategies to learn, such as elaboration and rehearsal, and routinely use regulatory strategies to manage their behaviors and learning environments, including help seeking and time management (Karabenick & Berger, 2013; Zimmerman & Martinez-Pons, 1988). In addition to using adaptive strategies, sophisticated self-regulated learners also seek to minimize personal displays of negative or maladaptive behaviors, such as avoiding help, self-handicapping, or procrastinating (Cleary & Chen, 2009; Flowers, Bridges, & Moore, 2012). In this study, we were interested in examining students’ motivation beliefs and their adaptive and maladaptive regulatory behaviors using self-report questionnaires and a teacher rating scale.
Assessment of SRL
Self-report questionnaires represent the most common type of SRL assessment. These measures typically involve having students use a Likert scale to retrospectively rate their regulatory beliefs, behaviors, and metacognition. Although desirable in many respects (e.g., ease of administration), self-report questionnaires have been criticized in the SRL literature because students are often unreliable reporters of their behaviors and because many of these measures lack adequate situational referents to guide students’ responses (Winne & Jamieson-Noel, 2002). To counter these potential limitations, SRL researchers have developed a variety of dynamic, contextualized assessment approaches in recent years, including think alouds in hypermedia (Azevedo, Johnson, Chauncey, & Graesser, 2011), behavioral traces (Winne & Perry, 2000), and contextualized structured interviews (Cleary, 2011). Although these innovations have been useful, there is a need to devote more attention to using teacher ratings as sources of student SRL behaviors and processes.
Teacher ratings of student SRL are important for several reasons. First, consistent with a multi-dimensional assessment approach, gathering SRL data from multiple sources (e.g., student reports, observations, teacher ratings) enables clinicians to converge assessment data and to facilitate hypothesis development (Sattler, 2008). In addition, because teachers spend much time with students and thus have many opportunities to directly observe student actions in the classroom (e.g., help seeking), they have the potential to provide unique information about student SRL processes. Finally, there is evidence showing that parents and teachers tend to be more accurate sources of students’ externalizing behaviors than students (Gould & Shaffer, 1985; Kamphaus & Frick, 2002).
Primary Objectives
Given the potential value yet paucity of SRL teacher rating scales, we developed the Self-Regulation Strategy Inventory–Teacher Rating Scale (SRSI-TRS) and examined its concurrent and predictive validity. Although a couple of SRL teacher rating scales are available (Zimmerman & Martinez-Pons, 1988), to our knowledge no scales have been developed to parallel a student SRL questionnaire (Cleary, 2006). Furthermore, we were interested in developing a measure that targeted student help-seeking behaviors across different situations, their use of self-motivation tactics to optimize effort in class, and their use of organization-related behaviors. We selected mathematics as the target domain in this study because of the strong link between student SRL and mathematics achievement (Pape et al., 2003) and because student motivation in mathematics remains an important issue in education circles (Ashcroft, Krause, & Hopko, 2007).
In this study, concurrent validity was examined by correlating the SRSI-TRS with a set of SRL self-report questionnaires. Given that the SRSI-TRS and all self-report measures were linked to the same academic class (i.e., mathematics) and due to the established link between students’ motivation beliefs (e.g., self-efficacy, interest) and their regulatory behaviors (Schunk & Zimmerman, 2008), we expected to observe high correlations between the SRSI-TRS and all self-report measures. In terms of predictive validity, we were primarily interested in examining the amount of variance in algebra test scores accounted for by the SRSI-TRS after controlling for prior math achievement and student responses on SRL questionnaires. We anticipated that the SRSI-TRS and the self-report measures would contribute unique variance to the prediction of students’ mathematics test grades after controlling for prior mathematics achievement.
Method
Participants and Procedures
A total of 128 ninth-grade students enrolled in one of four sections of an algebra course in an urban high school located in a school district in the Midwestern region of the United States were asked to participate. Eighty-seven students returned parental consent forms and completed the surveys (68% response rate). Fifty-three percent of the students were African American, 16% were Latino, 20% Caucasian, and 11% Asian American. The sample consisted of 56% females while 70% of all participating students were eligible for free or reduced-price lunch.
All students who participated in this study were enrolled in one of four sections of an algebra course taught by one algebra teacher. This teacher had five years of teaching experience, all of which occurred at the target high school. Given the highly diverse instructional format across mathematics teachers in the target school, we elected to focus on a sample of students who received identical curriculum and grading criteria in mathematics from the same teacher. Participating students returned parental consent forms in the Fall semester of the school year and completed the packet of self-report measures during a single class session. The teacher signed his or her consent form in the same semester and completed the SRSI-TRS within two weeks of the student assessment. Information pertaining to student achievement data and background information was provided by the school district at the end of the study.
Measures of Self-Regulation and Motivation Beliefs
SRSI-TRS
A TRS of student self-regulation was developed to parallel the student version of the SRSI−Self-Report (SRSI-SR; Cleary, 2006). Using the 28-item SRSI-SR as a framework, an original item pool of 20 items was initially developed to represent behaviors that students might exhibit during class activities, particularly those pertaining to help seeking and self-motivation. This initial item pool was given to three high school teachers (i.e., Language Arts, mathematics, science) to evaluate the readability and applicability of the items to a high school classroom environment. A focused feedback session with these teachers revealed that seven items were not suitable because they either referred to covert processes or to student behaviors that were most likely to be displayed outside of the school context (e.g., studying). The final 13-item scale was designed to examine teacher ratings of the frequency with which students engage in various help-seeking behaviors (e.g., ask about specific errors on tests, seek out and attend extra help sessions), self-motivation tactics (e.g., to push oneself to learn the details of course lessons), and organization behaviors (e.g., keeps class materials organized) within the mathematics classroom context. The measure utilized a 5-point Likert scale ranging from 1 (almost never) to 5 (almost always). An example item was, “The student asks questions in class when he or she does not understand something.” It should be noted that, in this study, we removed one item (i.e., attending extra help), because the algebra teacher indicated that he did not routinely provide extra help sessions. The composite score of the 12-item SRSI-TRS was shown to have high internal reliability (α = .97).
Maladaptive Regulatory Behaviors–SRSI-SR
The SRSI-SR is a 28-item self-report measure that includes three subscales: Seeking and Learning Information (SLI), Managing Environment and Behavior (MBE), and Maladaptive Regulatory Behavior (MRB; Cleary, 2006). For this study, however, we only used the eight-item MRB subscale given that most of the items on that scale corresponded closely to the SRSI-TRS. An example item was, “I avoid asking about things in class that I do not understand.” This measure utilized a 5-point Likert scale ranging from 1 (almost never) to 5 (almost always). High scores on this subscale indicate frequent displays of maladaptive academic behaviors. An alpha coefficient of .73 was obtained, which is consistent with prior research (Cleary, 2006).
Test-taking
The eight-item Test-Taking strategies subscale from the Learning and Study Strategies Inventory−High School Version (LASSI-HS) was used to examine student self-report of test-taking strategies (Weinstein & Palmer, 1990). This scale was selected because many items target behaviors that are likely to occur during classroom activities (i.e., asking questions during a test, discussion of test preparation tactics). A 5-point Likert scale ranging from 1 (not at all typical of me) to 5 (very much typical of me) was used. All items were modified slightly to reflect mathematics contexts. An example item included, “When I take a math test, I often realize that I have studied the wrong material.” Given that all items were negatively worded, high scores on this subscale represent maladaptive test preparation or test-taking skills. The reliability coefficient for this subscale (α = .85) was consistent with prior research (Weinstein & Palmer, 1990).
Mathematics interest
This self-report measure was an adaptation of an interest scale developed for science (Cleary et al., 2008). For this project, all items were re-phrased to reflect student interest and enjoyment in mathematics. An example item was, “Learning how to do mathematics is very interesting.” A 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree) was used. The coefficient alpha of .87 obtained in this study was similar to estimates in prior research (Cleary, 2006; Cleary et al., 2008).
Mathematics self-efficacy
A six-item self-report measure of mathematics self-efficacy was developed to assess the participants’ perceived capability to regulate their academic behavior and to perform well in their mathematics class. This measure was developed following general guidelines put forth by Bandura (2006) and used a Likert scale ranging from 0 to 100. The scale was broken down into 10-point increments (e.g., 0, 10, 20, etc.), with three primary anchors: not at all confident (0), somewhat confident (50) and completely confident (100). All items began with the phrase, “How confident are you that you can . . . ” This stem phrase was followed by several statements depicting performance expectations (e.g., “ . . . get at least a B in this math course”) or regulatory behaviors in mathematics (e.g., “. . . figure out how to correct a mistake on a mathematics problem without help from the teacher”). A Cronbach’s alpha value of .78 indicated acceptable internal consistency.
Measures of Mathematics Achievement
Prior mathematics skill
Student performance on a standardized academic achievement test, Measure of Academic Progress (MAP), served as an index of prior mathematics achievement (Northwest Evaluation Association [NWEA], 2005). This standardized test provided a measure of students’ mathematics and reading performance and is administered three times during the academic year. Student performance on the mathematics portion of MAP in September of ninth grade was used as an index of prior achievement.
Mathematics classroom test percentage
Data regarding the students’ mathematics achievement was calculated by dividing the total number of points earned on teacher-developed mathematics tests during the last semester of the school year by the total number of possible test points. The test scores reflected the actual number of points earned by students and were not “adjusted” due to a normative grading system, extra credit, or bonus points. Thus, the test scores represented a relatively objective measure of students’ mathematical knowledge and skills. There was an approximate 2-month lag between the SRL survey assessment session and the Spring semester when the classroom tests were administered.
Results
Pearson correlations and hierarchical linear regression analyses were used to address the research questions. Prior to statistical analysis, missing data and the kurtosis and skewness of all variables were examined and found to be within acceptable limits (Kline, 1998). There was no missing data for the survey items. Furthermore, given that there were no significant group differences across gender, ethnicity, and SES status across the core measure used in this study, all data were collapsed. However, 10 students were removed prior to the regression analysis due to unavailability of MAP scores or classroom test scores. One-tailed tests using a significance level of p < .05 were used in all analyses unless noted otherwise.
Concurrent Validity of SRSI-TRS
The first objective of this study was to examine whether the SRSI-TRS correlated with student ratings of their motivation beliefs and regulatory behaviors. We calculated bivariate correlations between the SRSI-TRS and four student self-report measures (see Table 1). Statistically significant correlations were observed between the SRSI-TRS and student reported mathematics interest (r =.32), SRSI-Maladaptive Regulatory Behaviors (MRB; r = −.41), and LASSI–Test Taking (r = −.42). All of these correlations were in the medium range (Cohen, 1988). In general, these results indicated that students who were rated by their teachers as being strategic during classroom activities also tended to self-report high levels of interest or enjoyment in mathematic class and low levels of maladaptive regulatory behaviors. Contrary to expectations, the SRSI-TRS did not relate to student mathematics self-efficacy.
Descriptive Statistics and Correlation Coefficients Between SRL Measures and Math Achievement.
Note. The self-efficacy scale used an 11-point Likert scale (0-10), while all other SRL measures utilized a 5-point Likert scale (1-5). High scores on the LASSI-Test-Taking and SRSI-SR Maladaptive subscales represent greater maladaptive skills, whereas high scores on all other measures represents adaptive functioning. SRL = self-regulated learning; MAP = Measure of Academic Progress; LASSI = Learning and Study Strategies Inventory; SRSI-SR = Self-Regulation Strategy Inventory-Self-Report; SRSI-TRS = Self-Regulation Strategy Inventory–Teacher Rating Scale.
Denotes statistically significant finding using one-tailed test at p < .05.
Predictive Validity of SRSI-TRS
The key objective of this study was to examine whether teacher ratings emerged as a significant predictor of students’ mathematics achievement after controlling for four self-report questionnaires and prior mathematics achievement. Given that we were also interested in evaluating whether student reports accounted for unique variance, we conducted a hierarchical regression analysis and reported changes in R2 across three models. We also reported semipartial correlations to convey the precise percentage of unique variance in mathematics achievement accounted for by each predictor after controlling for all other predictors (see Table 2).
Hierarchical Regression Analysis for Teacher and Student Predictors of Student Mathematics Classroom Test Scores.
Note. Step 1: Adjusted R2 = .125; ∆R2 = .136 Step 2: Adjusted R2 = .234; ∆R2 = .148 Step 3: Adjusted R2 = .325; ∆R2 = .094. sr2 = semipartial squared represents the proportion of unique variance in mathematics test scores accounted for by a specific predictor after controlling for all other variables. MAP = Measure of Academic Progress; LASSI = Learning and Study Strategies Inventory; SRSI-SR = Self-Regulation Strategy Inventory–Self-Report; SRSI-TRS = Self-Regulation Strategy Inventory–Teacher Rating Scale.
p = .05.
Denotes statistically significant finding using one-tailed test at p < .05.
In general, all predictors accounted for approximately 38% of the variance in mathematics achievement. Of greater importance was that each block of variables accounted for a medium increase in R2. For example, in the first block, students’ prior mathematical achievement accounted for approximately 14% of the variation in classroom test scores, F(1, 75) = 11.82, p < .01, ΔR2 = .136. In the second block, a significant and robust improvement in the prediction of mathematics achievement was observed, F(1, 71) = 3.67, p < .01, ΔR2 = .148, although the MAP and SRSI-MRB emerged as the only significant predictors. Adding the SRSI-TRS in the third block accounted for an additional 9% of the variance in mathematics test grades, F(1, 70) = 10.64, p < .01, ΔR2 = .094. In the final model, three variables emerged as significant predictors; the SRSI- TRS, SRSI-MRB, and prior achievement, with the TRS emerging as the strongest predictor (see semipartial correlations in Table 2).
Discussion
In this study, we hypothesized that a broad teacher measure of student SRL would correlate with student self-reports of their motivation processes and regulatory behaviors (e.g., seeking out help), and would also emerge as a significant predictor of mathematics achievement. Our hypotheses were largely confirmed, supporting a key general premise that different sources of student SRL can contribute unique variance to students’ academic achievement.
The SRSI-TRS accounted for a medium level of variation (9.4 %) in mathematics classroom test scores, after controlling for approximately 28% of the variance attributed to prior mathematics skills and student-reported motivation beliefs and regulatory behaviors. In a broad sense, these findings are consistent with prior research showing that teacher ratings can accurately predict a variety of academic outcomes (Perry & Meisels, 1996; Whitebread et al., 2009). Furthermore, the fact that a self-report measure of regulatory behaviors (i.e., SRSI-MRB) emerged as a significant predictor in the final regression model (see Table 2) is particularly noteworthy because many researchers have questioned the utility of self-report questionnaires relative to more objective or dynamic, event-based measures of SRL (Winne & Jamieson-Noel, 2002).
The second key finding in this study was that students who were identified by their teachers as being highly strategic and motivated during class activities were likely to self-report high levels of interest in mathematics activities and fewer maladaptive regulatory behaviors, such as avoiding help or procrastinating. A few important points pertaining to this latter finding are warranted. First, we showed that the SRSI-TRS correlated positively with students’ adaptive self-reported processes (i.e., mathematics interest) and negatively with their maladaptive self-reported regulatory behaviors. The fact that the SRSI-TRS correlated in the expected negative direction with two self-report measures targeting maladaptive regulatory behaviors further buttresses claims that the SRSI-TRS is in fact a measure of adaptive regulatory behaviors, such as seeking out help when confused, motivating oneself when struggling to learn, and being organized and prepared for class.
If one perceives teacher ratings of student behaviors as a more objective type of measure than self-reports, our results also appear to contradict SRL research showing that student reports of their behaviors do not correspond to more objective indicators of those behaviors. For example, Winne and Jamieson-Noel (2002) showed that student self-reports about their strategic behaviors during a studying activity exhibited very poor relations with real-time data about their behavioral traces during the studying activity. Furthermore, there is evidence in clinical contexts showing that student ratings of their behaviors often exhibit low correlations with teacher or parent reports of those same behaviors (Achenbach, McConaughy, & Howell, 1987; Loeber, Green, & Lahey, 1990).
Although we cannot make any claims that SRSI-TRS responses were “correct” or more accurate than student self-reports, there are a couple of explanations that support the observed relations in the current study. First, there was much overlap in the format and structure of the SRSI-TRS and the student measures. That is, both measures were contextualized to the algebra class and relied on retrospective, aggregated accounts of student SRL. Although speculative, a second explanation is that the nature of items in the SRL self-report questionnaires (i.e., focus on maladaptive regulatory behaviors) impacted the observed correlations. A few research studies have demonstrated that negatively worded self-report measures are sometimes more strongly correlated with teacher ratings of achievement and work habits (Bush, 1999; Malone, 1998). In our study, we surveyed students’ adaptive motivation beliefs but not their adaptive regulatory behaviors. As a result, we cannot draw any definitive conclusions regarding how the nature of the SRL questionnaires impacted the observed correlations. However, in light of prior research and the moderate concurrent validity coefficients observed between the teacher and student measures of SRL behaviors, future research should further explore this issue.
The lone unexpected finding of this study involved the non-significant relation between student self-efficacy and the SRSI-TRS. The interpretation of this finding is somewhat challenging because the self-efficacy measure was reliable and exhibited medium to large correlations with all other self-report scales and with both measures of mathematics achievement. Furthermore, our findings contradict the general finding in prior research that highly efficacious students in a particular domain exhibit more strategic behaviors in that context (Cleary & Zimmerman, 2001; Pajares & Urdan, 2006). One potential explanation, however, pertains to whether mathematics self-efficacy perceptions should be expected to relate to all types of classroom behaviors that are observable to a teacher. For example, six of the items on the SRSI-TRS targeted help-seeking which is an important regulatory strategy (Karabenick & Berger, 2013). Based on prior research, it is possible that some students in our study who exhibited high levels of confidence did not perceive the need to ask questions about tests or to seek out help when confused, whereas other confident students frequently sought out support from the teacher because of their inherent interest in learning (e.g., mastery goal orientation) or perhaps their high performance expectations (Karabenick & Berger, 2013). Although some studies have examined the relation between self-efficacy and help-seeking behaviors (Bong, 2008; Ryan & Shin, 2011), the precise nature of the relationship is inconsistent and needs to be studied in greater depth.
Limitations
One of the limitations of this study was the modest sample size. Although small sample sizes may reduce the likelihood of detecting a significant relation and/or may underestimate the predictive validity of measures, the fact that almost all of our hypotheses were supported reduces this concern. However, the external validity of our study was limited given that all student participants were enrolled in the ninth grade at a single school and were assessed specifically in relation to mathematics class and because only one teacher was used to provide responses on the SRSI-TRS.
As indicated previously, another limitation was the omission of a self-report measure of adaptive regulatory behaviors. Including measures that target student reports of adaptive and maladaptive SRL behaviors in future research would enable one to clarify whether the strong relation between student self-report questionnaires and teacher ratings observed in our study is moderated by the nature of the behaviors (i.e., adaptive versus maladaptive) targeted in the self-report scales. This area of future research is particularly fruitful given that most SRL research tends to focus more on the adaptive rather than the maladaptive components of SRL.
Conclusion
In this study, we found that student self-report SRL measures and a SRL-TRS collectively accounted for 24% of the variation in classroom-based mathematics tests after controlling for prior mathematics achievement. In the final model, the SRSI-TRS and the SRSI-MRB (maladaptive regulation) emerged as the key SRL predictors of student achievement. These results support the general premise that a multi-dimensional assessment approach is an important way to not only understand how students might regulate in specific contexts, but to also develop more robust explanations for student performance in schools. From our perspective, when practitioners encounter discrepancies between teacher ratings and student questionnaires, it is important for them to resolve such discrepancies by gathering additional data with other types of SRL assessment tools, such as direct observations, think-aloud protocols, behavioral traces, and SRL microanalytic interviews (Winne & Perry, 2000; Zimmerman, 2008).
Footnotes
Authors’ Note
This manuscript is based on an earlier version of a paper presented at the 2012 American Education Research Association conference.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
