Promise and Paradox

Abstract

We used self-report surveys to gather information on a broad set of non-cognitive skills from 1,368 eighth graders. At the student level, scales measuring conscientiousness, self-control, grit, and growth mindset are positively correlated with attendance, behavior, and test-score gains between fourth grade and eighth grade. Conscientiousness, self-control, and grit are unrelated to test-score gains at the school level, however, and students attending over-subscribed charter schools score lower on these scales than do students attending district schools. Exploiting admissions lotteries, we find positive impacts of charter school attendance on achievement and attendance but negative impacts on these non-cognitive skills. We provide suggestive evidence that these paradoxical results are driven by reference bias or the tendency for survey responses to be influenced by social context.

Keywords

non-cognitive skills accountability education policy education reform reference bias charter schools

Recent evidence from economics and psychology highlights the importance of traits other than general intelligence for success in school and in life (Almlund, Duckworth, Heckman, & Kautz, 2011; Borghans, Duckworth, Heckman, & ter Weel, 2008; Moffitt et al., 2011). Disparities in so-called non-cognitive skills appear to contribute to the academic achievement gap separating wealthy from disadvantaged students (Evans & Rosenbaum, 2008). Furthermore, non-cognitive skills may be more amenable to direct intervention than cognitive ability, particularly beyond infancy and early childhood (Cunha & Heckman, 2008; Dee & West, 2011; Heckman & Kautz, 2013). Understandably, popular interest in measuring and developing students’ non-cognitive skills has escalated (see, for example, Tough, 2012)

Non-cognitive is, of course, a misnomer. Every psychological process is cognitive in the sense of relying on the processing of information of some kind. Characteristic patterns of attending to and interpreting information underlie many if not most personality traits (Bandura, 1999; Mischel & Shoda, 1999). Moreover, emotion and personality influence the quality of one’s thinking (Baron, 1982) and how much a child learns in school (Duckworth & Seligman, 2005).

Why, then, does the term non-cognitive persist? Cognitive in this context is shorthand for cognitive ability and knowledge, constructs that can be reliably measured by standardized intelligence and achievement tests (Messick, 1979). Non-cognitive, therefore, has become a catchall term for traits or skills not captured by assessments of cognitive ability and knowledge. Recent discussions of non-cognitive skills have emphasized the importance of traits such as conscientiousness, self-control, and grit that appear to contribute to students’ ability to sustain effort at academically demanding tasks. Many educators prefer the umbrella term “social and emotional learning” (Durlak, Dymnicki, Taylor, Weissberg, & Schellinger, 2011), which is often used to encompass a broader range of competencies such as social awareness, empathy, and self-regulation. Others use “21st Century skills” (National Research Council, 2012) to emphasize the value of critical thinking and problem-solving skills.

Regardless of the label, educators and policymakers are increasingly interested in developing students’ non-cognitive skills in support of success both in school and later in life (see, for example, Yeager, Paunesku, Walton, & Dweck, 2013). For example, several high-performing charter management organizations have implemented comprehensive discipline systems aimed at molding student behavior in and out of school in pro-social and pro-academic directions (Lake et al., 2012). Some Knowledge Is Power Program (KIPP) charter schools go so far as to issue a regular “Character Growth Card” for each student that tracks the development of various non-cognitive skills. Related developments include efforts to address students’ social and emotional learning needs alongside traditional academic goals (Durlak et al., 2011). In August 2013, the U.S. Department of Education approved an application from a consortium of California school districts to implement a new school accountability metric that weights test-based outcomes as only 60% of overall performance, with the balance assigned to school climate indicators and measures of social-emotional development based on student self-reports.

As practice and policy race forward, however, research on non-cognitive skills remains in its infancy. There is little agreement on which skills are most important, how they can be reliably measured, and their malleability in school settings. Absent consensus on these points, educators cannot rely on available measures of non-cognitive skills or their underlying theories of personal development to assess and support individual students or to evaluate the success of schools, teachers, or interventions. As if to illustrate this dilemma, the California consortium applying to develop its own accountability system left the specific social-emotional measures used in school ratings to be defined later.¹

In this article, we draw on cross-sectional data from a large sample of students in the city of Boston to shed light on the relationships between four prominent non-cognitive skills, student behavior, and academic achievement, as well as on the ability of extant measures of those skills to capture school impacts on their development. We used self-report survey instruments to gather information on non-cognitive skills from more than 1,300 eighth-grade students across a wide range of the city’s public schools and linked this information to administrative data on the students’ demographics, attendance, behavior, and test-score performance. The schools attended by students in our sample include both open-enrollment public schools operated by the local school district and over-subscribed charter schools that have been shown to have large positive impacts on student achievement as measured by state math and English language arts (ELA) tests (Abdulkadiroglu, Angrist, Dynarski, Kane, & Pathak, 2011; Angrist, Pathak, & Walters, 2013).

The non-cognitive skills we measured include conscientiousness, self-control, grit, and growth mindset. Of the many non-cognitive attributes that psychologists have studied in students, conscientiousness and self-control have arguably the strongest evidence of predictive power over academic and life outcomes, even when controlling for cognitive ability and demographics (Almlund et al., 2011; Duckworth & Carlson, 2013; Poropat, 2009). We also examine two newer measures, grit and growth mindset, because of their current salience among educators seeking to influence non-cognitive skills to support academic achievement, post-secondary persistence and completion, and life outcomes. Grit refers to the tendency to sustain interest in, and effort toward, long-term goals (Duckworth, Peterson, Matthews, & Kelly, 2007), while growth mindset measures students’ implicit theory of intelligence—in particular, the extent to which they believe that their academic ability can improve with effort, rather than being fixed by factors outside of their control (Dweck, 2006).

Our work builds on the framework developed by Farrington et al. (2012) in their valuable synthesis of research on the role of “non-cognitive factors” in shaping academic performance. The authors refer to conscientiousness, self-control, and grit as measures of academic perseverance and to growth mindset as one of several aspects of academic mindset. They suggest that academic mindsets and academic perseverance interact to lead students to engage in more pro-academic behaviors, such as attending school, doing homework, and studying, resulting ultimately in higher achievement. We use our data to look for evidence consistent with these hypothesized relationships both within and across schools.

Our results highlight both the potential value of these factors in explaining pro-academic behavior and achievement and a challenging paradox that may be inherent to many available measures of non-cognitive skills. The promise is illustrated by the fact that our measures of non-cognitive skills are positively correlated with student attendance and behavior, state test scores, and test-score gains from fourth grade to eighth grade. The paradox is that schools in which students on average report higher levels of conscientiousness, self-control, and grit do not have higher average test-score gains than do other schools. In other words, the positive student-level relationships between these self-reported measures of non-cognitive skills and improvements in academic achievement dissipate when the measures are aggregated to the school level.

This paradox is especially apparent when comparing students attending over-subscribed charter schools and those attending open-enrollment district schools. Despite making larger test-score gains than students attending open-enrollment district schools, charter school students rate themselves lower, on average, on measures of conscientiousness, self-control, and grit. Exploiting data from school admissions lotteries, we replicate previous quasi-experimental findings indicating positive impacts of charter school attendance on math achievement within the students in our sample, but find large and statistically significant negative impacts on these non-cognitive skills. Finally, we also present longitudinal data showing marked declines in the same non-cognitive skills over time among students attending two over-subscribed charter middle schools.

This pattern is puzzling for two reasons. First, evidence gathered in the same study and reported in Finn et al. (2014) indicates that the test-score gains made by the charter school students in our data were not accompanied by gains in fluid reasoning skills, which are highly correlated with test-score levels and gains. Although this might seem to suggest that these students’ academic progress was supported by improvements in non-cognitive skills, we observe sharp declines in three non-cognitive skills that are also correlated with academic success. A second reason is the emphasis that the over-subscribed charter schools in our study, all of which subscribe to a “no excuses” approach to urban education, place on character development as a means to foster academic success (Seider, 2012). Indeed, Angrist, Pathak, and Walters (2013) present compelling evidence that it is this “no excuses” orientation that distinguishes Boston’s charter schools from their counterparts elsewhere in the state that have neutral or even negative impacts on student test scores.

Two competing hypotheses could explain this paradox. One is that the measures of non-cognitive skills are accurate and the charter schools, despite their success in raising test scores, and contrary to their stated goals, reduce students’ non-cognitive abilities along crucial dimensions such as conscientiousness, self-control, and grit. An alternative hypothesis is that the measures, all self-reported by students, are misleading because they are prone to reference bias—the tendency for survey responses to be influenced by the context in which the survey is administered. We find suggestive evidence supporting this alternative hypothesis, highlighting the importance of improved measurement of non-cognitive skills to capitalize on their promise as a tool for education practice and policy.

Measurement of Non-Cognitive Skills and the Perils of Reference Bias

Recognition of the importance of non-cognitive skills has, with few exceptions, preceded the development of valid and reliable measures thereof. Whereas performance tasks to assess how well children can read, write, and cipher are widely available, non-cognitive skills are typically assessed using self-report and, less frequently, informant-report questionnaires. Like standardized achievement tests, both types of questionnaires have the advantage of quick, cheap, and easy administration.

Questionnaires also have limitations. Most obviously, they are subject to faking and therefore to social desirability bias (Paulhus, 1991). When endorsing a questionnaire item such as “I am a hard worker,” a child (or her teacher or parent) might be inclined to choose higher ratings to seem more attractive to observers or to herself. To the extent that social desirability bias is uniform within a population under study, it can alter the absolute level of individual responses but not their rank order. If some individuals are more influenced by social pressure than others, however, their relative placement within the overall distribution of responses can change.

Less obvious but possibly more pernicious is reference bias, which occurs when individual responses are influenced by differing implicit standards of comparison. When considering whether “I am a hard worker” should be marked “very much like me,” a child must conjure up a mental image of “a hard worker” to which she can then compare her own habits. A child with very high standards might consider a hard worker to be someone who does all of her homework well before bedtime and, in addition, organizes and reviews all of her notes from the day’s classes. Another child might consider a hard worker to be someone who brings home her assignments and attempts to complete them, even if most of them remain unfinished the next day.

Reference bias was first documented in cross-cultural psychology, and culturally shared standards appear to be a primary influence on implicit standards of comparison. In studies of distinct societies, data from self-report surveys often conflict with the conclusions of cultural experts (Peng, Nisbett, & Wong, 1997). For instance, despite a widely acknowledged cultural emphasis on conscientious behavior, individuals in East Asian countries rate themselves lower in conscientiousness than do individuals in any other region (Schmitt, McRae, & Benet-Martinez, 2007). Heine, Buchtel, and Norenzayan (2008) demonstrate that self-reported conscientiousness at the country level (i.e., the average self-reported conscientiousness rating for citizens of a particular country) is negatively correlated with several objective proxies for conscientiousness, including postal workers’ efficiency, the accuracy of clocks in public banks, walking speed, and longevity. Within the United States, Naumann and John (2013) find that at the University of California, Berkeley, European American undergraduates rated themselves higher in conscientiousness than did their Asian American classmates, despite earning lower grade point averages (GPAs). This paradoxical finding disappeared when both groups were asked to complete the same questions with the explicit reference point of a “typical Asian American Berkeley student.”

Education researchers have documented similarly unexpected patterns in students’ responses to questionnaires administered during international assessments of student achievement. For example, Kyllonen and Bertling (2013) report that, in the 2003 Programme for International Student Assessment study, a five-item scale measuring self-confidence in math is positively correlated with math achievement among students within the same country but negatively correlated with math achievement at the aggregate level across countries. That is, students who are more confident in their abilities than their peers tend to achieve at higher levels, but students in countries where students are more confident on average tend to achieve at lower levels. Kyllonen and Bertling refer to this pattern, which is also evident for scales measuring interest in math and science, perceptions of teacher support, and general attitudes toward school, as the “attitude-achievement anomaly.” Although there is no logical reason why the individual-level and aggregate correlations between two variables need point in the same direction—assuming they must is the “ecological fallacy”—these patterns nonetheless cast doubt on the validity of cross-cultural comparisons of student attitudes (Van de Gaer, Grisay, Schulz, & Gebhardt, 2012).

Apart from national culture or ethnicity, what are likely influences on students’ implicit frames of reference when responding to surveys designed to gauge their non-cognitive skills? Because students (like adults) are unable to see the full distribution of human behavior, their peer groups and other aspects of their immediate social context are likely to shape how they evaluate themselves. It follows that the school environment in which they spend much of their waking lives could exert a powerful influence on students’ perspectives on their own attributes and abilities. We return to this possibility when interpreting our findings below.

Data and Measures

Sample

We collected data from a large sample of eighth-grade students attending 32 of the 49 public schools with an eighth-grade cohort in the city of Boston during the spring semester of the 2010–2011 school year. The schools that agreed to participate in the study included 22 open-enrollment district schools, 5 over-subscribed charter schools, 2 test-in exam schools, and 3 charter schools that were not over-subscribed at the time the students in our study entered middle school. Within these schools, we sampled all students for whom we obtained parental consent to participate in the study and who were in attendance on the day we collected our data.

We acquired school enrollment and demographic information, data on attendance and suspensions, and math and ELA test scores on the Massachusetts Comprehensive Assessment System (MCAS) from databases maintained by the Massachusetts Department of Elementary and Secondary Education. MCAS-scaled scores were standardized to have mean zero and unit variance by grade, subject, and year across all tested students in Massachusetts. We limit our analytic sample to the 1,368 of a total of 1,852 students who participated in the non-cognitive data collection for whom MCAS math and ELA scores were available in 2007 (when most students were in fourth grade) and 2011, making it possible to track their academic progress and school enrollment since they entered middle school.

Table 1 compares the demographic characteristics and academic indicators of students in our analytic sample with those of all eighth-grade students attending public schools in Boston, as well as with those of eighth graders attending schools participating in the study. The sampled students are demographically similar to all eighth-grade students attending public schools in the city and to eighth graders attending the same schools. However, the requirement that students return a parental consent form and be in attendance on the day we administered our survey did result in some notable differences between sampled students and non-sampled students. The eighth-grade test scores of sampled students are 0.24 and 0.21 standard deviations higher than the citywide average in math and ELA, respectively. Comparing the test scores of the sampled students with those of eighth graders attending the same schools indicates that more than half of this difference reflects positive selection into the study sample within participating schools. As we would expect, sampled students have slightly fewer suspensions and are absent less often than other students within the same schools. These differences, though small, may limit the generalizability of our findings.

Table 1

Mean Student Demographic Characteristics and Academic Indicators by School Type Among All Public Schools in Boston, Sampled Schools, and Sampled Students

	Public schools in Boston			Open-enrollment district schools		Over-subscribed charter schools
	All students	All students in sampled schools	Sampled students	All students in sampled schools	Sampled students	All students in sampled schools	Sampled students
Male	0.52	0.51	0.47	0.53	0.49	0.43	0.43
African American	0.41	0.40	0.38	0.41	0.39	0.53	0.50
White	0.12	0.12	0.13	0.08	0.08	0.17	0.18
Asian	0.09	0.07	0.09	0.03	0.05	0.02	0.01
Hispanic	0.37	0.39	0.39	0.46	0.46	0.27	0.30
Free/reduced price lunch	0.79	0.78	0.77	0.86	0.87	0.66	0.66
Limited English proficient	0.12	0.12	0.11	0.17	0.16	0.00	0.00
Special education	0.22	0.22	0.19	0.27	0.23	0.16	0.18
Eighth-grade math	−0.32	−0.24	−0.08	−0.63	−0.49	0.36	0.41
Eighth-grade ELA	−0.44	−0.36	−0.23	−0.67	−0.55	0.20	0.21
Fourth-grade math	−0.47	−0.46	−0.36	−0.74	−0.66	−0.42	−0.38
Fourth-grade ELA	−0.57	−0.55	−0.46	−0.85	−0.79	−0.45	−0.42
Days suspended	0.25	0.22	0.19	0.19	0.16	0.39	0.31
Days absent	11.85	11.03	9.00	12.95	10.61	7.88	7.14
Number of schools	49	32	32	22	22	5	5
Number of students	3,151	2,121	1,368	1,483	906	234	148

Note. All samples are restricted to students with valid 2011 and 2007 MCAS scores. Sampled schools are schools participating in non-cognitive skill data collection; sampled students are those with valid data on at least one non-cognitive skill. Math and ELA test scores are standardized to have mean zero and unit variance statewide. ELA = English language arts; MCAS = Massachusetts Comprehensive Assessment System.

Much of our analysis compares sampled students attending open-enrollment district and over-subscribed charter schools. Looking separately at these two groups, we see that positive selection with respect to academic indicators is more pronounced within the district schools. Specifically, the eighth-grade test scores of sampled students in district schools exceeded those of all students by 0.15 (math) and 0.11 (ELA) standard deviations, while the analogous differences in the over-subscribed charter schools were 0.05 (math) and 0.01 (ELA). This difference does not stem from substantially higher rates of study participation within the charter sector; the share of all eighth graders participating in the study was 63% in the over-subscribed charter schools, as compared with 61% in the open-enrollment district schools.

Table 1 also reveals that the eighth-grade test scores of students are considerably higher in the over-subscribed charter schools than in the open-enrollment district schools. Within our sample, students in these over-subscribed charter schools exceeded the statewide mean by 0.41 (math) and 0.21 (ELA) standard deviations, whereas students in open-enrollment district schools trailed the state average by −0.49 (math) and −0.55 (ELA) standard deviations. The students in our sample attending over-subscribed charter schools also experienced larger gains in test scores (relative to the state average) between fourth grade and eighth grade. Charter students gained 0.79 (math) and 0.63 (ELA) standard deviations relative to the state average over those 4 years, whereas their district peers gained 0.17 (math) and 0.25 (ELA) standard deviations. Sampled students in over-subscribed charter schools were also 10 percentage points more likely to be White, 16 percentage points less likely to be Hispanic, and 21 percentage points less likely to be eligible for a free or reduced price lunch than their counterparts in open-enrollment district schools. They had on average two thirds as many days absent as students in open-enrollment district schools but spent nearly twice as much time in suspension (0.31 vs. 0.16 days), suggesting greater use of suspensions as a disciplinary tool in the city’s charter sector.

Measures of Non-Cognitive Skills

All students participating in our study completed a battery of questionnaires designed to measure their non-cognitive skills along various dimensions. These questionnaires, which were administered in students’ regular classrooms, included items probing students’ conscientiousness, self-control, grit, and growth mindset that have been previously shown to produce internally reliable measures of each construct for adolescent students. After scoring student responses to each series of items based on the relevant rubric, we then standardized the scores to have a zero mean and unit variance within our analytic sample.

To assess students’ conscientiousness, we administered the Big Five Inventory (John & Srivastava, 1999), a well-established 44-item survey measuring each of the “Big Five” personality traits: neuroticism, extraversion, openness to experience, agreeableness, and conscientiousness.² Students endorsed items using a 5-category Likert-type scale, where 1 = strongly disagree and 5 = strongly agree. Each student’s conscientiousness score is calculated as the average of their responses to the nine items that comprise the conscientiousness scale (e.g., “I think I am someone who is a reliable worker”). Among the students in our data, this conscientious scale had an internal reliability score of .76.

Our measure of self-control is based on the Impulsivity Scale for Children, an eight-item survey developed to measure school-aged students’ impulsivity, defined as the “inability to regulate behavior, attention, and emotions in the service of valued goals” (Tsukayama, Duckworth, & Kim, 2013, p. 879). This survey asked students to indicate how often during the past school year they exhibited each of a set of behaviors indicative of a lack of self-control, with five response options ranging from “almost never” to “at least once a day.” The use of response categories specifying objective, discrete time periods was motivated by a desire to “avoid reference bias” in students’ responses (Tsukayama et al., 2013, p. 881). The survey included four items measuring interpersonal self-control (e.g., “I interrupted other students while they were talking”) and four items measuring intrapersonal self-control (e.g., “I forgot something I needed for class”). We calculated an overall self-control score for each student as the average of their (reverse-coded) responses to all eight items. This scale had an internal reliability of .83.

Students also completed the eight-item Short Grit Scale (Grit-S) developed by Duckworth and Quinn (2009) to measure trait-level persistence toward long-term goals.³ Students endorsed a series of items (e.g., “New ideas and projects sometimes distract me from old ones” and “I finish whatever I begin”) using a 5-category Likert-type scale, where 1 = not like me at all and 5 = very much like me. Students’ grit scores were then calculated as their mean response across all eight items. This scale had somewhat lower internal reliability (α = .64) among the students in our sample than the conscientiousness and self-control scales.

Finally, to probe students’ implicit theory of intelligence, we administered a set of three items measuring the extent to which students view intelligence as a fixed trait rather than one which can be improved with effort (Dweck, 2006). For example, students were asked to rate their agreement with the claim that “You have a certain amount of intelligence, and you really can’t do much to change it.” Following Blackwell, Trzesniewski, and Dweck (2007), we used a 6-category Likert-type scale, where 1 = strongly disagree and 6 = strongly agree. After reverse coding, we calculated each student’s mean response across these three items to create a scale with an internal reliability of .86. We refer to this scale as measuring the extent to which students have a growth mindset (as opposed to a fixed mindset).⁴

Student Perceptions of School Climate

The battery of questionnaires students completed also included a set of 10 items probing the climate at their school. These items were derived from a survey developed by the Tripod Project for School Improvement to measure various aspects of school or classroom climate. Each item asked students to respond to a descriptive statement about their school using a 5-category Likert-type scale, where 1 = totally untrue and 5 = totally true. The overall set included two items related to each of five dimensions of school climate: high expectations (e.g., “Teachers at this school demand that students work hard”), teacher strictness (e.g., “Teachers are very strict here”), clarity of rules (e.g., “Students understand what will happen to them if they break a rule”), negative peer effects (e.g., “In this school, some students try to keep others from working hard”), and student input (e.g., “Students in this school have a say in how things work”). We calculated school-level averages of students’ responses to the two items within each pair to provide a summary measure of the relevant aspect of each school’s climate as perceived by its students.

Achievement Gains

We also used our administrative data to generate measures of each student’s test-score gains in math and ELA between fourth grade and eighth grade. Specifically, we regressed students’ eighth-grade MCAS test scores in the relevant subject on a cubic polynomial of their fourth-grade test scores in both subjects and took the residual from that regression for each student. We use these residualized gain scores, which capture the extent to which a student’s eighth-grade performance in math and ELA exceeded expectations based on their performance 4 years earlier, to examine the relationship between non-cognitive traits and improvements in test-score performance over time.

Results

Student- and School-Level Correlations of Non-Cognitive Traits and Academic Indicators

Table 2 reports student-level Pearson product–moment correlations among the full set of non-cognitive skills included in our analysis, eighth-grade test scores in math and ELA, and annual counts of absences and suspensions in eighth grade.⁵ Given that conscientiousness, self-control, and grit are closely related constructs, it is unsurprising that they are highly interrelated, with correlations ranging from .43 to .66. Online Appendix Table A1 (available at http://epa.sagepub.com/supplemental) shows that the correlations among these three measures increase to between .59 to .95 when we use their internal reliabilities to adjust for error in the measurement of each construct (Spearman, 1904). Growth mindset is also positively and significantly correlated with each of these measures, but at lower levels ranging from .08 (conscientiousness) to .18 (grit).

Table 2

Correlation Matrix of Non-Cognitive Skills and Academic Indicators

	Conscientiousness	Self-control	Grit	Growth mindset	Eighth-grade math	Eighth-grade ELA	Days suspended	Days absent
Conscientiousness	1
Self-control	.47***	1
Grit	.66***	.43***	1
Growth mindset	.08**	.10***	.18***	1
Eighth-grade math	.05*	.13***	.03	.32***	1
Eighth-grade ELA	.05*	.10***	.04	.36***	.69***	1
Days suspended	−.06*	−.14***	−.12***	−.04	−.10***	−.11***	1
Days absent	−.06*	−.12***	−.03	−.10***	−.30***	−.25***	.13***	1

Note. Sample restricted to students with complete data on each indicator (n = 1,340). ELA = English language arts.

p < .05. **p < .01. ***p < .001.

As noted above, these non-cognitive skills are hypothesized to lead to pro-academic behaviors and higher student achievement. Among the four non-cognitive measures, growth mindset is most strongly related to eighth-grade test scores, with correlations of .32 (math) and .36 (ELA). Self-control is also significantly related to test scores, with correlations of .13 (math) and .10 (ELA). The correlations of both conscientiousness and grit with test-score levels are positive in both subjects but small and statistically insignificant.

Table 2 also confirms that these non-cognitive measures are related to suspensions and absences, the two behavioral indicators available in our administrative data. All four non-cognitive measures are negatively correlated with the number of days suspended and absent a student accumulated in eighth grade. All but two of those eight correlations are statistically significant.⁶ Because both absences and suspensions are not normally distributed among the students in our data, the strength of these relationships is difficult to interpret. Table 3 therefore compares average days absent and suspended, as well as the percent of students who were suspended at least once in eighth grade, for students in the top and bottom quartile of each non-cognitive skill. This analysis confirms the existence of substantively important differences in both behavioral indicators between students with relatively high and low self-reported non-cognitive skills. For example, students in the bottom quartile of self-control are absent 2.8 more days than students in the top quartile, spend 4 times as much time in suspension, and are almost 3 times as likely to have been suspended at least once. Noteworthy differences between bottom- and top-quartile students are also evident for conscientiousness (absences and suspensions), grit (suspensions), and growth mindset (absences).

Table 3

Absences and Suspensions by Non-Cognitive Skill Quartile

	Days absent			Days suspended			Percent suspended
	Bottom	Top	Difference	Bottom	Top	Difference	Bottom	Top	Difference
Conscientiousness	9.56	7.76	1.80**	0.20	0.09	0.11*	12.2	7.2	5.0*
	(8.63)	(8.00)	(0.66)	(0.68)	(0.34)	(0.04)	(32.8)	(25.9)	(0.2)
Self-control	10.30	7.42	2.80***	0.28	0.07	0.21***	15.8	5.7	10.1***
	(9.07)	(7.62)	(0.66)	(0.84)	(0.29)	(0.05)	(36.6)	(23.3)	(2.4)
Grit	8.99	8.23	0.76	0.28	0.09	0.19***	15.1	6.2	8.9***
	(8.62)	(7.92)	(0.64)	(0.88)	(0.38)	(0.06)	(35.9)	(24.1)	(2.4)
Growth mindset	10.48	7.73	2.75***	0.21	0.14	0.08	13.5	9.0	4.5
	(9.88)	(7.70)	(0.70)	(0.74)	(0.57)	(0.05)	(34.2)	(28.7)	(2.5)

Note. Standard deviations are reported in parentheses. Sample restricted to students with complete data on each indicator (n = 1,340).

p < .05. **p < .01. ***p < .001.

Table 4 examines the relationships between the non-cognitive and behavioral measures and residualized test-score gains, which capture students’ academic performance in eighth grade relative to expectations based on their performance in fourth grade. The first column confirms that each of the four non-cognitive measures is positively correlated with test-score gains in both math and ELA, although the correlation between self-control and ELA performance is not statistically significant. The relationships are strongest for growth mindset, which has correlations with test-score gains of .22 and .17 in math and ELA, respectively. It also confirms that suspensions and absences are negatively associated with achievement gains, suggesting that pro-academic behavior may mediate the relationship between non-cognitive skills and gains in student achievement over time.

Table 4

Student- and School-Level Relationships Between Non-Cognitive Skills and Test-Score Gains

	Overall, student level	Overall, school level	Within-school, student level
4th–8th MCAS math gains
Conscientiousness	0.106**	−0.118	0.154***
	(0.030)	(0.181)	(0.032)
Self-control	0.083*	−0.026	0.133***
	(0.036)	(0.183)	(0.030)
Grit	0.120**	0.030	0.166***
	(0.033)	(0.182)	(0.033)
Growth mindset	0.216***	0.662***	0.169***
	(0.038)	(0.137)	(0.043)
Days suspended	−0.018	0.164	−0.069**
	(0.041)	(0.180)	(0.024)
Days absent	−0.166***	−0.64***	−0.075**
	(0.029)	(0.140)	(0.023)
4th–8th MCAS ELA gains
Conscientiousness	0.084**	−0.159	0.115***
	(0.028)	(0.180)	(0.030)
Self-control	0.040	−0.063	0.083**
	(0.028)	(0.182)	(0.029)
Grit	0.079*	−0.055	0.104*
	(0.036)	(0.182)	(0.039)
Growth mindset	0.167***	0.482**	0.165***
	(0.041)	(0.160)	(0.036)
Days suspended	−0.017	0.148	−0.056*
	(0.034)	(0.181)	(0.024)
Days absent	−0.102**	−0.386*	−0.081**
	(0.037)	(0.168)	(0.027)
Observations	1,340	32	1,340

Note. Each cell presents results from a separate regression. Standard errors reported in parentheses are clustered by eighth-grade school. Within-school regressions include fixed effects for schools. Non-cognitive skills are standardized to have mean zero and unit variance in the study sample. MCAS = Massachusetts Comprehensive Assessment System; ELA = English language arts.

p < .05. **p < .01. ***p < .001.

The second column of Table 4, however, reveals that the positive relationships between test-score gains and three of the measures—conscientiousness, self-control, and grit—dissipate and even reverse when we aggregate the data to the school level and examine the correlation between school-average achievement gains and non-cognitive measures. Although none of the six correlations reported for these variables is statistically significant due to the reduced sample size, five are negative. In other words, although students who rate themselves higher in these areas make larger test-score gains, schools with higher average student ratings have, if anything, smaller average gains in achievement. Importantly, the same unexpected pattern is not evident for growth mindset. For this non-cognitive measure, school-level correlations with test-score gains are substantially stronger (0.66 and 0.48 in math and ELA, respectively) than the analogous student-level correlations. Our two behavioral indicators also exhibit strikingly different patterns. The negative relationship between suspensions and test-score gains becomes positive (though not statistically significant) when we aggregate to the school level, whereas the relationship between absences and test-score gains remains negative and grows substantially stronger.

The final column in Table 4, in which we use fixed effects to limit comparisons to students attending the same school, confirms that the student-level relationships between conscientiousness, self-control, and grit and test-score gains among students in the same school are consistently stronger than the same relationships across the sample as a whole. For example, the magnitude of the relationship between self-control and test-score gains increases by 60% in math and 108% in ELA. In contrast, the student-level relationship between growth mindset and test-score gains is modestly weaker when estimated within schools rather than across the full sample. We again find that suspensions follow the same pattern as conscientiousness, self-control, and grit, with a stronger negative relationship to test-score gains within schools compared with the overall student-level correlation. Absences follow the pattern of growth mindset whereby the within-school correlation is slightly attenuated.

On their own, the divergent student- and school-level correlations we observe for conscientiousness, self-control, and grit do not establish that these measures are biased by school context. It could be the case, for example, that teachers within a given school devote more attention and resources to their most conscientious students at the expense of their least conscientious students, leading conscientious students to learn more than their same-school peers. This would generate a positive within-school correlation between conscientiousness and test-score gains even if the two variables were unrelated at higher levels of aggregation. Yet, the lack of a positive school-level relationship could also reflect a tendency on the part of students in schools with more demanding academic environments to rate their non-cognitive skills more critically. To examine this possibility, we turn now to a more focused comparison of students attending over-subscribed charter and open-enrollment district schools.

Mean Non-Cognitive Traits and Test-Score Gains by School Type

Table 5 compares the mean test-score gains and non-cognitive traits for students attending the 22 open-enrollment district and 5 over-subscribed charter schools included in our sample. Consistent with the descriptive statistics in Table 1, mean residualized test-score gains between fourth grade and eighth grade are higher among students attending charter schools. The differences are substantial, at 0.67 standard deviations in math and 0.47 standard deviations in ELA.

Table 5

Mean Test-Score Gains and Non-Cognitive Skills by School Type

	Open-enrollment district school	Over-subscribed charter school	Difference
MCAS math gain (4th–8th)	−0.016	0.656	0.673**
	(0.664)	(0.654)	(0.114)
	[906]	[148]
MCAS ELA gain (4th–8th)	−0.013	0.453	0.466**
	(0.734)	(0.751)	(0.129)
	[906]	[148]
Conscientiousness	0.022	−0.069	−0.091
	(0.994)	(0.981)	(0.078)
	[890]	[145]
Self-control	0.001	−0.225	−0.226*
	(1.017)	(0.969)	(0.116)
	[891]	[145]
Grit	0.033	−0.101	−0.133
	(0.986)	(1.009)	(0.093)
	[888]	[145]
Growth mindset	−0.121	0.260	0.381**
	(1.016)	(0.95)	(0.104)
	[887]	[144]

Note. Statistical significance is for difference in mean for over-subscribed charter schools and traditional public schools. In the first two columns, standard deviations are reported in parentheses and sample sizes in brackets. Standard errors reported in parentheses in the third column are adjusted for clustering by school. MCAS scores are standardized to have mean zero and unit variance statewide; non-cognitive skills are standardized to have mean zero and unit variance in the study sample. MCAS = Massachusetts Comprehensive Assessment System; ELA = English language arts.

p < .05. **p < .01.

Despite the fact that sampled students attending charter schools experienced larger test-score gains than sampled students in district schools, the same students exhibit markedly lower average levels of self-control as measured by student self-reports. This statistically significant difference of −0.23 standard deviations is in the opposite direction of that expected based on the positive student-level correlations between self-control and test-score gains evident in Table 2. The average differences between the charter and district students in conscientiousness (−0.09) and grit (−0.13), although statistically insignificant, run in the same counter-intuitive direction.

Interestingly, the difference in average growth mindset between charter school and district school students follows a different pattern than the other three non-cognitive skills. Charter school students score 0.38 standard deviations higher, on average, which is consistent with the student-level correlation between growth mindset and test-score gains within the sample as a whole.

Lottery-Based Estimates of the Effect of Charter School Attendance

Simple comparisons of the outcomes of students attending charter and district public schools, such as those presented in Table 5, capture both any effects of attending a charter school on those outcomes and selection into charter schools based on characteristics correlated with the outcome. Although over-subscribed charter schools must admit students via lottery, applicants who seek to enroll in an academically demanding charter school likely differ from those who do not along both observed and unobserved dimensions. To better isolate the causal effect of attendance at the five over-subscribed charter schools in our sample, we exploit the lottery admissions process at these schools to restrict our comparisons to students who entered the admissions lottery of one or more of the over-subscribed charter schools.

Students apply individually to each charter school and are entered into independent admissions lotteries when the number of applications exceeds the number of available seats. We acquired records from the lotteries used to admit the students in our sample directly from the charter schools and matched these records to state administrative data on all public school students using names, year, and grade of application. Of 702 verified lottery participants, 497 appeared in the administrative data and had valid demographic data and test scores for both subjects in 2007 and 2011.⁷ A smaller subset of 200 students met those requirements and participated in the non-cognitive data collection. Among these students, 143 were offered admission to at least one charter school and 124 enrolled in a charter for at least 1 year. Although we can produce lottery-based estimates of charter school impacts on non-cognitive traits only in the latter sample, we present estimated impacts on test scores for both to compare the results across samples. To the extent that our estimated impacts on test scores are similar across the two groups, it should reduce concerns that our results are biased due to non-random sampling of successful and unsuccessful applicants.

Table 6 examines whether the demographic characteristics and fourth-grade test scores of students offered and not offered admission to a charter school were balanced within these two subsamples of lottery participants. We first note that within both subsamples, the share of applicants who were offered admission (68% in the administrative data sample and 71% in the non-cognitive sample) is very similar to the share among all lottery applicants (69%). F tests nonetheless reveal that students’ observed characteristics are jointly significant predictors of whether they were offered a seat in a charter school in both samples.

Table 6

Balance of Observed Characteristics in Admissions Lotteries for Over-Subscribed Charter Schools

	Administrative data sample				Non-cognitive sample
	Offer	No offer	Regression-adjusted difference	p value	Offer	No offer	Regression-adjusted difference	p value
Male	0.501	0.437	0.100	.099	0.420	0.368	0.054	.532
African American	0.590	0.513	0.013	.779	0.517	0.544	−0.124	.145
White	0.100	0.171	0.008	.828	0.112	0.211	0.001	.980
Asian	0.009	0.025	−0.016	.312	0.014	0.018	0.007	.691
Hispanic	0.277	0.278	−0.016	.760	0.350	0.228	0.107	.140
Free/reduced price lunch	0.687	0.563	0.107	.022	0.657	0.579	0.007	.926
Limited English proficient	0.032	0.070	−0.049	.129	0.021	0.035	−0.022	.530
Special education	0.195	0.209	−0.003	.956	0.140	0.123	0.009	.884
Fourth-grade MCAS math	−0.527	−0.415	−0.080	.306	−0.349	−0.095	−0.187	.091
Fourth-grade MCAS ELA	−0.520	−0.358	−0.127	.236	−0.356	0.074	−0.300	.110
Joint F-test statistic				.003				.000
Observations	339	158			143	57

Note. Administrative data sample includes all applicants to over-subscribed charter schools matched to valid 2011 and 2007 test scores in the Massachusetts Department of Elementary and Secondary Education state database. Study sample includes students in the administrative data sample with valid data on at least one non-cognitive outcome. The first two columns for each sample provide the mean of each variable for students receiving at least one and no offers of admission to an over-subscribed charter school. Regression-adjusted differences control for fixed effects for lottery applicant risk sets used to estimate charter attendance effects. p values are for the regression-adjusted difference. MCAS = Massachusetts Comprehensive Assessment System; ELA = English language arts.

In the larger sample of students matched to the administrative data, students offered a charter school seat are significantly more likely to be male and eligible for a free or reduced price lunch. The fourth-grade test scores of students offered a seat are also modestly lower in both math and ELA, though these differences are not statistically significant. Within the smaller non-cognitive sample, the differences in fourth-grade test scores are substantial in both subjects and marginally statistically significant in math (p < .09). These differences, which favor students not offered a charter school seat, may reflect the fact that positive selection into the non-cognitive sample was less pronounced in over-subscribed charter schools (see Table 1) and should bias our results against finding positive impacts of charter attendance on test scores. Given the imbalances between students offered and not offered a seat in a charter in both the administrative and survey data samples, we control for students’ observed characteristics and fourth-grade test scores throughout our lottery-based analysis of the effects of charter school attendance.

Following Abdulkadiroglu et al. (2011), we implement the lottery-based analysis via a Two Stage Least Squares (2SLS) regression model in which we first predict charter attendance for each student based on whether they were offered admission and use those predictions to generate an estimate of the effect of charter attendance on our outcomes of interest. The first-stage model is as follows:

YEAR S_{i} = γ_{c} O F F E R_{i c} + θ A_{i, t - 4} + τ X_{i} + \sum_{j} ρ_{j} d_{i j} + ξ_{i s},

where $YEAR S_{i}$ measures the number of years between fifth-grade and eighth-grade student i attended an over-subscribed charter school and $OFFER R_{ic}$ represents a vector of dummy variables indicating that the student was or was not offered a seat at over-subscribed charter school c. We include as controls cubic polynomials of lagged fourth-grade scores in math and ELA ( $A_{i, t - 4}$ ) and a vector of student demographic characteristics ( $X_{i}$ ) including gender, race, age, free and reduced priced lunch status, limited English proficiency, and special education status. The set of indicator variables $d_{i j}$ controls for lottery “risk sets,” or the unique combination of lotteries to which each student applied, indexed by j. Including these risk sets restricts our identifying variation to that which occurs among students who applied to the exact same set of charter schools, eliminating any bias that could result from comparing students who applied to different numbers or combinations of charter schools.

Within the administrative data sample, students offered a seat in at least one of the five over-subscribed charter schools spent an average of 2.1 years between fifth grade and eighth grade in one of those schools, as compared with 0.4 years among students not initially offered a seat. Among students in the non-cognitive sample, students offered and not offered a seat spent 2.4 and 0.6 years in charter schools, respectively.⁸ Online Appendix Table A2, which presents the first-stage regression results, confirms that the $OFFE R_{ic}$ indicators are relevant instruments for predicting variation in years of attendance, with joint F-test statistics of 35.5 and 16.0 for the administrative data and non-cognitive samples, respectively.

We implement the second stage of our 2SLS model as follows:

Y_{i s} = β {\hat{Years}}_{i} + α A_{i, t - 4} + λ X_{i} + \sum_{j} δ_{j} d_{i j} + ϵ_{i s},

where $Y_{i s}$ represents a given test score, non-cognitive, or behavioral outcome for student i in school s; ${\hat{YEARS}}_{is}$ captures each student’s predicted years of charter attendance based on the first-stage regression; and all other variables are as above. Parameter $β$ is the quantity of interest: the effect of 1 year’s attendance at any of the five over-subscribed charter schools.

We present the results of these analyses in Table 7. Consistent with Abdulkadiroglu et al. (2011), the first two columns show that, among the students in the administrative data sample, each additional year of charter attendance is estimated to increase eighth-grade math scores by 0.13 standard deviations. The estimated effect for ELA scores is positive and of non-negligible magnitude, but too imprecisely estimated to achieve statistical significance. We replicate these analyses in our non-cognitive sample and find very similar point estimates in both math and ELA; the math effect is statistically significant at the p < .1 confidence level.⁹

Table 7

Instrumental Variables Estimates of the Effects of a Year’s Attendance at an Over-Subscribed Charter School

	Administrative data sample	Non-cognitive sample
Eighth-grade MCAS math	0.129***	0.110^†
Eighth-grade MCAS math	(0.036)	(0.057)
Eighth-grade MCAS ELA	0.046	0.052
	(0.043)	(0.065)
	[497]	[200]
Conscientiousness		−0.157^†
		(0.078)
		[196]
Self-control		−0.211*
		(0.095)
		[196]
Grit		−0.119*
		(0.054)
		[195]
Growth mindset		−0.030
		(0.093)
		[195]
Days suspended	0.012	0.019
	(0.076)	(0.063)
	[431]	[200]
Days absent	−0.949	−1.42*
	(0.625)	(0.624)
	[431]	[200]

Note. Standard errors reported in parentheses are clustered by eighth-grade school. Sample sizes for each outcome are in brackets. Each cell presents results from a separate regression. Administrative data sample includes all applicants to over-subscribed charter schools matched to valid 2011 test scores in the Massachusetts Department of Elementary and Secondary Education state database. Study sample includes students in the administrative data sample with valid data on at least one non-cognitive outcome. All regressions control for student gender, race, age, LEP, SPED, and free/reduced price lunch; cubic polynomials in fourth-grade MCAS ELA and math scores; and fixed effects for lottery applicant risk sets. MCAS = Massachusetts Comprehensive Assessment System; ELA = English language arts; LEP = limited English proficiency; SPED = special education.

†

p < .1. *p < .05. **p < .01. ***p < .001.

Within that sample, we estimate that 1 year’s attendance at an over-subscribed charter school has a statistically significant negative effect on students’ self-reported self-control and grit and a marginally significant negative effect on self-reported conscientiousness. The estimated effect sizes are in the opposite direction of the achievement effects and of similar or even larger magnitude, ranging between −0.12 (grit) and −0.21 (self-control) standard deviations. These results are consistent with the descriptive patterns in Table 5, which show students in over-subscribed charter schools making larger achievement gains despite lower scores on these non-cognitive measures, and suggest that those patterns are not due merely to selection. Rather, it would appear that attending one of these charter schools adversely affects students’ non-cognitive abilities along these dimensions as assessed by student self-reports. We discuss our interpretation of this unexpected finding below.

Intriguingly, we estimate a near-zero effect of attending an over-subscribed charter schools on the degree to which a student in our non-cognitive sample has a growth mindset, despite the evidence in Table 5 that students in over-subscribed charter schools rated themselves substantially higher on this measure. Although the lottery-based estimate is not precise enough to rule out the possibility of moderate positive (or negative) effects, the null result suggests that the higher levels of growth mindset we observe among students in charter schools may be an artifact of selection. In other words, it may be that students who believe that their intelligence can be improved with effort are more likely to seek out a school with a demanding environment, but that attendance at such a school has no causal effect on their implicit theory of intelligence.

Finally, we examine the effect of attending an over-subscribed charter school on our two behavioral outcomes, suspensions and absences.¹⁰ We find that in both the administrative and non-cognitive samples, there is no effect of charter attendance on suspensions despite the apparent decrease in students’ conscientiousness, self-control, and grit. We also find that, if anything, attending an over-subscribed charter school decreases student absences. In the non-cognitive sample, we estimate that a year’s attendance at an over-subscribed charter school decreases absences by almost a day and half. If over-subscribed charter schools were, in fact, decreasing students’ conscientiousness, self-control, and grit, we would expect corresponding increases in suspensions and absences rather than the null and negative effects we observe.

Longitudinal Trends in Non-Cognitive Skills

We supplement our lottery analysis with longitudinal data tracking one grade cohort of students in each of two over-subscribed Massachusetts charter schools and in one open-enrollment district school over 3 years. Starting in fall 2009, we administered a battery of questionnaires similar to that used in our main analysis to the entering student cohort at three middle schools.¹¹ We then re-administered the same battery at the end of that school year and the 2 years that followed. One of the charter schools and the district school are in Boston and part of our larger study sample, whereas the second charter school is located in another school district but has a “no excuses” orientation similar to that of the over-subscribed charter schools in our sample.

Figure 1 plots average scores for those students in the 2009 entering cohort of each school for whom we have complete data across four time points.¹² Consistent with our lottery-based estimates of a negative impact of charter attendance on students’ conscientiousness, self-control, and grit as assessed by self-reports, we observe a steady decline in each of these traits among students attending the two charter schools as they progress through middle school. The aggregate declines over 3 years are substantial, reaching 0.65 standard deviations in conscientiousness, 0.78 standard deviations in self-control, and 0.59 standard deviations in grit.¹³

Figure 1.

Mean student non-cognitive skills in three middle schools over time.

Scores on these scales also appear to decline among students attending the open-enrollment district school, but by a smaller amount; for conscientiousness and grit, the changes between the first and final time points are statistically insignificant. The statistically significant 0.31 standard deviation decline in average self-control among district school students may reflect normative, developmental decreases in self-control associated with adolescence (Steinberg, 2007). For example, it is very similar to the 0.28 standard deviation decline Duckworth, Tsukayama, and May (2010, Table 1) observe over 4 years among students attending a magnet middle school. The average changes in self-control in “no excuses” charter schools, as perceived and reported by the students themselves, are roughly 2.5 times as large.

In contrast with the other three non-cognitive traits, we observe an increase of 0.49 standard deviations in growth mindset among students in over-subscribed charter schools.¹⁴ This gain is mirrored, however, by a similarly large increase among students at the district school. This pattern in the longitudinal data is again consistent with our lottery-based estimates, which showed no impact of charter school attendance on growth mindset.

Student Perceptions of School Climate by School Type

Our results thus far indicate that students attending over-subscribed charter schools experience sharp declines in several self-reported measures of non-cognitive skills, both over time and relative to students in open-enrollment public schools. These apparently adverse effects of charter school attendance, however, could be an artifact of reference bias. In other words, it could be that students use a higher bar when assessing their own conscientiousness, self-control, and grit when they attend schools that establish high expectations for student effort and a “no excuses” disciplinary culture. To assess the plausibility of this explanation, we examine whether students’ perceptions of the academic and disciplinary climate in open-enrollment district and over-subscribed charter schools differ in a way that could influence students’ self-ratings of their non-cognitive abilities.

Table 8 confirms that students attending over-subscribed charter schools perceive their schools as having very different academic and disciplinary climates than students attending open-enrollment district schools. Students in over-subscribed charter schools rate the work ethic expected of students, teacher strictness, and the clarity of rules in their school substantially higher than do students in district schools. For example, charter students’ ratings of expectations exceed those of their district counterparts by 0.57 on the 5-point scale used for these items, or 63% of a standard deviation of district students’ responses. The analogous differences observed for teacher strictness and clarity of rules are of comparable magnitude. Students in the over-subscribed charter schools also report substantially lower levels of negative peer effects and modestly lower levels of student input in their schools. In sum, the academic and disciplinary climates of the over-subscribed charter schools in our sample do appear to differ in ways that could lead their students to use a higher bar when assessing their conscientiousness, self-control, and grit.

Table 8

Student Perceptions of School Climate by School Type

	Open-enrollment district school	Over-subscribed charter school	Difference
High expectations	3.929	4.496	0.567**
	(0.900)	(0.669)	(0.149)
	[885]	[112]
Teacher strictness	3.526	4.107	0.581*
	(0.888)	(0.904)	(0.211)
	[878]	[112]
Clarity of rules	3.789	4.186	0.397*
	(0.938)	(0.854)	(0.154)
	[881]	[110]
Negative peer effects	2.738	2.252	−0.486**
	(0.962)	(0.796)	(0.122)
	[878]	[112]
Student input	2.514	2.264	−0.250*
	(0.924)	(0.851)	(0.093)
	[882]	[111]

Note. Statistical significance is for difference in mean for over-subscribed charter and traditional public schools. In the first two columns, standard deviations are reported in parentheses and sample sizes in brackets. Standard errors reported in parentheses in the third column are adjusted for clustering by school.

p < .05. **p < .01.

Discussion

Generations of parents have sought to instill in their children the virtues of diligence, self-discipline, and perseverance. These qualities are at the core of the fables and legends that societies around the world have developed to cultivate the skills, habits, and beliefs understood to be essential for human flourishing. In recent decades, scholars have begun to confirm this common wisdom by developing measures for these non-cognitive skills and examining their relationship with children’s success in school and in life. We add to this literature new evidence that four prominent and widely used measures of non-cognitive skills are positively correlated with achievement gains on standardized tests among a large and diverse sample of eighth-grade students attending distinctly different types of schools. Measures of conscientiousness, self-control, grit, and growth mindset were all positively correlated with attendance, behavior, and math and ELA test-score gains from fourth grade to eighth grade. Higher non-cognitive ability along the dimensions captured by these measures may help explain why some eighth-grade students attend school more consistently, are disciplined less, and score better than predicted by their fourth-grade achievement levels.

However, a paradox emerges when we juxtapose these results with four additional findings. First, the positive correlation between conscientiousness, self-control, and grit and test-score gains is not evident at the school level. Second, students in a set of over-subscribed charter schools where students make unusually large test-score gains report lower average levels of conscientiousness, self-control, and grit than students in open-enrollment district schools. Third, lottery-based analyses of the causal impact of attending these charter schools indicate negative effects on these non-cognitive traits as assessed by self-reports despite improved attendance. Finally, longitudinal data from two charter schools indicate marked declines in the same skills over time.

Although it is possible that the academic gains posted by these schools occur despite or even at the expense of adverse effects on important non-cognitive skills, an alternative explanation is that these paradoxical findings reflect reference bias. More specifically, students attending academically and behaviorally demanding charter schools may redefine upward their notion of what it means to demonstrate conscientiousness, self-control, and grit—and thus rate themselves more critically. In theory, such reference bias could be severe enough to distort the magnitude of any changes in the underlying traits and even to invert their sign.

We find support for this alternative hypothesis not only in our own data on students’ perceptions of school climate but also in other recent evaluations of “no excuses” charter middle schools. Most notably, in a lottery-based study that includes the three charter schools in our sample with high school grades, Angrist, Cohodes, Dynarski, and Pathak (2013) show that charter attendance increased Advanced Placement test-taking and performance and the likelihood of attending a 4-year post-secondary institution. Although not the exact same schools and sample, these findings are difficult to reconcile with an authentic reduction in students’ non-cognitive skills. Tuttle et al. (2013) find large positive effects of attending a KIPP middle school on student test scores and time spent on homework, but no effects on student-reported measures of self-control and persistence in school. Similarly, Dobbie and Fryer (2013) find that attending the Harlem Promise Academy reduced student-reported grit despite having positive effects on test scores and college enrollment and negative effects on teenage pregnancy (for females) and incarceration (for males). This parallel evidence from research in similar settings increases our confidence that reference bias stemming from differences in school climate offers the most likely explanation for our unexpected findings.

Importantly, we see the same paradoxical pattern across our measures of conscientiousness, self-control, and grit despite important differences in their measurement approaches. Items for conscientiousness and grit ask students to assess the degree to which broad statements describe them—a task that effectively requires an external frame of reference. The scale measuring self-control instead asks students to report on the frequency of specific behaviors in an attempt to reduce the influence of variation in students’ reference points. However, our findings suggest that this approach does not reduce the extent of reference bias in student self-reports. One possible explanation for this is that students’ school contexts may influence the extent to which they are aware of specific behaviors such as saying something rude or losing focus in class.

Our findings with respect to growth mindset are quite different. We find little evidence that our measure of this construct is subject to the same paradoxical patterns as the other three measures. It may be that growth mindset is less susceptible to reference bias than the other three skills we consider because it is a measure of internal beliefs, rather than behaviors benchmarked against external ideals. Coming to class prepared or controlling one’s emotions when provoked are desirable behaviors for which students may hold different standards for what is “good enough.” The items on our questionnaire probing growth mindset, in contrast, ask students to report the extent they agree with statements of personal belief about the nature of human intelligence.¹⁵

Taken at face value, our results suggest that the higher average growth mindset scores observed among students in Boston’s over-subscribed charter schools are due to selection and that attendance at a charter school has no impact on students’ academic mindset. Indeed, it seems plausible that students who already believe that their intelligence can be improved with effort would be more likely to seek out a rigorous academic environment. Such a pattern of selection could not account for the strong positive impacts on achievement of attending these schools documented by lottery-based studies, which make comparisons only among charter school applicants. It does suggest, however, that those positive impacts could be driven in part by peer effects arising from the schools’ success in attracting cohorts of students who share a common mindset conducive to academic success.

Finally, our results with respect to suspensions and absences suggest the importance of considering the role of school context even when developing measures of pro-academic behavior. Schools differ considerably in the degree to which they use suspensions as a disciplinary consequence. Thus, the number of times a student has been suspended is likely to be a context-specific measure. When comparing student- and school-level correlations between suspensions and test-score gains, we observe the same paradoxical pattern of results for suspensions that we do for conscientiousness, self-control, and grit. Our lottery analysis also suggests that the over-subscribed charter schools in our sample do not reduce the incidence of suspensions, despite generating large improvements in academic achievement. In contrast, student attendance may be less sensitive to contextual differences across schools. We find that the relationship between absences and test-score gains runs in the same direction within and between schools, and that attendance at an over-subscribed charter school reduces absenteeism.

Conclusion

The findings presented in this article suggest three broad conclusions. First, existing measures of four prominent non-cognitive skills based on self-reports help to explain the proximate outcomes of student behavior and test-score gains in middle school. Efforts by schools to measure and influence those skills may therefore be a promising way to support students’ academic success. To the extent that these skills prove to be malleable in school settings, interventions that target them may yield improvements in academic outcomes in addition to any long-term benefits that could accrue if school-induced changes in non-cognitive skills persist into adulthood.

Second, we have shown that schools can have significant, even if initially surprising, impacts on students’ non-cognitive skills as assessed through self-reports. If we are correct that the apparent negative effects of attending a “no excuses” charter school on conscientiousness, self-control, and grit are due to reference bias, then what these schools influence are the standards to which students hold themselves when evaluating their behavior. The consequences of this shift in normative standards for their actual behavior both within and outside the school environment deserve further research.

Finally, we have shown that findings about school impacts on non-cognitive skills based on self-reports may be misleading due to reference bias resulting from differences in school climate. This appears to be particularly true of skills reflected in behaviors readily observed in one’s peers and may be less of a problem for attributes like growth mindset that reflect internal, privately held beliefs. The challenges posed by reference bias may grow more severe over time to the extent that schools work explicitly to change students’ habits and thereby alter their normative standards.

Our study has several limitations. First, our evidence of the importance of reference bias is circumstantial rather than direct. Second, we have documented the potential problem posed by reference bias without offering a solution to overcome it. In particular, we have not examined whether teacher or parent ratings of students’ non-cognitive skills are less prone to reference bias than are self-reports. Other possible solutions could include the use of anchoring vignettes within surveys to establish consistent reference points (King, Murray, Salomon, & Tandon, 2004) or the development of behavioral indicators of non-cognitive skills that render reference points irrelevant (Hitt & Trivitt, 2013; Jackson et al., 2010).¹⁶ Additional research to document the conditions under which reference bias influences the measurement of students’ non-cognitive skills and validate strategies to correct it is also critical.

The current policy environment demands accountability, and accountability requires assessment. In the rush to embrace non-cognitive skills as the missing piece in American education, policymakers may overlook the limitations of extant measures. Our results raise important questions about the practice of assessing students’ non-cognitive skills based on the existing instruments that rely on student self-reports. In particular, evaluations of the effects of teacher, school, and family influences on non-cognitive skills could lead to false conclusions if the assessments used are biased by distinct frames of reference. Biased measures could similarly misguide scientific investigation of non-cognitive skills.

If we are correct that these measures show both promise and peril, it is imperative that the scholars and practitioners seeking to improve non-cognitive skills through schooling develop new, better measures that are less susceptible to reference bias. Unbiased measures of non-cognitive skills that are robust across a range of school settings could play a constructive role in supporting students’ academic and personal growth. We can and should measure students’ non-cognitive skills, but we should do so in full recognition of the flaws in our measures.

Footnotes

Acknowledgements

We thank Susanna Loeb, Jennifer Jennings, Josh Goodman, and three anonymous referees for helpful comments. Aaron Dow provided outstanding research assistance.

Authors’ Note

Any errors are the sole responsibility of the authors.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by funding from Bill and Melinda Gates Foundation.

Notes

Authors

MARTIN R. WEST is an associate professor of education at the Harvard Graduate School of Education and a faculty research fellow at the National Bureau of Economic Research.

MATTHEW A. KRAFT is an assistant professor of education at Brown University.

AMY S. FINN is a postdoctoral fellow in the Gabrieli Lab in the Department of Brian and Cognitive Sciences at the Massachusetts Institute of Technology.

REBECCA E. MARTIN is a PhD candidate in developmental cognitive neuroscience at Columbia University.

ANGELA L. DUCKWORTH is an associate professor of psychology at the University of Pennsylvania.

CHRISTOPHER F. O. GABRIELI is a lecturer at the Harvard Graduate School of Education and co-founder of Transforming Education.

JOHN D. E. GABRIELI is the Grover Hermann Professor in health sciences and technology and cognitive neuroscience at the Massachusetts Institute of Technology.

References

Abdulkadiroglu

Angrist

J. D.

Dynarski

S. M.

Kane

T. J.

Pathak

P. A.

(2011). Accountability and flexibility in public schools: Evidence from Boston’s charters and pilots. Quarterly Journal of Economics, 126, 699–748.

Almlund

Duckworth

A. L.

Heckman

J. J.

Kautz

T. D.

(2011). Personality psychology and economics. In Hanushek

E. A.

Machin

Woessmann

(Eds.), Handbook of the economics of education (Vol. 4, pp. 1–181). Amsterdam, The Netherlands: Elsevier.

Angrist

J. D.

Cohodes

S. R.

Dynarski

S. M.

Pathak

P. A.

(2013). Stand and deliver: Effects of Boston’s charter schools on college preparation, entry and choice (NBER Working Paper No. 19275). Cambridge, MA: National Bureau of Economic Research.

Angrist

J. D.

Pathak

P. A.

Walters

C. R.

(2013). Explaining charter school effectiveness. American Economic Journal: Applied Economics, 5(4), 1–27.

Bandura

(1999). Social cognitive theory of personality. In Pervin

L. A.

John

O. P.

(Eds.), Handbook of personality psychology (2nd ed., pp. 154–196). New York, NY: Guilford Press.

Baron

(1982). Personality and intelligence. In Sternberg

R. J.

(Ed.), Handbook of human intelligence (pp. 308–351). Cambridge, UK: Cambridge University Press.

Blackwell

L. S.

Trzesniewski

K. H.

Dweck

C. S.

(2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246–263.

Borghans

Duckworth

A. L.

Heckman

J. J.

ter Weel

(2008). The economics and psychology of personality traits. Journal of Human Resources, 43, 972–1059.

Cunha

Heckman

J. J.

(2008). Formulating, identifying and estimating the technology of cognitive and noncognitive skill formation. Journal of Human Resources, 43, 738–782.

10.

Dee

T. S.

West

M. R.

(2011). The non-cognitive returns to class size. Education Evaluation and Policy Analysis, 33, 23–46.

11.

Dobbie

Fryer

R. G.

Jr. (2013). The medium-term impacts of high-achieving charter schools on non-test score outcomes (NBER Working Paper No. 19581). Cambridge, MA: National Bureau of Economic Research.

12.

Duckworth

A. L.

Carlson

S. M.

(2013). Self-regulation and school success. In Sokol

B. W.

Grouzet

F. M. E.

Muller

(Eds.), Self-regulation and autonomy: Social and developmental dimensions of human conduct (pp. 208–230). New York, NY: Cambridge University Press.

13.

Duckworth

A. L.

Peterson

Matthews

M. D.

Kelly

D. R.

(2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101.

14.

Duckworth

A. L.

Quinn

P. D.

(2009). Development and validation of the Short Grit Scale (Grit-S). Journal of Personality Assessment, 91, 166–174.

15.

Duckworth

A. L.

Seligman

M. E. P.

(2005). Self-discipline outdoes IQ in predicting academic performance of adolescents. Psychological Science, 16, 939–944.

16.

Duckworth

A. L.

Tsukayama

May

(2010). Establishing causality using hierarchical linear modeling: An illustration predicting achievement from self-control. Social Psychological & Personality Science, 1, 311–317.

17.

Durlak

J. A.

Dymnicki

A. B.

Taylor

Weissberg

R. P.

Schellinger

K. B.

(2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432.

18.

Dweck

C. S.

(2006). Mindset: The new psychology of success. New York, NY: Random House.

19.

Evans

G. W.

Rosenbaum

(2008). Self-regulation and the income-achievement gap. Early Childhood Research Quarterly, 23, 504–514.

20.

Farrington

Roderick

Allensworth

Nagaoka

Keyes

T. S.

Johnson

Beechum

N. O.

(2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance. Chicago, IL: University of Chicago Consortium on Chicago School Research.

21.

Finn

A. S.

Kraft

M. A.

West

M. R.

Leonard

J. A.

Bish

C. E.

Martin

R. E.

. . . Gabrieli

J. D. E.

(2014). Cognitive skills, student achievement tests, and schools. Psychological Science, 25, 736–744.

22.

Heckman

J. J.

Kautz

(2013). Fostering and measuring skills: Interventions that improve character and cognition. In Heckman

J. J.

Humphries

J. E.

Kautz

(Eds.), The myth of achievement tests: The GED and the role of character in American life (pp. 341–430). Chicago, IL: University of Chicago Press.

23.

Heine

S. J.

Buchtel

E. E.

Norenzayan

(2008). What do cross-national comparisons of personality traits tell us? The case of conscientiousness. Psychological Science, 19, 309–313.

24.

Hitt

Trivitt

(2013). Don’t know? Or don’t care? Predicting educational attainment using survey response rates and coding speed tests as measures of conscientiousness (EDRE Working Paper 2013-05). Fayetteville: University of Arkansas.

25.

Jackson

J. J.

Wood

Bogg

Walton

K. E.

Harms

P. D.

Roberts

B. W.

(2010). What do conscientious people do? Development and validation of the Behavioral Indicators of Conscientiousness (BIC). Journal of Research in Personality, 44, 501–511.

26.

John

O. P.

Srivastava

(1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In Pervin

L. A.

John

O. P.

(Eds.), Handbook of personality: Theory and research (2nd ed., pp. 102–138). New York, NY: Guilford Press.

27.

King

Murray

C. J. L.

Salomon

J. A.

Tandon

(2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98, 191–205.

28.

Kyllonen

P. C.

Bertling

J. P.

(2013). Innovative questionnaire assessment methods to increase cross-country comparability. In Rutkowski

von Davier

Rutkowski

(Eds.), Handbook of international large-scale assessment: Background, technical Issues, and methods of data analysis (pp. 277-285). London, England: Chapman & Hall.

29.

Lake

Bowen

Demeritt

McCullough

Haimson

Gill

(2012). Learning from charter school management organizations: Strategies for student behavior and teacher coaching. Washington, DC: Center on Reinventing Public Education and Mathematica Policy Research.

30.

Messick

(1979). Potential uses of noncognitive measurement in education. Journal of Educational Psychology, 71, 281–292.

31.

Mischel

Shoda

(1999). Integrating dispositions and processing dynamics within a unified theory of personality: The cognitive-affective personality system. Handbook of Personality: Theory and Research, 2, 197-218.

32.

Moffitt

T. E.

Arseneault

Belsky

Dickson

Hancox

R. J.

Harrington

. . . Caspi

(2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academy of Sciences, 108, 2693–2698.

33.

National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills for the 21st century. Washington, DC: The National Academies Press.

34.

Naumann

L. P.

John

O. P.

(2013). Toward a domain-specific approach to cultural differences: The influence of cultural values and reference-group standards on self-reported personality. Unpublished manuscript.

35.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press.

36.

Peng

Nisbett

R. E.

Wong

N. Y. C.

(1997). Validity problems comparing values across cultures and possible solutions. Psychological Methods, 2, 329–344.

37.

Poropat

A. E.

(2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338.

38.

Schmitt

D. P.

McRae

R. R.

Benet-Martinez

(2007). The geographic distribution of big five personality traits: Patterns and profiles of human self-description across 56 nations. Journal of Cross-Cultural Psychology, 38, 173–212.

39.

Seider

(2012). Character compass: How powerful school culture can point students toward success. Cambridge, MA: Harvard Education Press.

40.

Spearman

(1904). “General intelligence,” objectively determined and measured. The American Journal of Psychology, 15, 201–292.

41.

Steinberg

(2007). Risk taking in adolescence: New perspectives from brain and behavioral science. Current Directions in Psychological Science, 16, 55–59.

42.

Tangney

J. P.

Baumeister

R. F.

Boone

A. L.

(2004). High self-control predicts good adjustment, less pathology, better grades, and interpersonal success. Journal of Personality, 72, 271–322.

43.

Tough

(2012). How children succeed: Grit, curiosity, and the hidden power of character. New York, NY: Houghton Mifflin Harcourt.

44.

Tsukayama

Duckworth

A. L.

Kim

(2013). Domain-specific impulsivity in school-age children. Developmental Science, 16, 879–893.

45.

Tuttle

C. C.

Gill

Gleason

Knechtel

Nichols-Barrer

Resch

(2013). KIPP middle schools: Impacts on achievement and other outcomes. Washington, DC: Mathematica Policy Research.

46.

Van de Gaer

Grisay

Schulz

Gebhardt

(2012). The reference group effect: An explanation of the paradoxical relationship between academic achievement and self-confidence across countries. Journal of Cross-Cultural Psychology, 43, 1205–1228.

47.

Yeager

D. S.

Paunesku

Walton

G. W.

Dweck

(2013, May). How can we instill productive mindsets at scale? A review of the evidence and an initial R&D agenda. White paper prepared for the White House Meeting on Excellence in Education: The Importance of Academic Mindsets, Washington, DC.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB