Promoting Deeper Learning in High School: Evidence of Opportunities and Outcomes

Abstract

Policymakers and practitioners are increasingly interested in students’ deeper learning skills, or the interpersonal and intrapersonal skills students need to succeed in school, careers, and civic life. This article presents evidence about whether the concept of deeper learning—applied across a variety of approaches—has potential merit as a means for education improvement. The analysis, based on 16 high schools implementing a school-wide approach to promoting deeper learning within the context of small schools, indicates that students who attended schools focused on deeper learning reported greater opportunities for deeper learning, greater competency in some deeper learning domains, had higher rates of graduating from high school, and were more likely to enroll in 4-year colleges than similar students who attended comparison schools.

Keywords

educational reform high schools learning processes/strategies quasi-experimental analysis

Passage of the Every Student Succeeds Act in 2015 prompted many education policymakers and administrators to revisit the definition of student success. While measures of student performance on standardized tests and rates of on-time graduation dominated the discussion of student success in the No Child Left Behind era, states and districts are beginning to focus on a range of student skills that prepare them for success in college and career. To successfully participate in an increasingly diverse democracy and engage in the evolving workplace, students need more than content knowledge in specific subject areas. They need to be able to communicate their ideas through a variety of media and to a variety of audiences, work together with others to solve problems, think creatively, and manage their own learning (Autor, Levy, & Murnane, 2003; Carnevale & Desrochers, 2003; National Research Council [NRC], 2008). They need to develop a disposition—or mind-sets—that empowers them to confront new challenges, take initiative, and persevere through difficulties and setbacks (Dweck, 2006; Finegold & Notabartolo, 2010). In certain education policy and reform circles, the term “deeper learning” is applied to this combination of a deeper understanding of core academic content, an ability to apply that understanding to novel problems and situations, and a range of competencies related to human interaction and self-management.

While the concept of deeper learning continues to evolve, it has been used to describe both (a) a set of outcome competencies or goals for students and (b) the process of developing deeper learning competencies and the ability to apply those competencies to new and varying situations. The William and Flora Hewlett Foundation—a leader in the national initiative to promote deeper learning in schools—focuses on a set of competencies within six interconnected dimensions of deeper learning seen as prerequisites for success in college, career, and civic life: (a) mastery of core academic content, (b) critical thinking and complex problem-solving skills, (c) effective communication skills, (d) collaboration skills, (e) an understanding of how to learn, and (f) academic mind-sets (Chow, 2010; Trilling, 2010; William and Flora Hewlett Foundation, 2013). Taking a slightly broader view, the NRC (2012) defines deeper learning as the “process through which an individual becomes capable of taking what was learned in one situation and applying it to a new situation” and grouped student skills into three domains of competence: the cognitive domain (e.g., mastery of content knowledge, critical thinking), the interpersonal domain (e.g., communication, collaboration), and the intrapersonal domain (e.g., academic mind-sets). The interpersonal and intrapersonal domains capture many of the competencies commonly referred to as “21st century” or “noncognitive” skills, and the three domains together align well with the dimensions of deeper learning identified by the Hewlett Foundation (see Table 1).

Table 1

Competency Domains and Dimensions for Deeper Learning

Domains from NRC framework	Competency dimensions from Hewlett Foundation
Cognitive domain	Deep content knowledge
Cognitive domain	Critical thinking and complex problem solving
Interpersonal domain	Collaboration
Interpersonal domain	Communication
Intrapersonal domain	Learning-to-learn
Intrapersonal domain	Academic mind-sets

Note. NRC = National Research Council.

In the United States, over 500 schools are associated with formal school networks that promote deeper learning competencies (Alliance for Excellent Education, 2011), and the number of schools that independently work to develop students’ interpersonal and intrapersonal skills is constantly increasing. Research on the effectiveness of such efforts, however, is still in its infancy. Previous research suggests that students who attend schools that promote deeper learning see improved academic achievement, as measured by GPA (Collins, Davis-Molin, & Conley, 2013) or test scores (Nichols-Barrer & Haimson, 2013). Other studies connect deeper learning exposure with improved college readiness and retention (Friedlaender, Burns, Lewis-Charp, Cook-Harvey, & Darling-Hammond, 2014) and with the development of interpersonal and intrapersonal skills (Collins et al., 2013; Guha et al., 2014). Although the positive associations between a wide range of student outcomes and deeper learning strategies are encouraging, these studies are largely correlational or descriptive in nature and limited to samples consisting of a single-school network or even a single school. In 2012, the NRC summarized the state of deeper learning research and highlighted the need for more rigorous research designs that go beyond the existing, primarily correlational, body of research to measure the impact of deeper learning.

This article presents findings from the Study of Deeper Learning: Opportunities and Outcomes—funded by the Hewlett Foundation—which was designed to produce the most rigorous evidence to date on the effects of implementing a school-wide approach to promoting deeper learning for high school students. The study aimed to determine whether students who attended high schools with a mature and at least moderately well implemented approach to promoting deeper learning within a small school context experienced greater deeper learning opportunities and outcomes than they would have if they had not attended these schools. In this article, we address the following research questions to test key components of a deeper learning theory of action (discussed in the next section):

Research Question 1: To what extent did students who attended schools with a deeper learning focus experience more opportunities to engage in the deeper learning process than they would have had they attended a more traditional school?

Research Question 2: To what extent did students who attended schools with a deeper learning focus demonstrate better deeper learning competencies than they would have had they attended a more traditional school?

Research Question 3: To what extent were students who attended schools with a deeper learning focus more likely to graduate from high school and pursue a postsecondary education than they would have had they attended a more traditional school?

In the following sections, we describe the Deeper Learning Community of Practice from which the study schools were selected and the theory of action that undergirds school-wide efforts to promote deeper learning to student outcomes. We then describe the study’s methods and findings. We conclude by discussing implications of the study findings, including limitations and directions for future research.

A Deeper Learning Community of Practice

In 2011–2012, the Hewlett Foundation identified 10 school networks, representing schools in 41 states, to participate in what would become the Deeper Learning Community of Practice (see http://deeperlearning4all.org/enabling-deeper-learning-in-schools for more information about the school networks). The purpose of the Community of Practice was to share strategies, tools, and lessons that both contribute to the work of the networks themselves and build the broader knowledge base about deeper learning. The main criterion for selecting networks to participate in this Community of Practice was simple: They needed to have experience in, and an explicit focus on, promoting a deep understanding of core academic content and the kinds of competencies reflected in the dimensions of deeper learning identified by the Hewlett Foundation, and they needed to do this across whole schools serving diverse populations of students (rather than targeting only certain students or teachers in a school). All networks involved in the Deeper Learning Community of Practice offered a range of assistance to their affiliated schools, although the types of assistance offered varied from network to network. The most common supports mentioned by network and school respondents in this study included the provision of curriculum and instructional materials, leadership and management support, and the provision of a variety of professional learning opportunities for teachers.

The Deeper Learning Community of Practice included networks that varied in terms of both structure and emphasis. For example, some networks operate K–12 schools while others include only high schools. Some networks work at the district level, while others focus on schools serving particular underserved student populations (e.g., recent immigrants). Some emphasize the use of technology as part of the learning process; for others, technology is less central. Networks also differ in the types of schools they include. The majority of the schools in these networks are part of traditional public school districts, but two networks consist of only charter schools. Despite these differences, all 10 networks have a well-established history of promoting deeper learning, and they all share an emphasis on providing educational opportunities for minority students and students from low-income families to prepare them for college and career. In addition, key to all 10 networks is the personalization of learning through small school and classroom sizes, which have been shown to independently benefit student outcomes (Bloom & Unterman, 2014).

High schools affiliated with these networks use a range of strategies to develop students’ deeper learning competencies. These schools typically support the implementation of instructional approaches aligned with deeper learning through the development of specific structural and cultural elements. In our research, we found that the strategies that schools commonly used to foster deeper learning included project-based learning, internship opportunities, collaborative group work, longer term cumulative assessments, and advisories (or regular meetings between an advisor and a student or a group of students to provide academic and social support; Huberman, Bitter, Anthony, & Day, 2014). In spite of different emphases on these strategies across schools, they are all intended to promote development in the cognitive, interpersonal, and intrapersonal competency domains. Many of the high schools in these networks enroll fewer than 400 students, thus capitalizing on the potential benefits of promoting deeper learning within a small school context.

Figure 1 depicts the study’s simplified theory of action for deeper learning, which links the high school strategies for fostering deeper learning to student college and career readiness. The theory of action includes the following key components:

Students attend a school with an explicit school-wide approach that focuses on promoting deeper learning; that is, a school where site leaders and decision makers have established strategies, structures, and cultures explicitly designed to foster deeper learning. The resulting approach includes essential elements of the network’s model and is distinct from more traditional education approaches.

Students experience greater opportunities for deeper learning than they would have otherwise. Examples of these experiences include (but are not limited to) opportunities for collaborative group experiences, opportunities to receive feedback from their teachers and peers, and opportunities for longer term, authentic assessment.

Students exposed to opportunities for deeper learning obtain transferable deeper learning competencies. For example, through increased opportunities for collaborative group experiences, students may develop collaboration and communication skills as well as increased engagement in their learning.

As a result of obtaining deeper learning competencies, students are more likely to be college and career ready. The deeper learning competencies students acquire in high school are expected to increase students’ likelihood of graduating from high school and students’ ability and confidence in pursuing a postsecondary education.

Figure 1.

Deeper learning theory of action.

Method

In contrast to an evaluation of a particular program or approach to deeper learning, this study focused on providing evidence to indicate whether high schools can promote deeper learning across a variety of approaches and a diversity of students. Ideally, we would estimate the effects of attending a network school with a random assignment design based on network school admission lotteries, and then follow students who won the admission lotteries and students who did not win longitudinally through high school and beyond. To measure students’ enrollment in postsecondary education, this study design would require us to follow students for a minimum of 4 years. Unfortunately, this ideal research design was not feasible because few network schools had admission lotteries and the number of students participating in those admission lotteries is too small for a sufficiently powered study.

Given such practical constraints, we used a quasi-experimental design with a matched comparison group for this study. First, to reduce the period of time necessary to observe students’ high school and postsecondary outcomes, we selected a sample of students who entered the ninth grade prior to the beginning of the study. Second, the design tries to approximate the ideal longitudinal random assignment design by selecting one matched comparison school for each network school to serve as the counterfactual school that students who attended a network school would have attended had they not attended the network school. While in reality a student might have multiple schooling options besides the network school, we selected a single comparison school for each network school to ease the burden of student tracking and data collection. Because this design feature may introduce biases relative to the ideal design, we used weighting and statistical adjustments based on a detailed set of pre–high school student characteristics to reduce these biases to the extent possible. In particular, we used inverse probability of treatment weighting (IPTW) to minimize selection bias due to differences between students who chose to attend a network school and students attending the comparison school.

In the remainder of this section, we describe the sample selection, data collection, measures, and analytic approach used for this study. More details about the study design and methods are provided in the online supplemental appendix (available in the online version of the journal).

Selection of Network Schools and Comparison Schools

For the study, we identified 20 high schools from the 10 networks in the Hewlett Foundation’s Deeper Learning Community of Practice that implemented the network approach school-wide and network representatives considered to be moderate or high implementers of the approach (based on criteria established by each network). To be included in the study, the selected “network schools” had to have existed for at least 4 years, had nonselective admissions, and served a student population in which at least 25% of students were eligible for free or reduced-price lunch (FRPL). We were unable to identify suitable comparison schools for four of the network schools selected for the study (due to either a lack of student-level administrative records or unique features of the network school), and so analyses conducted for this article include a total of 16 network schools for which we were able to identify a matched comparison school, representing nine of the deeper learning networks. These network schools were located in five California districts and New York City.

To examine the effects of attending each network school in the study sample, we selected a matched comparison school for each network school. All of the network schools included in this study are in large urban areas where students typically have more schooling options than their local neighborhood public high school. For the study, we used a single matched comparison school to serve as the counterfactual condition: the high school experience most students would likely have been exposed to had they not attended a network school. Each matched comparison school was located in the same geographic area as its paired network school (i.e., in the same school district or a neighboring district), and the school pair had similar incoming student populations (based on student demographics and achievement). Reflecting the prevalence of small schools among the network schools, the matched comparison schools did enroll more students than the network schools (1,350 students vs. 398 students, on average). This difference was driven by large comparison schools in California. (See the supplemental appendix in the online version of the journal for additional details about the school sample and matching process.) While the study includes a total of 15 matched pairs of network and comparison schools,¹ not all schools participated in each form of data collection. (For more information about the number of school pairs included in each analysis, see the “Data Collection” section.)

Student Sample

Based on district-provided student-level data, we selected the student sample from four cohorts of Grade 9 students who entered the selected network and comparison high schools in the 2007–2008 through 2010–2011 academic years. We then followed these students over time to administer student surveys and assessments (Cohorts 3 and 4 only), observe high school graduation, and collect data on enrollment in postsecondary institutions. The expected progression of each cohort and the timing of data collection are presented in Table 2.

Table 2

Grade Progression of Student Cohorts Included in the Study

School year	Cohort 1	Cohort 2	Cohort 3	Cohort 4
2007–2008	Grade 9
2008–2009	Grade 10	Grade 9
2009–2010	Grade 11	Grade 10	Grade 9
2010–2011	Grade 12 (G)	Grade 11	Grade 10	Grade 9
2011–2012	Post Year 1	Grade 12 (G)	Grade 11	Grade 10
2012–2013	Post Year 2	Post Year 1	Grade 12 (G)(S)	Grade 11 (S)
2013–2014	Post Year 3	Post Year 2	Post Year 1	Grade 12 (G)
2014–2015	Post Year 4	Post Year 3	Post Year 2	Post Year 1
2015–2016	Post Year 5 (P)	Post Year 4 (P)	Post Year 3 (P)	Post Year 2 (P)

Note. (G) = on-time high school graduation measured; (S) = student survey and cognitive assessment administered; (P) = postsecondary enrollment measured.

To improve the comparability between students attending network and comparison schools within any given matched school pair, we restricted the student sample in two ways. First, we restricted the sample to students who had data on Grade 8 characteristics, including middle school state standardized test scores, in the available district extant data. Second, we excluded students in comparison schools with background characteristics too dissimilar to the network school students to serve as reasonable comparisons. This exclusion was based on the propensity score distribution of network school students. Estimation of the propensity score is described in the “Analytic Approach” section, along with a description of the student sample characteristics.

Data Collection

Analyses presented in this article utilize four data sources: extant district administrative data on student background characteristics and high school graduation; postsecondary enrollment data from the StudentTracker service at the National Student Clearinghouse (NSC); a student survey developed by the study team to measure deeper learning opportunities and competencies; and the Organisation for Economic Co-Operation and Development’s (OECD) PISA-Based Test for Schools (PBTS) administered to a sample of students as part of the study. An overview of data collection from each source is presented in Table 3.

Table 3

Data Sources and Sample Sizes

Data source	Description	Target sample	Number of participating schools	Number of students with data
Student Survey	Measures students’ self-reported opportunities to engage in deeper learning, as well as interpersonal and intrapersonal competencies (such as self-efficacy)	Students in Cohorts 3 and 4 with parental consent, who were subsampled for data collection, and who were in school during 2013 data collection	22 schools,11 school pairs	1,762
OECD’s PISA-Based Test for Schools	Measures students’ higher order skills in reading, mathematics, and science	Students in Cohorts 3 and 4 with parental consent, who were subsampled for data collection, and who were in school during 2013 data collection	20 schools,10 school pairs	1,267
Graduation Data From District Data Systems	Measures students’ graduation from a high school within the district within 4 years of entering Grade 9	Students in Cohorts 1–4	27 schools,14 school pairs	20,435
Postsecondary Enrollment Data From the National Student Clearinghouse	Measures students’ postsecondary enrollment and the types of postsecondary institutions attended	Students in Cohorts 1–4	25 schools,13 school pairs	20,057

Note. Not all schools participated in each type of data collection. In addition, to maximize the similarity between matched network and comparison schools, five of the selected comparison schools were each matched with two network schools in the same district. Please see Supplemental Table S1 in the online version of the journal for a description of the demographic composition of study schools as well as a description of which schools participated in each type of data collection. OECD = Organization for Economic Co-Operation and Development; PISA = Program for International Student Assessment.

We received extant district administrative data and the NSC data for all Grade 9 students who entered network and comparison schools during the four cohort years (2007–2008 through 2010–2011). While we were able to collect student background characteristics and high school graduation records from all participating districts, we were unable to obtain graduation records for one network school (because the school was no longer affiliated with the district when we requested graduation data), and two of the districts were unable to provide us with postsecondary data from NSC. Therefore, analyses of graduation outcomes include 14 school pairs, and analyses of postsecondary outcomes include 11 school pairs.

A 1-hour student survey was designed to measure students’ opportunities for deeper learning within school and students’ deeper learning competencies (particularly, interpersonal and intrapersonal competencies). The survey included previously validated item sets from established surveys, including surveys from the Consortium of Chicago School Research (CCSR) and the National Survey of Student Engagement developed by Indiana University (Kuh, 2001). We supplemented these existing items with original items designed to address specific constructs important to this study. The survey was piloted prior to administration to test the validity of the scales.

We used the PBTS to measure student cognitive competency in three core content areas: reading/English language arts (ELA), mathematics, and science. We selected the PBTS because it includes a large number of test items focused on knowledge and application of core academic subjects at the high school level.

The student survey and PBTS were administered to a subsample of students in Cohorts 3 and 4 who were still enrolled in the sampled schools at the time of data collection, and who consented to participate in the data collection. Four of the network schools did not participate in the student survey or PBTS due to low consent rates, and one additional network school chose not to participate in the PBTS. Therefore, analyses of student survey data include 11 school pairs and analyses of PBTS data include 10 school pairs.

In the network schools, all consented students in Cohorts 3 and 4 were eligible to take the survey and PBTS. In the comparison schools, it was not feasible to administer the survey and PBTS to all eligible students given the size of the schools, so we identified a stratified random sample of consented students in Cohorts 3 and 4 for data collection, where the propensity score quintiles (defined by the distribution of students in network schools) were used as sampling strata (see the supplemental appendix in the online version of the journal for more details). Overall, 1,762 (76%) of the 2,329 sampled students (80% of sampled students in network schools and 73% of sampled students in comparison schools) completed the student survey. In addition, 1,267 (61%) of the 2,066 sampled students in schools participating in the PBTS (74% of sampled students in network schools and 54% of sampled students in comparison schools) completed the assessment.

Measures

Analyses presented in this article are based on measures that fall into five categories: (a) student background characteristics, (b) deeper learning opportunities, (c) deeper learning competencies, (d) high school graduation, and (e) postsecondary enrollment. These measures are briefly described in this section, with more details about the opportunity and competency measures based on the student survey provided in the online supplemental appendix (available in the online version of the journal).

Student Background Characteristics

We obtained student characteristics from participating school districts’ student-level administrative records. We used the extant data from students’ Grade 8 and Grade 9 years to identify students to be included in the study (i.e., first-time Grade 9 students) and to incorporate covariates in our analyses. For all study schools, we had data on students’ gender, race/ethnicity, English language learner status, and individual education plan (special education) status. Because study schools were located in multiple school districts, some data were not available for all study schools. However, as school pairs were constructed within districts (or neighboring districts), we had data on the same set of student background characteristics for the two schools in any given pair. In all but one school pair, we either had, depending on the district, parental education or eligibility for free or reduced-price lunch (FRPL status) as a measure of socioeconomic status. For all but two school pairs, we had standardized test score data for Grade 8 ELA and mathematics achievement. For the New York schools, we also had data on students’ attendance rate in Grade 8 and the students’ age at the start of Grade 9 (used to determine whether a student was “over-age,” an indicator for repeating a grade prior to high school).

Measures of Deeper Learning Opportunities

On the student survey, students were asked to respond to a set of Likert-type items asking about activities they engaged in as part of their core content classes (including English, mathematics, science, and social studies) during the school year. These items covered nine measures of deeper learning opportunities (see Table 4). Some items asked students to report the number of classes in which they engaged in a type of activity (0 = none of my classes; 1 = one of my classes; 2 = two of my classes; and 3 = three or more of my classes), and other items asked students to report how often they engaged in particular activities (0 = never; 1 = some of the time; 2 = most of the time; and 3 = all of the time). We used Rasch modeling to create scale scores from the survey items for each measure, and the scale scores were standardized to have a mean of zero and standard deviation of one in the full analytic sample of surveyed students.

Table 4

Measures of Deeper Learning Opportunities

Measure	Rasch reliability	M	SD	Minimum	Maximum
Opportunities for complex problem solving (Four items)	.90	0.15	1.15	−5.70	5.82
Description: The degree to which students engage in complex problem solving by analyzing ideas, judging the value and reliability of an idea or source, constructing new ideas, and applying knowledge to solve new problems.
Opportunities for creative thinking (Five items)	.79	0.91	2.31	−4.82	5.11
Description: The extent to which students have the opportunity to engage in creative thinking in their core academic classes, such as thinking of original solutions to problems and new ways to do things, creating new ideas, and using their imagination.
Opportunities to communicate (12 items)	.83	0.54	1.48	−4.74	5.00
Description: The extent to which students have the opportunity to practice written and oral communication skills.
Opportunities to collaborate (Four items)	.69	0.95	1.40	−4.49	4.73
Description: The degree to which students collaborate on assignments, provide feedback on each other’s work, and collaborate in other ways.
Opportunities to learn how to learn (Four items)	.52	1.81	1.79	−4.07	4.41
Description: The degree to which students practice monitoring and directing their own work and learning.
Opportunities to receive feedback(Six items)	.75	0.96	1.79	−4.45	4.78
Description: The degree to which students receive written and oral feedback on their work from teachers, peers, or others.
Assessments aligned with deeper learning(Nine items)	.77	0.74	1.53	−4.45	4.70
Description: The extent to which students engage in various forms of assessment including assessments of problem solving, communication, and collaboration.
Opportunities for interdisciplinary learning(Four items)	.78	−0.40	2.35	−5.30	5.33
Description: The degree to which students engage in interdisciplinary learning, where two or more disciplines are combined to enhance inquiry and knowledge generation.
Opportunities for real-world connections(11 items)	.84	0.58	1.77	−4.95	5.28
Description: The degree to which students engage in instructional activities that emphasize real-world connections.

Measures of Deeper Learning Competencies

On the student survey, students were asked to respond to a set of Likert-type items asking about the extent to which they agreed with different statements that cover eight measures of interpersonal and intrapersonal competencies (see Table 5). Response options ranged from 0 (strongly disagree or never or almost never true) to 3 (strongly agree or always or almost always true). To create scales from the survey items for each measure, we used the same Rasch modeling approach that was used to measure the deeper learning opportunities. Measures of students’ cognitive competencies were based on their PBTS scores, which measure critical thinking and problem-solving skills as well as mastery of content knowledge in ELA, mathematics, and science.

Table 5

Measures of Interpersonal and Intrapersonal Competencies

Measure	Rasch reliability	M	SD	Minimum	Maximum
Creative thinking skills(Five items)	.77	1.77	2.30	−6.12	5.89
Description: The extent to which a student perceives that he or she can think of original ideas and solutions.
Collaboration skills(10 items)	.83	2.19	2.08	−4.02	6.10
Description: The extent to which a student perceives that he or she works well in a group (e.g., positive personal interactions; the ability to pay attention, share ideas, be prepared, and do his or her part) and cooperates to identify or create solutions.
Academic engagement(10 items)	.74	0.79	1.17	−5.70	6.00
Description: The degree to which a student agrees that he or she has “interest and engagement in learning” and participates actively in classroom learning activities.
Motivation to learn(Five items)	.75	1.57	2.09	−5.71	5.61
Description: The degree to which a student is motivated to do well academically and to become more knowledgeable, measured by a student’s “perceived importance of coursework as well as preference for challenge and mastery goals.”
Self-efficacy(Seven items)	.84	2.49	2.62	−6.84	6.92
Description: The degree to which a student tends to view himself or herself “as capable of meeting task demands in a broad array of contexts.”
Locus of control(Five items)	.73	2.16	2.20	−5.62	5.70
Description: The extent to which a student feels he or she has control over what happens to them, rather than their circumstances being controlled by chance or fate.
Perseverance(Five items)	.79	2.59	2.61	−6.46	6.34
Description: The degree to which a student agrees that he or she maintains effort and interest despite failure, adversity, and plateaus in progress.
Self-management(11 items)	.81	0.75	1.53	−5.28	5.44
Description: The extent to which a student feels he or she is able to independently manage his or her work and schedules to meet goals.

Measure of High School Graduation

For the high school graduation outcome, we focused on “on-time” high school graduation, which was defined as graduating from high school within 4 years of entering ninth grade, including the summer after a student’s fourth year of high school. A student’s graduation status was based on student records provided by school districts in which the participating schools were located. As a result, our definition of high school graduation is limited to students who graduated from a high school within the same school district they attended in the ninth grade. Any students who did not have a graduation record (including students who dropped out, students who took longer than 4 years to graduate, and students who transferred outside the district or to a private school) were classified as “not on-time graduates.” We counted students who transferred outside the participating districts as “not on-time graduates” because some of the district data systems did not reliably distinguish students who transferred from those who dropped out. Due to our focus on first-time ninth-grade student cohorts and our inclusion of transfer students as “not on-time graduates,” the graduation rates presented in this article do not reflect official graduation rates for the schools included in the study.

Measures of Postsecondary Enrollment

We used student records from the StudentTracker service from the NSC to define postsecondary enrollment as enrolling in any postsecondary institution by fall 2015, when the youngest student cohort in the study were in their second year after expected high school graduation and the oldest cohort were in their fifth year after expected high school graduation. To further explore postsecondary enrollment, we separated the “any” enrollment measure into a measure of whether students enrolled in a 2-year institution and a measure of whether students enrolled in a 4-year institution by fall 2015. We requested NSC data for all Cohort 1 to 4 students within our study schools, including those who were not observed to graduate from high school within the district. As such, the postsecondary analyses included students who may have transferred to another district prior to graduating from high school.

Analytic Approach

Students were not randomly assigned to network and comparison schools included in this study; therefore, claims about a network school’s effects on student opportunities and outcomes could be biased by preexisting differences between network school students and comparison school students. We employed two strategies to take observed preexisting differences into account in estimating the effects of network schools: weighting and regression-based covariate adjustment. We outline our analytic approach here and provide more details in the online supplemental appendix (available in the online version of the journal).

Weighting for Bias Reduction

We employed propensity score weighting to match the sample of students attending the comparison school in each pair as closely as possible to the sample of students attending the network school in the pair. We also used weights to account for student persistence between Grade 9 entry and the year of data collection, select a subsample of students in large comparison schools for the study survey and PBTS, and adjust for survey nonresponse in the impact analyses.

To weight the sample for student selection into network schools, we estimated a student’s probability of attending a network school instead of the matched comparison school and used the estimated propensity score to create an IPTW applied to all analyses of opportunities and outcomes (Hirano et al., 2003). To estimate the propensity scores, we ran a separate logistic regression model for each school pair and student cohort, based on the student characteristics that were available for a given school pair. The estimated propensity scores were then used to calculate IPTW for the comparison school students, and network school students’ weights were set to 1. With this weighting method, the comparison group was weighted to represent the network group when estimating the effect of attending a network school, and therefore our results can be interpreted as average treatment effects on the treated (ATT). The IPTW was included in all analyses to account for group differences in measured pre–high school characteristics (including both demographic characteristics and Grade 8 achievement test scores).

The subsample of students for whom we have survey and PBTS data may differ in important ways from the full sample of students entering Grade 9. Such differences could influence generalizability of the findings and could bias effect estimates. To reduce potential bias from sources of attrition (lack of persistence/consent and nonresponse), as well as data collection sampling, we calculated persistence/consent,² response, and sampling weights. For analyses of student survey and PBTS measures, we combined these three weights with the IPTW. This combined weight captures measured baseline differences between network and comparison school students, as well as differences between student survey respondents and the target Grade 9 student cohorts. The analyses of high school graduation and postsecondary enrollment only utilize the IPTW.

To assess the quality of the final analytic weight, we examined the degree to which network and comparison school students in the analysis samples had similar background characteristics after applying the final weight (to deal with potential selection bias due to the measured preexisting differences). Tables 6 and 7 show average student characteristics for the network and comparison school students before and after weighting for the samples included in analyses of student survey data and PBTS scores, respectively. The difference in the total number of students between network and comparison schools reflects the fact that the network schools are smaller than the comparison schools, particularly in California. For each characteristic, we report the standardized mean difference (SMD), averaged across pairs, as a summary of baseline covariate balance. For a given pair and characteristic, the SMD is defined as the mean difference between network and comparison school students, standardized by the pooled standard deviation for all Grade 9 students in the full sample. Although network school students were slightly lower achieving and consisted of fewer Hispanic students (more Black students) than comparison schools, on average, the SMD is below 0.25 standard deviations across all the measured student background characteristics, which is a common threshold for baseline balance (What Works Clearinghouse, 2017).³ Given consent and attrition, however, balance for some student characteristics was worse for the analytic sample than the baseline sample, even after applying the weights. This was particularly the case for race/ethnicity indicators, individualized education plan (IEP) status, and student cohort. To account for imbalance that remains after weighting, and to improve the precision of the estimated effects, we used covariate adjustment in addition to the weights in the outcome models.

Table 6

Network and Comparison Student Characteristics for the Survey Sample (Cohorts 3 and 4), Before and After Weighting

Student characteristic	Unweighted sample of all students in Cohorts 3 and 4			Unweighted survey sample			Weighted survey sample
Student characteristic	Network mean	Comparison mean	SMD	Network mean	Comparison mean	SMD	Network mean	Comparison mean	SMD
Number of students	1,617	7,957	—	687	1,075	—	687	1,075	—
Propensity score	0.31	0.22	0.84	0.31	0.25	0.51	0.31	0.31	0.06
Grade 8 math test scores	−0.02	0.05	−0.08	0.06	0.15	−0.10	0.03	0.10	−0.06
Grade 8 ELA test scores	−0.09	0.00	−0.10	0.00	0.16	−0.19	−0.03	−0.01	−0.02
ELL status	31.5%	27.6%	0.10	30.5%	26.8%	0.09	30.7%	30.6%	0.01
IEP status	6.2%	6.2%	0.02	5.7%	3.7%	0.10	6.0%	3.6%	0.11
Gender (female)	52.7%	50.9%	0.04	54.8%	57.6%	−0.06	54.4%	53.4%	0.02
Race/ethnicity (Black)	15.2%	13.8%	0.08	15.1%	9.7%	0.18	15.1%	10.8%	0.14
Race/ethnicity (Hispanic)	58.9%	54.6%	0.05	56.7%	61.7%	−0.17	57.6%	64.2%	−0.20
Race/ethnicity (White)	20.2%	20.7%	0.03	22.4%	18.7%	0.16	21.8%	17.7%	0.17
Cohort (Grade 12)	50.3%	50.0%	0.01	41.1%	53.0%	−0.24	43.8%	52.6%	−0.18

Note. Survey Sample = 11 school pairs; 9,574 students entering Grade 9; 1,762 survey respondents. Test scores are reported in standardized z-score units, where the standardization was based on state- and year-specific means and standard deviations for each test. Unweighted means and SMDs were calculated separately for each school pair, using the population of incoming Grade 9 students in network and comparison schools. Weighted means and SMDs were calculated separately for each pair, using the weighted sample of survey respondents. The results shown in the table are based on an equally weighted average across the pair-specific means and SMDs. SMD = standardized mean difference; ELA = English language arts; ELL = English language learner; IEP = individualized education plan.

Table 7

Network and Comparison Student Characteristics for the PBTS Sample (Cohorts 3 and 4), Before and After Weighting

Student characteristic	Unweighted sample of all students in Cohorts 3 and 4			Unweighted PBTS sample			Weighted PBTS sample
Student characteristic	Network mean	Comparison mean	SMD	Network mean	Comparison mean	SMD	Network mean	Comparison mean	SMD
Number of students	1,413	6,968	—	570	697	—	570	697	—
Propensity score	0.31	0.23	0.81	0.31	0.25	0.46	0.31	0.30	0.04
Grade 8 math test scores	−0.02	0.03	−0.05	0.09	0.13	−0.05	0.08	0.10	−0.02
Grade 8 ELA test scores	−0.08	−0.01	−0.07	−0.01	0.18	−0.22	0.00	0.04	−0.05
ELL status	31.3%	27.2%	0.11	28.2%	27.2%	0.05	28.9%	31.6%	−0.06
IEP status	5.7%	6.0%	0.02	5.4%	3.4%	0.10	5.8%	2.4%	0.15
Gender (female)	51.2%	51.0%	0.00	53.2%	57.2%	−0.08	53.0%	57.1%	−0.08
Race/ethnicity (Black)	13.7%	12.9%	0.07	14.1%	9.3%	0.18	13.5%	9.6%	0.15
Race/ethnicity (Hispanic)	60.0%	57.2%	0.01	57.2%	62.7%	−0.19	58.5%	65.4%	−0.22
Race/ethnicity (White)	20.9%	21.3%	0.04	22.7%	19.6%	0.16	22.0%	18.9%	0.17
Cohort (Grade 12)	50.6%	50.0%	0.01	41.2%	48.7%	−0.15	43.4%	55.1%	−0.23

Note. PBTS sample = 10 school pairs; 8,381 students entering Grade 9; 1,267 PBTS respondents. Test scores are reported in standardized z-score units, where the standardization was based on state- and year-specific means and standard deviations for each test. Unweighted means and SMDs were calculated separately for each school pair, using the population of incoming Grade 9 students in network and comparison schools. Weighted means and SMDs were calculated separately for each pair, using the weighted sample of survey respondents. The results shown in the table are based on an equally weighted average across the pair-specific means and SMDs. The PBTS sample excludes one school pair in which the network school did not participate in PBTS data collection. PBTS = PISA-Based Test for Schools; SMD = standardized mean difference; ELA = English language arts; ELL = English language learner; IEP = individualized education plan.

Within-Pair Effect Estimation

To estimate the effects of enrolling in a network school instead of a comparison school, we first conducted pair-by-pair analyses.⁴ The analysis method is considered doubly robust (Funk et al., 2011) because it accounts for observed differences in network and comparison students in two ways: (a) through propensity score weighting and (b) through regression-based covariate adjustment. To apply both the propensity score weight and the regression-based covariate adjustment, we used the following weighted ordinary least squares (OLS) regression model:

Y_{i j} = β_{0 j} + β_{1 j} T_{i j} + β_{3 j} X_{i j} + e_{i j},

where $Y_{i j}$ is a given opportunity or competency measure for student i in school pair j; $T_{i j}$ is a dichotomous indicator for whether the student enrolled in the network school ( $T_{i j} = 1$ ) or the comparison school ( $T_{i j} = 0$ ) in the fall of Grade 9; and $X_{i j}$ is a vector of available student background characteristics, as well as dichotomous indicators for student cohorts. The main parameter of interest is $β_{1 j}$ , which is the effect of enrolling in a network school instead of the matched comparison school in a given school pair. For the high school graduation and postsecondary enrollment outcomes, we used logistic regression models rather than OLS models.

Analyses of the cognitive competencies measured with the PBTS data used a variation on this model that took into account the measurement error associated with the PBTS scores. Accounting for measurement error was particularly important for analyses of PBTS data because different students received different forms of the PBTS and each student responded to only a subset of items within each subject area. As part of the PBTS scoring conducted by the test vendor, each student received an ability score and the standard error for their score. For analyses of PBTS scores, we used a two-level, variance-known, hierarchical linear model (Raudenbush & Bryk, 2002). The first level of the model accounted for the error associated with each PBTS score (based on the standard error for each student’s score), while the second level mirrored the equation above for estimating within-pair effects.

Averaging Pair-Specific Effect Estimates

The main results presented in the article are estimates of the effects of attending a network school, averaged across the pairs for which we have data. We view the results as pertaining only to the particular schools included in our sample and not to a wider population. Thus, we used a fixed-effects meta-analysis approach (Hedges & Vevea, 1998) to calculate the average effect across the school pairs:

\bar{E S} = \frac{\sum_{j = 1}^{J} w_{j} {\hat{β}}_{j}}{\sum_{j = 1}^{J} w_{j}},

where $\hat{β_{j}}$ is the estimated network effect for pair j, $w_{j}$ is the inverse of the variance of pair j’s estimate (i.e., one divided by the standard error squared), and J is the number of pairs included in the analysis. This calculation is the precision-weighted mean effect of the pair-specific effect estimates, where estimates with more precision (less error variance) receive more weight in the average.

Subgroup Analysis

In addition to estimating effects for the full sample of students attending network schools, we examined whether the effects of network school enrollment differed across student subgroups. In particular, we tested whether effects differed by gender, cohort, FRPL status, and prior ELA achievement (dichotomized into below vs. above average⁵). To test whether effects differed significantly across subgroups, we estimated a model similar to the model described above, adding the interaction of network enrollment and the dichotomous subgroup indicator.

Findings

In this section, we present the results for each research question. First, we describe overall average differences in students’ opportunities for deeper learning between students who attended network schools and students who attended comparison schools. Next, we examine average differences in deeper learning competencies as measured by the student survey and the PBTS. Finally, we test whether rates of high school graduation and probabilities of enrolling in postsecondary education significantly differ between students who attended network schools and students who attended comparison schools. We focus on the average effects across school pairs. In the online supplemental appendix (available in the online version of the journal), we provide the pair-specific effect estimates.

Effects on Deeper Learning Opportunities

Our first research question addresses a fundamental assumption that underpins the deeper learning initiative: that students in schools with at least moderately well implemented approaches to promoting deeper learning actually experience greater opportunities to engage in deeper learning across their classes. This question is critical in that it asks whether students’ experiences in the network schools were significantly different from what comparable students experienced in other schools.

Our analyses of student-reported deeper learning opportunities suggest that, on average, students in participating network schools did experience significantly more opportunities for deeper learning than their matched counterparts in the paired comparison schools. These significant differences held true for all nine opportunity measures (see Figure 2). We report estimated differences between students in network schools and comparison schools in terms of effect sizes. As shown in Figure 2, the average effect sizes ranged from 0.21 (for opportunities for assessments aligned with deeper learning) to 0.55 (for opportunities to collaborate).

Figure 2.

Average effects of attending a network school on student opportunities for deeper learning.

Effects on Deeper Learning Competencies

If students attending network schools were exposed to more opportunities for deeper learning than they would have if they had attended a more traditional school, the next logical question is whether students attending network schools demonstrated better deeper learning competencies compared with students in comparison schools.

Our analyses of PBTS scores and student-reported interpersonal and intrapersonal competencies suggest that, on average, students in participating network schools did exhibit greater deeper learning competencies than their matched counterparts in the paired comparison schools (see Figure 3). In particular, students in network schools scored 0.10 to 0.12 standard deviations higher on the three PBTS cognitive competency assessments (reading, mathematics, and science) than students in the comparison schools.⁶ Students in network schools also reported significantly higher levels of interpersonal and intrapersonal competencies for four of the eight measures than students in comparison schools: collaboration skills, academic engagement, motivation to learn, and self-efficacy. Effect sizes for these four competencies ranged from 0.12 (motivation to learn) to 0.20 (academic engagement). There were no statistically significant differences between students in network and comparison schools with respect to creative thinking skills, locus of control, perseverance, or self-management.

Figure 3.

Average effects of attending a network school on student deeper learning competencies.

Effects on High School Graduation

The deeper learning initiative’s emphasis on more ambitious academics, coupled with its focus on developing a set of skills that many believe are critical for success in college and career, may mean that instruction focused on deeper learning can motivate students to engage in their studies, be successful in school, and graduate at higher rates. We found that students who attended a network school were more likely to graduate from high school on time than were students in comparison schools. As shown in Figure 4, approximately 65% of students in network schools graduated within 4 years from a high school in the same district, compared with 58% of similar students who attended comparison high schools.⁷

Figure 4.

High school graduation and postsecondary enrollment rates for network school students and comparison school students.

Effects on Postsecondary Enrollment

The deeper learning competencies that students acquire in high school are expected to improve students’ ability and confidence in pursuing a postsecondary education. Among the students included in our analysis, those who attended a network school were significantly more likely to enroll in a postsecondary institution by the end of 2014 than those who attended a comparison school (see Figure 4). Overall, 53% of the students who attended network schools enrolled in a postsecondary institution compared with 50% of the students who attended comparison schools. This difference is primarily due to the significantly higher 4-year college enrollment rate among network school students than comparison school students (22% vs. 18%). There was no significant difference in the percentage of network and comparison school students who attended a 2-year postsecondary institution by the end of 2014.

Differential Effects by Subgroups

The consistent positive effects across a range of student outcomes suggest that mature network schools were successful, on average, in providing greater opportunities for students to engage in deeper learning and helping students achieve better educational outcomes. These results, however, raise the question of whether attending a network school benefited all students, or whether the effects varied by subgroups of students. We did not find consistent subgroup effects across the opportunities and outcome measures to suggest that the benefits of attending a network school were concentrated among specific types of students. We did, however, find some subgroup differences that warrant future investigation. In particular, results show that the impact of attending a deeper learning network school on students’ opportunities for deeper learning was significantly stronger for female students than for male students, but we did not find significant subgroup differences by gender on student outcome measures. Results from all the subgroup analyses are provided in the online supplemental appendix (available in the online version of the journal).

Discussion

Summary of Findings

As schools adapt practices to conform to the Every Student Succeeds Act and evolving state content standards and graduation requirements, it is important to consider how focusing on the development of deeper learning skills can have far-reaching effects on student outcomes. The findings described in this article support two key assumptions underlying efforts to implement deeper learning initiatives in high schools: that high schools focused on deeper learning—when the approach is at least moderately well implemented—can provide greater opportunities for students to engage in deeper learning activities, and produce better outcomes for students from diverse backgrounds. Our analyses of student opportunities for deeper learning demonstrated that students attending the network schools experienced more opportunities for deeper learning across their core content classes than comparable students attending comparison schools. Our analyses of student outcomes demonstrated that students in the network schools exhibited greater cognitive competencies, and were ultimately more likely to graduate from high school on time and attend a 4-year college than their counterparts in comparison schools. In addition, students in the network schools exhibited greater interpersonal and intrapersonal competencies on some measures, though not all of them. Taken together, these findings highlight the potential benefits of school-wide efforts to promote deeper learning.

Furthermore, the findings were relatively consistent across multiple measures of opportunities and outcomes, across a wide range of models and approaches to deeper learning, and held for students with different background characteristics and prior achievement levels. This is nontrivial. Prior research has shown that schools often struggle to implement programs and instructional initiatives across classrooms and for all students. This study demonstrates that it is possible to implement a deeper-learning-focused instructional approach (in many different ways) that offers opportunities for a wide range of students.

Study Limitations

Although our study design used strict school and student selection criteria so that we can draw meaningful inferences from the results, it is important to acknowledge some potential limitations to internal and external validity. Where internal validity is concerned, network students self-selected into the network schools and may have differed in some unmeasured ways from students with otherwise similar characteristics and prior performance who did not choose to attend a network school. To the extent that these unmeasured characteristics are associated with the outcomes, independent of the measured characteristics in the propensity score models, the estimated effects of network school attendance will be biased. Furthermore, nonconsent and nonresponse for the student survey and PBTS may have weakened group equivalence and increased selection bias. To attribute the opportunity and outcome differences between students in network and comparison schools to network school enrollment, we must assume that the covariates included in the propensity score weighting and attrition weights adequately account for the pre–high school differences between students in network schools and comparison schools.

We also acknowledge that, in addition to the implementation of instruction focused on deeper learning, most of the network schools in our study were smaller in size than their matched comparison school. Although the smaller school size was considered by school leaders as an essential component of the deeper learning model, facilitating the personalization of instruction and supports, our analysis cannot separate the effects of attending a deeper learning network school from the effects of attending a small school.⁸

Where external validity is concerned, we only included schools that implemented the network approaches at a moderate or high level. In addition, the school sample did not include some of the network schools that had implemented the models to the highest standard (due to the application of selection criteria relating to school size, grade range, or ongoing participation in other studies, for example). These school inclusion criteria mean that findings from this study cannot be generalized to all schools trying to implement approaches to deeper learning. Furthermore, the study focused only on schools in two specific state contexts (California and New York), which may limit the extent to which the findings generalize to schools under different educational contexts (e.g., states with different high school graduation requirements and college accessibility).

Directions for Future Research

As initiatives and instruction focused on deeper learning become more widespread in the American education system, it is important that research continues to document how, and for whom, experiences with deeper learning affect students’ educational experiences and outcomes. To encourage continued research in this emerging area of reform, we conclude with suggestions for future research that can expand upon the results presented in this article. First, we need a better understanding of how the results from this study are applicable to different high school contexts. Because we chose schools that were implementing network approaches to fostering deeper learning at least moderately well, we do not know what results would have been obtained had we sampled network schools that implemented network approaches with lower levels of fidelity. In addition, we do not know the effectiveness of deeper learning practices and approaches that are employed in schools that are not associated with a deeper learning network. Practices commonly associated with deeper learning network schools (e.g., advisories, opportunities for collaborative group work, opportunities to relate academic content to real-world contexts) can be found in a variety of school settings, and so future research should examine the effectiveness of these approaches outside of a network school setting.

Second, future research should identify strategies that schools and districts may use to provide deeper learning opportunities to students at scale. While we know that the network schools in our sample provided opportunities for many students to engage in activities intended to promote deeper learning, more research is needed to determine how network schools reach and sustain their level of implementation, and what types of support may be necessary for traditional schools that are not associated with deeper learning networks to implement these practices. For example, principal interviews in both network schools and nonnetwork schools suggest that traditional schools are less likely to incorporate project-based learning, collaborative group work, and internship opportunities into instruction than network schools. In addition, learning goals directly related to competencies such as academic mind-sets and learning how to learn are not as common in traditional schools as in network schools (Huberman et al., 2014). All the network schools in our study received assistance from a school network, and it is likely that similar kinds of support would be needed, whether from network organizations, local districts, or states, for additional schools to engage in deeper learning activities.

Because adopting a school-wide model of deeper learning may not always be feasible, researchers should identify specific approaches or strategies that are particularly effective at providing opportunities to engage in deeper learning. As the variety of network schools in our study indicate, there can be multiple approaches to promoting deeper learning within schools. While most of the network schools used strategies such as project-based learning, internships, collaborative group work, longer term cumulative assessments, and advisories, they did it with different degrees of intensity and emphasis. For example, project-based learning was integral to daily instruction in slightly over a third of the schools in our sample and used more sporadically in others. Also, advisory classes had different numbers of students (from 15 to 30 students), ran for different amounts of time (between 30 and 60 minutes), and happened with different frequencies (from every day to once or twice a week), depending on the school (Huberman et al., 2014). Thus, further research on the implementation and effectiveness of deeper learning approaches is clearly needed to pinpoint and scale up the practices that best prepare students for success.

Third, as the concept of deeper learning is still evolving, our survey measures of opportunities and competencies may not fully capture students’ deeper learning experiences. For example, the survey measures did not examine the quality of the opportunities nor the frequency with which they were experienced within classes. While our results suggest that students in network schools had different learning experiences than students in schools with more traditional approaches, they also suggest that more investigation could be done to better understand and define these experiences.

Fourth, while we found that attending a network school had positive effects on the cognitive, intrapersonal, and interpersonal competencies we measured, we do not know which of these competencies (or which combination of competencies) are crucial for college and career success. To draw stronger conclusions about the lasting effects of attending a school focused on deeper learning, students will need to be followed over a longer time frame.

Fifth, our results provide some indication that the network schools promoted equity in certain student outcomes, but the evidence is not consistent across outcomes. For example, we found that students with lower prior achievement benefited more from attending network schools than students with higher prior achievement in terms of postsecondary enrollment, but the effects of attending network schools on the cognitive competency measures were similar for students with different levels of prior achievement. On the contrary, the results indicate that higher achieving students benefited more than lower achieving students on some measures of intrapersonal skills. More work is required to better understand the relationships between deeper learning opportunities, outcomes, and equity.

Sixth, our study focused on exposure to deeper learning within high schools. As efforts to extend deeper learning to elementary and middle school students expand, it will be important to study the extent to which approaches to deeper learning affect student outcomes, and whether there are cumulative effects of being exposed to deeper learning opportunities throughout the K–12 experience.

Although the results presented in this article are limited by methodological constraints and the study’s scope, to date, they provide one of the most comprehensive looks at how attending a school focused on deeper learning affects students’ high school experiences, academic mind-sets, and academic outcomes. Given the promising findings, we hope this article sets the stage for future work to explore how, where, and for whom a focus on deeper learning in the classroom may have long-lasting effects on students’ lives.

Supplemental Material

DS_10.3102_0162373719837949 – Supplemental material for Promoting Deeper Learning in High School: Evidence of Opportunities and Outcomes

Supplemental material, DS_10.3102_0162373719837949 for Promoting Deeper Learning in High School: Evidence of Opportunities and Outcomes by Jordan Rickles, Kristina L. Zeiser, Rui Yang, Jennifer O’Day and Michael S. Garet in Educational Evaluation and Policy Analysis

Footnotes

Acknowledgements

We acknowledge the many people who helped make this study possible. We thank the thousands of students, teachers, principals, and network and district staff who agreed to provide responses to the study’s many data collections. We thank Kristi Kimball for her initiation of the project and Marc Chun and Barbara Chow for their consistent support of this study. We are also grateful to James Kemple and the staff at the Research Alliance for New York City Schools for their expert analysis in this collaboration, and members of the American Institutes for Research project team who helped with data collection and analysis, particularly Mette Huberman and Mengli Song for their feedback on earlier drafts of this paper. In addition, we would like to thank the reviewers and the editor for their thoughtful comments and suggestions for improving the article. The statements, findings, and conclusions here are those of the authors and do not necessarily represent the viewpoints of these organizations or individuals.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We extend our appreciation to the William and Flora Hewlett Foundation for the grant that made this study possible.

Notes

Authors

Jordan Rickles is a principal researcher at the American Institutes for Research. He specializes in research design and quantitative methodology for estimating the effects of educational practices, policies, and programs.

Kristina L. Zeiser is a senior researcher at the American Institutes for Research. Her research focuses on the application of quantitative methods to examine the transition from high school to postsecondary education, with a particular focus on the relationship between students’ high school experiences and their postsecondary outcomes.

Rui Yang is a senior researcher at the American Institutes for Research. His major research areas include impact evaluation, measurement of noncognitive skills, and cross-cultural comparisons.

Jennifer O’Day is an institute fellow at American Institutes for Research. Her research focuses on system learning and improvement, with particular attention to strategies for promoting equitable access and outcomes. She is also the chair of the California Collaborative on District Reform, which for 13 years has joined district leaders, researchers, state policy makers and other stakeholders in ongoing dialogue and collective action to improve teaching and learning for California’s diverse student population.

Michael S. Garet is an institute fellow and vice president at American Institutes for Research. His areas of specialization include methodological issues in evaluating the impacts of educational interventions, the analysis of large-scale survey and assessment data, and the analysis of organizational structure and process. His recent work has focused on teacher quality and professional development in reading and mathematics.

References

Alliance for Excellent Education. (2011). A time for deeper learning: Preparing students for a changing world (Policy brief). Retrieved from http://all4ed.org/reports-factsheets/a-time-for-deeper-learning-preparing-students-for-a-changing-world/

Autor

Levy

Murnane

(2003). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics, 118, 1279–1333.

Bloom

H. S.

Unterman

(2014). Can small high schools of choice improve educational prospects for disadvantaged students? Journal of Policy Analysis and Management, 33(2), 290–319.

Carnevale

Desrochers

(2003). Standards for what? The economic roots of K–16 reform. Princeton, NJ: Educational Testing Service.

Chow

(2010, October 5). The quest for deeper learning. Education Week. Retrieved from http://www.edweek.org/ew/articles/2010/10/06/06chow_ep.h30.html

Collins

Davis-Molin

Conley

(2013). Journey toward deeper learning: An evaluation of the Roadtrip Nation Experience in the San Jose PLUS academies. Eugene, OR: Educational Policy Improvement Center.

Dweck

(2006). Mindset. New York, NY: Random House.

Finegold

Notabartolo

A. S.

(2010). 21st-century competencies and their impact: An interdisciplinary literature review. Retrieved from https://hewlett.org/wp-content/uploads/2016/11/21st_Century_Competencies_Impact.pdf

Friedlaender

Burns

Lewis-Charp

Cook-Harvey

C. M.

Darling-Hammond

(2014). Student-centered schools: Closing the opportunity gap. Palo Alto, CA: Stanford Center for Opportunity Policy in Education.

10.

Funk

Westreich

Wiesen

Sturmer

Brookhard

Davidian

(2011). Doubly robust estimation of causal effects. American Journal of Epidemiology, 173, 761–767. Retrieved from http://aje.oxfordjournals.org/content/early/2011/03/08/aje.kwq439.full.pdf+html

11.

Guha

Adelman

Arshan

Bland

Caspary

Padilla

Biscocho

(2014). Taking stock of the California Linked Learning District Initiative: Fourth-year evaluation report. Menlo Park, CA: SRI International.

12.

Hedges

L. V.

Vevea

J. L.

(1998). Fixed- and random-effects models in meta-analysis. Psycho-logical Methods, 3(4), 486.

13.

Hirano

Imbens

G. W.

Ridder

(2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4), 1161–1189.

14.

Huberman

Bitter

’Anthony

J. O.

Day

(2014). The shape of deeper learning: Strategies, structures, and cultures in deeper learning network high schools (Report #1: Findings from the Study of Deeper Learning: Opportunities and Outcomes). Washington, DC: American Institutes for Research.

15.

Kuh

G. D.

(2001). Assessing what really matters to student learning inside the National Survey of Student Engagement. Change: The Magazine of Higher Learning, 33(3), 10–17.

16.

National Research Council. (2008). Research on future skill demands: A workshop summary. Washington, DC: The National Academies Press.

17.

National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: The National Academies Press.

18.

Nichols-Barrer

Haimson

(2013). Impacts of five Expeditionary Learning middle schools on academic achievement. Cambridge, MA: Mathematica Policy Research.

19.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Sage.

20.

Trilling

(2010). Defining competence in deeper learning (Draft report to the William and Flora Hewlett Foundation). Menlo Park, CA: Hewlett Foundation.

21.

William and Flora Hewlett Foundation. (2013). Deeper learning defined. Retrieved from http://www.hewlett.org/library/hewlett-foundation-publication/deeper-learning-defined

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.69 MB