Effects of a Summer Mathematics Intervention for Low-Income Children

Abstract

Prior research suggests that summer learning loss among low-income children contributes to income-based gaps in achievement and educational attainment. We present results from a randomized experiment of a summer mathematics program conducted in a large, high-poverty urban public school district. Children in the third to ninth grade (N = 263) were randomly assigned to an offer of an online summer mathematics program, the same program plus a free laptop computer, or the control group. Being randomly assigned to the program plus laptop condition caused children to experience significantly higher reported levels of summer home mathematics engagement relative to their peers in the control group. Treatment and control children performed similarly on distal measures of academic achievement. We discuss implications for future research.

Keywords

summer learning loss online learning mathematics

Low-income children score more than a standard deviation below high-income children on mathematics achievement tests (Duncan & Magnuson, 2011; Reardon, 2011). Because children’s mathematical knowledge is cumulative (Hiebert & Wearne, 1996), weak foundations of early mathematical knowledge built up in the elementary and middle school grades may block low-income children’s opportunities for future success in advanced mathematics coursework, a critical gatekeeper to STEM career pathways (National Council of Teachers of Mathematics, 2000).

Summer learning loss among low-income children is a widely documented problem. Research examining seasonal patterns in children’s learning often indicates that achievement gaps in reading and mathematics between high- and low-income children grow primarily while children are on summer vacation. In an early landmark study, Heyns (1978) found that among middle school children in Atlanta, socioeconomic status (SES)–based achievement gaps grew more quickly while children were on summer vacation than during the school year. In a longitudinal study examining the achievement of a panel of Baltimore children from the first through the ninth grade, Alexander, Entwisle, and Olson (2007) found that low-SES children’s summer learning losses in reading and mathematics accumulated over the elementary and middle school years. They concluded that low-SES children’s cumulative summer deficits contributed substantially to SES-based differences in high school course “tracking,” high school completion, and 4-year college attendance (Alexander et al., 2007). More recently, the finding that the SES achievement gap widens during summer vacation has been replicated in nationally representative data from the Early Childhood Longitudinal Study, Kindergarten Class of 1998–1999 (ECLS-K; Downey, von Hippel, & Broh, 2004). Further, also using ECLS-K data, Burkam, Ready, Lee, and LoGerfo (2004) found that approximately 25% of the SES-based summer math gap could be explained by differential participation in summer activities, most notably whether the child used a computer for educational activities during the summer.

Policy Responses to Low-Income Children’s Summer Learning Loss

Summer learning loss among low-income children poses both equity and achievement challenges for educators and policymakers. For example, many urban school districts have developed programs aimed at ameliorating summer learning losses among low-income children, with most efforts focused on literacy and mathematics (examples include Chicago, New York, and Boston; see, for example, Jacob & Lefgren, 2004; Mariano & Martorell, 2013; Matsudaira, 2008). In an update of Cooper, Charlton, Valentine, Muhlenbruck, and Borman’s (2000) review of the summer school literature, a recent meta-analysis of the K–8 summer mathematics program literature from the past decade (Quinn, Lynch, & Kim, 2014) indicates that low-income students who attend school-based summer mathematics programs outscore their counterparts who do not attend such programs by an average of .11 SD on subsequent mathematics assessments. Whereas this meta-analysis includes school-based summer mathematics programs, including summer school and camps, it includes no studies of home-based summer math programs, as none were found in the literature. Yet as noted below, there are both theoretical and practical reasons to hypothesize that the effects of school-based and home-based summer interventions could differ (Quinn et al., 2014). Most notably, school-based programs generally offer the support of classroom teachers who lead and scaffold instruction; in addition, many school-based programs provide additional resources such as enrichment activities (McCombs et al., 2011). By contrast, home-based summer interventions, such as home-based summer book reading programs, generally do not involve support from classroom teachers, and instead rely on parents and family members or the students themselves to motivate and support summer learning activities (e.g., Allington et al., 2010).

Noting that summer school programs are expensive, requiring school teachers, staff, and facilities (McCombs et al., 2011), education researchers and policymakers have recently called for the investigation of home-based and online summer academic interventions as low-cost alternatives to summer school (Allington et al., 2010; Walters & Sorensen, 2013). Computer-based summer interventions hold out potential promise, both for their impacts on achievement and for their cost-effectiveness. Online and computer-based programs have the potential to provide students with customized instruction and rapid feedback (Balacheff & Kaput, 1996). Recent meta-analyses of both elementary (Slavin & Lake, 2008) and secondary (Slavin, Lake, & Groff, 2009) mathematics interventions provided during the regular school year find that computer-assisted instruction programs are effective on average at improving student math achievement, although effect sizes were relatively modest (+0.19 SD for elementary students; +0.08 SD for secondary students). In addition, computer-based summer interventions may have the potential to reduce costs for hiring summer school teachers and keeping schools open.

Home-based summer interventions may also provide cost-effective policy alternatives. In the domain of literacy, researchers have found that, on average, home-based literacy interventions appear to be equally as effective and more cost-effective than summer school at improving reading outcomes (Kim & Quinn, 2013). In addition, a growing body of evidence suggests that children’s engagement with mathematics at home may be beneficial for children’s mathematics attitudes and achievement. The importance of the home environment for children’s mathematical development is evident in the fact that the income-based mathematics achievement gap is already present at school entry (Lee & Burkam, 2002; Starkey, Klein, & Wakeley, 2004). Researchers have found that home math inputs, including the amount of math-related talk parents engage in with their children, predict preschool children’s later math achievement, even after controlling for family SES (Gunderson & Levine, 2011; Levine, Ratliff, Huttenlocher, & Cannon, 2012; Levine, Suriyakham, Rowe, Huttenlocher, & Gunderson, 2010). In an experimental study conducted with a sample of mostly middle- to upper-middle class families during the academic year, parent–child home reading of topical math passages on a tablet increased children’s math achievement across the school year (Berkowitz et al., 2015). In a review of the causal evidence on parent involvement in homework, Patall, Cooper, and Robinson (2008) found that parent homework involvement improved children’s homework completion rates and decreased the rate of homework problems such as negative affect about homework or receiving a homework-related school punishment—both of which could potentially benefit children’s achievement in the longer term. Hoover-Dempsey et al. (2001) suggest that through activities such as modeling, reinforcement, and instruction, parent involvement in children’s homework may influence proximal outcomes such as positive attitudes toward learning, positive perceptions of self-competence, and a belief in the importance of effort for success, which in turn may shape children’s achievement outcomes.

However, evidence to support the larger idea that offering students free online math instruction to do at home, without teachers, is likely to be effective is lacking. For example, some proponents have suggested that programs such as Khan Academy, which has over 10 million users each month and support from funders such as Google and the Gates Foundation, could transform schooling by at least partially replacing classroom instruction with web-based instructional video clips and online practice (e.g., Wagner, 2011). However, to date there has been no rigorous evaluation of Khan Academy except as a supplement to classroom teachers’ instruction (which we discus below; Snipes, Huang, Jaquet, & Finkelstein, 2015).

Despite policy interest and the success of home-based summer interventions for low-income children in reading, we can identify no prior studies of home-based summer mathematics interventions. In the current study, we conducted a randomized experiment of a home-based summer mathematics program aimed at improving children’s summer home mathematics engagement and reducing summer learning loss. The current research is needed to address the gap in our understanding of how low-touch, home-based mathematics interventions may affect low-income children’s summer math participation and learning outcomes.

Summer Mathematics Intervention

Tenmarks is an online mathematics program in which participating students complete “worksheets” of math questions adjusted to their skill level. The Tenmarks program is of policy interest for several reasons. First, over 1 million students in all 50 U.S. states and over 126 countries have used Tenmarks, either through their schools or through personal subscriptions (Tenmarks, 2012). The program is also reportedly used by teachers in over 85% of U.S. school districts (Tenmarks, 2015). However, despite widespread use and time expenditures, to date there has been no evaluation of the program’s effectiveness. Second, the Tenmarks program is low cost (and at time of writing is available free on Amazon.com) and is relatively typical of free math practice programs available on the Internet. Thus, the Tenmarks program represents a low-cost summer math intervention that is in mass use, but whose effects have not been examined.

Research Questions

In this study we examine whether being randomly assigned to an offer of a free summer-long subscription to an online mathematics program (Tenmarks) or to an offer of the Tenmarks program plus a free laptop computer caused students to experience higher levels of summer home and family mathematics engagement, and/or higher subsequent mathematics achievement and attitudes, compared with their peers in the control group. Given the voluntary nature of this summer intervention, we further explore what factors predicted program participation, and whether the benefits of offering the summer mathematics intervention varied depending on students’ background characteristics, family resources, and/or mathematics attitudes or intrinsic motivation for doing mathematics.

Method

Research Design

To estimate the causal impact of the intervention on students’ subsequent family and home math engagement and math achievement and attitudes, we randomly assigned students to either (a) the math program only condition; (b) the math program plus a free laptop computer condition; or (c) the control group. We randomly assigned students at the individual level to improve the power of the study design and to assess the efficacy of the program when delivered to individual students (Burkam et al., 2004).

Site

The study was conducted in a large, urban school district in the northeast region of the United States. More than 80% of the students in the district are non-White, and more than three quarters of the students are eligible for free or reduced price school lunches.

Sample

We purposefully sampled schools that were located in the highest poverty neighborhoods where students were most at-risk of summer learning loss. Four schools serving high populations of minority youth and high proportions of students eligible for free or reduced price school lunch participated in the study. Participating schools included one elementary school (School 1), one middle school (School 2), one middle/high school (School 3), and one high school (School 4). To be included in the study sample, principals and teachers had to sign a memorandum of understanding agreeing to implement the study procedures for obtaining active parental consent, to administer assessments, and to random assignment of students to conditions. A total of 263 students consented to participate.

Table 1 provides descriptive statistics on the study sample in each school, as well as a comparison of demographic characteristics of the study sample with students in the overall school, district, and state. Compared with the district overall, sample students were similar in proportion minority (87% in the sample; 87% in the district overall) and somewhat less likely to be free/reduced lunch eligible (59% in the sample; 70% in the district overall). Sample control group students in Grades 4 and 8 performed similarly to district students in Grades 4 and 8 on a set of math items from the National Assessment of Educational Progress (NAEP; described below; 51% in the sample vs. 48% in the district overall). In a broader context, sample control students answered fewer of the NAEP items correctly than children in the state overall (state overall = 58%) and a similar number to the national average (U.S. overall = 51%).

Table 1

Descriptive Statistics of Selected Baseline Characteristics of Participants, by School

School	School description		n (% of total sample)	Grades	% Internet access at baseline	% subsidized meals	% minority	% female	% NAEP math correct
School 1	Public, Grades K–8	Sample	65 (.33)	3–8	.63	.74	.98	.56	.50^a
School 1	Public, Grades K–8	School overall				.91	.99	.50	—
School 2	Public, Grades 6–8	Sample	99 (.38)	6–8	.36	.56	.96	.59	.39^a
School 2	Public, Grades 6–8	School overall				.70	.97	.50	—
School 3	Public exam school, Grades 7–12	Sample	87 (.33)	7–8	.97	.41	.74	.72	.59^a
School 3	Public exam school, Grades 7–12	School overall				.53	.72	.57	—
School 4	Public, Grades 9–12	Sample	12 (.05)	9	.33	.6	.83	.45	.25^a
School 4	Public, Grades 9–12	School overall				.76	.99	.49	—
Sample overall						.59	.87	.62	.51^b
District overall						.70	.87	.48	.48
State overall						.35	.33	.49	.58
U.S. overall									.51

Note. NAEP = National Assessment of Educational Progress.

Scores for control group.

Scores for fourth and eighth graders in the control group only, to allow for comparisons with same-grade test-takers.

No statistically significant differences between experimental and control groups were found on any of the measured baseline covariates. Table 2 provides descriptive statistics for covariates in the overall sample and by experimental group.

Table 2

Descriptive Statistics for Covariates and Assessments, for Full Sample and by Experimental Group

Variable	Scale (range in current sample)	Full sample	Control group	Tenmarks only group	Tenmarks plus laptop group
Spring math benchmark score	% correct (.03−1.00)	0.62	.64	.63	.60
Internet	% yes	0.63	.62	.62	.64
Gender	% female	0.62	.69	.61	.57
Grade	(3–9)	6.40	6.41	6.37	6.42
Race (%)	White	.11	.13	.07	.15
	Black	.38	.34	.37	.42
	Asian	.10	.09	.10	.09
	Hispanic	.31	.31	.37	.25
	Other	.02	.01	.01	.02
	Not reported	.09	.13	.08	.07
Free lunch	% Eligible	.59	.60	.60	.56
Parent supervision	M = 0, SD = 1 (−3.55, 1.24)	0	.02	.08	−.09
Academic effort	M = 0, SD = 1 (−3.98, 1.42)	0	−.03	.14	−.11
Mathematics intrinsic motivation	M = 0, SD = 1 (−2.39, 1.28)	0	−.05	−.03	.08
Mathematics confidence	M = 0, SD = 1 (−3.71, 1.33)	0	−.05	.10	−.04
Spring math benchmark score present		.82	.77	.80	.89
Fall math benchmark score present		.84	.81	.83	.87
Fall national assessment-based math score present		.74	.66	.75	.81

Program Description

Key components of the Tenmarks program were curriculum materials that adjusted content to children’s individual skill levels as they worked, embedded text and video “hints” that students could click on for assistance, and digital games that children could unlock as rewards for completing worksheets. The program developers intended for students to complete three worksheets each week for 10 weeks. See Figure 1 for a hypothesized logic model for the Tenmarks program. It was hypothesized that participating in an online summer mathematics practice program would increase low-income students’ summer home math engagement and that Tenmarks worksheet completion would improve students’ knowledge of mathematics and distal outcomes of mathematics test scores. In addition, given the voluntary nature of the intervention, which occurred over summer vacation, we hypothesized that children’s background characteristics, home resources, and affective characteristics may moderate the program’s effectiveness, such that children with greater access to home resources and higher levels of academic effort, mathematics confidence, and intrinsic motivation for doing mathematics would participate at higher levels and thus obtain greater benefits.

Figure 1.

Hypothesized Tenmarks intervention logic model.

The control condition was “business as usual.” Control students were free to participate in whatever other summer activities were available to them. Approximately 43% of control students reported participating in other summer programs, and 16% reported attending summer school. These summer activities are discussed further below.

Measures

We use data from five sources for most analyses presented in this report: (a) parent registration forms; (b) student presurveys; (c) district administrative records; (d) student postsurveys, which included a national assessment-based mathematics test; and (e) program usage data for the Tenmarks website. First, all parents completed a registration form before their children began the program; this included demographic information such as the child’s grade and whether the family had home Internet access at baseline. Second, in the last week of school prior to summer vacation, and prior to random assignment, teachers administered the presurvey. The presurvey included items measuring students’ academic effort, intrinsic motivation for doing mathematics, math confidence, and parent supervision; the response rate was 97%. Third, district administrative records were used to collect students’ demographic data, as well as scores on district curriculum-based mathematics assessments administered before and after summer vacation. (More detail on these assessments is provided below.) Spring and fall math scores were present in the district data for 82% and 84% of students, respectively. Fourth, after summer vacation and the Tenmarks program had concluded, approximately 6 weeks into the school year, teachers administered the student postsurvey. The postsurvey included a 30-item national assessment-based math test composed of NAEP items (described below), and measures including students’ family home math engagement, intrinsic motivation for doing math, and summer activity participation. The response rate for the postsurvey was 74%. Last, we collected program usage data provided by the Tenmarks website developers indicating how many times each child logged into the program and how many online math worksheets each child completed.

An attrition analysis indicated that students who had and did not have district assessment scores generally did not differ demographically, by experimental group, or on a range of pretreatment measures; the exception was that students who were eligible for free lunch were more likely to have usable scores (p < .05). Students who submitted the postsurvey also generally did not differ from nonrespondents on a range of pretreatment measures. However, postsurvey nonrespondents were more likely to be in the control condition than in the program plus laptop condition (p < .05). In addition, they were less likely to be from School 3 than from Schools 1 or 2. We conducted analyses using both imputed scores estimated using multiple imputation and ordinary least squares regression with listwise deletion. For the multiple imputation analysis, we utilized Stata’s mi impute chained routine, which employs an iterative imputation technique which imputes multiple variables using chained equations, a series of univariate imputation methods with fully conditional specification of prediction equations (StataCorp, 2013b), using 10 multiply imputed data sets. We present results using the complete case analysis; the results from the imputed data are in the appendix.

Outcome Variables

We describe each of the outcome variables that we use in our analyses below.

Intrinsic motivation for doing mathematics

Both before and after the intervention, we measured children’s intrinsic motivation for mathematics using the Intrinsic Motivation Inventory (IMI) Interest/Enjoyment subscale, which has been used in prior educational evaluations and validated for use as the child self-report measure of intrinsic motivation for an activity (Plant & Ryan, 1985; Ryan, 1982). The scale is comprised of seven items and operationalized as the degree to which students enjoyed doing math, thought that math was fun to do, thought that math was boring (reverse-coded), felt that math did not hold their attention (reverse-coded), would describe math as very interesting, felt that math was quite enjoyable, and agreed that while they were doing math, they were thinking about how much they enjoyed it (Cronbach’s α = .92 at both pre- and postsurvey; Plant & Ryan, 1985; Ryan, 1982). The IMI score is a composite variable estimated with principal components analysis composed of the child’s responses to the scale items. Higher values of this variable indicate greater intrinsic motivation for doing math.

Summer home and family mathematics engagement

The postsummer survey included four items that asked children about their summer mathematics home activities and family involvement, adapted from the Literacy Habits Survey (Paris et al., 2004), which has been used in prior evaluations of summer programs to measure summer home literacy involvement (Kim, 2007; Kim & White, 2008; Paris et al., 2004). The items were adapted to reflect a focus on mathematics (Cronbach’s α = .72). Children selected one of four responses: less than once a month; once or twice a month; once or twice a week; and almost every day. The following items comprised this scale:

During summer vacation, how often did you talk about math with someone in your home?

During summer vacation, how often did your parents (or someone in your family) help you do math at home?

During summer vacation, how often did your parents encourage or tell you to do math?

During summer vacation, how often did you do math at home?

Students’ scores on this index are comprised of a composite variable estimated with principal components analysis, composed from the four survey items. In addition, the survey included a single, binary item which asked students whether their mother or father did any math with them this summer; students selected yes or no.

Mathematics enjoyment

The postsummer survey included three items related to students’ level of mathematics enjoyment, drawn from the NAEP Mathematics Student Questionnaire (U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 2011; Cronbach’s α = .88). Students selected from four responses: strongly agree, agree, disagree, and strongly disagree. The items measured the extent to which students agreed that math was fun and they did not want to give it up; they liked math, and math was one of their favorite subjects. Students’ scores on this index are comprised of a composite variable estimated with principal components analysis, composed from the four survey items.

Achievement measures

Two achievement assessments were administered. Both math tests are designed to assess student mastery of either national or local district curriculum standards. The first was a district curriculum-based math assessment administered to all students in the school district in September and June. This is a computer-adaptive test, vertically aligned using item response theory to allow for comparisons across time points. We collected students’ demographic information and scores from the June and September administrations of this exam, to capture differences in students’ scores that emerged over the summer vacation. Students completed the June test within a few weeks of the end of the school year (before random assignment) and completed the September test in the second week of the subsequent fall semester.

Because the district’s curriculum-based math assessments are secure instruments, we were unable to inspect the items for alignment with the Tenmarks intervention. As a result, we administered a second, national assessment-based mathematics test to participants as a fall posttest. This assessment was comprised of 30 randomly selected publicly released items from the NAEP, representing a range of content areas and difficulty levels.¹ Students in Grades 4, 5, and 6 received a test form containing items from the fourth-grade NAEP, and students in Grades 7, 8, and 9 received a test form containing items from the eighth-grade NAEP; in this way, all students saw assessment items within two grade levels of their own. Using publicly released NAEP items allows us to measure students’ achievement against NAEP benchmarks of “what students should know and be able to do in a given grade” in the United States (Gorman, 2010).

Covariates

In addition to the baseline levels of the variables described above, we collected information on the following covariates.

Parent supervision

The presurvey included three parent supervision items adapted from the National Education Longitudinal Study of 1998 (NELS:88), which asked students to rate (on a scale including “never,” “rarely,” “sometimes,” and “often”) how often their parents or guardians checked on whether they had done their homework, were home when they returned from school, and limited the amount of time they could spend watching TV (Institute of Education Sciences, 2011). Students’ scores on this index are comprised of a composite variable estimated with principal components analysis, composed from the survey items. Although Cronbach’s α for this scale was low (.41), the NELS items have been used in prior survey research and research on parent involvement (Sui-Chu & Willms, 1996).

Mathematics confidence

The presummer survey included four items related to students’ level of mathematics confidence, adapted from the High School Longitudinal Study of 2009 (HSLS:09; Institute of Education Sciences, 2011; Cronbach’s α = .85). Students selected from four responses: strongly agree, agree, disagree, and strongly disagree, indicating the degree to which they felt confident that they could do an excellent job on tests in their math class; felt certain that they could understand the hardest material in their math book; felt certain that they could master the skills being taught in their math class; and felt confident that they could do an excellent job on assignments in their math class. Students’ scores on this index are comprised of a composite variable estimated with principal components analysis, composed from the four survey items.

Academic effort

Students’ scores are comprised of a composite variable estimated with principal components analysis of the extent of students’ academic effort, composed from seven presurvey items such as “I work very hard at school” (which students rated on a scale including “never,” “some of the time,” “half of the time,” “most of the time, and “all of the time”) The items are adapted from Fryer (2011; Cronbach’s α = .54).

Demographic information

This included students’ race, gender, free lunch eligibility, and home Internet access at baseline.

Procedures

Within 2 weeks of the end of the school year (but before random assignment was conducted), students completed the spring district curriculum-based math assessment as part of their regular instructional regimen. In the last week of school in June, just prior to summer vacation but also prior to random assignment, teachers administered the presummer survey to students in their classrooms.

After random assignment, Tenmarks staff members visited each school to give each student who was randomly assigned an offer of the Tenmarks online math program login credentials for the Tenmarks website, entitling each student to a free Tenmarks subscription. The staff members also provided a brief demonstration of the website’s features. At the same time, students who were randomly assigned a free laptop computer received instructions on how to pick up the computer. To receive a computer, students were required to participate in an afterschool or evening computer training, accompanied by a parent or guardian. Although all children in the program plus laptop condition were assigned to receive a laptop, our analyses are intent-to-treat estimates as we did not (nor could we) ensure compliance.

Each week throughout the summer, a Tenmarks staff member randomly selected one student from each school who had completed Tenmarks worksheets to receive a small gift card. Students also received weekly text messages encouraging them to log into Tenmarks. As approximately one third of students reportedly lacked home Internet access at baseline, text messages also provided locations of public libraries, community centers, and open school buildings with Internet-equipped computers for student use. No control group students logged into the program, according to information provided by Tenmarks.

The following fall, during the second week of school, students completed the fall district curriculum-based math assessment. Approximately 6 weeks into the fall term, teachers administered the national assessment-based math test and postsurvey.

Results

The results section is organized as follows. First, we provide a descriptive picture of summer mathematics learning loss, intrinsic motivation loss, and activity participation over the summer vacation period for the sample. Next, we present intent-to-treat estimates of the intervention’s impacts on summer mathematics engagement and student outcomes. Last, we describe the effects of intervention dosage on program outcomes and explore what factors predicted program participation.

Summer Achievement and Summer Motivation Loss

First, we examined descriptively whether sample children experienced losses in mathematics achievement or intrinsic motivation for doing mathematics over the summer.

Following Cooper, Nye, Charlton, Lindsay, and Greathouse (1996), we estimated the effects of summer vacation on achievement by calculating for each sample in the control group the standardized mean difference (Cohen’s d). This metric allows us to express the difference in students’ achievement scores in the fall relative to their scores in the spring, irrespective of the specific test metric. Children in the control group on average experienced summer losses on the district assessment (d = −0.48 SD). This estimate is somewhat larger than the average summer loss effect size for math computation reported in Cooper et al. (1996) (d = −.32), and larger and oppositely signed than Cooper et al.’s finding that summer vacation had a positive effect on math application (d = +0.17). As the district assessment scores were not broken out by problem type or mathematics subdomain, we cannot tell whether summer losses were greater for computation than application problems.

Using the same procedures as above, we estimated the relationship between summer vacation and children’s intrinsic motivation for doing mathematics, calculating the standardized mean difference between control group students’ scores at pre- and postsummer on the IMI scale. On average, children experienced decreases in their intrinsic motivation for doing mathematics over the testing period. (d = −.32 SD).

Summer Activity Participation

We begin with a brief summary of children’s reported summer activities. Over half (53%) of children reported that they did not attend a summer camp or summer program during summer vacation. Free/reduced price lunch–eligible children were somewhat less likely to report attending a summer program (44%) compared with higher income children (51%). Six percent of the children who did report attending a summer program said that it was Tenmarks. Approximately 16% of children reported that they attended summer school, and 10% reported that they attended a summer school that had a math component. Summer camp and summer school attendance did not differ significantly by experimental group.

Estimated Impacts on Summer Mathematics Engagement and Student Outcomes

In Table 3, we present a taxonomy of fitted regression models in which we estimate the intent-to-treat impact of offering students a chance to participate in the online summer mathematics program or offering the same opportunity plus a free laptop computer, on the proximal and distal outcomes and the two mediators shown in the intervention logic model (Figure 1). To estimate the causal impact of the experimental condition, for each outcome, we fit a regression model with the outcome as the dependent variable and the two treatment conditions as predictors. Models control for relevant variables measured at baseline: the family/home math engagement model (M.1.1) includes a control for baseline level of parent supervision; the math assessment models include controls for prior spring math score and, for the national assessment-based math test, student grade level (M.2.1 and M.3.1); the intrinsic motivation model (M.4.1) includes a control for baseline level of intrinsic motivation; and the math enjoyment model (M.5.1) includes controls for baseline levels of intrinsic motivation and academic effort. We also fit all models with school fixed effects, to account for the nesting of students within schools at baseline; the results are similar to those below. The results from the main effects models are similar in the ordinary least squares and multiple imputation analyses; in the interaction models, the statistical significance of some estimates varies across models, which we note below. See the appendix for the imputation results.

Table 3

Results of Fitting a Taxonomy of Regression Models for Family/Home Math Engagement, NAEP Math Test, District Mathematics Benchmark Assessment, Mathematics Intrinsic Motivation, and Mathematics Enjoyment as a Function of Treatment Assignment and Student Background Characteristics

	Model M.1.1 Family home math engagement	Model M.2.1 National assessment-based math assessment	Model M.3.1 District curriculum-based math assessment	Model M.4.1 Mathematics intrinsic motivation	Model M.5.1 Mathematics enjoyment
Intercept	−.22 (.14)	.13 (.11)	.01 (.09)	−.01 (.11)	.08 (.11)
Tenmarks only	.19 (.19)	.03 (.15)	−.07 (.13)	.03 (.15)	.01 (.15)
Tenmarks + Laptop	.39* (.18)	−.09 (.14)	.04 (.12)	.06 (.15)	−.08 (.15)
Baseline parent supervision	.20** (.07)
Spring math score		.70*** (.00)	.73*** (.05)
Elementary age		−.65*** (.15)
Baseline mathematics intrinsic motivation				.67*** (.06)	.67*** (.07)
Baseline academic effort					−.18* (.07)
Adj. R²	.05	.40	.50	.45	.40
n	170	176	196	152	158

Note. NAEP = National Assessment of Educational Progress. Standard errors in parentheses.

p < .05. **p < .01. ***p < .001.

Impacts on Summer Home and Family Mathematics Engagement

Compared with their counterparts in the control group, children who were offered the intervention scored higher on the measure of summer home and family mathematics engagement. We find that being randomly assigned to an offer of the most intensive treatment, the program plus laptop condition, caused children to report levels of family/home math engagement .19 SD higher than their peers in the control group, and this difference was statistically significant (B = .39, p = .04; β = .19; Model M.1.1). For example, the proportion of students who reported doing math at home “almost every day” during summer vacation was 14% in the control group, 18% in the program only treatment group, and 30% in the program plus laptop treatment group. This result appeared to be driven by children in the program plus laptop condition reporting that they did math more frequently at home (B = .47, p < .01, β = .21) and that their parents encouraged or told them to do math more frequently (B = .61, p = .02, β = .26); the differences between groups in frequency of talking about math at home and parents helping with math were not significant. The point estimate for the program only condition was also positive, but was not statistically significant (B = .19, p = .32, β = .09). In addition, results for the single item asking whether the child’s father or mother did any math with them this summer were not significant.

We examined the possibility that the impact of the intervention varied by participant characteristics by considering interactions between baseline characteristics and experimental group. For the family and home math engagement outcome, we found that there was a statistically significant interaction between the program plus laptop computer condition and home Internet access (B = .91, p = .02), suggesting that the effects of receiving an offer of the program plus laptop on family/home math engagement depended on whether the child had access to the Internet at home (see Table 4, Model M.1.2, and Figure 2). To further explore this issue, we fit separate models examining the impact of the intervention offer on family/home math engagement for children who did and did not have Internet access. Among children who had access to the Internet at baseline, the effect of the program plus laptop condition was significant and positive (B = .75, p < .01), whereas the effect for children who lacked Internet access at baseline was oppositely signed and not significant (B = −.18, p = .56). However, this interaction was not significant in the imputation model. Interactions with students’ grade level, prior math achievement, gender, free lunch eligibility, and baseline levels of academic effort, math intrinsic motivation, parent supervision, and math confidence were not significant. Results from additional moderator analyses are available from the authors on request.

Table 4

Results of Fitting Selected Interaction Models for Family/Home Math Engagement and NAEP Math Test Outcomes

	Model M.1.1 Family home math engagement	Model M.1.2 Family home math engagement (Internet × Treatment)	Model M.1.3 Family home math engagement (Race × Treatment)	Model M.2.1 National assessment-based math assessment	Model M.2.2 National assessment-based math assessment (Grade Level × Treatment)	Model M.2.3 National assessment-based math assessment (Parent Supervision × Treatment)
Intercept	−0.22 (.14)	0.05 (.22)	−0.52 (.16)	0.13 (.11)	−0.01 (.12)	−0.03 (.12)
Tenmarks only	0.19 (.19)	0.20 (.30)	0.44^† (.23)	0.03 (.15)	0.33* (.16)	0.31 (.16)^†
Tenmarks + Laptop	0.39* (.18)	−0.16 (.30)	0.85*** (.23)	−0.09 (.14)	0.01 (.16)	0.01 (.16)
Spring math score				0.70*** (.00)	0.70*** (.00)	0.70*** (.06)
Elementary age				−0.65*** (.15)	−0.14 (.24)	−0.13 (.23)
Elementary Age × Tenmarks Only					−1.29*** (.33)	−1.10** (.34)
Elementary Age × Tenmarks Plus Laptop					−0.31 (.32)	−0.21 (.33)
Internet		−0.46 (.28)
Internet × Tenmarks Only		0.04 (.38)
Internet × Tenmarks Plus Laptop		0.91* (.38)
Black			0.98** (.29)
Black × Tenmarks Only			−0.86* (.38)
Black × Tenmarks Plus Laptop			−1.31** (.38)
Baseline parent supervision	0.20** (.07)	0.20** (.07)	0.20** (.07)			0.04 (.10)
Baseline Parent Supervision × Tenmarks Only						−0.34* (.14)
Baseline Parent Supervision × Tenmarks Plus Laptop						−0.10 (.14)
Adj. R²	.05	.08	.10	.40	.45	.47
n	170	170	170	176	176	169

Note. NAEP = National Assessment of Educational Progress. Standard errors in parentheses.

†

p < .1. *p < .05. **p < .01. ***p < .001.

Figure 2.

Estimated family/home mathematics engagement as a function of treatment assignment and baseline home Internet access.

Impacts on Achievement and Affective Outcomes

Treatments’ impacts on the distal achievement outcomes were statistically insignificant. For the national assessment-based math test, the effects of the program only condition (B = .03, p = .86, β = .01) and the program plus laptop condition (B = −.09, p = 0.55, β = −.04) were not significant (Model M.2.1).

Examining interactions between participant characteristics and experimental group, we found that first, there was a statistically significant interaction between the program only treatment and student grade level (elementary vs. middle/high) (β = −1.29, p < .01), suggesting that the impact of the program only on students’ national assessment-based math scores depended on students’ grade level (see Model M.2.2 and Figure 3). To further understand this difference, we fit separate models for older (Grades 6–9) and younger (Grades 3–5) children. We found that for older children (n = 135), the program only condition had a positive effect on national assessment-based math scores (B = .33, p = .04, β = .16). Among older children, the .16 impact is large enough to offset approximately a third of the loss in summer math skills overall (which was reported earlier in the results). On the other hand, for younger children, the effect of the program only was negative (B = −.96, p = .01, β = −.44). For the program plus laptop condition, the effects were not significant for either group (older: B = .01, p = .95, β = .00; younger: B = −.31, p = .32, β = −.15). Lastly, for the program only condition, there was a significant interaction with baseline level of parent supervision, such that the relationship between the program only condition and national assessment-based math scores depended on baseline level of parent supervision (see Model M.2.4 and Figure 4; the significance of relationship is marginal in the imputation model). Students in this condition who reported lower levels of parent supervision at baseline had higher adjusted fall national assessment-based math scores.

Figure 3.

Estimated fall national assessment-based math scores as a function of treatment assignment and grade level.

Figure 4.

Estimated fall national assessment-based math scores as a function of treatment assignment and baseline level of parent supervision (low = 25th percentile; high = 75th percentile).

The effects of treatment on the district curriculum-based fall mathematics assessment were also statistically insignificant (program plus laptop, B = .04, p = .78, β = .02; program only, B = −.07, p = .59, β = −.03; Model M.3.1). No significant interactions of measured covariates with treatment assignment were found. We also fit models with student intrinsic motivation for doing math and math enjoyment as outcomes, but results of these analyses were not statistically significant (see Models M.4.1 and M.5.1), nor were significant interactions between covariates and treatment assignment found.

Intervention Dosage

The number of math “worksheets” that participants completed over the summer provides an indication of participants’ take-up of the intervention. Each worksheet included 10 math problems, with embedded hints and videos students could click for support. Approximately 60% of participants who were offered a subscription to Tenmarks completed at least one worksheet over the summer, and the average number of worksheets completed was 15.39 (SD = 24.07). Among students who completed any worksheets, the average number completed was 26.10 (SD = 23.88). As the program developers intended for students to complete three worksheets per week for 10 weeks, the average student completed slightly over half of the dosage the developers recommended, and approximately 40% of students received no dosage. Students’ actual usage of the materials was thus substantially less frequent than what the developers intended. The number of worksheets children in the treatment groups completed was correlated with their reported home and family mathematics engagement (program plus laptop: r = .27, p = .03; program only: r = .22, p = .08).

To explore whether take-up of the intervention caused children to experience improved outcomes, we conducted an instrumental variables analysis using the number of worksheets completed as the take-up measure. The instrumental variables approach provides an estimate of the treatment effect on the treated; when participants are assigned randomly to conditions, treatment assignment may be used as an instrument for participation in the intervention (Angrist, Imbens, & Rubin, 1996). We utilized Stata’s ivregress routine, with the 2sls option (StataCorp, 2013a), with random assignment to treatment serving as an instrument for participation. The second-stage, or outcome, equations included a mediator variable measuring the number of worksheets completed, reflecting students’ actual participation in the intervention.

The instrumental variables results did not indicate that increased participation in the intervention caused increased mathematics achievement scores (district curriculum-based measure: B = −.02, p = .91; national assessment-based measure: B = −.04, p = .79), although in line with the findings above, it did cause increased home and family mathematics engagement (B = .34, p < .05).

What Predicted Participation?

Given the voluntary nature of the intervention, which occurred over the summer vacation period, it is also of interest to understand what factors predict student take-up and participation. We fit a series of models using zero-inflated Poisson regression to explore which factors predicted the number of times students logged into Tenmarks, among those in the treatment groups. The logit model component of the Poisson regression used student-reported home Internet access at baseline to predict the latent binary outcome of whether the student was a certain zero.

We found that girls logged in fewer times than boys (B = −.45, p = .02) and that low-income children (i.e., eligible for free or reduced price school lunch) logged in marginally fewer times than middle-income children (B = −.39, p = .06). On the other hand, children with higher baseline scores on the parent supervision index logged in significantly more frequently (B = .21, p = .03), and children with Internet access logged in marginally more often (B = .37, p = .10). Number of logins was not significantly predicted by children’s grade level, prior math achievement, or any of the affective/motivational constructs measured at baseline (academic effort, intrinsic motivation for doing mathematics, or mathematics confidence).

Limitations

The limitations of the current study suggest several potentially productive areas for future research. First, we note that an important limitation of the current study is its relatively small sample size. The effects of summer programs on academic achievement generally are expected to be small; in recent studies, school-based summer math programs had an average effect size of +0.08 SD. Furthermore, these programs tended to be significantly more intensive than the current intervention, involving substantial classroom instructional time and teacher interaction. The current study’s relatively small sample size clearly limited its power to detect small effects, as well as the precision of the estimates obtained. In the future, replication studies with larger sample sizes are needed to estimate the effects of the intervention with greater precision.

Second, as noted above, all participating students volunteered for the study; although they were demographically relatively similar to others in their schools and district, still they may have been unusually motivated to participate in the program. Future studies could include larger samples, and examine contexts in which the program is mandatory.

Third, as is generally the case with surveys and self-report measures, the child self-report measures of home and family math engagement used in this study are probably susceptible to self-report and social desirability bias. Summer time-diary studies that include measures of children’s summer math activities are rare (for an exception, see Gershenson, 2013) and would be helpful for confirming the current results. In prior research with a nationally representative sample, parent self-reports of children’s summer activities, such as the use of computers for educational activities, were a significant predictor of children’s summer learning loss (Burkam et al., 2004). In addition, time diary studies would be useful for capturing a more fine-grained range of summer mathematics activities. Although the current research utilized a measure adapted from the Literacy Habits Survey (Paris et al., 2004), which has been used in summer home-based literacy intervention research, it is possible that a time diary could have captured a richer range of informal family mathematical involvement, such as that involved in measuring and estimating in household chores.

Discussion and Conclusions

In summary, this study utilized random assignment to examine the impacts of a summer online mathematics program on children’s summer home and family mathematics engagement and mathematics achievement.

We found that the more intensive treatment condition, an offer both of the Tenmarks program and a free laptop computer, caused children to report significantly higher levels of summer home and family mathematics engagement compared with children in the randomly chosen control group. The intervention thus appears to have succeeded at improving students’ summer home involvement in mathematics. This is despite the fact that the intervention was relatively “low touch”; the more intensive variant was comprised of an offer of a free online summer program plus a laptop valued at a few hundred dollars. The results suggest that low-income children’s reported home math engagement can be increased with the provision of a relatively low-touch intervention.

This increased engagement, however, did not translate into main effects of improved distal achievement outcomes. While the lack of significant findings may be related to power limitations due to sample size, several limitations of the current intervention may also have limited its impacts on achievement. As we discuss below, these limitations imply one of two options. On the one hand, perhaps modifications to the program may be possible that would increase the program’s impact. On the other hand, although the program may be relatively easy to implement, the lack of positive effects on student achievement may call into question the program’s underlying theory of action.

The first possibility is that modifications to the program may be possible that would increase the program’s impact. First, the math activities included in the Tenmarks software may not have been interesting enough to attract students’ attention and elicit engagement during summer vacation, which children often associate with leisure activities. The low worksheet completion rate suggests this possibility. From a student engagement perspective, some policymakers have argued that academic summer programs may benefit from recognizing American summer culture, differentiating themselves from typical school year programming with activities that emphasize student engagement, inquiry, and curiosity (McCombs et al., 2011).

Family resource constraints may also have prevented the predominantly low-income families in the sample from reaping the full potential benefits of a digital intervention during summer vacation. The intervention required Internet access, which 36% of students lacked at baseline. As school districts explore online alternatives for summer learning, summer credit recovery, and supplemental educational services for low-income populations (Heinrich & Nisar, 2013; Walters & Sorensen, 2013), ensuring that children from low-income families have access to technological supports outside of school time is an important concern. However, in the current study, even students who did have Internet access did not garner significant achievement benefits, suggesting that ensuring that all students have digital access will not be enough to make the treatment effective.

Furthermore, parent interactions around mathematics during the summer vacation period may have needed more structure to be effective. While the child-report measures suggest that the amount of home mathematics engagement increased as a result of the program plus laptop treatment, parents and guardians may not have known how to translate their intentions to encourage their children into effective strategies for supporting their children’s mathematics learning. Parents’ mathematics skills may be remembered from their own schooling, and mismatched with contemporary mathematics curriculum content (Peressini, 1998; Remillard & Jackson, 2006). During summer vacation, when the daily flow of structured mathematics instruction and support from teachers is turned off, parents may find it particularly challenging to support home mathematics practice effectively. Some evidence suggests that home-based summer reading programs are more effective when they provide specific instructions for parent interactions. For example, the National Reading Panel (2000) found little evidence for the effectiveness of home-based summer reading programs in which students were merely provided books and asked to read silently alone, with little or no feedback from parents. By contrast, in studies where children were provided with instructions on how to read aloud to their parents and discuss books with family members (e.g., Kim, 2006), low-SES students enjoyed sizable reading gains, perhaps due to the comprehension scaffolding and feedback they received from interacting with parents.

If this is the case, one fruitful avenue for future research may be to explore strategies for helping parents make home mathematics engagement over the summer more effective, perhaps with improved curriculum materials. One possible model might provide parents with structured materials and instruction in providing their children with one-on-one tutoring during summer vacation, perhaps in coordination with a digital intervention. Substantial research supports the efficacy of mathematics tutoring interventions (e.g., Cohen, Kulik, & Kulik, 1982; Fryer, 2011; Ritter, Barnett, Denny, & Albin, 2009). Whereas most research on math skills tutoring has been conducted in schools, with either peers or other adults as tutors, a meta-analysis of the literature on parent tutoring in math (Erion, 2006) found that parent tutoring has an overall positive impact on students’ math achievement. For example, Thurston and Dasta (1990) found that parent tutoring improved students’ knowledge of math facts, and this translated to improved school performance. However, research is lacking on how best to structure parental tutoring during summer vacation, when children are not exposed to the routine supports of daily mathematics instruction. Future research could help school districts seeking to reduce achievement gaps via summer remediation to support children’s experiences in these programs more effectively.

However, as noted above, a second possibility is that the online intervention was simply ineffective at teaching children mathematics, due to flaws in the intervention’s underlying theory of action. In this viewpoint, perhaps the intervention offer was successful at encouraging children to engage in more mathematics than they otherwise would have, but the intervention was inadequate to translate this effort into improved mathematics skills in the way envisioned in the program logic model. Prior research on computer-assisted instruction suggests that even under relatively ideal conditions, in which students often spent several sessions each week during the academic year completing math exercises in a computer lab fully equipped with the needed technology and staffed by a teacher or paraprofessional, effect sizes on student achievement were relatively modest, at +0.19 in elementary school (Slavin & Lake, 2008) and +0.08 in secondary school (Slavin et al., 2009). By contrast, in the current intervention, students experienced difficulties with access to technology; they likely also experienced distractions, as many other summer leisure pursuits called for their attention. Under these conditions, students may have had minimal motivation to expend time and effort on completing math worksheets. Perhaps most importantly, students in the current intervention lacked support from a teacher. Worksheets and videos alone, unconnected to school instruction, may simply have been inadequate to teach children mathematics under these conditions. It may be the case that online interventions such as Tenmarks could be more beneficial with substantial changes to the program’s logic model, recasting the online materials as a supplement to a more traditional summer program with teacher scaffolding. In a recent random assignment study of the Elevate Math summer program, in which seventh-grade students spent 3 hours each day receiving math instruction from a certified teacher, plus 1 hour each day using Khan Academy, treatment group students experienced significant improvements in algebra readiness (+0.7 SD) relative to the control group (Snipes et al., 2015). Although it is unclear whether the usage of Khan Academy was instrumental to these gains, this study nonetheless suggests the potential for using online materials as a supplement to classroom-based summer school math instruction.

A related possibility is that the materials were effective for some students and not others. The supplemental interaction analysis suggested that the program only condition had a positive effect on older students’ national assessment-based math scores, but a negative effect on younger students’ scores. Although this result may simply be stochastic, it suggests the possibility of greater intervention effects for older students. Some research indicates that children left to learn mathematics with limited teacher involvement may develop mathematical misconceptions (Erlwanger, 1973). This problem may have been compounded for younger children in the current intervention, who in addition to lacking access to a teacher over the summer months, may have struggled with the reading load required in the Tenmarks program’s instructions and word problems. Research conducted with literacy apps has suggested that in some cases, young children’s learning may even be harmed by the digital format, perhaps because of distracting interactive elements which interrupt their ability to pay attention to the content (Parish-Morris, Mahajan, Hirsh-Pasek, Golinkoff, & Collins, 2013). Younger children may experience greater success when they receive more scaffolded help from parents. For example, in a study conducted with higher income children and their parents during the academic year, Berkowitz et al. (2015) found that first graders who responded to numerical story problems delivered via app with a parent experienced math learning gains.

On the other hand, older students may have been more accustomed to the online learning format and better able to read and interpret the content, aiding learning gains. This pattern of different effects for older and younger children found for the national assessment-based outcome in the program only condition, however, was not statistically significant for the program plus laptop condition, and was not detected on the district mathematics assessment. It is possible that the district assessments, which were used as general benchmark tests, may not have been well aligned to the Tenmarks program content; however, we could not investigate this issue due to the secure nature of the district tests. The hypothesis that home-based mathematics programs that require independent student use may be more effective for older students, who need less scaffolding to do this work on their own, merits follow-up. In addition, strategies to better support younger students’ summer mathematics learning, such as increased parent scaffolding, merit attention.

Returning to the program’s logic model, we also note the absence of some of the moderator effects that we hypothesized. Children’s program participation was not significantly predicted by any of the child-level affective/motivational constructs measured at baseline. On the other hand, several family and home resource measures did predict children’s participation. Higher income children logged into the program marginally more often than did children eligible for a subsidized lunch, and children with home Internet access and higher levels of parent supervision also logged in more frequently. These differences suggest the relative importance of home and family resources in shaping children’s summer activities and mathematics engagement.

Future Directions

This study raises several questions for future research. First, our finding that a relatively low-touch intervention increased low-income children’s summer mathematics participation suggests that the summer vacation period may represent an underutilized opportunity to increase low-income students’ engagement with math. Although participation was lower than what the program developers intended, many children did participate over the course of the summer, and as a result they did more math than they otherwise would have. However, this increased participation did not translate into improved overall distal achievement outcomes. As this is the first study we know of that has examined a home-based summer math intervention, these findings are preliminary. As suggested above, future design research, in line with that which has been conducted in literacy (e.g., Kim, 2006, 2007; Allington et al., 2010), is needed to develop curricula and intervention supports that would help low-income children to translate their increased time spent on summer mathematics into improved mathematics achievement.

In addition, our preliminary finding that children had lower intrinsic motivation for doing mathematics after the summer vacation period suggests that it may be productive to explore how children’s attitudes and orientations toward mathematics and STEM develop or decline during summer vacation, an extended period away from the “resources faucet” of schools. Because intrinsic motivation is an important predictor of children’s STEM attainment (Gottfried, Marcoulides, Gottfried, & Oliver, 2013), it is important to understand whether and how summer vacation periods may contribute to STEM motivation losses later in children’s academic careers. The findings from the current study thus point toward avenues for future research, to improve our understanding of effective strategies to reduce low-income children’s summer learning loss in math.

Footnotes

Appendix

Results of Fitting Regression Models With Multiple Imputation

Acknowledgements

The authors are grateful to Allison Atteberry, Greg Duncan, Alejandro Ganimian, Drew Gitomer, Heather Hill, Andrew Ho, Christopher Jencks, Richard Murnane, Jon Star, John Willett, and participants in the Harvard Inequality and Social Policy seminar for helpful comments. Thanks to Florine Oyakhire and Natalie Orozco for their assistance with data collection.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a grant from the Eric M. Mindich Research Fund for the Foundations of Human Behavior at Harvard University and by the Harvard Institute for Quantitative Social Science.

Notes

Authors

KATHLEEN LYNCH is a doctoral student at the Harvard Graduate School of Education. Her research interests include education policy and strategies to reduce educational inequality, particularly in mathematics.

JAMES S. KIM studies the effectiveness of literacy reforms and interventions in improving student outcomes. He is an expert in conducting randomized field trials to evaluate, improve, and scale evidence-based literacy reforms.

References

Alexander

K. L.

Entwisle

D. R.

Olson

L. S.

(2007). Lasting consequences of the summer learning gap. American Sociological Review, 72, 167–180.

Allington

R. L.

McGill-Franzen

Camilli

Williams

Graff

Zeig

… Nowak

(2010). Addressing summer reading setback among economically disadvantaged elementary students. Reading Psychology, 31, 411–427. doi:10.1080/02702711.2010.505165

Angrist

J. D.

Imbens

G. W.

Rubin

D. B.

(1996). Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91, 444–455.

Balacheff

Kaput

J. J.

(1996). Computer-based learning environments in mathematics. In Bishop

A. J.

Clements

Keitel

Kilpatrick

Laborde

(Eds.), International handbook of mathematics education (pp. 469–501). Dordrecht, The Netherlands: Springer. doi:10.1007/978-94-009-1465-0_14

Berkowitz

Schaeffer

M. W.

Maloney

E. A.

Peterson

Gregor

Levine

S. C.

Beilock

S. L.

(2015). Math at home adds up to achievement in school. Science, 350, 196–198.

Burkam

D. T.

Ready

D. D.

Lee

V. E.

LoGerfo

L. F.

(2004). Social-class differences in summer learning between kindergarten and first grade: Model specification and estimation. Sociology of Education, 77, 1–31. doi:10.1177/003804070407700101

Cohen

P. A.

Kulik

J. A.

Kulik

C. C.

(1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237–248.

Cooper

Charlton

Valentine

J. C.

Muhlenbruck

Borman

G. D.

(2000). Making the most of summer school: A meta-analytic and narrative review. Monographs of the Society for Research in Child Development, 65, 1–127.

Cooper

Nye

Charlton

Lindsay

Greathouse

(1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 66, 227–268.

10.

Downey

von Hippel

P. T.

Broh

B. A.

(2004). Are schools the great equalizer? Cognitive inequality during the summer months and the school year. American Sociological Review, 69, 613–635.

11.

Duncan

G. J.

Magnuson

(2011). The nature and impact of early achievement skills, attention skills, and behavior problems. In Duncan

G. J.

Murnane

R. J.

(Eds.), Whither opportunity? Rising inequality, schools, and children’s life chances (pp. 47–70). New York, NY: Russell Sage.

12.

Erion

(2006). Parent tutoring: A meta-analysis. Education and Treatment of Children, 29, 79–106.

13.

Erlwanger

S. H.

(1973). Benny’s conception of rules and answers in IPI Mathematics. Journal of Children’s Mathematical Behavior, 1(2), 7–26.

14.

Fryer

R. G.

(2011). Injecting successful charter school strategies into traditional public schools: Early results from an experiment in Houston (NBER Working Paper No. 17494). Cambridge, MA: National Bureau of Economic Research.

15.

Gershenson

(2013). Do summer time-use gaps vary by socioeconomic status? American Educational Research Journal, 50, 1219–1248. doi:10.3102/0002831213502516

16.

Gorman

(2010). An introduction to NAEP (NCES Publication No. 2010468). Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2010468

17.

Gottfried

A. E.

Marcoulides

G. A.

Gottfried

A. W.

Oliver

P. H.

(2013). Longitudinal pathways from math intrinsic motivation and achievement to math course accomplishments and educational attainment. Journal of Research on Educational Effectiveness, 6, 68–92. doi:10.1080/19345747.2012.698376

18.

Gunderson

E. A.

Levine

S. C.

(2011). Some types of parent number talk count more than others: Relations between parents’ input and children’s cardinal-number knowledge. Developmental Science, 14, 1021–1032.

19.

Heinrich

C. J.

Nisar

(2013). The efficacy of private sector providers in improving public educational outcomes. American Educational Research Journal, 50, 856–894. doi:10.3102/0002831213486334

20.

Heyns

(1978). Summer learning and the effects of schooling. New York, NY: Academic Press.

21.

Hiebert

Wearne

(1996). Instruction, understanding, and skill in multidigit addition and subtraction. Cognition and Instruction, 14, 251–283. doi:10.1207/s1532690xci1403_1

22.

Hoover-Dempsey

K. V.

Battiato

A. C.

Walker

J. M.

Reed

R. P.

DeJong

J. M.

Jones

K. P.

(2001). Parental involvement in homework. Educational psychologist, 36, 195–209.

23.

Institute of Education Sciences. (2011). National Assessment of Educational Progress (NAEP) mathematics student questionnaire, grade 8. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/bgq/student/BQ11_student_math_grade8.pdf

24.

Jacob

B. A.

Lefgren

(2004). Remedial education and student achievement: A regression-discontinuity analysis. The Review of Economics and Statistics, 86, 226–244. doi:10.1162/003465304323023778

25.

Kim

J. S.

(2006). Effects of a voluntary summer reading intervention on reading achievement: Results from a randomized field trial. Educational Evaluation and Policy Analysis, 28, 335–355.

26.

Kim

J. S.

(2007). The effects of a voluntary summer reading intervention on reading activities and reading achievement. Journal of Educational Psychology, 99, 505–515. doi:10.1037/0022-0663.99.3.505

27.

Kim

J. S.

Quinn

D. M.

(2013). The effects of summer reading on low-income children’s literacy achievement from kindergarten to grade 8: A meta-analysis of classroom and home interventions. Review of Educational Research, 83, 386–3431. doi:10.3102/0034654313483906

28.

Kim

J. S.

White

T. G.

(2008). Scaffolding voluntary summer reading for children in grades 3 to 5: An experimental study. Scientific Studies of Reading, 12, 1–23.

29.

Lee

Burkam

(2002). Inequality at the starting gate: Social background differences in achievement as children begin school. Washington, DC: Economic Policy Institute.

30.

Levine

S. C.

Ratliff

K. R.

Huttenlocher

Cannon

(2012). Early puzzle play: A predictor of preschoolers’ spatial transformation skill. Developmental Psychology, 48, 530–542.

31.

Levine

S. C.

Suriyakham

L. W.

Rowe

M. L.

Huttenlocher

Gunderson

E. A.

(2010). What counts in the development of young children’s number knowledge? Developmental Psychology, 46, 1309–1319.

32.

Mariano

L. T.

Martorell

(2013). The academic effects of summer instruction and retention in New York City. Educational Evaluation and Policy Analysis, 35, 96–117. doi:10.3102/0162373712454327

33.

Matsudaira

J. D.

(2008). Mandatory summer school and student achievement. Journal of Econometrics, 142, 829–850. doi:10.1016/j.jeconom.2007.05.015

34.

McCombs

J. S.

Augustine

C. H.

Schwartz

H. L.

Bodilly

S. J.

McInnis

Lichter

D. S.

Cross

A. B.

(2011). Making summer count: How summer programs can boost children’s learning. Santa Monica, CA: RAND.

35.

National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author.

36.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: National Institute of Child Health and Human Development.

37.

Paris

S. G.

Pearson

P. D.

Cervetti

Carpenter

Paris

A. H.

DeGroot

Flukes

(2004). Assessing the effectiveness of summer reading programs. In Borman

Boulay

(Eds.), Summer learning: Research, policies, and programs (pp. 122–161). Mahwah, NJ: Lawrence Erlbaum.

38.

Parish-Morris

Mahajan

Hirsh-Pasek

Golinkoff

R. M.

Collins

M. F.

(2013). Once upon a time: Parent–child dialogue and storybook reading in the electronic era. Mind, Brain, and Education, 7, 200–211.

39.

Patall

E. A.

Cooper

Robinson

J. C.

(2008). Parent involvement in homework: A research synthesis. Review of Educational Research, 78, 1039–1101. doi:10.3102/0034654308325185

40.

Peressini

D. D.

(1998). The portrayal of parents in the school mathematics reform literature: Locating the context for parental involvement. Journal for Research in Mathematics Education, 29, 555–582.

41.

Plant

R. W.

Ryan

R. M.

(1985). Intrinsic motivation and the effects of self-consciousness, self-awareness, and ego-involvement: An investigation of internally controlling styles. Journal of Personality, 53, 435–449. doi:10.1111/j.1467-6494.1985.tb00375.x

42.

Quinn

Lynch

Kim

J. S.

(2014, March). Replicating the moderating role of income status on summer school effects across subject areas: A meta-analysis. Paper presented at the spring conference of the Society for Research on Educational Effectiveness (SREE), Washington, DC.

43.

Reardon

S. F.

(2011). The widening academic achievement gap between the rich and the poor: New evidence and possible explanations. In Murnane

Duncan

(Eds.), Whither opportunity? Rising inequality and the uncertain life chances of low-income children (pp. 91–116). New York, NY: Russell Sage.

44.

Remillard

J. T.

Jackson

(2006). Old math, new math: Parents’ experiences with standards-based reform. Mathematical Thinking and Learning, 8, 231–259.

45.

Ritter

Barnett

Denny

Albin

(2009). The effectiveness of volunteer tutoring programs for elementary and middle school students: A meta-analysis. Review of Educational Research, 79, 3–38.

46.

Ryan

R. M.

(1982). Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43, 450–461. doi:10.1037/0022-3514.43.3.450

47.

Slavin

R. E.

Lake

(2008). Effective programs in elementary mathematics: A best-evidence synthesis. Review of Educational Research, 78, 427–515. doi:10.3102/0034654308317473

48.

Slavin

R. E.

Lake

Groff

(2009). Effective programs in middle and high school mathematics: A best-evidence synthesis. Review of Educational Research, 79, 839–911. doi:10.3102/0034654308330968

49.

Snipes

Huang

C.-W.

Jaquet

Finkelstein

(2015). The effects of the Elevate Math summer program on math achievement and algebra readiness (REL 2015–096). Washington, DC: Regional Educational Laboratory West, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/edlabs

50.

Starkey

Klein

Wakeley

(2004). Enhancing young children’s mathematical knowledge through a pre-kindergarten mathematics intervention. Early Childhood Research Quarterly, 19, 99–120.

51.

StataCorp. (2013a). Single-equation instrumental-variables regression. College Station, TX: Stata Press.

52.

StataCorp. (2013b). Stata multiple-imputation reference manual: Release 13. College Station, TX: Stata Press.

53.

Sui-Chu

E. H.

Willms

J. D.

(1996). Effects of parental involvement on eighth-grade achievement. Sociology of Education, 69, 126–141. doi:10.2307/2112802

54.

Tenmarks. (2012). Recognition. Retrieved from http://www.tenmarks.com/tmother/awards

55.

Tenmarks. (2015). Your priority is your students. Available from https://www.tenmarks.com

56.

Thurston

L. P.

Dasta

(1990). An analysis of in-home parent tutoring procedures: Effects on children’s academic behavior at home and in school and on parents’ tutoring behaviors. Remedial and Special Education, 11, 41–52.

57.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2011). NAEP Mathematics Student Questionnaire. Retrieved from https://nces.ed.gov/nationsreportcard/bgquest.aspx

58.

Wagner

(2011, April 4). Can Sal Khan reform education in America? The Huffington Post. Retrieved from http://www.huffingtonpost.com/2011/04/04/the-khan-academy-and-educ_n_844390.html

59.

Walters

Sorensen

(2013, September). Can at-risk students get back on track? Results from a rigorous study of online and F2F algebra for credit recovery: Implementation challenges and results. Paper presented at the spring meeting of the Society for Research on Educational Effectiveness, Washington, DC.