Re-Revisiting the Marshmallow Test: A Direct Comparison of Studies by Shoda,Mischel,and Peake (1990) and Watts,Duncan,and Quan (2018)

Differences in Measures and Bivariate Correlations

Watts et al. describe their study as a “conceptual replication” because the underlying population as well as the achievement tests and delay-of-gratification measures substantially differed from those used by Shoda et al. In many respects, this obstructs a direct comparison of results, as exemplified in the following section. Most importantly, the delay-of-gratification measure was based on a different experimental protocol, as the children in Watts et al.’s study had to wait for only 7 min as opposed to 15 min in Shoda et al.’s study. This variation of the task produced a heavily compressed distribution in the ability-to-delay-gratification measure (SD = 3.08 min) compared with the Shoda et al. sample (SD = 6.23 min). The reason is that Watts et al.’s data contain large fractions of censored observations (55% of children waited until the interviewer returned), whereas in the diagnostic condition of the original marshmallow study, censoring was very low (Mischel, Ebbesen, & Raskoff Zeiss, 1972). Watts et al. acknowledge and address the censoring problem but argue that it “does not affect the conclusion” (p. 1169) because the delay-of-gratification effect was driven only by children who waited less than 2 min. However, this statement hinges on nonmonotonicities in regression models with a substantial set of control variables (for a discussion on the control variables, see below). It does not hold with respect to the bivariate relationships among waiting times and achievement (Watts et al., Table 4, Models 1 and 4) that were used to compare effect sizes across studies. Above and beyond that concern, these nonmonotonicities are not informative per se about the extent of censoring bias in correlation coefficients. In Monte Carlo simulations, detailed below, we found that the bivariate correlations in Watts et al. were considerably downward biased.

To provide some guidance for a more direct comparison of the bivariate correlations for achievement measures reported by Shoda et al. and Watts et al., we used Monte Carlo simulations to illustrate the downward-biasing effect of censoring on correlation coefficients (see Table 1). We answer the following question: Assuming true bivariate correlations of the size reported by Shoda et al., how does censoring affect the estimated correlation coefficients? The corresponding confidence intervals reflect degrees of uncertainty for samples of the size reported by Shoda et al. and Watts et al., respectively. The simulation results indicate that in the presence of 55% censoring, as in the full sample of Watts et al., the expected value of the correlations in Shoda et al., ρ(35) = .42 for SAT Verbal, ρ(35) = .57 for SAT Quantitative, would also be much lower, ρ′(35) = .349 for SAT Verbal, ρ′(35) = .474 for SAT Quantitative. Two-sided t tests indicate that these coefficients are not statistically different from the correlation, ρ(918) = .30, reported by Watts et al., p = .761 for the comparison with ρ′(35) for SAT Verbal and p = .253 for the comparison with ρ′(35) for SAT Quantitative.

OPEN IN VIEWER

Table 1.

Results of Monte Carlo Simulations Depicting the Degree of Downward Bias in Pearson Correlation Coefficients With Censored Data

	Pearson correlation coefficients under censoring Simulation results for 10,000 replications
Censoring	ρ = .42			ρ = .57
	M	95% confidence interval			95% confidence interval
		N = 918	N = 35	M	N = 918	N = 35
None	.420	[.367, .473]	[.135, .693]	.570	[.526, .613]	[.332, .795]
30%	.391	[.337, .445]	[.101, .672]	.531	[.486, .576]	[.283, .768]
35%	.384	[.330, .439]	[.091, .665]	.522	[.476, .567]	[.270, .760]
40%	.377	[.322, .431]	[.084, .660]	.511	[.465, .558]	[.259, .753]
45%	.368	[.314, .423]	[.075, .654]	.500	[.453, .547]	[.247, .746]
50%	.359	[.304, .414]	[.063, .645]	.488	[.441, .535]	[.229, .734]
55%	.349	[.294, .405]	[.049, .636]	.474	[.426, .522]	[.209, .722]
60%	.338	[.282, .394]	[.037, .628]	.459	[.410, .507]	[.194, .712]
65%	.325	[.269, .381]	[.024, .618]	.442	[.392, .491]	[.175, .699]
70%	.311	[.254, .367]	[.006, .603]	.422	[.372, .472]	[.149, .680]

Note: The table reports results based on data that mimic the correlation for achievement measures reported by Shoda, Mischel, and Peake (1990; ρ = .42 for SAT Verbal and ρ = .57 for SAT Quantitative) in a sample of the size reported by Watts, Duncan, and Quan (2018) and Shoda et al., respectively, and with censored waiting times. In the Monte Carlo simulations (based on 10,000 Monte Carlo replications), we drew two correlated random variables from a standard normal distribution (waiting times and SAT Verbal/Quantitative scores) and censored the waiting-time variable. Waiting times in Watts et al. were censored for 55% of the children (45% for children whose mothers did not have a college degree and 68% for children whose mothers did complete college).

Moreover, the results in Table 1 indicate that any of the other bivariate correlations reported by Watts et al. would be substantially larger in the absence of censoring. For a more detailed description of the Monte Carlo study, see “Description of the Monte Carlo Simulations” in the Supplemental Material available online.

Following Shoda et al., Watts et al. also analyzed associations between delay of gratification and behavioral problems. The results are difficult to compare for two reasons. First, the constructs used to measure behavioral problems in both studies differ more widely than the achievement measures. Regarding academic achievement, the most important difference is a high-stakes test (parent-reported SAT scores) in Shoda et al.’s study versus a low-stakes academic-achievement measure (subtests of the Woodcock-Johnson Psycho-Educational Battery Revised) in Watts et al.’s study. For behavioral problems, however, Shoda et al. relied on measures that, by the nature of their construction, are very closely linked to self-control and delay of gratification. The Adolescent Coping Questionnaire contains many questions about self-control and delay of gratification (those were also the most related), and when using the California Child Q-Set personality inventory, the authors deliberately extracted those 11 subitems from the 100 item that were deemed “highly related” to delay of gratification (one item even reads “Is unable to delay gratifications, cannot wait for satisfactions”). Watts et al.’s main analysis, by contrast, draws on a behavioral composite score of externalizing and internalizing behaviors that comprises all 100 items (rated on a 3-point scale) from the Child Behavior Checklist. Second, Watts et al., Shoda et al., and Mischel, Shoda, and Peake (1988) all report very mixed results as regards the relation of delay of gratification with different measures of behavioral problems. Therefore, the absence of a significant relation between waiting time and externalizing and internalizing behavior reported by Watts et al. does not necessarily contradict Shoda et al.’s findings.

Adding Controls for Family Background and Early Cognitive Ability

In addition to reporting bivariate correlations, Watts et al. also studied how the relation changes if covariates are added as control variables in a multivariate regression framework. We appreciate the authors’ idea to explore the potential effect of interventions targeted at delay of gratification after the age of 36 or 54 months. However, we disagree that the analysis necessarily helps “to assess how much bias might be contained in longitudinal bivariate correlations between gratification delay and later outcomes as a result of failure to control for characteristics of children and their environments” (Watts et al., p. 1165). Instead, Watts et al.’s approach to control for ability in all multivariate models can lead to an underestimation of the effect of interest whenever the measures of ability themselves are a function of (early) delay of gratification (see Angrist & Pischke, 2009, Chapter 3.2.3, and our explanations below). This may be the case, for example, if both characteristics develop jointly or if the measurement of cognitive ability is affected by delay of gratification. A reanalysis of Watts et al.’s data reveals that conditioning only on predetermined child background variables produces estimates that are more similar to the ones reported by Mischel et al. (1989) and Shoda et al.

To illustrate the above argument, consider the following model:

y i = β 0 + β DG DG i + X i ′ β + ε i ,

(1)

where y i is a later achievement measure, DG i is the delay-of-gratification measure, and X i is a vector of control variables. Assume that A i is a cognitive-ability measure which depends on DG i as follows:

A i = α i + α DG DG i

(2)

If we estimated Equation 1 and included A i as a control variable (as Watts et al. did, e.g., in the analyses reported in their Table 4), the overall effect of DG i on y i would be as follows: d y i d DG i = β DG + β A α DG . Thus, given that β A and α DG tend to be positive, β DG likely underestimates the true overall effect. There are several reasons for a potential dependence of A i on DG i . Most importantly, if cognitive ability and delay of gratification develop jointly and interactively in children (e.g., see Cunha & Heckman, 2007), Equation 2 holds and a ceteris paribus interpretation of reduced form effects in observational data is difficult. In addition, even if A i develops independently of DG i , the measure of cognitive ability might nevertheless be affected by noncognitive traits, such as delay of gratification (see, e.g., Borghans, Ter Meijers, & Ter Weel, 2008; Carlson & Moses, 2001; Duckworth & Seligman, 2006). Besides, note that even if cognitive ability were measured earlier than delay of gratification, Equation 2 may hold if delay of gratification is reasonably correlated over time.

To determine the effect of delay of gratification on later achievement in multivariate models with predetermined control variables only, we conducted a new analysis. Our empirical approach followed that of Watts et al., but we controlled only for exogenous and predetermined variables such as sex, race, socioeconomic status, and parental ability, that is, excluding any variables that tend to serve as a proxy for early (cognitive) ability in children, except those that were administered before birth. We preregistered the analysis before we had access to the data (see https://osf.io/5jpt4/). Under a matching-on-observables assumption, this analysis unveiled the effect of delay of gratification on later achievement holding fixed characteristics that were determined at birth. As displayed in the upper part of Table 2, conditioning on predetermined variables reduces the coefficients by much less compared with Watts et al.’s analyses (one third instead of two thirds). The coefficient of waiting time remains highly significant (p < .001) when analyses control for all available predetermined variables. Moreover, if we standardize all variables in a way that allows for a direct comparison of the estimates to the correlation coefficients reported by Shoda et al., effect sizes increase even further (lower part of Table 2).

OPEN IN VIEWER

Table 2.

Associations Between Delay of Gratification at Age 54 Months and Later Measures of Academic Achievement for Children of Mothers Without College Degrees

	Achievement composite
	Grade 1		Age 15
Variable	(1)	(2)	(3)	(4)
Variables standardized among all children as in Watts, Duncan, and Quan (2018; mothers with and without a college degree)
Delay minutes	0.279* (0.038)	0.181* (0.036)	0.236* (0.037)	0.137* (0.036)
Controls: predetermined background characteristics	No	Yes	No	Yes
Variables standardized within sample (only mothers without college degree)
Delay minutes	0.305* (0.041)	0.198* (0.040)	0.286* (0.045)	0.166* (0.043)
Controls: predetermined background characteristics	No	Yes	No	Yes

Note: n = 552. For details on the estimations, see Watts, Duncan, and Quan (2018). The table shows standardized coefficients (with standard errors in parentheses). Columns 1 and 3 show estimates for models that contain only delay of gratification as a covariate. In the bottom half of the table, the variables are standardized within the sample, that is, among children of mothers without a college degree. Consequently, the coefficients in Columns 1 and 3 in the bottom part of the table are equivalent to Pearson correlation coefficients and can therefore be compared with the Pearson correlation coefficients reported by Shoda, Mischel, and Peake (1990). Columns 2 and 4 report estimates for models that controlled for the following predetermined background characteristics: sex of child, race dummies, log of family income, mother’s age at birth (years), mother’s education (years), and mother’s Peabody Picture Vocabulary Test score.

*

p < .05.

Discussion and Conclusion

We strongly appreciate Watts et al.’s contribution to the literature and their initiative to replicate Shoda et al.’s analysis in a larger and more diverse sample. However, given the fundamental differences across studies and in light of the active academic and public debate about child inequality, early ability measures, and intervention possibilities, we suggest a more cautious interpretation. Because of fundamental differences in measures, downward-biased estimates, and difficulties in interpreting effect sizes in empirical models with other ability measures as control variables, the new findings should not be interpreted as a falsification of the original marshmallow-test studies. Instead, our reanalysis of the data reveals that conditioning only on predetermined variables yields estimates that are much closer to the ones reported in the original studies.

Watts et al. provide useful suggestive evidence that the early environment shapes a child’s ability to delay gratification (see the correlations in Table 7 in Watts et al.’s article), which is in accordance with a body of related evidence (e.g., Deckers, Falk, Kosse, Pinger, & Schildberg-Hörisch, 2017; Falk & Kosse, 2016). Nonetheless, a detailed understanding of the dynamic process describing how parental background factors and investments form children’s delay of gratification, cognitive achievement, and outcomes requires more refined structural models that allow for simultaneity, self-productivities, and cross-productivities (Cunha & Heckman, 2007). In addition, we agree with Watts et al. that targeted intervention studies are needed to provide truly exogenous variation in investments and thus in delay of gratification. Arguably, such interventions would take most effect if they did not teach waiting strategies but instead provided an enhanced social environment to children who are deprived by the accident of birth (Heckman, 2008, 2013).

Supplemental Material