Does Self-Control Training Improve Self-Control? A Meta-Analysis

Inclusion criteria

Studies were eligible for inclusion if they (1) implemented at least one training procedure that contained the repeated control of dominant responses, (2) included at least one control group, (3) allocated participants randomly to conditions, (4) measured at least one self-control-related outcome variable in a different domain than the domain in which the training occurred, (5) assessed the outcome variable(s) at least 1 day after the last training session, ³ and (6) included samples of mentally healthy adults. We decided to only include studies with random allocation to conditions because only random allocation allows for a causal interpretation of training effects. For studies that contained conditions and/or outcomes irrelevant to our research question, we only included the conditions and/or outcome variables that matched all criteria. In case of ambiguity about the relevance of the chosen outcome variable(s), we generally followed the arguments of the original study authors. For a detailed documentation of all decisions that were made, see the documentation available on the Open Science Framework (https://osf.io/v7gxf/).

Search strategy

We conducted a systematic literature search using three online citation-database providers—namely, EBSCO, ProQuest, and ISI Web of Science. In EBSCO, we searched the databases PsycINFO, ERIC, PsycARTICLES, and PSYNDEX, using the exact search term (TI self regulat* OR TI self control OR TI inhibit* OR TI willpower) AND (TI training OR TI practic* OR TI exercis* OR TI improv*). For ISI Web of Science, the exact search term was TITLE: ([self regulat* OR self control OR inhibition OR willpower] AND [training OR practic* OR exercis* OR improv*]). This search was restricted to entries tagged as “psychology.” In ProQuest, we searched for ([“self regulat*” OR “self control” OR “inhibition” OR “willpower”] AND [“training” OR “practice” OR “exercise”] AND “psychology”). All databases were searched from 1999 onward, the publication year of the first self-control training study (Muraven, Baumeister, & Tice, 1999). Additionally, we issued calls for unpublished data through the mailing lists of three scientific societies (SPSP, EASP, SASP) and personally corresponded with researchers that are active in the field. Finally, the literature search was complemented by unsystematic searches and reference harvesting from included studies and relevant overview articles.

Screening

Titles and abstracts of 4,075 records were screened by the second author for relevance to the present work. Of these, 4,026 were excluded. Forty-nine full-text articles were assessed for eligibility according to the inclusion criteria. Twenty-eight were included in the final database. The PRISMA flow chart in Figure 1 provides details about these steps.

OPEN IN VIEWER

Fig. 1.

PRISMA flow chart of the literature search and study coding.

Study coding

We coded several potential moderator variables of self-control training effects. One potential moderator pertains to the type of training that was implemented, some pertain to the study level, and some pertain to level of the outcome used. For further potential moderators and respective analyses, please see the Supplemental Material available online. The second author and a research assistant coded all potential moderators explained in the remainder of this section (see documentation on the OSF for details). Interrater reliability was examined using intraclass correlation (ICC) for continuous moderators—ICC(1,1) (Shrout & Fleiss, 1979) and Cohen’s Kappa for categorical moderators (Cohen, 1968). Interrater reliability for the study coding was high by common standards (Cicchetti, 1994), mean κ = 0.83, mean ICC(1,1) = 0.92.

Effect size coding

We computed Hedges’ g effect sizes and respective variances (Var _g ) for all effects (Hedges, 1981). Hedges’ g is similar to Cohen’s d but corrects for small sample bias. Two design types were prevalent: pretest-posttest-control designs (PPC) and posttest-only-control designs (POC). For continuous dependent variables, we first computed Cohen’s d and its variance Var _d and then applied Hedges’ correction factor for small sample bias to compute g and Var _g . For PPC designs, Cohen’s d was defined as the difference of mean improvement between the training group and the control group, divided by the pooled pretest standard deviation (SD):

d P P C = ( M T r e a t [ P O S T ] − M T r e a t [ P R E ] ) − ( M C t r l [ P O S T ] − M C t r l [ P R E ] ) ( ( n T r e a t − 1 ) × S D T r e a t [ P R E ] 2 + ( n C t r l − 1 ) × S D C t r l [ P R E ] 2 ) × 1 n T r e a t + n C t r l − 2 .

Thus, the numerator in the Cohen’s d fraction was a difference of differences—that is, the difference of the mean improvement (M_post – M_pre) between the two conditions. Standardizing by pooled pretest SD rather than pooled posttest SD or pooled total SD has been shown to yield a more precise estimate of the true effect, as interventions typically cause greater variation at posttest (Morris, 2008).

For POC designs, Cohen’s d was defined as the difference in means divided by the pooled posttest standard deviation.

d P O C = M T r e a t [ P O S T ] − M C t r l [ P O S T ] ( ( n T r e a t − 1 ) × S D T r e a t [ P O S T ] 2 + ( n C t r l − 1 ) × S D C t r l [ P O S T ] 2 ) × 1 n T r e a t + n C t r l − 2 .

For noncontinuous variables, appropriate effect sizes for the respective scale level were computed and then transformed to Hedges’ g (Hedges, 1981). When possible, effect sizes were computed from descriptive statistics and sample sizes. We contacted the authors if required information was missing in the manuscript. Eighteen out of 23 responded to our inquiry. If authors did not respond or could not provide the required information, we approximated the effect size as closely as possible using the information provided in the original manuscript.

Some studies included more than one treatment group or control group (e.g., using self-control training tasks and/or control tasks from different domains). When multiple treatment and/or control groups were implemented, we compared each treatment group separately against each control group. For studies that included multiple outcomes, we computed one effect size per outcome for each comparison. For example, a study reporting two treatment groups, two control groups, and three outcomes would contribute a total of 12 effect sizes (2 treatments × 2 controls × 3 outcomes). Some studies reported multiple measurements of the same outcomes after training. In these cases, we only included the measurement temporally most proximate to the training phase (exception: follow-up moderator analysis; see next paragraph).

For the moderator analysis “follow-up,” we contrasted outcome variables measured directly after the training (posttraining, see above) with later measurement occasions (follow-up). If a study included both posttraining and follow-up measurements, we included effect sizes for both time points. When multiple training and/or control groups were implemented, we combined them, respectively, before computing the effect sizes, as type of training/control group was not of interest in this particular analysis.

Meta-analytic procedure

We deviated from the path of data analysis outlined in the preregistration because we followed valuable reviewer suggestions made in the editorial process (i.e., reliance on the robust variance estimation, RVE, approach; see below). All analyses were conducted using random effects models because self-control training interventions, control groups, and outcome variables varied considerably between studies. Hence, it was unreasonable to expect one true, “fixed” population effect.

Conventional meta-analytical techniques assume that effect sizes are statistically independent. Including multiple effect sizes stemming from multiple outcomes or comparisons per study violates this assumption (Lipsey & Wilson, 2001). Several approaches have been proposed to address this issue and to arrive at a set of independent effect sizes (for an overview, see Marín-Martínez & Sánchez-Meca, 1999). One widely used approach averages and adjusts effect sizes based on the correlation of the combined effect sizes (Borenstein, Hedges, Higgins, & Rothstein, 2009). More specifically, the effect size variance estimate is more strongly reduced if the combined outcomes are weakly correlated compared to when they are highly correlated. This reflects the idea that uncorrelated outcomes contain broader informational value than highly correlated outcomes. One downside of this approach is that averaging effect sizes leads to a loss of information because analyses on the level of effect sizes are no longer possible. To illustrate, consider a study reporting treatment effects on reading and mathematics achievement. Averaging these effect sizes delivers one study summary effect. The single summary effect prohibits a moderator analysis investigating effects of the treatment on different outcomes such as reading versus mathematic achievement across several studies in the meta-analysis.

The recently developed RVE approach for meta-analysis (Hedges, Tipton, & Johnson, 2010) solves this issue. It permits conducting random effects meta-regression on dependent effect sizes, thus offering many advantages over the previously described averaging approach. Unfortunately, there are some drawbacks to RVE as well. First, RVE estimates the correlation matrix of dependent effect sizes rather than accounting for it directly. It will therefore generally yield less precise results than approaches that incorporate the empirical correlation structure (e.g., the procedure proposed by Borenstein et al., 2009). Second, because the approach is relatively novel, the validity of some key meta-analytical techniques has not yet been validated in the RVE context, such as regression-based tests for small-study effects, Trim and Fill procedures, or power analyses. Third, although it is possible to calculate point estimates of true variance in the effect sizes in RVE (i.e., I²), there are currently no significance tests of these estimates available. Hence, researchers must rely on conventions when interpreting the true variance of effect sizes (Higgins & Green, 2011).

Considering the respective (dis)advantages of the Borenstein approach and the RVE approach, we adopted the following threefold strategy for the present meta-analysis: First, we computed the global summary effect of self-control training based on RVE and provide the parallel estimate based on the Borenstein approach for converging evidence. Second, all moderator analyses were run based on RVE. Third, all tests to detect and correct for small study effects were run based on the Borenstein approach, as the validity of these procedures has not yet been investigated in the RVE context. We also ran these analyses within the RVE approach for converging evidence. These latter analyses should be interpreted with caution, however. Please refer to the Supplemental Material available online for details of the Borenstein approach. We relied on the MAd package to implement the approach (Del Re & Hoyt, 2014).

Small-study effects and publication bias

Publication bias results if studies with certain characteristics (e.g., significant effects, large effect sizes) are systematically more likely to be submitted for publication by authors and/or accepted for publication by journals than studies with nonsignificant or negligible effect sizes. If this happens, the published literature is not representative of the full body of research and overestimates the population effect size (Ioannidis, 2008). Publication bias is a pervasive problem in the social sciences including psychology (Bakker et al., 2012; Franco, Malhotra, & Simonovits, 2014, 2016).

When a given literature is affected by publication bias, there will likely be a negative relationship between studies’ effect sizes and their precision (or sample size): More precise studies with larger samples yield smaller effect sizes. This relationship is found in many meta-analyses (Levine et al., 2009). Small studies are more likely than larger studies to be excluded from the published literature due to nonsignificance or to be influenced by questionable data analysis methods that lead to significant findings at the cost of a factually increased Type I error (e.g., p-hacking; Simmons, Nelson, & Simonsohn, 2011). Therefore, several statistical methods to detect and correct for publication bias investigate the relation between effect size and precision. These assume that in an unbiased literature small studies (on average) should be no more likely to deliver strong effects than larger studies.

It is important to note that a negative relationship between effect size and precision may also result from unproblematic causes other than publication bias. For example, smaller studies may have used other populations that may be more strongly affected by the intervention. Further, it is possible that certain particularly effective interventions are more readily applied in small than in large studies. Also, experimental manipulations may be more rigorously (and therefore more effectively) applied in small than in large studies (Sterne et al., 2011). These kinds of small-study effects reflect true heterogeneity of effect sizes. This heterogeneity may be quantified and potentially explained by statistical analyses such as moderator analyses. Importantly, they are not a problematic sign of publication bias. In case of an empirically negative association of effect size and study precision, meta-analysts therefore need to reflect about possible reasons for this relationship with respect to the specific body of research that is being investigated.

We applied two methods to detect publication bias (Funnel plot, Egger’s regression test) and two further methods to correct for publication bias (Trim and Fill; Precision Effect Estimation With Standard Error, PEESE). In the way they have been developed and validated, these techniques require statistical independence, so we applied them to the set of independent effect sizes (Borenstein approach). However, the logic of Egger’s regression test and PEESE can be readily extended to RVE. We report both approaches for these procedures, but caution is warranted in interpreting the RVE variants until the techniques have been thoroughly validated in RVE.

Characteristics of included studies

The search identified 4,075 articles, of which 28 were eligible for inclusion, contributing a total of 33 studies. See Figure 1 for a PRISMA flow chart and the documentation on the Open Science Framework (https://osf.io/v7gxf/) for (a) a list of excluded studies with reasons for exclusion and (b) full references for all included studies. Out of these 33 studies, 10 were unpublished as of December, 14, 2016. Publication dates ranged from 1999 to 2016 (Mdn = 2014). Self-control training was operationalized through a diverse set of training paradigms. For instance, participants were prompted to use their nondominant hand for everyday tasks, to complete multiple sessions of computerized inhibitory control tasks, or to control their diet. The majority of training procedures lasted 2 weeks (m = 19 effect sizes). In total, the analysis included data from 2,616 participants who were on average 21.63 years old. The average total sample size per study was n = 79, comprising mostly student samples (k = 27 studies) and females (M_female = 67%). A wide array of outcomes was used to measure self-control-related constructs after (k = 16) or both before and after the training (k = 17). Nine studies also included a follow-up measurement. Training effects were predominantly evaluated through inhibitory control tasks (m = 18) or in the domains of physical persistence (m = 15), health behavior (m = 16), and affect and well-being (m = 29).

Main analyses

Moderator analyses

Descriptive statistics, confidence intervals, and inferential statistics of all categorical moderator variables are provided in Table 1. Numbers of effect sizes per group (m) are provided in parentheses. Results of the meta-regressions for continuous moderators are provided in Table 2.

OPEN IN VIEWER

Table 1.

Results of Moderation Analyses for Categorical Moderators

Moderator	Summary effect and 95% CI						kstudy	meffects	Test of moderation			I2
	g	LL	UL	t	df	p			Statistic	df	p
Treatment-level moderator
Type of training									HTZ = 1.11	7.37	.421	54.85%
Inhibitory control task	0.21	−0.02	0.44	2.04	9.41	.070	11	56
Handgrip	0.37	—	—	5.21	3.66	—	5	21
Nondominant hand	0.42	0.25	0.59	5.58	9.25	<.001	11	56
Posture regulation	0.23	—	—	2.55	2.53	—	4	11
Diet regulation	−0.01	—	—	−0.02	2.61	—	4	28
Study-level moderators
Publication status									t = 1.76	16.47	.098	56.48%
Published	0.37	0.24	0.51	5.83	20.53	<.001	23	131
Unpublished	0.13	−0.16	0.41	1.01	8.52	.338	10	27
Research group									t = 2.49	12.53	.028	55.61%
Strength model	0.51	0.29	0.74	5.42	7.20	<.001	9	22
Other	0.22	0.08	0.36	3.19	21.81	.004	24	136
Control group quality									t = 1.73	20.79	.099	57.43%
Active control group	0.23	0.08	0.39	3.10	19.70	.006	22	79
Inactive control group	0.43	0.23	0.64	4.68	11.02	<.001	13	79
Outcome-level moderators
Type of outcome									HTZ = 1.55	10.40	.259	62.76%
Affect and well-being	0.30	−0.12	0.71	1.87	4.70	.124	6	29
Health behavior	0.12	−0.21	0.45	1.01	4.01	.368	6	16
Inhibition	0.17	−0.26	0.59	0.90	8.30	.395	11	18
Inhibition after ego depletion	0.48	0.10	0.86	3.33	4.54	.024	6	9
Physical persistence	−0.06	−0.42	0.29	−0.46	5.28	.665	8	15
Subjectivity of outcome measurement									t = 0.30	26.07	.588	59.79%
Other	0.32	0.13	0.51	3.50	21.90	.002	26	80
Subjective	0.26	0.14	0.39	4.44	13.93	<.001	18	78
Lab-based versus real-world behavior									t = −0.88	16.32	.392	59.35%
Lab-based	0.32	0.16	0.48	4.18	24.35	<.001	29	79
Real-world	0.23	0.05	0.40	2.93	10.00	.015	12	79
Stamina versus strength									t = −2.84	17.52	.011	56.50%
Stamina	0.60	0.33	0.87	4.83	11.79	<.001	16	29
Strength	0.21	0.07	0.34	3.14	23.92	.004	28	129
Maximum versus realized potential									t < 0.01	27.75	.997	59.36%
Maximum	0.30	0.02	0.58	2.26	15.91	.038	21	54
Realized	0.30	0.19	0.40	5.91	19.74	<.001	23	104
Follow-up									t = 1.12	9.69	.291	61.22%
Follow-up	0.18	−0.02	0.39	2.16	6.74	.069	28	9
Posttraining	0.31	0.16	0.45	4.32	27.00	<.001	74	31

Note: df = associated small sample corrected degrees of freedom; g = effect size; k_study = number of studies that contributed to the respective moderator level; LL = lower limit of the 95% CI; m_effect = number of effect sizes in the respective moderator category; p = p value associated with the t value and df in the same row; t = t value associated with the g value in the same row testing statistical significance in the respective moderator level; UL = upper limit of the 95% CI. Statistic (test of moderation): t value for single parameter tests or Hotelling-T-approximated (HTZ) test statistic for multiple parameter tests. Significant test statistics indicate significance of the overall model. I² reflects the proportion of true variance in the total observed variance of effect sizes after accounting for the respective moderator. For some moderator models the values for I² can become be larger than for the global summary-effect model because of missing values or differences in effect size computation. Note that for three subgroups in the type of training analysis, degrees of freedom fell below 4. Significance tests for the summary effects should thus not be interpreted. Accordingly, we did not report CI₉₅ and p values for the respective subgroups.

OPEN IN VIEWER

Table 2.

Results of Moderation Analyses for Continuous Moderators

	Meta-regression		Test of moderation
Moderator	Intercept	Slope	t	df	p	I2
Study-level moderators
Length of training	0.25	0.003	0.44	4.01	.682	60.58%
Gender ratio	0.04	0.008	2.02	13.27	.064	55.83%

Note: Test of Moderation, t value and corresponding small-sample corrected degrees of freedom. Significant t values indicate significant moderation. I² reflects the proportion of true variance in the total observed variance of effect sizes. For some moderator models the values for I² can become be larger than for the global summary-effect model because of missing values or differences in effect size computation.

Treatment-level moderator

Type of training

Five types of training procedures were applied in at least five studies. The most effect sizes originated from studies that used repeated sessions of computerized inhibitory control training (g = 0.21, m = 56), followed by training procedures prompting participants to use their nondominant hand for everyday tasks (g = 0.42, m = 49). Other common procedures required participants to repeatedly press and squeeze a hand strength training device until failure (g = 0.37, m = 21), to continuously regulate their posture by sitting and walking upright (g = 0.23, m = 11), or to continuously regulate their diet (g = −0.01, m = 8). Despite substantial descriptive differences, the overall analysis between the subgroups was not significant, HTZ(7.37) = 1.11, p = .421 (Fig. 2).

OPEN IN VIEWER

Fig. 2.

Moderation by type of training, HTZ[7.37] = 1.11, p = .421. g = Hedges’ g summary effect within the respective subgroup; k = number of studies in a subgroup; m = number of effect sizes in a subgroup; p = p value testing Hedges’ g against zero. Black dots represent individual effect sizes. The thick black horizontal lines represent the meta-analytic summary effects within the subgroups. The thin black horizontal lines represent the borders of the 95% CI around the subgroup summary effect. The dashed grey horizontal line represents the null effect at g = 0. For informational purposes, the sample size that was used to calculate the respective effect size is depicted on the x axis, but the moderating role of this attribute is not investigated in this analysis. Circle size represents the weight of the respective effect size in the meta-analytic RVE mixed-effects model depicted here. Diet: control one’s diet; handgrip: repeated use of a handgrip squeezer; inhibition: computerized inhibition control training procedures; non-dominant hand: use of non-dominant hand for everyday tasks; posture: keep an upright posture in everyday life. Note that for three subgroups in this analysis, degrees of freedom fell below 4. The corresponding significance tests for the summary effects should thus not be interpreted. Accordingly, we did not report CI₉₅ and p values for the respective subgroups.

Study-level moderators

Length of training

The majority of studies used a training procedure with a duration of 2 weeks (m = 19; 58%). Thus, there was little variability in training duration, precluding a meaningful test of this moderator. Consequently, there was no significant moderation effect of the length of the training duration, b₁ = 0.003, t(4.01) = 0.44, p = .682 (Fig. S2 in the Supplemental Material available online).

Publication status

On average, effect sizes were almost three times larger for published (g = 0.37, m = 131) than for unpublished studies (g = 0.13, m = 27). This difference was close to conventional levels of statistical significance, t(16.47) = 1.76, p = .098 (Fig. S3 in the Supplemental Material available online).

Research group

Significantly larger effects were found by the “strength model research group” (g = 0.51, m = 22) compared to other research groups (g = 0.22, m = 136), t(12.53) = 2.49, p = .028 (Fig. 3).

OPEN IN VIEWER

Fig. 3.

Moderation by research group, t(12.53) = 2.49, p = .028. g = Hedges’ g summary effect in subgroup; k = number of different studies within subgroup; m = number of effect sizes within subgroup; p = p value testing Hedges’ g against zero. Black dots represent individual effect sizes. Thick black horizontal line, meta-analytic summary effect within subgroup; thin black lines, 95% CI; dashed grey line, null effect at g = 0. The associated sample size for each effect size is depicted on the x axis for informational purposes. Circle size represents effect size weight for the subgroup analysis.

Control group quality

Descriptively smaller effects were evident in studies with active control groups (g = 0.23, m = 79) compared to studies with inactive control groups (g = 0.43, m = 79). The difference was close to statistical significance, t(20.79) = 1.73, p = .099 (Fig. S4 in the Supplemental Material available online).

Gender ratio

We imputed two missing values for this moderator by fitting the linear model based on all but the respective two effect sizes and then entering the two effect sizes in the regression equation, thus predicting the missing values from the effect sizes. The moderating effect of the percentage of males in the study samples was close to statistical significance, b₁ = 0.008, t(13.27) = 2.02, p = .064, such that Hedges’ g was predicted to increase by Δg = 0.08 per 10% more males in the sample (Fig. 4). Percentages ranged from 0% to 64% across studies, so any interpretation of this slope should be limited to this range.

OPEN IN VIEWER

Fig. 4.

Moderation by gender ratio. The line represents the weighted RVE meta-regression of effect size on gender ratio, b₁ = 0.008, t(13.27) = 2.02, p = .064. Circle size represents effect size weight.

Outcome-level moderators

Type of outcome

In total, the included studies featured 94 unique dependent variables. We grouped these variables into theoretically homogeneous clusters. Note that degrees of freedom for significance tests of subgroup summary effects are dependent on the number of studies and effect sizes within the respective cluster. Significance tests are only interpretable when df > 4 (Tipton & Pustejovsky, 2015). Additionally, small clusters in subgroup analyses can bias tests of other clusters and the full model because they tend to increase imbalance in categorical predictors. Thus, it was necessary to exclude small clusters from the analysis to arrive at a model for which all parameters are interpretable. To do so, we sequentially removed clusters with the lowest degrees of freedom, until all degrees of freedom for the single parameter tests were four or larger. This procedure retained five outcome clusters in the final model. These were affect and wellbeing (g = 0.30, m = 29), inhibitory control (g = 0.17, m = 18), physical persistence (g = −0.06, m = 15), health behavior (g = 0.12, m = 16), and inhibitory control after depletion (g = 0.48, m = 9). The difference between these outcome clusters was not significant, HTZ(10.40) = 1.55, p = .259 (Fig. 5).

OPEN IN VIEWER

Fig. 5.

Moderation by type of outcome, HTZ(10.40) = 1.55, p = .259. g = Hedges’ g summary effect in subgroup; k = number of different studies within subgroup; m = number of effect sizes within subgroup; p = p value testing Hedges’ g against zero. Black dots represent individual effect sizes. Thick black horizontal line, meta-analytic summary effect within subgroup; thin black lines, 95% CI; dashed grey line, null effect at g = 0. The associated sample size for each effect size is depicted on the x axis for informational purposes. Circle size represents effect size weight for the subgroup analysis.

Lab-based versus real-world behavior

Effect sizes for outcomes that were measured in the lab (g = 0.32, m = 79) were not significantly different from outcomes that reflect real-world behavior (g = 0.23, m = 79), t(16.32) = −0.88, p = .392 (Fig. S5 in the Supplemental Material available online).

Stamina versus strength

Effects for outcomes that were preceded by an effortful task (stamina; g = 0.60, m = 29) were remarkably larger than for outcomes that were not preceded by an effortful task (strength; g = 0.21, m = 129), t(17.52) = −2.84, p = .011 (Fig. 6).

OPEN IN VIEWER

Fig. 6.

Moderation by strength versus stamina, t(17.52) = −2.84, p = .011. g = Hedges’ g summary effect in subgroup; k = number of different studies within subgroup; m = number of effect sizes within subgroup; p = p value testing Hedges’ g against zero. Black dots, individual effect sizes; thick black horizontal line, meta-analytic summary effect within subgroup; thin black lines, 95% CI; dashed grey line, null effect at g = 0. The associated sample size for each effect size is depicted on the x axis for informational purposes. Circle size represents effect size weight for the subgroup analysis.

Maximum versus realized potential

Whether outcomes reflected maximum self-control potential (g = 0.30, m = 54) or realized self-control potential (g = 0.30, m = 104) had no effect on effect sizes, t(27.75) < 0.01, p = .997 (Fig. S6 in the Supplemental Material available online).

Follow-up

The distribution of the time-lags between the last day of the training and the time of outcome measurement was discontinuous with very large variance and therefore inept for a regression analysis. We therefore ran a categorical moderation analysis comparing posttest shortly after training with follow-up measurements (see Effect Size Coding). The follow-up measurements took place Mdn = 9.5 days after the last day of training (M = 42, SD = 65, min. = 3.5, max. = 184). Outcome measures that were assessed directly after the training yielded descriptively larger effect sizes (g = 0.31, m = 74) compared to outcomes measured at later time points (g = 0.18, m = 28). This difference was not significant, t(9.69) =1.12, p = .291 (Fig. S7 in the Supplemental Material available online).

Multiple moderators

Testing multiple moderators simultaneously allows estimating the unique moderating role of each predictor while controlling for the overlap with other moderators. For this analysis, it was necessary to select a subset of moderators in order to avoid overfitting the model. Several moderators had to be excluded a priori from this process (e.g., due to missing values or restricted variance; please see the Supplemental Material available online for a full list of excluded moderators and reasons for exclusion).

As outlined in the Methods section, we employed two approaches to select the most appropriate moderators for this combined analysis: One approach relied on the findings from the bivariate moderator analyses; the second approach was a data-driven bottom-up approach seeking to explain a high degree of heterogeneity with a small number of predictors.

Results of the bivariate analyses suggested entering four moderators with p values of p = .100 or smaller in the respective bivariate analysis into the combined model: control group quality, stamina versus strength, research group, and gender ratio. The data-driven bottom-up approach delivered converging evidence: We fitted multimoderator models for all possible combinations of predictor variables, resulting in 2⁹ = 512 models, and retrieved the 100 models that explained the greatest amount of true heterogeneity (i.e., reduction in I²). Figure S8 in the Supplemental Material available online reports the relative importance of the nine examined moderators and can be interpreted akin to a Scree plot in factor analysis. There was a relatively large gap in importance between the fifth (gender ratio) and sixth (subjectivity of outcome measurement) most important moderators—suggesting entering the first five moderators in the combined analysis. Four of these five moderators match those identified in the bivariate analyses. Maximum versus realized potential emerged as an additional important moderator despite being far from significance in the bivariate analysis (p = .996). This suggests that this moderator binds residual variation in the other predictors and thereby contributes to explaining heterogeneity (suppression effect; Conger, 1974). In summary, the approach based on the bivariate analyses and the data-driven bottom-up approach provided converging evidence for the relevance of four moderators, and the latter approach unveiled the contribution of one additional moderator potentially acting as a suppressor variable.

The full model including all five predictors was significant, HTZ(13.46) = 3.32, p = .036 (Table 3). The model explained ΔI² = 13.87% more true effect size variance than the intercept-only model. The moderator stamina versus strength again emerged as significant (p = .027). For research group, there still was a trend toward significance (p = .097). The p value for control group quality was almost unchanged compared to the bivariate analysis (p = .097). By contrast, gender ratio did not border on significance anymore (p = .169). The alleged suppressor variable, maximum versus realized potential, was also not significant (p = .248). These findings suggest that three of the four moderators that were at least marginally significant in the bivariate tests tended to explain unique portions of effect size heterogeneity, even when controlling for the influence of the other most potent moderators.

OPEN IN VIEWER

Table 3.

Summary of RVE Mixed-Effects Meta-Regression Model Predicting Effect Sizes From Multiple Moderators

Variable	b	SE(b)	t	df	p
Intercept	0.175	0.169	1.04	13.57	.317
Control group quality (inactive)	0.207	0.116	1.78	16.25	.094
Stamina versus strength (stamina)	−0.387	0.155	−2.50	13.20	.027
Research group (strength model)	0.205	0.114	1.80	12.17	.097
Self-control potential (realized)	0.174	0.146	1.20	16.84	.248
Gender ratio	0.006	0.004	1.45	13.71	.169

Note: Categorical predictors were dummy coded with 0 and 1. The moderator level coded as 1 is indicated in parentheses. b = regression coefficient; df = corresponding small-sample corrected degrees of freedom; p = p value associated with the t value and df in the same row; SE(b) = standard error of regression coefficient; t = t value testing whether the regression coefficient in the same row is significantly different from zero. The full model was significant, HTZ(13.46) = 3.32, p = .036, I² = 45.24%.

Note that in this regression, shared variance between predictors contributes to the overall model fit but is not assigned to any predictor specifically. Hence, to the extent that a predictor has a causal claim for parts of the nonassigned shared variance, even nonsignificant predictors may be important for the overall model. Nonsignificance of predictors should therefore not be overinterpreted as indicating that this predictor is unimportant in explaining heterogeneity.

Small-study effects and publication bias

Trim and Fill

After the previously reported bias-detection techniques, we turned to bias-correction techniques. The Trim and Fill method indicated that four studies were missing on the left of the mean meta-analytic effect size in order to obtain a fully symmetrical funnel plot (Fig. 7). Imputing these studies and adding them to the model delivered a bias-corrected random-effects summary estimate of g = 0.24, SE_g = 0.051, CI₉₅ [0.14, 0.34], p < .001, that can be most adequately compared to the corresponding uncorrected summary effect size estimate based on independent effect sizes (g = 0.28). This analysis suggests a moderate degree of small-study effects and potentially publication bias.

OPEN IN VIEWER

Fig. 7.

Funnel plot after Trim and Fill bias correction. Note that this analysis is based on the study-level effect sizes (Borenstein approach). Compared to the original funnel plot (see the Supplemental Material available online), four studies were imputed to achieve symmetry (i.e., white circles). This resulted in a bias-corrected summary effect size of g = 0.24, CI₉₅ [0.14, 0.34] that is slightly smaller than the original (Borenstein approach) estimate of g = 0.28, CI₉₅ [0.19, 0.38].

Small-study effects and publication bias

The Trim and Fill method indicated a moderate degree of bias and delivered a corrected effect size estimate of g = 0.24. By contrast, PEESE indicated a much greater degree of bias and delivered an estimate of g = 0.13 that was not significantly different from zero. Note that an association between effect size and study precision (as detected by Trim and Fill and PEESE) can result from publication bias, p-hacking, and other biases, but it may also partly or completely be due to mundane reasons that cause small-study effects. For example, in the medical sciences, samples that are particularly receptive to an intervention due to a certain health condition may show particularly strong effect sizes. Such samples may also be difficult to recruit and therefore form smaller sample sizes than samples consisting of more readily available (and less susceptible) participants. Concerning the present database, we were unable to come up with analogous mundane reasons for small-study effects in the self-control training literature. Given how the field worked for many years (e.g., difficulty to publish nonsignificant findings), we deem it likely that there is publication bias in the investigated literature, but the severity of this bias is difficult to estimate. This is because none of the currently available techniques performs consistently well under conditions typical for (social) psychological literatures including heterogeneity and publication bias (Gervais, 2015; Inzlicht et al., 2015). Thus, the degree to which the bias-corrected estimates are biased themselves is unknown.

Mechanisms underlying training effects

The present meta-analysis suggests that self-control training may lead to slight improvements in self-control in other domains. The strength model postulates that the repeated control of dominant responses strengthens the “self-control muscle” (Baumeister & Vohs, 2016b). This metaphor is vivid and descriptive, but it is of limited explanatory value for the observed effects because it does not specify the psychological mechanisms explaining training success. What do we know about mechanisms underlying training effects? One may approach this question from two perspectives. First, one may try to identify the crucial elements in a self-control training that make it effective. Second, one may think about the psychological processes that mediate self-control training effects.

The strength model claims that the repeated exertion of self-control by overcoming a dominant response is the driving “ingredient” of the self-control training. However, effect sizes stemming from studies with inactive control groups were almost 50% larger than those from studies with active control conditions. In studies with inactive control groups, various mechanisms besides the repeated control of dominant responses can cause an intervention effect (e.g., demand effects, greater engagement with the study by the active intervention group). What is more, even in the subset of studies with active control groups, few control groups closely matched the training condition, allowing for other than the focal mechanism to drive training effects. Thus, the net training effect due to the control of dominant responses may still be smaller than indicated by the training effect obtained for the studies employing active control groups (g_active = 0.23, CI₉₅ [0.08, 0.39]).

With regard to the mediating psychological processes, surprisingly little is known. Some studies investigated changes in self-efficacy, awareness of the concept of self-control, and implicit theories about willpower as possible mechanisms but did not find evidence for mediation (Job, Friese, & Bernecker, 2015; Klinger, 2013; Muraven, 2010a, 2010b). In one study, self-control training reduced academic effort avoidance in university students, which partly mediated the effect of training on participants’ grade point average (Job et al., 2015). This study suggests that motivational variables might play a mediating role. Future research has to test whether changes in effort avoidance may account for training effects in other domains than academic achievement.

One hitherto unexplored possibility is that training and control conditions differentially affect participants’ expectations, thus allowing for placebo effects without actual changes in the trained constructs (Boot, Simons, Stothart, & Stutts, 2013; Foroughi, Monfort, Paczynski, McKnight, & Greenwood, 2016). Expectations regarding possible improvements on the dependent variables may differ between groups if they are not measured or, better, experimentally controlled—even in studies with active control groups. Hence, more knowledge is needed about how participants believe the (training or control) intervention is affecting them. What do participants believe their training regimen to be good for? What are their ideas about the researchers’ goals for the study, and which expectations about improvement on the measured constructs do participants hold?

In sum, little is known about the crucial elements of a training intervention. The literature to date does not deliver conclusive evidence that exerting self-control by repeatedly overriding dominant responses is the dominant causal mechanism that improves self-control over time and across domains. Even less is known about the psychological processes that are affected by a self-control training and lead to improved self-control performance.

How to move forward?

We will briefly discuss recommendations for future work concerning both methodological and theoretical developments. On the methodological level, future research should, first, conduct direct, high-powered, and preregistered replications. The set of the present 33 studies is very diverse, containing no close replications that would bolster confidence in obtained findings. Second, it will be important to more consistently use pre-post designs to increase statistical power. Based on the mean parameters evidenced by the current meta-analysis (g = .30, N_average = 79, α = .05, r_pre-post = .70 within control groups), power for studies with pre-post designs is adequate (1-β = .92). However, in post-only designs the same parameters result in a poor power of 37%, even with a one-tailed test. Note that it is possible that the true training effect is smaller than g = .30, which further increases demands on sample size. Third, future studies should employ (a) longer and (b) more varying training durations as well as (c) more consistently include follow-up measurements with (d) varying time lags. Only 9 of the analyzed 33 studies included a follow-up measurement (median time lag, 9.5 days). Effect sizes posttraining were considerably larger (g = 0.31) than at follow-up (g = 0.18). Although nonsignificant, this difference raises concerns about the practical utility of self-control training in the way it has been implemented to date. Researchers may want to consider ways to foster more sustainable self-control training, for example, by reminding participants of the training principles or implementing brief training refreshments after the main training period.

On the theoretical level, self-control training should only lead to performance improvements in activities that actually require self-control for a given person. This is not the case if a person has no goal to control a behavior. In this case, enacting such behaviors does not constitute a self-control failure. People who strive to achieve a certain goal or change a specific behavior—but are unsatisfied with their success in doing so (e.g., alcohol or nicotine consumption, eating behavior)—are the ones who are most likely to profit from a self-control training. For these people, a self-control training may constitute a welcome means to work on the goal and provide a motivational boost by conveying the possibility that the training may help to achieve the respective goal (even if the person has no elaborate idea about how the training may do so). Ideally, a training sets in motion recursive motivational processes that help to build and keep up adaptive routines that may then contribute to lasting changes in behavior (Walton, 2014).

In addition, it will be important to control for differences in expectations about the consequences of a training regimen because different expectations may drive training effects (Boot et al., 2013). Such placebo effects are interesting in their own regard, but they limit researchers’ ability to draw causal conclusions about a proposed working mechanism of self-control training. However, from the perspective of people who are interested in self-control improvements, making progress toward goal attainment is more pressing than identifying the underlying processes. If placebo effects do the trick and do so reliably, one may pragmatically advocate to let them do it. Researchers may interpret such, at first, poorly understood effects as an opportunity to investigate the underlying (motivational) processes in depth and apply this knowledge to new training interventions.

Limitations

The present work suffers from some limitations that future research may want to ameliorate. First, with 33 studies the available evidence on self-control training is still moderate. In light of the analyses presented here, it is premature to draw far-reaching conclusions. Several moderator analyses delivered substantial descriptive differences that did not reach significance, potentially due to low power.

A second limitation is that we could not calculate publication bias-corrected effect size estimates to the extent we had initially planned. Some techniques proved very unsatisfactory in simulation studies in that they severely underestimated true effect sizes under almost all realistic conditions (PET; Gervais, 2015; Inzlicht et al., 2015). Several other recently introduced techniques appear promising (Simonsohn et al., 2014; van Assen, van Aert, & Wicherts, 2015) but cannot be applied in a reasonable way to the current literature. These procedures rely exclusively on significant and published effect sizes with only one reported p value per study entering the computation. For the present meta-analysis, this would have led to an excessive loss of information (see Note 5). Also, they assume a homogeneous distribution of effect sizes, an assumption clearly not valid in the present literature. Future developments in meta-analytic techniques may be able to deliver valid publication bias-corrected effect size estimates for literatures with similar characteristics as the present one.