Drawing Generalizable Conclusions From Multilevel Models: Commentary on Van de Calseyde and Efendić (2022)

Reanalysis of presented models

Van de Calseyde and Efendić (2022) present findings from multilevel models that include condition as a lone predictor and random intercepts for participants and items. Recall that condition was manipulated between participants, and so random participant-level intercepts are appropriate in that case. However, effects of condition are present across items, and so maximal random effects should include item-level random slopes for condition. For these data, models with item-level slopes trigger singularity warnings in the lme4 package (Bates et al., 2015) in R statistical software, which was used by Van de Calseyde and Efendić. However, such models converge without issue in the Bayesian brms package (Bürkner, 2017), which allows users to specify models in lme4 syntax and subsequently estimate parameters via Markov chain Monte Carlo algorithms (most notably, the No-U-Turn Sampler [NUTS]; Hoffman & Gelman, 2014) in Stan probabilistic language (Stan Development Team, 2022). Users may additionally declare custom priors and incorporate other elements into their models that are not possible in lme4 (see the next subsection on the omnibus analysis), but my primary aim is to demonstrate the usefulness of Bayesian estimation even in the absence of those additional specifications. (In a departure from lme4 syntax, I did have to adjust the “iter” argument, for analyses pertaining to Experiment 1a, and “adapt_delta” argument, for analyses pertaining to the remaining experiments, in brms to facilitate model convergence and prevent divergent transitions [Betancourt, 2016] for the models in Table 1.)

Table 1 presents the results of a reanalysis of the Van de Calseyde and Efendić (2022) data with Bayesian models that contain random intercepts or else a maximal random-effects structure (i.e., random intercepts for participants and random intercepts and slopes for items; note that maximal random effects also entail estimates of pairwise correlations between the intercept and slopes corresponding to a given grouping variable). My reanalyses entailed all models that evaluated the key claim that adopting a disagreeing perspective enhances inner-crowd wisdom relative to the self and agree conditions. These models were estimated for each individual experiment as well as an aggregation of data from Experiments 2, 3, and 4 (see footnote 5 in the original article).

The third column of Table 1 contains p values from the random-intercept models that are reported by Van de Calseyde and Efendić (2022). The fourth column displays 95% credible intervals (CIs) from random-intercept models estimated in Stan via the brms package. Note that the results reported by Van de Calseyde and Efendić are taken from models that are fit to subsets of the data that are relevant to each comparison of interest rather than to all the data from a given experiment. In contrast, my reanalyses estimated all the effects of interest simultaneously. (Estimating effects from all the data is the more appropriate choice because the full data set is necessary for estimating item-level intercepts and slopes.) This difference in data sets yields only one inconsistent outcome, which is for the agree versus self condition in Experiment 3. However, a model fit to all the data from Experiment 3 in the lme4 package replicates the pattern of results from my brms model (see notes for Table 1). Thus, the random-intercept models that are estimated in brms all yield the same patterns of findings that are found with lme4.

The key outcomes of interest are displayed in the final column of Table 1, which contains 95% CIs from models with maximal random-effects structures. Again, the difference between these models and the random-intercept models is the addition of a random slope of condition for items. The use of maximal random effects removes every positive result that is reported by Van de Calseyde and Efendić (2022)—that is, for every analysis that found a benefit of the disagree condition relative to either control condition, that benefit is no longer present.

Why does the addition of random slopes so strongly impact the inferences drawn from these models? It is important to note that the uncertainty of the estimates increases substantially in the maximal models (see Tables S2 and S3 in the Supplemental Material for posterior means and standard deviations). This increased uncertainty arises from the fact that the effect of experimental condition at the item level is quite inconsistent, as can be seen in Figure S1 in the Supplemental Material, and the inconsistency of this effect is captured only by the maximal models. This sensitivity of the maximal models to the inconsistency of the experimental manipulation consequently introduces substantially more uncertainty into the population-level estimates relative to those from the random-intercept models.

Omnibus analysis

As a final analysis, I combined the data from all the experiments to evaluate whether the disagree condition might generate superior inner-crowd wisdom relative to the self condition when all the relevant data are analyzed simultaneously. This analysis proceeded differently than for the ones that are summarized in Table 1. First, I used mean-squared proportional error (MSPE) as my outcome of interest to account for the different scales of the items (i.e., deviations between guesses and a true value are divided by the true value prior to squaring; Fiechter & Kornell, 2021; Lorenz et al., 2011; Rauhut & Lorenz, 2011; van Dolder & van den Assem, 2018). Second, I modeled MSPE across participants’ two guesses (see preceding citations as well as Vul & Pashler, 2008) rather than differential MSPE. This model therefore had three predictors: guess, experimental condition, and the guess-by-condition interaction (this last predictor gauged the differential reduction in error over guesses and was of primary interest). The choice to model MSPE across guesses arose from the fact that differential squared error is not described particularly well by any readily implemented linear or generalized linear model. In contrast, MSPE is better described by a hurdle-gamma model (i.e., a gamma model that allows zeros in the response variable; Heilbron, 1994) than is differential MSE by an ordinary linear model (R² = .29 for my omnibus models versus .04 to .06 for the models in Table 1; R² is calculated according to Gelman et al., 2019).

Third, in addition to the group-level effects estimated for participants and items, I also estimated (a) random intercepts and slopes for each experiment as well as (b) separate participant-level effects for people assigned to either experimental condition. These choices were both motivated by implementing maximal random effects. Fourth, I evaluated a set of three Gaussian priors (µ = 0; σ = {0.5, 1, 2}) on all population-level effects across three different models. These weakly informative priors allowed me to calculate Bayes factors via Savage-Dickey ratios (Wagenmakers et al., 2010) for each effect of interest. One benefit of Bayes factors over 95% CIs is that they allow analysts to quantify evidence in favor of either the alternative or null hypothesis. I will present Bayes factors as a ratio of evidence in favor of the null relative to the alternative (i.e., BF₀₁). Following recommendations from Gallistel (2009), I evaluated whether BF₀₁ ever suggests support for the alternative hypothesis across these three reasonable priors (see Figure S3 in the Supplemental Material for how these priors correspond to variance in participant means); to consistently find evidence in favor of the null (i.e., BF₀₁ > 1) suggests convincing support for that hypothesis.

All estimates from these hurdle-gamma models are presented in Table S4 in the Supplemental Material. For present purposes, it is sufficient to note that every Bayes factor for the guess-by-condition interaction supports the null hypothesis (from narrowest to widest prior, BF₀₁ = {2.95, 6.16, 11.44}; also see Figure S2 in the Supplemental Material). Thus, after aggregating every relevant data point from Van de Calseyde and Efendić (2022), I am still unable to find any generalizable evidence to support the claim that adopting a disagreeing perspective enhances inner-crowd wisdom relative to making a second guess without perspective taking. In fact, a set of Bayes factors that is calculated across three reasonable priors supports the conclusion that there is no difference between those conditions.