The Connection Is in the Data

Conference Abstracts as an Indicator of Publication Bias

Bialystok and her colleagues argue first that our analysis of conference abstracts is not a reliable measurement of publication bias. This method, however, is neither new (e.g., Scherer, Dickersin, & Langenberg, 1994) nor rare (e.g., Song et al., 2009); it has been widely used in, among other disciplines, epidemiology (e.g., Petticrew et al., 2008), health technology (e.g., Dundar et al., 2006), stroke research (e.g., Brazzelli, Lewis, Deeks, & Sandercock, 2009), medical interventions (e.g., Peinemann, McGauran, Sauerland, & Lange, 2008), pediatrics (e.g., Zamakhshary, Abuznadah, Zacny, & Giacomantonio, 2006), orthopedics (e.g., Harris, Mourad, Kadir, Solomon, & Young, 2006), and veterinary medicine (e.g., Snedeker, Totton, & Sargeant, 2010), and it is endorsed by the Cochrane Collaboration (Scherer, Langenberg, & von Elm, 2007; Young & Hopewell, 2011). Hence, we are not the first to use this method to examine a potential publication bias.

It is true that conference abstracts may include only preliminary results or interpretations that are corrected in final articles and that publications therefore may be argued to be improved versions of abstracts. Moreover, studies described in conference abstracts may not be published, because they present only preliminary data with small sample sizes or because they were meant to be pilot studies of a new task. Indeed, we found that half of all studies described in the conference abstracts were not published. We do not know why several studies were not eventually published. Crucially, we carefully analyzed potential background differences between abstracts supporting and abstracts challenging bilingual-advantage theories to ensure that null or negative results were not less likely to be published because of quality differences. We did not find any differences in terms of year of conference abstract, type of tasks used, number of participants, or power to detect an effect. This strongly suggests that it was not the quality of the study that led to publication differences between positive and null or negative results.

Bialystok and her colleagues then argue that in order to determine the exact rejection rate, it would be valuable to determine whether unpublished conference results had been submitted to a journal. We agree that this could have added value and, as already underlined in our article, we sent out a short ad hoc questionnaire to all first authors of unpublished abstracts. Unfortunately, 33 of the 52 authors contacted did not reply or refused to fill in the questionnaire. Of the 6 respondents from abstracts supporting a bilingual advantage, 2 indicated that they had not submitted their results to a journal. Of the 13 respondents from abstracts challenging an advantage, 8 indicated that they had not submitted their results. This suggests that more than half of the null or negative findings had not been submitted. However, we clearly do not have enough data to draw reliable conclusions, and in our article, we acknowledged that we do not know at which level this discrepancy in publication chances occurred.

Bilingual Disadvantages in Conference Abstracts

The second point raised by Bialystok and her colleagues concerns the relatively low number of studies finding a bilingual disadvantage in our overview of abstracts. In our article, we clearly acknowledged that an equal number of disadvantages and advantages should be found if there is no advantage at all. Nevertheless, it is difficult to assess whether bilingual disadvantages are rarely found or rather results showing such disadvantage are not even submitted to conferences: Indeed, in the absence of a theoretical motivation for monolingual executive-control advantages, a bilingual disadvantage is likely to be interpreted as the result of a Type I error.

Bilingual Effects on Verbal Executive-Control Tasks

Bialystok and her colleagues’ third comment concerns the inclusion of tasks with verbal materials. They argue that bilinguals may have a disadvantage on purely lexical tasks. However, as is clear from our article (p. 100), we did not include conference abstracts that discuss lexical tasks without an executive-control component. We did include executive-control tasks with verbal materials, in which, according to the studies mentioned by Bialystok et al. themselves (e.g., Bialystok, 2009), bilinguals should outperform monolinguals. These tasks have revealed no bilingual effects in some studies (e.g., Paap & Liu, 2014) but bilingual advantages in others (e.g., Bialystok, Craik, & Luk, 2008; Filippi, Leech, Thomas, Green, & Dick, 2012). Accordingly, we do not see good theoretical reasons to exclude verbal tasks. However, when we do exclude all studies using verbal materials from our analysis (leaving 78 abstracts and 125 tasks in the funnel plot shown in Fig. S1; see the original Supplemental Material available online), the results remain the same. The funnel plot still shows an asymmetry (z = 6.61, p < .0001), and the publication rates are similar to the ones found in the original analysis: 70% of studies fully supporting an advantage were published, compared with 55% of the mainly supporting, 41% of the mainly challenging, and 23% of the fully challenging studies. The outcome is thus not affected by the nature of the materials.

Prejudices in Bilingual-Advantage Research

Bialystok and her colleagues state that bilingual advantages have been found in “a recent upsurge of research” (p. 945) showing that “these changes are real” (p. 945). We agree that there are many interesting and compelling data supporting an advantage (e.g., Pliatsikas, Moschopoulou, & Saddy, 2015). Yet evidence for a bilingual advantage is far from accepted wisdom. Indeed, the number of studies challenging such an advantage has recently been increasing (e.g., Lawton, Gasquoine, & Weimer, 2014; Paap, Johnson, & Sawi, 2014).

Publication biases may be worsened by researchers’ own prejudices or agenda. Bialystok and her colleagues have doubted the value of null results, asserting that “the considerable literature that reports group differences between monolingual and bilingual participants is greatly more informative than the attempted replications that fail to find significance” (Kroll & Bialystok, 2013, p. 502). In their Commentary, they go even further and claim, “imagine the state of journals if [studies that show some effect and no effect] were published with the same frequency” (p. 944). We disagree with the logic underlying this supposition. If one wants to understand the nature of a phenomenon, such as the potential bilingual advantage, one needs to have access to all good studies regardless of their outcome (cf. Spellman, 2012). In their title, Bialystok and her colleagues question the connection between biases and evidence. The connection is in the data; we should consider them all.