Comment: Alternatives to Wood et al.’s Conclusions

Using sophisticated techniques, such as Egger’s test, Duval and Tweedie’s trim-and-fill procedure, and Ferguson and Brannick’s tandem method, Wood et al. (2014) find good evidence of publication bias for studies of symmetry preferences, but not for masculinity, dominance, and health preferences. Consequently, their conclusion that evidence for cyclic shifts in these latter preferences is an artifact of publication bias relies heavily on comparisons of effect sizes in published and unpublished studies. However, larger effect sizes in published than unpublished studies are expected if a true effect exists and better (e.g., more rigorous or theoretically compelling) studies are more likely to be assessed by peer review as warranting publication. After all, although publication bias certainly occurs in science, effect size and significance are not the only criteria for editorial decisions.

Wood et al. (2014) attempt to minimize concerns about the quality of their unpublished studies by testing for effects of aspects of methodological quality that they coded. While testing for effects of methodological differences among studies is important, many of these analyses have low power or employed problematic codings. For example, many studies that experimentally manipulated stimuli along a single physical dimension reported significant (i.e., above-chance) discrimination along that dimension in either critical conditions or overall. Although these results show, by definition, that the manipulations were detectable, Wood et al. coded these studies as showing no clear evidence that the manipulation could be detected (the same code given to studies reporting explicit evidence that stimulus manipulations could not be detected, e.g., Cavanaugh, 2008). Additionally, methodological problems with some of the studies suggest they should be excluded from the meta-analysis completely. For example, studies that asked women to trade off preferences for several different attributes are inappropriate tests for cyclic shifts in preferences for individual traits because preferences for each of the traits included are, by definition, confounded.

Wood et al.’s (2014) conclusion that evidence for cyclic shifts in mate preferences are research artifacts also relies heavily on a series of meta-regressions. However, the subjective decisions necessary when conducting meta-regressions can be problematic and require clear justification (Thompson & Higgins, 2002). This is not the case in Wood et al. where careful examination of Figure 2C reveals that effect sizes from different conditions in studies that manipulated relationship context were averaged. For one such study (Jones et al., 2005, Study 3), although cyclic shifts differed significantly between contexts, the between-phase effect sizes for the long-term (g = 0.20) and the short-term (g = −0.47) conditions were averaged to produce an effect size of g = −0.135 that is unrepresentative of either effect. Problematic decisions like this could influence the meta-regressions’ findings and should, at least, be more transparent.

Concerns like these are particularly worrying in light of the results of another meta-analysis of cyclic shifts in mate preferences (Gildersleeve, Haselton, & Fales, in press). This meta-analysis also includes a large number of published and unpublished studies but reports significant overall effects of fertility on masculinity and dominance preferences, finding little evidence that publication bias can account for the significant overall effects. Studies in which fertility was a continuous variable also show an overall effect of fertility, suggesting that cyclic effects are not simply an artifact of undisclosed degrees of freedom in how researchers allocate days to high-fertility and low-fertility conditions.

Because Gildersleeve et al. (in press) did not examine cyclic shifts in disease avoidance, I will briefly comment on Wood et al.’s (2014) conclusions regarding health preferences. Wood et al. report significant overall effects of cycle phase on health preferences when published and unpublished studies were combined in a single analysis and when published studies were analyzed separately. They report no significant effect of cycle phase when unpublished studies were analyzed separately, although effect sizes in unpublished and published studies did not differ significantly (p = .096). They dismiss the significant overall effect because they claim to show clear evidence of publication bias.

Wood et al. (2014) conducted a comprehensive series of publication bias tests on all traits, including Egger’s test, Duval and Tweedie’s trim-and-fill procedure, and Ferguson and Brannick’s tandem method. For health preferences, they report an effect for Egger’s test that they describe as marginally significant (p = .095) and that no other tests were significant. This is arguably weak evidence for publication bias, at best, particularly since they interpret similar p-values as null results elsewhere (e.g., when discussing Roney, Simmons, & Gray [2011] and Swaddle & Reierson [2002]).

Wood et al. (2014) also report a significant relationship between number of days included in the fertile phase and cycle effect sizes from health preference studies. Given the original rationale for cyclic shifts in health preferences (that disease avoidance tracks changes in progesterone, not fertility; Jones et al., 2005), it is unclear why this result is problematic. Because progesterone is uniformly low during the follicular phase and rises only after ovulation (Gilbert, 2000), including more days from the early follicular phase will increase the probability that women tested in this phase are correctly allocated to the low-progesterone condition. Given that some research also implicates progesterone in cyclic shifts in masculinity (Puts, 2006) and symmetry (Garver-Apgar, Gangestad, & Thornhill, 2008) preferences, this might also explain the relationships Wood et al. report between number of days included in the fertile phase and cycle effect sizes for masculinity and symmetry preferences.

Together, these points suggest that it is unreasonable to discount the significant overall effect of cycle phase for health preferences, particularly given converging evidence for links between progesterone and protective behaviors (e.g., Conway et al., 2007; Derntl, Hack, Kryspin-Exner, & Habel, 2013; Derntl, Kryspin-Exner, Fernbach, Moser, & Habel, 2008; Fleischman & Fessler, 2011).