The Crowd Within and the Benefits of Dialectical Bootstrapping

Dialectical Bootstrapping: Does It Have Surplus Value?

We defined the gain obtained by averaging the responses of a given participant as the “median decrease in error of the average of the two estimates relative to the first estimate” (H&H, p. 234). W&A criticized this accuracy change measure. First, they reported that participants’ first and second estimates in our reliability condition ¹ were, on average, identical in 20% of cases, but that first and second estimates were identical in merely 1% of cases in our dialectical condition. Furthermore, W&A reported that our accuracy change measure was confounded with the proportion of identical first and second estimates. Second, W&A noted that they “prefer to measure accuracy change independently of the proportion of identical first and second responses” (p. 115). Third, they noted that our measure has “awkward statistical properties” (p. 115). Finally, when they used a measure that decoupled accuracy change and the proportion of identical responses, accuracy gains did not differ between our dialectical and reliability conditions. From this, W&A concluded that there is no evidence that “encouraging people to alter their responses more often than they would if not given special instructions yields more accurate average responses” (p. 116).

There are various accuracy (and accuracy change) measures, and opinions about their respective merits differ—because of statistical considerations (Armstrong & Collopy, 1992) or because different measures imply different loss functions (Winkler, 2003). Using a robust measure of accuracy change on the item level, we found that dialectical bootstrapping results in accuracy gains that go beyond reliability gains; using a measure of accuracy change at the participant level, W&A found no such advantage. But how persuasive are W&A’s two key reasons to prefer their measure?

First, an alleged weakness of our measure is that it has awkward statistical properties (p. 115)—presumably because it includes a ratio of two variables (W&A cited p. 22 of Pohl, 2007, where Pohl discussed the “awkward statistical properties” of a “quotient of two variables”). W&A, however, also analyzed two variants of our measure that employed either a more reliable denominator or no denominator, and in both cases, the findings obtained with our original measure were confirmed (see their online Supplemental Material).

Second, W&A conjectured that it is better to measure accuracy change independently of the proportion of identical responses, but what is wrong when a genuine psychological fact—that people hesitate to alter their opinion—enters the accuracy analysis? W&A motivated their independence requirement by reference to hindsight-bias research, in which cases of perfect recall must be separated from cases of reconstruction (e.g., Hoffrage, Hertwig, & Gigerenzer, 2000)—because hindsight bias can occur only in the latter cases. This analogy with the hindsight bias, however, is misleading. The goal of dialectical bootstrapping is to increase accuracy by fostering independence between repeated estimates, and the low proportion of identical estimates indicates that this goal was met. Therefore, we disagree with W&A’s stipulation that a measure that gauges accuracy change independently of the proportion of identical responses is preferable.

Conclusion

W&A and we agree that the crowd within works. The question is whether, when, and how dialectical bootstrapping can foster its potential. We are grateful for W&A’s comments. Their reanalysis highlights the necessity of including different accuracy measures in future studies and analyzing whether the results obtained using them converge (and if not, why not). When there is no good reason to prefer one measure over others, aggregating them may be one solution to their plurality (Armstrong & Collopy, 1992, p. 75).

Does dialectical bootstrapping improve accuracy beyond mere reliability gains? Clearly, as W&A showed, when one employs the consider-the-opposite strategy (as in H&H), the advantage of dialectical bootstrapping depends on the accuracy measure. This, however, should not be taken as a general verdict on the dialectical-bootstrapping framework, which we explicitly did “not confine . . . to the consider-the-opposite strategy” (H&H, p. 236). There are many ways to leverage “people’s capacity to construct conflicting realities” (H&H, p. 236), and thus to achieve dialectical bootstrapping. For instance, we are currently exploring the extent to which averaging different non-Bayesian strategies makes people more Bayesian and the extent to which averaging holistic and analytical judgments improves accuracy. The dialectical-bootstrapping framework poses a wealth of questions, including questions concerning how to design successful and robust dialectical techniques, the ecological conditions under which dialectical bootstrapping pays, and whether people intuitively use this strategy. The work on how and when to poll the crowd in one’s head has just begun.