Quantifying Accuracy Improvement in Sets of Pooled Judgments

Results

For each participant and item, i, H&H subtracted the absolute difference between the average of the participant’s two responses, R ¯ , and the true response, T, from the absolute difference between the first response, R₁, and T; the result of this subtraction of difference scores was normalized using the absolute difference between R₁ and T. The median of these 40 normalized difference scores was the participant’s accuracy change score, A_diff:

A diff = M d n i = 1 i = 40 ( | R 1 , i − T i | − | R i ¯ − T i | | R 1 , i − T i | )

The mean value of A_diff was significantly higher in the dialectical-bootstrapping condition (M = 0.046, SE = 0.008) than in the control condition (M = 0.010, SE = 0.008), t(99) = 3.12, p = .002.

H&H did not report a further difference between the conditions that was confounded with A_diff. In the dialectical-bootstrapping condition, the second response (R₂) to an item matched the participant’s first response to the item only 0.7% of the time (SE = 1.4%). In the control condition, participants’ first and second responses matched 20.2% (SE = 1.3%) of the time, significantly more often than in the dialectical-bootstrapping condition, t(99) = 10.3, p < .001. When R₁ equals R₂, A_diff is 0, and in fact, the median A_diff in the control condition was 0 for 29 of the 51 participants; thus, the mean A_diff across all participants was close to 0. In the dialectical-bootstrapping condition, only 2 of the 50 participants had a median A_diff of 0. The confounding of these two measures is reflected in the significant correlation between the proportion of identical first and second responses (i.e., p(R₁ = R₂)) and A_diff, r(99) = −.307, p = .002.

We prefer to measure accuracy change independently of the proportion of identical first and second responses. Instead of using a median value of the differences in accuracy, such as A_diff, we analyzed a pair of median accuracy values for each participant. In addition, we used absolute measures of accuracy instead of relative (normalized) measures because in hindsight-bias research, relative measures that have similarities to A_diff have “awkward statistical properties” (Pohl, 2007, p. 22).

For each participant, we took a pair of values: the median absolute error of R₁ across the 40 items, A₁, and the median absolute error of R ¯ , A_avg:

A 1 = M d n i = 1 i = 40 ( | R 1 , i − T i | )

A avg = M d n i = 1 i = 40 ( | R 1 , i − T i | )

We analyzed these data using a mixed-design analysis of variance including the independent variables of response (R₁ vs. R ¯ ; within subjects) and condition (dialectical bootstrapping vs. control; between subjects). R ¯ (M = 125.6, SE = 4.8) was significantly more accurate than R₁ (M = 130.7, SE = 4.6), F(1, 99) = 14.8, p < .001. More important, the response-by-condition interaction was not significant, F(1, 99) = 0.00, p = .98: The magnitude of the effect in the dialectical-bootstrapping condition, d = 0.38, was not significantly different from the magnitude of the effect in the control condition, d = 0.39.

These paired accuracy measures (A₁ and A_avg) provide no support for the effectiveness of the dialectical-bootstrapping instructions beyond that of the control instructions. Tellingly, the accuracy gain shown by these paired accuracy measures (i.e., A₁ – A_avg) was unrelated to p(R₁ = R₂), r(99) = –.003, p = .97. A robust Wald test showed that this correlation was significantly lower than that between p(R₁=R₂) and A_diff, χ²(1, N = 101) = 4.85, p = .03. Means for all of the measures we have discussed are shown in Table 1.

Results for an additional control condition and several alternative accuracy measures are included in the Supplemental Material available online. Every accuracy measure that was similar to A_diff in that the median of a set of difference scores was used (whether the value was normalized for item difficulty, as in A_diff, or not) was significantly correlated with p(R₁=R₂), and every variation that was similar to A₁ and A_avg in using a set of paired accuracy scores for each person (again, whether the values were normalized or not) was not significantly correlated with p(R₁ =  R₂). Only the accuracy measures that were correlated with p(R₁=  R₂) showed significant differences between the conditions.

OPEN IN VIEWER

Table 1.

Mean Values for Various Dependent Measures

	Measure
Condition	A 1	A avg	A1 – Aavg	A diff	p(R1 = R2)
Dialectical bootstrapping	130.9	125.8	5.1	0.046	.007
Control	130.5	125.4	5.1	0.010	.202

Note: R₁ = first response; R₂ = second response; R ¯ = average of R₁ and R₂; A₁ = median absolute error of R₁; A_avg = median absolute error of R ¯ ; A_diff = median relative difference between the accuracy of R₁ and the accuracy of R ¯ ; p(R₁ = R₂) = proportion of items for which R₁ and R₂ were identical.

Discussion

H&H concluded that the accuracy gained by making a second response and then averaging the pair of responses was significantly greater for participants in the dialectical-bootstrapping condition than for those in the control condition. This conclusion was based on an accuracy change measure that was confounded with the proportion of identical first and second responses. Participants in the dialectical-bootstrapping condition were instructed to “assume that your first estimate is off the mark” and to “make a second, alternative estimate” (H&H, p. 234). Observing a difference between the conditions when using a measure that was confounded with the difference in the proportion of identical responses therefore served only as a manipulation check.

Using measures that are independent of each other is important in many fields of research. For example, in hindsight-bias research, measures of the percentage of perfect recall must be separated from measures of retrieval bias (Pohl, 2007).

When the data were analyzed using measures of accuracy that were uncorrelated with the proportion of identical first and second responses, the difference between the conditions disappeared. People may have some awareness of when they cannot improve upon their first response, and in these cases, they will change their response only if explicitly instructed to do so. There is no evidence in H&H’s data that encouraging people to alter their responses more often than they would if not given special instructions yields more accurate average responses. Dialectical instructions are not needed to achieve the wisdom of many in one mind.