People routinely make decisions based on predictions made by others (e.g., political pundits, market analysts), so it is in their best interest to identify high-quality forecasts. Experts characterize good forecasting as minimization of continuous error (i.e., predictions close to the eventual outcome). By contrast, the present work reveals that laypeople typically see good forecasts as those that correctly predict an event's categorical outcome (e.g., the winning team). Using within-subjects, between-subjects, and incentive-compatible designs, 15 studies demonstrate this “pick-the-winner-picker heuristic” as well as its psychological mechanism: People evaluate forecasts by assigning separate weights to (1) categorical correctness and (2) continuous error minimization, depending on the overall importance of the categorical and continuous dimensions for that situation. Thus, in the common case when the categorical dimension matters most (e.g., sports contests), people prize forecasts that accurately predicted the categorical outcome (e.g., the winner, not the margin of victory). However, when the categorical dimension's stakes are experimentally reduced, an attenuation is observed. Although this describes how people typically evaluate forecasts, crucially, a dimension's importance is not necessarily related to its diagnosticity of forecaster skill or reliability. Accordingly, the pick-the-winner-picker heuristic may constitute a normative mistake, while framing manipulations help debias judgments.
AmrheinValentinGreenlandSanderMcShaneBlake (2019), “Scientists Rise Up Against Statistical Significance,” Nature, 567 (7748), 305–7.
2.
AndréQuentinReinholtzNicholas (2024), “Pre-Registered Interim Analysis Designs (PRIADs): Increasing the Cost-Effectiveness of Hypothesis Testing,” Journal of Consumer Research, 51 (4), 845–65.
3.
ArkesHal R. (1991), “Costs and Benefits of Judgment Errors: Implications for Debiasing.,” Psychological Bulletin, 110 (3), 486–98.
4.
AtanasovPavelHimmelsteinMark (2023), “Talent Spotting in Crowd Prediction,” in Judgment in Predictive Analytics, SeifertMatthias, ed. Springer, 135–84.
5.
AwtreyEliThornleyNicoDannalsJennifer E.BarnesChristopher M.UhlmannEric Luis (2021), “Distribution Neglect in Performance Evaluations,” Organizational Behavior and Human Decision Processes, 165, 213–27.
6.
BagchiRajeshInceElise Chandon (2016), “Is a 70% Forecast More Accurate Than a 30% Forecast? How Level of a Forecast Affects Inferences About Forecasts and Forecasters,” Journal of Marketing Research, 53 (1), 31–45.
7.
BhuiRahulJiaoPeiran (2023), “Attention Constraints and Learning in Categories,” Management Science, 69 (9), 5394–404.
8.
BotchkarevAlexei (2019), “A New Typology Design of Performance Metrics to Measure Errors in Machine Learning Regression Algorithms,” Interdisciplinary Journal of Information, Knowledge, and Management, 14, 45–76.
9.
BrierGlenn W. (1950), “Verification of Forecasts Expressed in Terms of Probability,” Monthly Weather Review, 78 (1), 1–3.
10.
BroadbentD.E. (1958), Perception and Communication. Pergamon Press.
DawesRobyn M. (1971), “A Case Study of Graduate Admissions: Application of Three Principles of Human Decision Making,” American Psychologist, 26 (2), 180–88.
13.
DawesRobyn M. (1979), “The Robust Beauty of Improper Linear Models in Decision Making,” American Psychologist, 34 (7), 571–82.
14.
DeCosterJamieIselinAnne-Marie R.GallucciMarcello (2009), “A Conceptual and Empirical Examination of Justifications for Dichotomization,” Psychological Methods, 14 (4), 349–66.
15.
DietvorstBerkeley J. (2025), “Understanding People’s Preferences for Predictions: People Prioritize Being Right over Minimizing How Wrong They Are in Expectation,” SSRN(July 4), https://doi.org/10.2139/ssrn.4886144.
16.
DietvorstBerkeley J.BhartiSoaham (2020), “People Reject Algorithms in Uncertain Decision Domains Because They Have Diminishing Sensitivity to Forecasting Error,” Psychological Science, 31 (10), 1302–14.
17.
DillonRobin L.TinsleyCatherine H. (2008), “How Near-Misses Influence Decision Making Under Risk: A Missed Opportunity for Learning,” Management Science, 54 (8), 1425–40.
18.
DingaRichardPenninxBrenda W.J.H.VeltmanDick J.SchmaalLianneMarquandAndre F. (2019), “Beyond Accuracy: Measures for Assessing Machine Learning Models, Pitfalls and Guidelines,” bioRxiv (August 22), https://doi.org/10.1101/743138.
FisherMatthewKeilFrank C. (2018), “The Binary Bias: A Systematic Distortion in the Integration of Information,” Psychological Science, 29 (11), 1846–58.
FoxCraig R.TannenbaumDavidÜlkümenGüldenWaltersDaniel J.ErnerCarsten (2016), “Skill Versus Luck and Attributions of Uncertainty: Validating an Epistemic-Aleatory Rating Scale (EARS),” working paper, UCLA Anderson School of Management.
25.
FuchsEster R.AdlerE. ScottMitchellLincoln A. (2000), “Win, Place, Show: Public Opinion Polls and Campaign Contributions in a New York City Election,” Urban Affairs Review, 35 (4), 479–501.
26.
GaertigCeliaSimmonsJoseph P. (2018), “Do People Inherently Dislike Uncertain Advice?” Psychological Science, 29 (4), 504–20.
GigerenzerGerdToddPeter M. (1999), “Fast and Frugal Heuristics: The Adaptive Toolbox,” in Simple Heuristics That Make Us Smart. Oxford University Press, 3–34.
29.
GneitingTilmann (2011), “Making and Evaluating Point Forecasts,” Journal of the American Statistical Association, 106 (494), 746–62.
30.
GneitingTilmannRafteryAdrian E. (2007), “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102 (477), 359–78.
31.
GodesDavid (2025), “Will the Truth Free Us from Misinformation?” Johns Hopkins Carey Business School Research Paper No. 24-08, SSRN (March 4), https://doi.org/10.2139/ssrn.4606265.
32.
GoodellJohn W.VähämaaSami (2013), “US Presidential Elections and Implied Volatility: The Role of Political Uncertainty,” Journal of Banking & Finance, 37 (3), 1108–17.
33.
GranzierRiakoPonsVincentTricaudClemence (2023), “Coordination and Bandwagon Effects: How Past Rankings Shape the Behavior of Voters and Candidates,” American Economic Journal: Applied Economics, 15 (4), 177–217.
34.
GrimmerJustinKnoxDeanWestwoodSean (2024), “Assessing the Reliability of Probabilistic US Presidential Election Forecasts May Take Decades,” OSF Preprints(August 26), https://osf.io/6g5zq.
35.
HalawiDannyZhangFredYueh-HanChenSteinhardtJacob (2024), “Approaching Human-Level Forecasting with Language Models,” arXiv (February 28), https://doi.org/10.48550/arXiv.2402.18563.
HimmelsteinMarkZhuSophie MaPetrovNikolayKargerEzraHelmerJessicaLivnatSivanBennettAmoryHedleyPageTetlockPhil (2024), “The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability,” PsyArXiv (November 21), https://osf.io/a7kdx.
38.
HodsonTimothy O. (2022), “Root-Mean-Square Error (RMSE) or Mean Absolute Error (MAE): When to Use Them or Not,” Geoscientific Model Development, 15 (14), 5481–87.
39.
HogarthRobin M.KarelaiaNatalia (2005), “Ignoring Information in Binary Choice with Continuous Variables: When Is Less ‘More’?” Journal of Mathematical Psychology, 49 (2), 115–24.
JohnsonSamuel G.B.MerchantThomasKeilFrank C. (2020), “Belief Digitization: Do We Treat Uncertainty as Probabilities or as Bits?” Journal of Experimental Psychology: General, 149 (8), 1417–34.
44.
JoseVictor Richmond R.NauRobert F.WinklerRobert L. (2009), “Sensitivity to Distance and Baseline Distributions in Forecast Evaluation,” Management Science, 55 (4), 582–90.
45.
KahnemanDaniel (1973), Attention and Effort. Prentice Hall.
46.
KahnemanDanielFrederickShane (2002), “Representativeness Revisited: Attribute Substitution in Intuitive Judgment,” in Heuristics and Biases: The Psychology of Intuitive Judgment, GilovichThomasGriffinDaleKahnemanDaniel, eds. Cambridge University Press, 49–81.
KoutsandreasDiamantisSpiliotisEvangelosPetropoulosFotiosAssimakopoulosVassilios (2022), “On the Selection of Forecasting Accuracy Measures,” Journal of the Operational Research Society, 73 (5), 937–54.
LefgrenLarsPlattBrennanPriceJoseph (2015), “Sticking with What (Barely) Worked: A Test of Outcome Bias,” Management Science, 61 (5), 1121–36.
52.
LohrSteveSingerNatasha (2016), “How Data Failed Us in Calling an Election,” New York Times(November 10), B1, https://nyti.ms/2eES827.
53.
MartignonLauraHoffrageUlrich (2002), “Fast, Frugal, and Fit: Simple Heuristics for Paired Comparison,” Theory and Decision, 52 (1), 29–71.
54.
MartignonLauraKatsikopoulosKonstantinos V.WoikeJan K. (2008), “Categorization with Limited Resources: A Family of Simple Heuristics,” Journal of Mathematical Psychology, 52 (6), 352–61.
55.
McKenzieCraig R.M.AminMarsha B. (2002), “When Wrong Predictions Provide More Support Than Right Ones,” Psychonomic Bulletin & Review, 9 (4), 821–28.
56.
MeehlPaul E. (1954), Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. University of Minnesota Press.
57.
MislavskyRobertGaertigCelia (2024), “Premature Predictions: Accurate Forecasters Are Not Viewed as More Competent for Earlier Predictions,” Journal of Experimental Psychology: General, 153 (1), 159–70.
58.
MolinaRenatoRudikIvan (2022), “The Social Value of Hurricane Forecasts,” National Bureau of Economic Research Working Paper w32548, https://www.ssrn.com/abstract=4859400.
59.
MorayN. (1967), “Where Is Capacity Limited? A Survey and a Model,” Acta Psychologica, 27, 84–92.
60.
MoriartyMark M.AdamsArthur J. (1984), “Management Judgment Forecasts, Composite Forecasting Models, and Conditional Efficiency,” Journal of Marketing Research, 21 (3), 239–50.
PachauriRajendra K.MeyerLeo A., eds. (2014), “Climate Change 2014: Synthesis Report,” contribution of Working Groups I, II, and III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, IPCC, Geneva, Switzerland, https://archive.ipcc.ch/report/ar5/syr/.
64.
PayneJohn W.BettmanJames R.JohnsonEric J. (1988), “Adaptive Strategy Selection in Decision Making,” Journal of Experimental Psychology: Learning, Memory, and Cognition, 14 (3), 534–52.
65.
PricePaul C.StoneEric R. (2004), “Intuitive Evaluation of Likelihood Judgment Producers: Evidence for a Confidence Heuristic,” Journal of Behavioral Decision Making, 17 (1), 39–57.
ReynaValerie F. (2012), “Risk Perception and Communication in Vaccination Decisions: A Fuzzy-Trace Theory Approach,” Vaccine, 30 (25), 3790–97.
68.
ReynaValerie F.BrainerdCharles J. (1991), “Fuzzy-Trace Theory and Framing Effects in Choice: Gist Extraction, Truncation, and Conversion,” Journal of Behavioral Decision Making, 4 (4), 249–62.
69.
ReynaValerie F.BrainerdCharles J. (1995), “Fuzzy-Trace Theory: An Interim Synthesis,” Learning and Individual Differences, 7 (1), 1–75.
70.
ReynaValerie F.BrainerdCharles J. (2011), “Dual Processes in Decision Making and Developmental Neuroscience: A Fuzzy-Trace Model,” Developmental Review, 31 (2–3), 180–206.
71.
ReynaValerie F.NelsonWendy L.HanPaul K.DieckmannNathan F. (2009), “How Numeracy Influences Risk Comprehension and Medical Decision Making,” Psychological Bulletin, 135 (6), 943–73.
72.
RoystonPatrickAltmanDouglas G.SauerbreiWilli (2006), “Dichotomizing Continuous Predictors in Multiple Regression: A Bad Idea,” Statistics in Medicine, 25 (1), 127–41.
73.
ShahAnuj K.OppenheimerDaniel M. (2008), “Heuristics Made Easy: An Effort-Reduction Framework,” Psychological Bulletin, 134 (2), 207–22.
SimonHerbert A. (1990), “Invariants of Human Behavior,” Annual Review of Psychology, 41 (1), 1–20.
76.
SlovicPaul (1975), “Choice Between Equally Valued Alternatives,” Journal of Experimental Psychology: Human Perception and Performance, 1 (3), 280–87.
77.
SollJack B.MilkmanKatherine L.PayneJohn W. (2015), “A User’s Guide to Debiasing,” in The Wiley Blackwell Handbook of Judgment and Decision Making, KerenGideonWuGeorge, eds. Wiley, 924–51.
78.
SongChiUngBoulierBryan L.SteklerHerman O. (2009), “Measuring Consensus in Binary Forecasts: NFL Game Predictions,” International Journal of Forecasting, 25 (1), 182–91.
79.
TeigenKarl HalvorLøhreErikHohleSigrid Møyner (2018), “The Boundary Effect: Perceived Post Hoc Accuracy of Prediction Intervals,” Judgment and Decision Making, 13 (4), 309–21.
80.
TetlockPhilip E.GardnerDan (2015), Superforecasting: The Art and Science of Prediction. Crown Publishers.
81.
TsetsosKonstantinosChaterNickUsherMarius (2012), “Salience Driven Value Integration Explains Decision Biases and Preference Reversal,” Proceedings of the National Academy of Sciences, 109 (24), 9659–64.
82.
TverskyAmos (1972), “Elimination by Aspects: A Theory of Choice,” Psychological Review, 79 (4), 281–99.
83.
TverskyAmosKahnemanDaniel (1974), “Judgment Under Uncertainty: Heuristics and Biases: Biases in Judgments Reveal Some Heuristics of Thinking Under Uncertainty,” Science, 185 (4157), 1124–31.
84.
TverskyAmosSattathShmuelSlovicPaul (1988), “Contingent Weighting in Judgment and Choice,” Psychological Review, 95 (3), 371–84.
YanivIlanFosterDean P. (1995), “Graininess of Judgment Under Uncertainty: An Accuracy-Informativeness Trade-Off,” Journal of Experimental Psychology: General, 124 (4), 424–32.
90.
YanivIlanFosterDean P. (1997), “Precision and Accuracy of Judgmental Estimation,” Journal of Behavioral Decision Making, 10 (1), 21–32.
91.
YatesJ. FrankPricePaul C.LeeJu-WheiRamirezJames (1996), “Good Probabilistic Forecasters: The ‘Consumer’s’ Perspective,” International Journal of Forecasting, 12 (1), 41–56.
92.
YeungSaiwing (2014), “Framing Effect in Evaluation of Others’ Predictions,” Judgment and Decision Making, 9 (5), 445–64.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
