Large AI Models Have a Prioritization Problem: Policy Implications and Solutions

Abstract

Artificial intelligence (AI) models trained on large corpora must prioritize some information over others. Distilling vast data into simple user-friendly representations can lead to a prioritization problem, in which large AI models neglect crucial information in their outputs. This prioritization problem manifests for newsfeed-ranking algorithms and generative AI such as large language models (LLMs). This paper discusses two flavors of this problem: (1) models trained to prioritize coherence or engagement may inadvertently prioritize misleading information over accurate information, and (2) models trained to prioritize prevalent information will underrepresent the heterogeneity of voices, reflecting cultural perspectives that dominate the training data. The prioritization problem can critically undermine knowledge, but it is not inevitable. Policymakers can address the prioritization problem by modifying the training material, training objectives, institutional safeguards, and messaging around large AI models.

Keywords

artificial intelligence large language models culture social media technology

Get full access to this article

View all access options for this article.

References

Open AI, Achiam

Adler

Agarwal

Ahmad

Akkaya

Aleman

F. L.

Almeida

Altenschmidt

Altman

Anadkat

Avila

Babuschkin

Balaji

Balcom

Baltescu

Bao

Bavarian

Belgum

Zoph

(2024). GPT-4 Technical Report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774

Agarwal

Beygelzimer

Dudík

Langford

Wallach

(2018). A reductions approach to fair classification. International Conference on Machine Learning, 80, 60–69. https://doi.org/10.48550/arXiv.1803.02453

Atari

Xue

M. J.

Park

P. S.

Blasi

Henrich

(2023). Which humans? https://osf.io/preprints/psyarxiv/5b26t/

Berman

Katona

(2020). Curation algorithms and filter bubbles in social networks. Marketing Science, 39(2), 296–316. https://doi.org/10.1287/mksc.2019.1208

Binz

Akata

Bethge

Brändle

Callaway

Coda-Forno

Dayan

Demircan

Eckstein

M. K.

Éltető

(2025). A foundation model to predict and capture human cognition. Nature, 644(8078), 1002–1009. https://doi.org/10.1038/s41586-025-09215-4

Birhane

Isaac

Prabhakaran

Diaz

Elish

M. C.

Gabriel

Mohamed

(2022). Power to the people? Opportunities and challenges for participatory AI. Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1–8. https://doi.org/10.1145/3551624.3555290

Brady

W. J.

Crockett

M. J.

Van Bavel

J. J.

(2020). The MAD model of moral contagion: The role of motivation, attention, and design in the spread of moralized content online. Perspectives on Psychological Science, 15(4), 978–1010. https://doi.org/10.1177/1745691620917336

Brady

W. J.

Jackson

J. C.

Doyle

Baier

(2025). Engagement-based algorithms disrupt human social norm learning. https://osf.io/mgdwq/download

Brady

W. J.

Jackson

J. C.

Lindström

Crockett

M. J.

(2023). Algorithm-mediated social learning in online social networks. Trends in Cognitive Sciences, 27(10), 947–960. https://doi.org/10.1016/j.tics.2023.06.008

10.

Brady

W. J.

McLoughlin

Doan

T. N.

Crockett

M. J.

(2021). How social learning amplifies moral outrage expression in online social networks. Science Advances, 7(33), eabe5641. https://doi.org/10.1126/sciadv.abe5641

11.

Brady

W. J.

Wills

J. A.

Jost

J. T.

Tucker

J. A.

Van Bavel

J. J.

(2017). Emotion shapes the diffusion of moralized content in social networks. Proceedings of the National Academy of Sciences, 114(28), 7313–7318. https://doi.org/10.1073/pnas.1618923114

12.

Brinkmann

Baumann

Bonnefon

J.-F.

Derex

Müller

T. F.

Nussberger

A.-M.

Czaplicka

Acerbi

Griffiths

T. L.

Henrich

Leibo

J. Z.

McElreath

Oudeyer

P.-Y.

Stray

Rahwan

(2023). Machine culture. Nature Human Behaviour, 7(11), 1855–1868. https://doi.org/10.1038/s41562-023-01742-2

13.

Brown

Mann

Ryder

Subbiah

Kaplan

J. D.

Dhariwal

Neelakantan

Shyam

Sastry

Askell

(2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://doi.org/10.48550/arXiv.2005.14165

14.

Burton

J. W.

Lopez-Lopez

Hechtlinger

Rahwan

Aeschbach

Bakker

M. A.

Becker

J. A.

Berditchevskaia

Berger

Brinkmann

Flek

Herzog

S. M.

Huang

Kapoor

Narayanan

Nussberger

A.-M.

Yasseri

Nickl

Almaatouq

Hertwig

(2024). How large language models can reshape collective intelligence. Nature Human Behaviour, 8(9), 1643–1655. https://doi.org/10.1038/s41562-024-01959-9

15.

Centola

(2022). The network science of collective intelligence. Trends in Cognitive Sciences, 26(11), 923–941. https://doi.org/10.1016/j.tics.2022.08.009

16.

Choudhury

(2023). Generative AI has a language problem. Nature Human Behaviour, 7(11), 1802–1803. https://doi.org/10.1038/s41562-023-01716-4

17.

Chu

C. Y. C.

Chang

J.-J.

Lin

C.-C.

(2025). Why does AI hinder democratization? Proceedings of the National Academy of Sciences, 122(19), e2423266122. https://doi.org/10.1073/pnas.2423266122

18.

Cinelli

De Francisci Morales

Galeazzi

Quattrociocchi

Starnini

(2021). The echo chamber effect on social media. Proceedings of the National Academy of Sciences, 118(9), e2023301118. https://doi.org/10.1073/pnas.2023301118

19.

Common Crawl. (2025). In Wikipedia. https://en.wikipedia.org/w/index.php?title=Common_Crawl&oldid=1272405656

20.

Costello

T. H.

Pennycook

Rand

D. G.

(2024). Durably reducing conspiracy beliefs through dialogues with AI. Science, 385(6714), eadq1814. https://doi.org/10.1126/science.adq1814

21.

Dattakumar

Jagadeesh

(2003). A review of literature on benchmarking. Benchmarking: An International Journal, 10(3), 176–209. https://doi.org/10.1108/14635770310477744

22.

Derex

Boyd

(2016). Partial connectivity increases cultural accumulation within groups. Proceedings of the National Academy of Sciences, 113(11), 2982–2987. https://doi.org/10.1073/pnas.1518798113

23.

Devlin

Chang

M.-W.

Lee

Toutanova

(2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805). arXiv. http://arxiv.org/abs/1810.04805.

24.

Dillion

Tandon

Gray

(2023). Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7), 597–600. https://doi.org/10.1016/j.tics.2023.04.008

25.

Farrell

(2025). AI As governance. Annual Review of Political Science, 28(1), 375–392. https://doi.org/10.1146/annurev-polisci-040723-013245

26.

Farrell

Gopnik

Shalizi

Evans

(2025). Large AI models are cultural and social technologies. Science, 387(6739), 1153–1156. https://doi.org/10.1126/science.adt9819

27.

Foffano

Scantamburlo

Cortés

(2023). Investing in AI for social good: An analysis of European national strategies. AI & SOCIETY, 38(2), 479–500. https://doi.org/10.1007/s00146-022-01445-8

28.

Fujiwara

Sasaki

Nakamura

Watanabe

(2024). Measuring the interpretability and explainability of model decisions of five large language models. Open Science Framework: Charlottesville, VA, USA. https://osf.io/d4ntw/download.

29.

Gallegos

I. O.

Rossi

R. A.

Barrow

Tanjim

M. M.

Kim

Dernoncourt

Zhang

Ahmed

N. K.

(2024). Bias and fairness in large language models: A survey. Computational Linguistics, 50(3), 1097–1179. https://doi.org/10.1162/coli_a_00524

30.

Gerosa

Trinkenreich

Steinmacher

Sarma

(2024). Can AI serve as a substitute for human subjects in software engineering research? Automated Software Engineering, 31(1), 13. https://doi.org/10.1007/s10515-023-00409-6

31.

Gibney

(2022). Open-source language AI challenges big tech’s models. Nature, 606(7916), 850–851. https://doi.org/10.1038/d41586-022-01705-z

32.

Gibney

(2025). Can researchers stop AI making up citations? Nature, 645(8081), 569–570. https://doi.org/10.1038/d41586-025-02853-8

33.

Gillespie

(2018). Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.

34.

Glickman

Sharot

(2025). How human–AI feedback loops alter human perceptual, emotional and social judgements. Nature Human Behaviour, 9(2), 345–359. https://doi.org/10.1038/s41562-024-02077-2

35.

Guo

Yang

Zhang

Song

Wang

Zhu

Zhang

Z. F.

Gou

Shao

Gao

Liu

Zhang

(2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633–638. https://doi.org/10.1038/s41586-025-09422-z

36.

Henrich

Boyd

(1998). The evolution of conformist transmission and the emergence of between-group differences. Evolution and Human Behavior, 19(4), 215–241. https://doi.org/10.1016/S1090-5138(98)00018-X

37.

Huang

Zhong

Feng

Wang

Chen

Peng

Feng

Qin

Liu

(2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2), 1–55. https://doi.org/10.1145/3703155

38.

Jackson

J. C.

Yam

K. C.

Tang

P. M.

Liu

Shariff

(2023a). Exposure to robot preachers undermines religious commitment. Journal of Experimental Psychology: General, 152(12), 3344. https://doi.org/10.1037/xge0001443

39.

Jackson

J. C.

Yam

K. C.

Tang

P. M.

Sibley

C. G.

Waytz

(2023b). Exposure to automation explains religious declines. Proceedings of the National Academy of Sciences, 120(34), e2304748120. https://doi.org/10.1073/pnas.2304748120

40.

Janis

I. L.

(1972). Victims of groupthink: A psychological study of foreign-policy decisions and fiascoes. https://psycnet.apa.org/record/1975-29417-000?utm_source=livewiremarkets.com&utm_medium=referral

41.

Lee

Frieske

Ishii

Bang

Y. J.

Madotto

Fung

(2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730

42.

Kalai

A. T.

Nachum

Vempala

S. S.

Zhang

(2025). Why Language Models Hallucinate (arXiv:2509.04664). arXiv. https://doi.org/10.48550/arXiv.2509.04664

43.

Kapoor

Bommasani

Klyman

Longpre

Ramaswami

Cihon

Hopkins

Bankston

Biderman

Bogen

Chowdhury

Engler

Henderson

Jernite

Lazar

Maffulli

Nelson

Pineau

Skowron

Narayanan

(2024). On the Societal Impact of Open Foundation Models (arXiv:2403.07918). arXiv. https://doi.org/10.48550/arXiv.2403.07918

44.

Kleinberg

Raghavan

(2021). Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22), e2018340118. https://doi.org/10.1073/pnas.2018340118

45.

Knott

Pedreschi

Chatila

Chakraborti

Leavy

Baeza-Yates

Eyers

Trotman

Teal

P. D.

Biecek

Russell

Bengio

(2023). Generative AI models should include detection mechanisms as a condition for public release. Ethics and Information Technology, 25(4), 55. https://doi.org/10.1007/s10676-023-09728-4

46.

Köbis

Rahwan

Rilla

Supriyatno

B. I.

Bersch

Ajaj

Bonnefon

J.-F.

Rahwan

(2025). Delegation to artificial intelligence can increase dishonest behaviour. Nature, 646(8083), 126–134. https://doi.org/10.1038/s41586-025-09505-x

47.

Koch

B. J.

Peterson

(2024). From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution (arXiv:2404.06647). arXiv. https://doi.org/10.48550/arXiv.2404.06647

48.

Korinek

Vipra

(2025). Concentrating intelligence: Scaling and market structure in artificial intelligence*. Economic Policy, 40(121), 225–256. https://doi.org/10.1093/epolic/eiae057

49.

Lazar

Lifshitz

Ayoubi

Emuna

(2025). Would archimedes shout “eureka” with algorithms? The hidden hand of algorithmic design in idea generation, the creation of ideation bubbles, and how experts can burst them. Academy of Management Journal, 68(5), 881–906. https://doi.org/10.5465/amj.2023.1307

50.

Lazer

Friedman

(2007). The network structure of exploration and exploitation. Administrative Science Quarterly, 52(4), 667–694. https://doi.org/10.2189/asqu.52.4.667

51.

Lewis

M. P.

(2009). Ethnologue: Languages of the world. SIL international.

52.

Teney

Yang

Wen

Xie

Wang

(2024). Culturepark: Boosting cross-cultural understanding in large language models. Advances in Neural Information Processing Systems, 37, 65183–65216. https://doi.org/10.52202/079017-2082

53.

Lika

Kolomvatsos

Hadjiefthymiades

(2014). Facing the cold start problem in recommender systems. Expert Systems with Applications, 41(4), 2065–2073. https://doi.org/10.1016/j.eswa.2013.09.005

54.

Liu

Jin

Wong

D. F.

Wen

Sun

Chen

Xie

Wang

(2025a). CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries (arXiv:2501.01282). arXiv. https://doi.org/10.48550/arXiv.2501.01282

55.

Liu

Yuan

Yang

Zhou

Liu

Guan

Wang

(2025b). Large language models and causal inference in collaboration: A comprehensive survey. Findings of the Association for Computational Linguistics: NAACL, 2025, 7668–7684. https://doi.org/10.18653/v1/2025.findings-naacl.427

56.

Loru

Nudo

Di Marco

Santirocchi

Atzeni

Cinelli

Cestari

Rossi-Arnaud

Quattrociocchi

(2025). The simulation of judgment in LLMs. Proceedings of the National Academy of Sciences, 122(42), e2518443122. https://doi.org/10.1073/pnas.2518443122

57.

J. G.

Song

L. L.

Zhang

L. D.

(2025). Cultural tendencies in generative AI. Nature Human Behaviour, 9(11), 2360–2369. https://doi.org/10.1038/s41562-025-02242-1

58.

Matz

S. C.

Horton

C. B.

Goethals

(2025). The Basic B*** Effect: The Use of LLM-based Agents Reduces the Distinctiveness and Diversity of People’s Choices (arXiv:2509.02910). arXiv. https://doi.org/10.48550/arXiv.2509.02910

59.

McLoughlin

K. L.

Brady

W. J.

Goolsbee

Kaiser

Klonick

Crockett

M. J.

(2024). Misinformation exploits outrage to spread online. Science, 386(6725), 991–996. https://doi.org/10.1126/science.adl2829

60.

Messeri

Crockett

M. J.

(2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002), 49–58. https://doi.org/10.1038/s41586-024-07146-0

61.

Mitchell

(2020). Artificial Intelligence. Picador Paper.

62.

Mitchell

Krakauer

D. C.

(2023). The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences, 120(13), e2215907120. https://doi.org/10.1073/pnas.2215907120

63.

Mökander

Schuett

Kirk

H. R.

Floridi

(2024). Auditing large language models: A three-layered approach. AI and Ethics, 4(4), 1085–1115. https://doi.org/10.1007/s43681-023-00289-2

64.

Molho

Peña

Singh

Derex

(2024). Do institutions evolve like material technologies? Current Opinion in Psychology, 60, 101913. https://doi.org/10.1016/j.copsyc.2024.101913

65.

Montag

Spapé

Becker

(2025). Can AI really help solve the loneliness epidemic? Trends in Cognitive Sciences, 29(10), 869–871. https://doi.org/10.1016/j.tics.2025.08.002

66.

Nisbett

R. E.

Peng

Choi

Norenzayan

(2001). Culture and systems of thought: Holistic versus analytic cognition. Psychological Review, 108(2), 291. https://doi.org/10.1037/0033-295X.108.2.291

67.

Padmakumar

(2024). Does Writing with Language Models Reduce Content Diversity? (arXiv:2309.05196). arXiv. https://doi.org/10.48550/arXiv.2309.05196

68.

Puryear

(2025). Rising Moralization in Social Media Discourse. https://repository.uncw.edu/items/ac618300-6698-4696-835a-52819dec3fc1.

69.

Radford

Narasimhan

Salimans

Sutskever

(2018). Improving language understanding by generative pre-training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

70.

Shiiku

Marjieh

Anglada-Tort

Jacoby

(2025). The dynamics of collective creativity in human-AI hybrid societies (arXiv:2502.17962). arXiv. https://doi.org/10.48550/arXiv.2502.17962

71.

Smaldino

P. E.

Moser

Pérez Velilla

Werling

(2024). Maintaining transient diversity is a general principle for improving collective problem solving. Perspectives on Psychological Science, 19(2), 454–464. https://doi.org/10.1177/17456916231180100

72.

Smith

M. G.

Bradbury

T. N.

Karney

B. R.

(2025). Can generative AI chatbots emulate human connection? A relationship science perspective. Perspectives on Psychological Science, 20(6), 1081–1099. https://doi.org/10.1177/17456916251351306

73.

Sucholutsky

(2025). Using LLMs to advance the cognitive science of collectives. Nature Computational Science, 5(9), 704–707. https://doi.org/10.1038/s43588-025-00848-z

74.

Summerfield

Argyle

L. P.

Bakker

Collins

Durmus

Eloundou

Gabriel

Ganguli

Hackenburg

Hadfield

G. K.

(2025). The impact of advanced AI systems on democracy. Nature Human Behaviour, 1–11. https://doi.org/10.1038/s41562-025-02309-z

75.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A. N.

Kaiser

Ł. u.

Polosukhin

(2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, https://doi.org/10.48550/arXiv.1706.03762

76.

Wang

Morgenstern

Dickerson

J. P.

(2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3), 400–411. https://doi.org/10.1038/s42256-025-00986-z

77.

Wang

Shu

Culotta

(2021). Enhancing Model Robustness and Fairness with Causality: A Regularization Approach (arXiv:2110.00911). arXiv. https://doi.org/10.48550/arXiv.2110.00911

78.

Welbl

Glaese

Uesato

Dathathri

Mellor

Hendricks

L. A.

Anderson

Kohli

Coppin

Huang

P.-S.

(2021). Challenges in Detoxifying Language Models (arXiv:2109.07445). arXiv. https://doi.org/10.48550/arXiv.2109.07445

79.

Whittlestone

Nyrup

Alexandrova

Cave

(2019). The role and limits of principles in AI ethics: Towards a focus on tensions. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 195–200. https://doi.org/10.1145/3306618.3314289

80.

Widder

D. G.

West

Whittaker

(2023). Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI (SSRN Scholarly Paper 4543807). Social Science Research Network, 1–27. https://doi.org/10.2139/ssrn.4543807

81.

Wikipedia:Multilingual statistics. (2024). In Wikipedia. https://en.wikipedia.org/w/index.php?title=Wikipedia:Multilingual_statistics&oldid=1261711959.

82.

Ren

Lin

Sun

(2018). Diversity-promoting GAN: A cross-entropy based generative adversarial network for diversified text generation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3940–3949. https://doi.org/10.18653/v1/D18-1428

83.

Xue

Constant

Roberts

Kale

Al-Rfou

Siddhant

Barua

Raffel

(2021). mT5: A massively multilingual pre-trained text-to-text transformer (arXiv:2010.11934). arXiv. https://doi.org/10.48550/arXiv.2010.11934

84.

Yam

K. C.

Tan

Jackson

J. C.

Shariff

Gray

(2023). Cultural differences in people’s reactions and applications of robots, algorithms, and artificial intelligence. Management and Organization Review, 19(5), 859–875. https://doi.org/10.1017/mor.2023.21