Automating Content Analysis With Multiple LLM Agents: Impacts of Agent Attributes and Human

Abstract

Emerging research in computational social science has applied LLMs to automate content analysis, often by prompting a single model to act as a human coder. While a single LLM may suffice for a few manifest variables, it still falls short on diverse latent constructs. And the impact of LLM agent attributes on measurement outcomes remains unclear, limiting their validity for communication research. Drawing upon the literature on interacting agents and communication, this study examines the impact of agent diversity, agent open-mindedness, and human–AI collaboration (HAIC) in a multi-LLM-agent system for automated content analysis. The results demonstrate reliable and accurate measurement of four communication variables across three datasets, with improved performance following agent discussion. Additionally, agent open-mindedness, but not agent diversity, significantly affects measurement outcomes. These results highlight the potential of multi-LLM-agent systems for automated content analysis and suggest the importance of considering agent attributes and values in system design.

Keywords

LLM automated content analysis multiagent simulation experiment

Introduction

Amid rapid advancements in information technology, artificial intelligence, and the proliferation of digital data, social and communication scientists are increasingly leveraging computational methods to automate traditional manual methodologies (Jansen et al., 2023; Van Atteveldt & Peng, 2018). In particular, an emerging stream of research focuses on leveraging large language models (LLMs) to automate content analysis and textual analysis (Zhao, Ma, & Ma, 2026; Chew et al., 2023; Törnberg, 2023; Ziems et al., 2024). Using the transformer’s attention mechanism and extensive training data, LLMs provide contextually rich, precise responses that enhance the accuracy of content analysis across contexts. For example, Ziems et al. (2024) showed that through prompt engineering, LLMs can reliably measure different variables such as political ideology and persuasiveness.

However, several challenges remain. First, most automated content analysis methods are variable-based (e.g., sentiment analysis) and lack the versatility to measure the wide range of constructs required for communication research. While a single LLM may be adequate for automating the measurement of manifest variables such as topic and sentiment, their accuracy still falls short of human expert coders when assessing latent, complex concepts such as emotional support or narrative perspective (Zhao, Ma, & Ma, 2026). Second, existing LLM-based automated content analysis often overlooks how agent attributes and values—some of which are inherently embedded in LLMs through their extensive training data—may introduce bias into the analysis. As a result, the impact of LLM agent attributes and their interactions on measurement outcomes remains underexamined. Last, LLMs, due to inherent randomness in their outputs, often lead to varied responses that limit reproducibility and reliability, warranting further study to enhance the replicability and robustness of LLM-based content analysis.

To provide a versatile, valid, and reliable tool for automated content analysis, this study examines the impact of agent attributes and human–AI collaboration, building upon a multi-LLM-agent system (Zhao et al., 2025). Note that “multi-LLM-agent” here refers to multiple instances of the same base model differentiated through prompt-based personas. Thus, observed attribute effects reflect prompt-induced rather than architectural heterogeneity. Specifically, this study contributes to the literature on computational social science (Hilbert et al., 2019; Van Atteveldt & Peng, 2018), content analysis (Krippendorff, 2019; Neuendorf, 2017), LLMs (Brown et al., 2020; Duan et al., 2024), and interacting agents (Waldherr et al., 2021; Wooldridge, 2009) in several important ways. First, it systematically evaluates a multi-LLM-agent system for valid and reliable measurement of diverse communication variables across datasets, using multiple replications to account for LLM randomness. Second, it examines how prompted modifications to LLM agent attributes, such as diversity and open-mindedness, influence their discussions and measurement outcomes. This not only contributes to LLMs and interacting agent research but also provides insights into optimizing LLM agent configurations for improved measurement quality. Third, this study examines how different modes of human–AI collaboration (HAIC)—persuasion vs. supervision—influence outcomes given different agent attributes. This unique focus on HAIC and its relational dynamics extends beyond the traditional human-in-the-loop approach (Mosqueira-Rey et al., 2023), which limits humans to intermittent oversight rather than fostering an ongoing, interactive conversation. By incorporating different forms of interaction between AI agents and human experts, our approach explores HAIC as a means to further improve measurement performance and potentially alleviate LLM bias.

Our experiments, conducted using three datasets and four measurement tasks with GPT-4o-mini with multiple repetitions¹, support the effectiveness of multiple LLM agents to measure diverse communication concepts with strong reliability and accuracy, with agent discussion, attributes, and human–AI collaboration affecting measurement outcomes. The following section discusses the literature on automated content analysis, LLMs, multiagent framework, and agent-based modeling, which lay the foundation for our work.

Literature Review

Content Analysis Within the Landscape of Computational Social Science

Content analysis is the objective and systematic quantification of communication symbols following predefined coding guidelines and the analysis of the relationships between these quantified values using statistical methods (Holsti, 1969; Krippendorff, 2019; Neuendorf, 2017). Content can be presented in multiple formats, such as text, images, and videos (Riffe et al., 2023). This study focuses on textual content. Quantitative content analysis can measure manifest content (Berelson, 2000), capturing straightforward, widely recognized meanings, such as the frequency of specific keywords or mentions of topics. It can also analyze latent content, capturing connotative, “between-the-lines” meanings such as discrete emotions (Bantum & Owen, 2009), frames (Matthes & Kohring, 2008), morality (Weber et al., 2021), and sexualization (Lynch et al., 2016).

Social scientists rely on reliability and validity to ensure the quality of content analysis. Reliability refers to the consistency of observations using the same measurement instrument across different times, places, and coders, often evaluated through metrics that account for agreement among multiple human coders during training. Such metrics typically adjust for the likelihood of random agreement, ensuring that reliability reflects true coder consistency rather than chance (Krippendorff, 2019). Validity concerns whether a variable, defined by relevant theories, can be reasonably operationalized and measured in content analysis. This suggests that while the measurement often does not fully represent communication reality, it seeks to reduce ambiguity in the representation (Riffe et al., 2023). Central to achieving reliability and validity is the manual coding process (for details, see Appendix A.1), a resource-intensive step that plays a crucial role in content analysis (Krippendorff, 2019; Neuendorf, 2017; Riffe et al., 2023).

While traditional, manual content analysis is well-established and valued for its objectivity and rigor, its scalability and generalizability can be constrained by labor-intensive, time-consuming procedures (Van Atteveldt & Peng, 2018), as well as reliance on the subjective perspectives of a small group of specialized scholars (Wu et al., 2021). For instance, Tankard (2001) lamented the early “lone-scholar” approach, which made framing identification “a rather subjective process.” (p. 98) To overcome these challenges, computational methods such as crowdcoding (Guo et al., 2020; Van Atteveldt et al., 2021), lexicons (Boyd et al., 2022; Hutto & Gilbert, 2014), machine learning (Van Atteveldt et al., 2021; Zhao & Wong, 2024), and large language models (LLMs) (Zhao, Ma, & Ma, 2026; Ziems et al., 2024) have been applied to automate content analysis. Specifically, Ziems et al. (2024) demonstrated that LLMs can perform zero-shot (i.e., without data training) text classification through prompt engineering and reliably measure variables like political ideology and persuasiveness, functioning as data annotators similar to human coders. However, in these studies, LLMs only act as a single coder and their potential mistakes and biases can only be partially mitigated through human-crafted prompts. This study tests different agent attributes and their interactions with human–AI collaboration modes in affecting measurement outcomes. Below we explain how and why LLMs can be used to automate content analysis in detail.

How and Why Can LLMs Be Used to Automate Content Analysis?

An LLM works by iteratively generating the most probable next word based on the previous words that were provided as input. It works similarly to a sentence or paragraph completion engine. For example, when a string of “What is the capital of the USA” is provided as the input to an LLM, it is likely to proceed as follows. First, it may output a highly probable next word, for example, “The.” Second, this word is appended to the end of the original question, and then the whole string is sent into the LLM again for the next word prediction, which may produce the word “capital.” The process will repeat a few times to further generate the rest of the words such as “of,” “the,” “USA,” “is,” “Washington,” and “DC.” When the input to an LLM is “Determine the sentiment (i.e., happy or unhappy) of this tweet ‘im completely fed up by the coursework’,” LLM will likely output “unhappy” as the response to completing the sentence. This is an example of zero-shot prompting (Chew et al., 2023; Liu et al., 2023; Törnberg, 2023; Ziems et al., 2024) for automated content analysis, in which LLM uses its own knowledge of natural language learned from training data to provide an answer. LLMs can provide correct answers for simple tasks with manifest content, which in part made LLMs widely popular in the past few years. However, LLMs have difficulties in answering questions with latent content if deeper reasoning is required (Ji et al., 2023; Ziems et al., 2024). LLMs may also provide biased answers if the training data contains biased examples (Abid et al., 2021; Ji et al., 2023; Lucy & Bamman, 2021).

An LLM can not only answer questions based on instructions but also take into account context such as a given persona when answering. The contextual information can be concatenated before or after the intended question, forming an augmented input for the LLM’s consumption. The LLM will iteratively conduct the next word prediction based on the intended questions taking into account the input persona. For example, when automating an LLM as a human-like agent to conduct sentiment analysis, one can augment the prompt with persona descriptions for LLM to role-play (Gao, 2023; Zhao et al., 2025), for example, “a social science researcher who analyzes texts by emphasizing more on logical reasoning than relying on rhetorical methods.” This persona may additionally include attributes such as background, skill level, and personal traits. By incorporating such nuanced details into the input, the LLM can emulate human characteristics and behaviors through role-playing (Zhao et al., 2025). Thanks to their extensive training in a wide array of training texts, LLMs are adept at drawing from this vast repository of knowledge to mirror these personas convincingly (Chen et al., 2024).

LLMs perform extremely well due to the attention mechanism in the transformer architecture, a large number of trainable parameters, and the vast amount of training data (Brown et al., 2020). First, the self-attention mechanism repeatedly refines the numerical representation of each input word to improve its contextualization based on other input words. Within a computer, a word is represented by a vector of numbers. Traditional tokenization algorithms such as Word2Vec or WordPiece have a unique vector representation for every word, leading to ambiguities when the word is used in different contexts. Transformers, instead, can adaptively update their vector representation by also including a weighted sum of the vector representations of all other words appearing at the input. Second, GPTs 3 and 4, the major commercial successes of transformers, were built on scaling up their prior generation with 10 $\times$ and $100 \times$ more parameters, respectively. This allows modern GPTs to capture the fine granularity of information present in the training data. Third, GPT training used tons of training data, from source code to books, to references to highly rated answers on Reddit.

What happens inside an LLM? At a basic level, an LLM is a function approximator that “learns” by adjusting its parameters to predict the likelihood of any word sequence it has encountered during training. When a training sentence is presented to the LLM, its learning engine will update the LLM’s parameters so that the approximated likelihood evaluator will deem the training sentence highly probable. Once the learning engine finishes passing all training examples to LLM for parameter update, LLM as a likelihood evaluator should be well trained to know what sentences are likely (e.g., a sentence from a newspaper) and unlikely (e.g., a random string of words), and can subsequently conduct high-quality iterative sentence/paragraph completion tasks.

Interacting Agents and Multiagent Framework

In computational social science, the multiagent framework and agent-based modeling are two pivotal approaches involving interacting agents, each with distinct emphases and applications. Multiagent research focuses on the design and coordination of a few sophisticated agents with advanced decision-making capabilities, often applied to solve practical problems in fields such as artificial intelligence and distributed systems (Wooldridge, 2009). This approach emphasizes the strategic interaction and cooperation among a small group of agents, each capable of complex behaviors that impact outcomes in various scenarios. Past studies have examined interaction modes including debate (Du et al., 2023; Liang et al., 2023) and consultancy (Kenton et al., 2024). In contrast, agent-based modeling (ABM) employs numerous, simpler agents to simulate emergent phenomena, modeling interactions among entities to reveal system-wide patterns (Gürcan, 2024). This method is particularly effective for understanding complex social dynamics and providing insights into the collective behaviors of the crowd (Gilbert, 2019; Waldherr et al., 2021). Recent works have utilized ABM for sustainability behavior study (Piatti et al., 2024), human society simulation (Park et al., 2023), and academic research collaboration analyses (Hutson & Ratican, 2023).

While our theoretical framework benefits from both lines of scholarship, our focus is the LLM-powered multiagent framework and its application to automated content analysis, where each agent’s nuanced capabilities and attributes significantly impact the outcomes of interactions. Computer science research within this framework has examined how agents’ memory capacity, decision-making strategies, and learning mechanisms, impact collective outcomes (Chen et al., 2018; Dehkordi et al., 2023). Their findings reveal that agent capacities and attributes can impact each agent’s effectiveness in a given context, influencing the overall system’s performance in tasks ranging from simple information retrieval to complex problem-solving. However, the multiagent framework is underexamined in communication and social science, limiting their ability to harness AI interactions and sophisticated decision-making.

Key Agent Attributes in Multi-LLM-Agent Discussion in Content Analysis

Research on interacting agents within complex social systems suggests that designing heterogeneous agents to interact in a simulated environment following simple rules can help assess how these interactions generate patterns observed at the group level (Waldherr et al., 2021). For instance, past research has systematically modified agent attributes to analyze collective outcomes and assess the sensitivity and robustness of observed behaviors in group discussion (Shugars, 2021), opinion dynamics (Sohn & Geidner, 2016), and team efficiency (Lazer & Friedman, 2007). On the other hand, the content analysis literature discusses the role of coder attributes in shaping coding outcomes, but primarily from a standardization standpoint to minimize bias and enhance intercoder reliability. While coder diversity can bring unique perspectives to data interpretation (Church et al., 2019), the focus remains on standardized training to align understanding and ensure uniform application of coding rules (Krippendorff, 2019; Neuendorf, 2017; Riffe et al., 2023). Together, these studies suggest the importance of balancing agent/coder heterogeneity and maintaining consistency when applying the multiagent framework to content analysis. Therefore, we focus on two specific agent attributes, diversity and open-mindedness, and discuss their impacts in multi-LLM-agent content analysis in the following sections.

Diversity of LLM Agents

The social science literature on team/dyadic diversity (e.g., cognitive or skill diversity) emphasizes how the presence of diverse viewpoints contributes to decision-making and task performance. Individuals with different backgrounds, expertise, or experiences often approach tasks with unique cognitive frameworks and informational resources, which can help to identify new opportunities and potential challenges that more homogenous teams may overlook. With a variety of perspectives, diverse team members could make more innovative and high-quality decisions, and more comprehensive approaches to problem-solving (Van Knippenberg et al., 2020).

For example, more diverse backgrounds can contribute to a richer exchange of perspectives, fostering creative solutions through the integration of multiple viewpoints. Research has shown that teams with greater demographic diversity tend to exhibit higher levels of team performance, as members can draw upon a broader range of cultural insights and experiences to generate novel ideas and solutions (Tshetshema & Chan, 2020). Additionally, the range of abilities and expertise present in a team can improve task performance by allowing individuals to draw on complementary strengths and tackle complex problems more effectively (Bell et al., 2011). Thus, cognitive diversity can lead to difference in thinking styles or problem-solving approaches, which contribute to team performance because it encourages critical discussion and reduces the risk of groupthink. Teams that engage in constructive debate and challenge assumptions often achieve more rigorous outcomes (Patrício & Franco, 2022).

Open-Mindedness of LLM Agents

The social science literature considers open-mindedness fundamental in democratic deliberation, defining it as a willingness to consider others’ opinions by relaxing one’s attachment to personal views (Barabas, 2004) and carefully evaluating opposing perspectives and arguments (Gastil, 2000). Shugars (2021)’s agent-based modeling study examined open- vs. close-mindedness as agents’ willingness to accept others’ views. Her findings indicate that open-minded and skeptical agents performed similarly, suggesting this willingness has limited impact on deliberative outcomes. Meanwhile, the interacting agents literature suggests that adaptability in automated agents allows them to adjust beliefs, attitudes, and decisions based on user interactions and environmental changes (Hilbert et al., 2019; Waldherr et al., 2021). This is substantiated by the interaction adaptation theory (Burgoon et al., 1995), which posits that agents not only process information individually but also adaptively respond to the dynamics of other agents’ behaviors.

Emerging literature shows that different LLMs display varying levels of open-mindedness (Brown et al., 2020; Duan et al., 2024; Wei et al., 2022). LLMs trained on a broad and diverse dataset could allow the model to adapt effectively to prompts and user needs due to its exposure to a wide range of ideas, topics, and contexts (Brown et al., 2020). By responding to diverse prompt structures, LLMs can also modify their output to suit specific task requirements (Wei et al., 2022). Duan et al. (2024) serendipitously observed that LLMs like Llama-2-70b-chat demonstrated high adaptability as they more readily gave up and accepted unfair decisions, while GPT-4 and Mistral were more “stubborn,” struggling to reach agreements and negotiating for fairer outcomes. Therefore, this study manipulates LLM agent adaptability through prompting and examines its impacts on LLM-agent-powered content analysis outcomes.

Human–AI Collaboration: Persuasion vs. Supervision

Human–AI collaboration (HAIC) is crucial in automated systems (Renner, 2020; Shoshitaishvili et al., 2017), as effective collaboration can balance the strengths of machine and human agency (Sundar, 2020). Namely, as machines become more agentic, they take on roles traditionally occupied by humans, such as analytics and decision-making. However, human intervention offers a way to balance machine agency with human oversight and refine system outputs to ensure alignment with human insights and reduce potential algorithmic biases (Jolfaei et al., 2022; Xu et al., 2023). Recent work on human–AI teaming suggests that effective collaboration depends not only on whether humans intervene, but also on how the collaborative relationship is structured (Vaccaro et al., 2024). In particular, shared mental models can help humans and AI systems coordinate more effectively by making reasoning processes and expectations more mutually intelligible (Kaur et al., 2019). In a systematic review and meta-analysis, Vaccaro et al. (2024) found that human–AI combinations are not inherently superior, highlighting the importance of theorizing when and how collaboration improves performance. This aligns with emerging discussions of hybrid or “centaur” human–AI systems, which stress the importance of structuring and integrating human and AI roles within collaborative workflows (Borghoff et al., 2025; Saghafian & Idan, 2024).

This study examines the impact of relational dynamics given different agent configurations. Grounded in social influence theories (Cialdini & Cialdini, 2007; French, 1959), there are two types of relational dynamics: in the “persuader” mode, LLM agents treat human experts as collaborative peers and can accept or reject their suggestions; in the “supervisor” mode, human experts act as absolute authorities, and LLM agents must follow their instructions. Persuader mode may encourage stronger alignment with the human expert’s reasoning, because agents must engage with and evaluate the expert’s input rather than merely comply with it. In this sense, persuasion may better support the development of a shared mental model between human and AI collaborators (Kaur et al., 2019). By contrast, supervisor mode may offer a more efficient coordination structure by reducing interactional overhead and enabling direct execution of expert guidance. These relational dynamics may interact with agent attributes to affect coding outcomes. For instance, when a human expert acts as a persuader, agent attributes likely play a role, as agents can actively engage with, resist, or accept the human’s input depending on their open-mindedness or specific demographic/exercise characteristics. As such, this study investigates how different human–AI relational dynamics (supervisor vs. persuader) interact with various configurations of LLM agents. To achieve this, we implement human intervention to facilitate more dynamic exchanges between human and LLM agents.

Summary of Research Questions

Based on the literature review, our first research question asks the extent to which it can automate reliable and valid measures across communication areas. Based on the literature on interacting agents and content analysis, the next question asks about the role of agent discussion in affecting measurement outcomes.

RQ1: To what extent does the multi-LLM-agent system automate valid and reliable measures of various communication concepts?

RQ2: How does agent discussion impact the reliability and validity of measures for various communication concepts?

In addition, this study investigates the impacts of systematic modifications to agent attributes and HAIC modes. The literature on team diversity suggests that compared to homogeneous groups, heterogeneous groups can propose more creative, high-quality solutions through integrating multiple perspectives. However, in content analysis, a certain degree of shared coder background and understanding might be essential for the standardized application of coding rules for intercoder reliability. Thus, LLM agent diversity might affect agent discussion and measurement outcomes. Additionally, the literature on the role of open-mindedness in collective discussion and agent simulations suggests that each agent’s open-mindedness could affect discussion outcomes. During content analysis discussions, a certain degree of coder open-mindedness is essential. Coders should be flexible in responding to feedback, which helps align interpretations, reduce personal biases, and enhance intercoder reliability. However, when coders are overly adaptable, they may quickly reach a consensus without deliberation, which can lead to oversimplification and reduce the complexity and richness captured in the analysis. Thus, the following two research questions are proposed to examine the impacts of agent diversity and open-mindedness:

RQ3: How does cognitive diversity (high vs. low) of LLM agents affect measurement reliability and validity?

RQ4: How does open-mindedness (high–high, high–low, vs. low–low) of LLM agents affect measurement reliability and validity?

Last, recognizing the significance of HAIC, we distinguish between two modes of human intervention in multi-LLM-agent coding. In supervision mode, the expert operates in a post-hoc verification role, providing guidance that agents are expected to follow. In persuasion mode, the expert operates in an interactive calibration role, offering reasoning that agents can evaluate during deliberation. This distinction is important for content analysis because disagreement resolution often depends not only on correcting outputs, but also on aligning coders’ reasoning through explanation, argumentation, and negotiated interpretation. We therefore ask:

RQ5: How do different HAIC modes (persuasion vs. supervision) affect measurement reliability and validity given different agent attributes?

Methods

A Multi-LLM-Agent System for Content Analysis

Content analysis traditionally relies on coder training to ensure validity and reliability (see Appendix A.1 for details). Figure 1 illustrates the design of a multi-LLM-agent system that simulates several key phases of coder training (Zhao et al., 2025), which serves as the basis for this study. First, LLM agents are simulated to resemble human coders, with distinct personas crafted through prompts. Following the guidelines provided in an initial codebook, two agents are independently prompted to code each text in a given sample, mirroring traditional independent coding. All agents are reminded in their prompts to strictly follow the codebook without relying on any external knowledge. Second, in each round of agent training, when the agents disagree on their coding results, they are prompted to engage in discussions to exchange rationales and resolve discrepancies.

Figure 1.

Workflow of multi-LLM-agent discussion andcodebook update

Note that they might accept or reject each other’s response. These discussions continue until a consensus or three discussion rounds are completed. Agent training includes multiple training rounds, each with several discussion rounds to address discrepancies, and continues until a threshold of intercoder reliability is achieved, similar to the process of manual content analysis. At the end of each training round, agents are prompted to update the codebook based on discussions. This serves to increase clarity and reduce ambiguities in the guidelines for greater application consistency. This updated codebook then serves as the guideline for the next training round, ensuring continuous improvement. After achieving reliability, the system analyzes millions of texts.

Datasets and Variables

We conducted experiments using three datasets across four different tasks, each focused on identifying distinct constructs using different textual data. These measurement tasks include two multiclass classification tasks, where a text only belongs to one of several predefined categories, and two multilabel classification tasks, where a text may belong to multiple categories simultaneously. For all datasets, the ground-truth labels were established through manual content analysis following best practices (Krippendorff, 2019; Riffe et al., 2023). Multiple trained experts independently coded the texts; depending on the dataset, the final ground truth was based either on consistently coded cases or on consensus reached through disagreement resolution.

Product Incidents Sentiment (PIS)

This dataset from Zhao and Wong (2024) consists of 200 Twitter/X user comments from various product recalls and corporate incidents from 2014 to 2017, such as the Target Data Breach, Samsung Note 7 Explosion, and Volkswagen Emission Scandal. The task is to detect the primary user sentiment—positive, neutral, or negative—toward the company and its product or service during these incidents. The data were labeled by three human experts with a Krippendorff’s alpha of 0.75, and the ground truth was based on the 200 tweets labeled consistently by all three experts.

Cancer Narratives (CN)

This dataset is based on Ma et al. (2023) and includes 60 Facebook posts from five major breast cancer non-profit organizations worldwide from 2016 to 2021. The first task involves identifying one or more cancer narrative events (CN-E): prevention, detection, treatment, or survivorship. The second task focuses on determining the narrator’s perspective (CN-P), which could be a cancer survivor, family or friends, the organization, journalist, or mixed. Ground-truth labels were established through iterative expert manual coding and consensus-based disagreement resolution. Three human experts achieved Krippendorff’s alphas ranging from .77 to 1.00.

Team Communication Tactics (TCT)

This dataset uses team discussion transcripts from Zhan and Hample (2023), with ground-truth annotations newly developed by the research team for this study. It contains 55 transcripts from 2017 of four-person role-play teams engaging in organizational decision-making discussions. The task involves identifying one or more team communication tactics, including task coordination, opinion solicitation, information sharing, problem evaluation, and opinion statement. The overall agreement rate was above 0.90, and two human experts resolved all disagreements through discussion.

System and LLM Agents Setup

We used GPT-4o-mini via application programming interface (API) calling to simulate coders through predefined personas in Appendix A.4. The GPT-4 family is one of the most powerful LLM models (as of the 4th quarter of 2024 when the experiments were conducted), which supports sophisticated natural language processing and human-like interactions. When combined with tailored prompts, GPT-4 allows for reasoning and dynamic adaptability—key aspects for replicating the intricacies of human discussions and achieving consensus (Hackl et al., 2023; Nori et al., 2023). Our pilot studies confirmed its capability through prompting to adapt coder attributes, decision-making styles, and demographic traits, aligning with the diversity found in real-world human coders. We chose to use GPT-4o-mini over GPT-4o due to the significantly lower cost of the former, which made it possible to process the large volume of text and run a hundred model repetitions required for this study, despite a minor reduction in performance as reported in Zhao et al. (2025). The system design has each agent independently coding 20 textual entries per training round across 3–5 rounds, with up to 3 discussion sessions per round. However, given the current focus on measurement outcomes, we limit codebook updates to non-categorical changes, such as adding examples or explanations for existing categories. More implementation details can be found in Appendix A.2.

Experimental Design

Experiment 1—Agent Diversity

Drawing from social science methodology and linguistic psychology literature (Babbie, 2020; Denzin & Lincoln, 2011; Hyland, 2005), we developed two distinct personas to manipulate cognitive diversity between agents. One persona reflected structured, rule-based thinking, characterized by explicit signposting of analytical steps, adherence to codebooks or checklists, and language emphasizing objectivity, replication, and precision. The other reflected contextual and interpretive thinking, emphasizing nuance, perspective, and lived experience, with a focus on meaning-making and empathetic interpretation rather than strict rule compliance. Based on these personas, two experimental conditions were established: high vs. low agent diversity. Details of these personas and their pairings are provided in Appendix A.4.

Experiment 2—Agent Open-Mindedness

Experiment 2 implemented open-mindedness as a separate prompt-based manipulation, independent of the diversity personas used in Experiment 1. Drawing on the literature (Barabas, 2004; Shugars, 2021), we developed two prompts of similar length and format to represent high versus low agent open-mindedness. An agent with high open-mindedness was prompted with the following: “You are open-minded and tend to adjust your outputs and justifications based on persuasive arguments and facts presented by others.” An agent with low open-mindedness received the opposite prompt: “You are close-minded and tend not to adjust your outputs and justifications based on persuasive arguments and facts presented by others.” We then configured paired agent open-mindedness as high–high, high–low, and low–low to create different conditions.

Experiment 3—HAIC Modes: Persuasion vs. Supervision

Human experts can intervene at two stages of coder training—during agent discussions and when updating the codebook—using two modes of human–AI collaboration (HAIC). Each mode is defined by distinct prompts given to the AI agents. In the persuasion mode, human experts acted as peers, offering advice agents could accept or reject: “A human social scientist has provided advice on your response. Consider this advice carefully as additional guidance. You may accept or reject it when updating your answer.” In the supervision mode, experts served as authorities whose instructions agents were required to follow: “A human social scientist expert has issued instructions regarding your response. You MUST follow these instructions when updating your answer.” A factorial design crossed HAIC mode (persuasion vs. supervision) with agent open-mindedness levels (high–high, high–low, low–low), resulting in six experimental conditions, which two social scientists applied to their assigned datasets.

Manipulation Check

Agent Diversity

A panel of three experts in social science and LLMs reviewed the prompts and agreed upon their content validity. For the manipulation check, we analyzed agent logs using both established and custom lexicons. LIWC’s “Cognitive Processes” category showed no significant difference between agents, likely because both agents’ responses reflect the cognitive style typical in content analysis. However, the agents exhibited different reasoning styles, as measured by the structured and contextual wordlists we developed based on the literature. Structured agents used significantly more explicit, rule-based terms (e.g., “consistency,” “precise,” “explicitly,” “objective”) than contextual agents (M = 5.84, M proportion = 2.6% vs. M = 4.02, M proportion = 1.8%; $t = 7.17$ , $p < . 001$ ). Conversely, contextual agents used more interpretive and experiential terms (e.g., “nuance,” “intuitive,” “perspective,” “experience”) than structured agents (M = 7.07, M proportion = 3.1% vs. M = 5.95, M proportion = 2.6%; $t = - 2.95$ , $p = . 004$ ). These results confirm the effectiveness of the manipulation check.

Agent Open-Mindedness

Agents with open-minded instructions were more willing to update their coding in response to the discussion, supporting the effectiveness of the manipulation at the condition level. These results should be interpreted as an overall behavioral pattern rather than a precise individual-level test, because the manipulation check uses a simple behavioral indicator and LLM outputs contain inherent randomness. In the high–high condition, Agent 1 changed codes an average of 0.45 times per text (SD = 0.83) and Agent 2 changed 0.62 times (SD = 0.99). In the low–low condition, changes were much less frequent (Agent 1: M = 0.03, SD = 0.17; Agent 2: M = 0.36, SD = 0.49). The mixed (high–low) condition was in between (Agent 1: M = 0.30, SD = 0.73; Agent 2: M = 0.28, SD = 0.69).

Evaluation Metrics

Intercoder Reliability

We use disagreement proportion and Cohen’s kappa to quantify intercoder reliability among LLM agents. The disagreement proportion measures the level of inconsistency among agents applying codebook rules to a dataset. It is defined as $p_{disagree} = N_{disagree} / N$ , where $N$ is the total number of text examples and $N_{disagree}$ is the number of text examples for which all agents fail to reach a consensus label. Cohen’s kappa $(κ)$ assesses the intercoder reliability beyond simply observing agreement but adjusts for the baseline level of agreement that could occur by chance. It is defined as $κ = (1 - p_{disagree} - p_{e}) / (1 - p_{e})$ , where $p_{e} = \sum_{i = 1}^{K} p_{i}^{2}$ is the expected likelihood that the agents might agree by random chance, $K$ is the total number of categories in the labeling scheme, and $p_{i}$ is the proportion of all examples assigned to category $i$ . The value of $κ$ ranges from $-$ 1 to 1, where $κ = 1$ indicates perfect agreement among agents, $κ = 0$ means agreement is no better than chance, and $κ < 0$ indicates potential systematic disagreement among agents.

Validity

We use classification accuracy as a proxy for validity. It offers an indication of how well the agents are labeling the data with reference to ground-truth labels. These ground-truth labels were established prior to the experiments through expert manual coding, either by retaining consistently coded cases or by resolving disagreements to produce final labels, depending on the dataset. The classification accuracy is defined as $p_{accuracy} = N_{correct} / N$ , where $N_{correct}$ is the number of examples for which all agents agree on the ground-truth label. Note that for a classification problem with each label containing multiple categories, we calculate the classification accuracy scores for all $K$ categories and average them to obtain the final classification accuracy.

LLM Randomness

To account for the randomness of LLM agent behaviors, we conduct multiple replications of our experiments and report the averages and sample standard deviations for each metric. The averages for the respective metrics better estimate the intercoder reliability and validity, whereas the sample standard deviations reflect the variability among different LLM runs. The latter quantifies intrinsic randomness from LLMs, relative to the variation caused by our modifications to agent attributes. However, in examining different HAIC modes, multiple repetitions were not feasible due to limited human expert resources and the substantial time to apply different HAIC modes to a dataset (approximately 10 hours).

Results

RQ1 examines the extent to which the system could automate valid and reliable measures of various communication concepts, and RQ2 investigates the role of agent discussion in affecting measurement validity and reliability. To answer the questions, the system was run in 5 repetitions for all datasets, measuring the mean intercoder reliability and validity metrics at two stages: the pre-discussion phase (independent coding) and the post-discussion phase (agent discussions on inconsistent coding results). Note that the reported metrics represent averages across multiple rounds of training in a dataset. Overall, the results show an average of 0.23 improvement in Cohen’s kappa and 0.11 in accuracy across datasets. As shown in Figure 2, before agent discussion, Cohen’s kappa ranged from 0.57 to 0.90, and accuracy ranged from 0.43 to 0.80 across datasets. These variations indicate low to medium intercoder reliability and validity, which might be affected by the complexity of each coding task. For example, as it might be relatively challenging to identify all narrative events in a Facebook post, Cohen’s kappa for narrative events between LLM coders was relatively low at 0.59. Agent discussion improved measurement reliability and validity, as post-discussion Cohen’s kappa ranged from 0.87 to 0.97, and accuracy ranged from 0.74 to 0.86 across datasets, demonstrating moderate to high intercoder reliability and validity. For detailed metrics with means and standard deviations, see Appendix A.3.

Figure 2.

Improved Accuracy (ACC), disagreement percentage (DP), and Cohen’s kappa (CK) after agent discussion across datasets and tasks

RQ3 and RQ4 investigate the impacts of LLM agent diversity and open-mindedness on measurement reliability and validity. To address the inherent randomness and significant variance in LLM agent behavior, we conducted 20 repetitions for each condition of diversity (high vs. low) and open-mindedness (high–high, high–low, vs. low–low), totaling 100 repetitions. Post-discussion metrics were recorded to evaluate measurement performance. We retain PIS as the focal dataset for RQ3 and RQ4 because sentiment detection is a relatively standardized annotation task and its ground-truth labels were based on consistently coded cases, providing a cleaner benchmark for isolating the effects of agent attributes. A series of independent-samples

t

-tests show that agent open-mindedness (Table 1), but not agent diversity (Table 2), significantly affected sentiment coding outcomes after agent discussion. For agent open-mindedness, compared with the low–low condition, the high–low condition had a significantly lower disagreement proportion,

t (38) = - 6.63

p < . 001

, and a significantly higher Cohen’s kappa,

t (38) = 6.53

p < . 001

. The high–high condition also had a significantly lower disagreement proportion,

t (38) = - 2.69

p = . 011

, and a significantly higher Cohen’s kappa,

t (38) = 3.73

p < . 001

, than the low–low condition. In addition, the high–low condition showed significantly higher accuracy,

t (38) = 2.08

p = . 044

, and lower disagreement proportion,

t (38) = - 2.05

p = . 047

, than the high–high condition, although the difference in Cohen’s kappa was not statistically significant.

Table 1.

Post-Discussion Performance by Agent Open-Mindedness

Condition/Comparison	ACC	DP	CK
High–high	0.83 (0.03)	0.02 (0.01)	0.96 (0.02)
High–low	0.85 (0.02)	0.02 (0.01)	0.97 (0.01)
Low–low	0.84 (0.02)	0.03 (0.01)	0.93 (0.02)
High–high vs. High–low, $t$	$-$ 2.08*	2.05*	$-$ 1.70
High–high vs. Low–low, $t$	$-$ 1.26	$-$ 2.69*	3.73***
High–low vs. Low–low, $t$	1.10	$-$ 6.63***	6.53***

Note. ACC = accuracy; DP = disagreement proportion; CK = Cohen’s kappa. Values are means (SDs) based on 20 runs per condition. * $p < . 05$ . ** $p < . 01$ . *** $p < . 001$ .

Table 2.

Post-Discussion Performance by Agent Diversity

Condition/Comparison	ACC	DP	CK
Low	0.86 (0.03)	0.02 (0.03)	0.96 (0.05)
High	0.85 (0.02)	0.02 (0.01)	0.95 (0.03)
High vs. Low, $t$	$-$ 1.40	0.68	$-$ 0.85

Note. ACC = accuracy; DP = disagreement proportion; CK = Cohen’s kappa. Values are means (SDs) based on 20 runs per condition; all $t$ tests were nonsignificant.

For robustness check, we conducted two additional tests. First, we repeated the sentiment-dataset analyses using a comparable LLM, Mistral-Small-3, and observed largely similar patterns (for details, see Appendix A.5). Second, we repeated these analyses on the other two datasets, and these results were weaker and less consistent overall, suggesting that the effects may be task-dependent (for details, see Appendix A.6).

RQ5 examines how different modes of human–AI collaboration (HAIC: persuasion vs. supervision) affect measurement reliability and validity. As shown in Table 3, in the persuasion mode, an average accuracy of 0.87 and Cohen’s kappa of 0.99 across four measurement tasks indicate satisfactory validity and strong reliability. Compared to the no-HAIC condition, the persuasion mode of HAIC led to accuracy improvements ranging from 0.02 to 0.13, while Cohen’s kappa improvements ranged from 0 to 0.13 across datasets. Compared to the no-HAIC condition, the supervision mode of HAIC led to accuracy improvements ranging from 0.03 to 0.17, while Cohen’s kappa improvements ranged from 0.03 to 0.13 across datasets. These results indicate that both modes of HAIC enhanced validity and reliability, with supervision showing a marginal advantage in average accuracy overall, although the difference was small and varied across datasets.

Table 3.

Measurement Outcomes With and Without Human–AI Collaboration (HAIC)

	CN-E			CN-P			PIS			TCT
	ACC	DP	CK	ACC	DP	CK	ACC	DP	CK	ACC	DP	CK
W/o HAIC	0.79	0.04	0.89	0.74	0.07	0.87	0.86	0.02	0.97	0.77	0.01	0.89
W/ HAIC: Persuasion	0.89	0	1	0.87	0	1	0.88	0.02	0.97	0.85	0	1
W/ HAIC: Supervision	0.91	0	1	0.91	0	1	0.89	0	1	0.84	0	0.99

Note. ACC = accuracy; DP = disagreement proportion; CK = Cohen’s kappa. CN-E = cancer narrative events; CN-P = cancer narrative perspective; PIS = product incidents sentiment; TCT = team communication tactics. The “W/o HAIC” baseline is taken from the post-discussion performance under default agent personas reported in Table 5.

We further explore how HAIC affects measurement outcomes at different levels of agent open-mindedness. Table 4 shows that, with HAIC, variations in agent open-mindedness generally had a limited impact on accuracy and Cohen’s kappa across datasets. Reliability reached a perfect or near-perfect level in most cases, and most conditions produced only small differences in validity and reliability. In the persuasion mode, measurement outcomes were broadly similar across open-mindedness conditions, although the high–high condition tended to yield somewhat stronger performance. In the supervision mode, the effects of agent open-mindedness were likewise limited, with only minor variation across conditions. Overall, these results suggest that agent open-mindedness did not have a consistent effect on HAIC outcomes.

Table 4.

Human–AI Collaboration (HAIC) Mode, Agent Open-Mindedness, and Outcomes

HAIC Mode	Open-	CN-E			CN-P			PIS			TCT
HAIC Mode	Mindedness	ACC	DP	CK	ACC	DP	CK	ACC	DP	CK	ACC	DP	CK
Persuasion	High–high	0.91	0	1	0.88	0	1	0.89	0.02	0.96	0.87	0	1
	High–low	0.86	0	1	0.85	0	1	0.86	0.03	0.95	0.83	0	1
	Low–low	0.90	0.01	0.99	0.88	0	1	0.88	0	1	0.85	0	1
Supervision	High–high	0.91	0	1	0.93	0	1	0.90	0	1	0.84	0	1
	High–low	0.90	0	1	0.90	0	1	0.90	0	1	0.80	0.01	0.98
	Low–low	0.91	0	1	0.90	0	1	0.88	0	1	0.88	0	1

Discussion

Our results confirm the reliability and accuracy of automated measures of various communication concepts through a multi-LLM-agent system, with agent discussion playing a key role in enhancing measurement quality. We also found that LLM agent open-mindedness and different modes of HAIC can further improve measurement outcomes. The results are discussed in detail below.

Multi-LLM Agents as a Promising Framework

Our results confirm that multiple LLM agents can be used to reliably and accurately measure various communication concepts, including sentiment, team communication tactics, and narrative events and perspectives, across different areas of communication. These tasks include categorizing text into multiple types simultaneously (multilabel) or selecting one type from several options (multiclass). Agent discussion was found to enhance both measurement reliability and accuracy, with greater improvements observed for tasks with lower reliability and accuracy prior to discussion. This highlights the potential of using LLMs as multiple collaborative agents for automated measurement. Namely, when one LLM struggles with complex coding tasks, two working together can achieve better results through collaborative deliberation. This discussion process probably helps clarify coding guidelines, align interpretations, and thereby boost measurement quality. For example, in response to the Facebook post, “Rest in peace to Jill Cohen, a powerful breast cancer advocate and friend, who passed away after 17 years of fighting breast cancer. Our hearts are with her family and friends,” two agents initially coded the narrative perspective differently: as a breast cancer organization vs. the cancer survivor’s family and friends. After discussion, they resolved their disagreement, with Agent Nick adjusting his result based on Claire’s argument that the tone and content suggest the narrator represents a community or organization honoring her impact. Overall, these findings extend the multiagent framework in computer science (Du et al., 2023; Kenton et al., 2024; Liang et al., 2023), often applied to controlled environments and straightforward text processing, by addressing complex, nuanced content coding in social science communities requiring significant deliberation. This integration of LLMs and multiagents in automated content analysis offers a substantial potential for transformative advancements in social science methodologies and theory development.

Agent Attributes Matter to a Degree

Our findings also speak to the role of agent open-mindedness in multi-LLM-agent content analysis. Extending Shugars’ (2021) work on open-mindedness in rule-based agent deliberation, our results suggest that prompt-induced open-mindedness can affect post-discussion outcomes in an LLM-agent setting. Specifically, in the focal sentiment task, agent open-mindedness primarily affected post-discussion reliability. Compared with the low–low condition, both conditions with at least one open-minded agent showed stronger post-discussion reliability, reflected in lower disagreement proportions and higher Cohen’s kappa. This reliability pattern was also replicated using a comparable LLM. These findings suggest that LLM agent open-mindedness could support coordination during deliberative coding by making agents more willing to reconsider their initial judgments and resolve disagreements after discussion. During our experiments, agents with low open-mindedness were sometimes found to resist revising their coding decisions after hearing the other agent’s reasoning, whereas conditions with at least one open-minded agent were generally better able to resolve disagreements. Meanwhile, accuracy differences were mostly nonsignificant, although the high–low condition showed higher accuracy than the high–high condition in one comparison. This may suggest that open-mindedness is most useful when balanced with some judgment stability. In a few high–high cases, GPT-4o-mini agents appeared to “overshoot,” repeatedly changing their coding decisions across discussion rounds and sometimes moving away from the benchmark labels. Overall, these results show that, in an LLM-agent setting, open-mindedness improves post-discussion reliability, while its effects on validity appear more limited and context-dependent.

We found limited evidence that agent diversity affected measurement outcomes, contrasting with communication literature suggesting that team diversity enhances performance through unique information processing pathways (Van Knippenberg et al., 2020). Although we designed agent personas with opposite cognitive traits, prompt-based agent persona diversity did not significantly affect accuracy across either GPT-4o-mini or Mistral-Small-3. One possible explanation is that the prompt-based diversity manipulation may have been too limited to induce genuinely distinct perspectives that would improve measurement validity. Because the two agents relied on the same underlying model and training data within each model-specific analysis, their responses may still have reflected largely shared representations despite different persona prompts. For instance, even if we prompt an agent to simulate 20 years of experience in health communication with quantitative expertise, it may still lack the field-specific knowledge and experiences of human experts. Alternatively, the benefits of agent diversity may not be reflected in measurement precision, as diversity might be more advantageous for creative tasks. For example, diverse agents were sometimes more proactive and creative in codebook updates, but these behaviors were not quantified as measurement outcomes in this study.

Utility of Human–AI Collaboration

We found that human–AI collaboration (HAIC) enhanced post-discussion measurement reliability and accuracy, with greater benefits observed for datasets with initially lower performance. This finding highlights the value of HAIC in automated systems, where human input can help structure collaboration and improve coordination between humans and AI (Kaur et al., 2019; Vaccaro et al., 2024). The slightly lower improvement in datasets with initially higher performance may be attributed to a ceiling effect, as post-discussion accuracy and reliability were already relatively high, limiting the potential benefits of further human intervention. However, we did not find strong evidence that one HAIC mode consistently outperformed the other across datasets. Although supervision sometimes yielded slightly stronger results, the pattern varied by task, suggesting that the relative utility of persuasion versus supervision may be context-dependent. This extends persuasion and social influence theories (Cialdini & Cialdini, 2007; French, 1959) to a multiagent framework involving both LLM and human agents, not by identifying a single superior mode, but by showing that different relational structures of human input may matter in different ways. For more standardized tasks (e.g., sentiment detection), direct supervision may help by giving agents clearer guidance and reducing ambiguity in how instructions are implemented. By contrast, for more interpretive tasks (e.g., team communication tactics classification), persuasion may sometimes be useful because it encourages agents to engage with the expert’s reasoning and align their representations more closely with the human’s mental model. Further, we observed that when human experts intervene, the influence of agent open-mindedness becomes more limited overall, likely because expert input helps structure and constrain the discussion process. These results suggest that human intervention can partially reduce the importance of agent attributes in determining measurement outcomes. As such, selecting optimal agent personas may be especially important in situations where human expertise is unavailable.

Theoretical and Practical Implications

Our findings contribute to the interdisciplinary literature of automated content analysis, LLMs, and interacting agents by assessing the utility of a multi-LLM-agent system for valid and reliable measurement of diverse communication constructs, and by demonstrating how human–AI collaboration might further enhance measurement for some complex constructs. By examining the effects of various agent attributes through prompting, our study sheds light on how LLM agent design influences measurement outcomes. The influence of LLM agent open-mindedness on reliability suggests that open-minded agents are more adaptable to nuanced interpretations, aligning better with the interpretative needs of content analysis, which often requires sensitivity to nuanced context in coding latent constructs. Although agent diversity was not found to be significant in this study, examining agent values and their alignment represents an important direction for future research, as the cultural dimensions of LLMs remain underexamined. The agent attributes examined here were operationalized through prompting, meaning that the “multi-LLM-agent” system in this study consisted of multiple agent instances derived from a single base model and differentiated through prompt-based personas. This design offers a practical and experimentally controlled way to isolate the effects of assigned agent attributes. At the same time, the observed effects reflect prompt-induced variation rather than deeper epistemic, cultural, or experiential heterogeneity or architecturally heterogeneous ensembles. Future research should examine whether more authentic heterogeneity can emerge through differences in model families, training data, or fine-tuned sub-models, and whether such variation produces different coding outcomes. Future studies could also examine additional agent attributes, such as ideology, and investigate how these attributes may influence outcomes beyond measurement quality, including the meaningfulness and creativity of the codebook. Further, having human experts supervise LLM agents and intervene in agent discussions and/or codebook updates can further enhance system performance, albeit sometimes incrementally. HAIC may be valuable for datasets that exhibit relatively low performance, as well as for those where potential LLM bias—such as bias related to cultural representation—may be present. Future research should more systematically examine how LLM biases may be reflected in agent discussion and how HAIC can be used to address these biases. This work also makes an equitable contribution to the social science community by leveling the playing field, enabling resource-limited researchers to leverage large datasets and the multi-LLM framework for large-scale content analysis.

Limitations and Future Work

First, our results are based primarily on the commercial model of GPT-4o-mini, although we also conducted a comparative analysis with Mistral-Small-3 and found comparable results. GPT-4o-mini was chosen for its cost-effectiveness—particularly given the large volume of text and the need to run a hundred repetitions—this decision necessarily involved a trade-off in performance. In practical applications, model selection varies depending on the balance between computational cost-efficiency and task effectiveness. At the same time, these results should not be taken to establish broad generalizability across all LLMs, as newer or larger models may show different performance patterns, levels of variability, or sensitivity to agent attributes. Future research should therefore examine whether these patterns hold as model capabilities continue to evolve. Second, isolating variation due to LLM randomness is challenging. Consequently, we conducted one hundred repetitions to test the impact of agent attributes only on a single dataset. Given this limitation, our findings on agent attributes should be generalized with caution, and future research should develop improved methods to account for LLM variability. Future research should also examine whether the observed patterns hold across additional datasets, languages, domains, and content-analytic tasks, and whether similar patterns emerge when the same agent configurations are implemented with different LLMs. Further, this study focuses solely on quantifiable aspects of measurement quality, namely intercoder reliability and accuracy. However, content analysis may also be assessed through qualitative criteria such as codebook update quality. Future research should investigate how multiple agents with varied attributes update and refine codebooks.

Supplemental Material

Supplemental Material - Automating Content Analysis With Multiple LLM Agents: Impacts of Agent Attributes and Human–AI Collaboration

Supplemental Material for Automating Content Analysis With Multiple LLM Agents: Impacts of Agent Attributes and Human–AI Collaboration by Xinyan Zhao, Chengshuai Zhao, Mordecai Mengesteab, Chau-Wai Wong, Mengqi Zhan, Zhen Tan, Tianlong Chen in Social Science Computer Review.

Footnotes

ORCID iDs

Xinyan Zhao

Chau-Wai Wong

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

Note

Author Biographies

Xinyan Zhao (Ph.D., University of Maryland) is an Associate Professor in the Hussman School of Journalism and Media at the University of North Carolina at Chapel Hill. Dr. Zhao’s research focuses on computational strategic communication, social media, and social networks, specifically in disaster, crisis, and health contexts. Her methodological work applies computational methods and LLM-based approaches to social science research.

Chengshuai Zhao is a computer science Ph.D. student in the School of Computing and Augmented Intelligence at Arizona State University. Zhao’s research focuses on data mining, AI for science, representation learning, and large language models. His work aims to develop more generalizable, secure, and trustworthy systems capable of uncovering knowledge at the frontier of human perception.

Mordecai Mengesteab is an undergraduate student in the Department of Computer Science at North Carolina (NC) State University. His research focuses on multiagent systems with interests in agent coordination, reasoning, and evaluation. He conducts research under Dr. Chau-Wai Wong at NC State University and is an ML/AI R&D intern at SAS.

Chau-Wai Wong (Ph.D., University of Maryland) is an Associate Professor in the Department of Electrical and Computer Engineering, the Forensic Sciences Cluster, and the Secure Computing Institute at North Carolina State University. Dr. Wong’s foundational research spans machine learning, multimedia forensics, statistical signal processing, and data analytics. He leverages these tools to tackle emerging applications in computational social science, generative models, and federated learning.

Monica Zhan (Ph.D., University of Maryland) is an Associate Professor in the Department of Communication at the University of Texas at Arlington. Dr. Zhan’s research focuses on organizational communication, corporate communication, and employee communication, specifically in how internal practices and emerging algorithmic technologies shape employee perceptions and workplace outcomes. Her methodological work applies quantitative and mixed-methods approaches.

Zhen Tan (Ph.D., Arizona State University) is an Assistant Professor in the School of Computing at Stevens Institute of Technology. His research focuses on trustworthy and interpretable machine learning, large language models, and human-centered AI, with interests in explainability, evaluation, and scientific discovery.

Tianlong Chen (Ph.D., University of Texas at Austin) is an Assistant Professor in the Department of Computer Science at the University of North Carolina at Chapel Hill. Dr. Chen’s research focus on building accurate, efficient, and trustworthy AI systems. His recent researches lie in large language models, multimodal learning, agentic systems, and AI for Biomedical Science.

References

Abid

Farooqi

Zou

(2021). Persistent anti-Muslim bias in large language models. In AAAI/ACM conference on AI (pp. 298–306). Ethics, and Society.

Babbie

E. R.

(2020). The practice of social research. Cengage Au.

Bantum

E. O.

Owen

J. E.

(2009). Evaluating the validity of computerized content analysis programs for identification of emotional expression in cancer narratives. Psychological Assessment, 21(1), 79–88. https://doi.org/10.1037/a0014643

Barabas

(2004). How deliberation affects policy opinions. American Political Science Review, 98(4), 687–701. https://doi.org/10.1017/s0003055404041425

Bell

S. T.

Villado

A. J.

Lukasik

M. A.

Belau

Briggs

A. L.

(2011). Getting specific about demographic diversity variable and team performance relationships: A meta-analysis. Journal of Management, 37(3), 709–743. https://doi.org/10.1177/0149206310365001

Berelson

(2000). Content analysis in communication research. In Marris

Thornham

(Eds.), Media studies: A reader (2 edition, pp. 200–209): New York University Press.

Borghoff

U. M.

Bottoni

Pareschi

(2025). Human-artificial interaction in the age of agentic ai: A system-theoretical approach. Frontiers in Human Dynamics, 7, 1579166. https://doi.org/10.3389/fhumd.2025.1579166

Boyd

R. L.

Ashokkumar

Seraj

Pennebaker

J. W.

(2022). The development and psychometric properties of LIWC-22, (Vol. 10). University of Texas at Austin.

Brown

T. B.

Mann

Ryder

Subbiah

Kaplan

Dhariwal

Neelakantan

Shyam

Sastry

Askell

Agarwal

(2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

10.

Burgoon

J. K.

Le Poire

B. A.

Rosenthal

(1995). Effects of preinteraction expectancies and target communication on perceiver reciprocity and compensation in dyadic interaction. Journal of Experimental Social Psychology, 31(4), 287–321. https://doi.org/10.1006/jesp.1995.1014

11.

Chen

Wang

Yuan

Zhang

Shi

Xie

Yang

Zhu

Chen

Ren

Xiao

(2024). From persona to personalization: A survey on role-playing language agents. arXiv preprint arXiv:2404.18231.

12.

Chen

Liu

(2018). Dynamic pricing strategies for perishable product in a competitive multi-agent retailers market. Journal of Artificial Societies and Social Simulation, 21(2), 12. https://doi.org/10.18564/jasss.3710

13.

Chew

Bollenbacher

Wenger

Speer

Kim

(2023). Llm-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306.14924.

14.

Church

Dunn

Prokopy

(2019). Benefits to qualitative data quality with multiple coders: Two case studies in multi-coder data analysis. Journal of Rural Social Sciences, 34(1), 2. https://egrove.olemiss.edu/jrss/vol34/iss1/2

15.

Cialdini

R. B.

Cialdini

R. B.

(2007). Influence: The psychology of persuasion, (Vol. 55). Collins New York.

16.

Dehkordi

M. A. E.

Lechner

Ghorbani

Nikolic

Chappin

Herder

P. M.

(2023). Using machine learning for agent specifications in agent-based models and simulations: A critical review and guidelines. Journal of Artificial Societies and Social Simulation, 26(1), 9. https://doi.org/10.18564/jasss.5016

17.

Denzin

N. K.

Lincoln

Y. S.

(2011). The sSage handbook of qualitative research. Sage.

18.

Torralba

Tenenbaum

J. B.

Mordatch

(2023). Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325.

19.

Duan

Zhang

Diffenderfer

Kailkhura

Sun

Stengel-Eskin

Bansal

Chen

(2024). GTBench: Uncovering the strategic reasoning limitations of LLMs via game-theoretic evaluations. arXiv preprint arXiv:2402.12348.

20.

French

J. R.

(1959). The bases of social power. In: Studies in social power. University of Michigan Press.

21.

Gao

(2023). Prompt engineering for large language models. Available at SSRN 4504303.

22.

Gastil

(2000). By popular demand: Revitalizing representative democracy through deliberative elections. Univ of California Press.

23.

Gilbert

(2019). Agent-based models. Sage Publications.

24.

Guo

Mays

Lai

Jalal

Ishwar

Betke

(2020). Accurate, fast, but not always cheap: Evaluating “crowdcoding” as an alternative approach to analyze social media data. Journalism & Mass Communication Quarterly, 97(3), 811–834. https://doi.org/10.1177/1077699019891437

25.

Gürcan

Ö.

(2024). LLM-augmented agent-based modelling for social simulations: Challenges and opportunities. In: HHAI 2024: Hybrid human AI systems for the social good, Malmö, Sweden, 10-14 June 2024, pp. 134–144.

26.

Hackl

Müller

A. E.

Granitzer

Sailer

(2023). Is GPT-4 a reliable rater? Evaluating consistency in GPT-4’s text ratings. In: Frontiers in education (Vol, 8, pp. 1272229). Frontiers Media SA.

27.

Hilbert

Barnett

Blumenstock

Contractor

Diesner

Frey

Gonzalez-Bailon

Lamberso

Pan

Tai-Quan

Cuihua

Paul

E. S.

Wouter van

Waldherr

Zhang

Jonatha

J. H. Z.

(2019). Computational communication science: A methodological catalyzer for a maturing discipline. International Journal of Communication, 13, 3912–3934. https://ijoc.org/index.php/ijoc/article/view/10675.

28.

Holsti

O. R.

(1969). Content analysis for the social sciences and humanities. Addison-Wesley (content analysis).

29.

Hutson

Ratican

(2023). Leveraging generative agents: Autonomous AI with simulated personas for interactive simulacra and collaborative research. Journal of Innovation and Technology, 1(15), 2805–5179. https://doi.org/10.61453/joit.v2023no15

30.

Hutto

Gilbert

(2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. International AAAI Conference on Web and Social Media, 8(1), 216–225. https://doi.org/10.1609/icwsm.v8i1.14550

31.

Hyland

(2005). Stance and engagement: A model of interaction in academic discourse. Discourse Studies, 7(2), 173–192. https://doi.org/10.1177/1461445605050365

32.

Jansen

B. J.

Jung

S.-g.

Salminen

(2023). Employing large language models in survey research. Natural Language Processing Journal, 4, 100020. https://doi.org/10.1016/j.nlp.2023.100020

33.

Zheng

Chen

(2023). Is ChatGPT a good personality recognizer? A preliminary study. arXiv preprint arXiv:2307.03952.

34.

Jolfaei

Usman

Roveri

Sheng

Palaniswami

Kant

(2022). Guest editorial: Computational intelligence for human-in-the-loop cyber physical systems. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(1), 2–5. https://doi.org/10.1109/tetci.2021.3139998

35.

Kaur

Williams

Lasecki

W. S.

(2019). Building shared mental models between humans and ai for effective collaboration. In CHI conference on human factors in computing systems extended abstracts. Association for Computing Machinery (ACM).

36.

Kenton

Siegel

N. Y.

Kramár

Brown-Cohen

Albanie

Bulian

Agarwal

Lindner

Tang

Goodman

N. D.

Shah

(2024). On scalable oversight with weak LLMs judging strong LLMs. arXiv preprint arXiv:2407.04622.

37.

Krippendorff

(2019). Content analysis: An introduction to its methodology. Sage publications.

38.

Lazer

Friedman

(2007). The network structure of exploration and exploitation. Administrative Science Quarterly, 52(4), 667–694. https://doi.org/10.2189/asqu.52.4.667

39.

Liang

Jiao

Wang

Yang

Shi

(2023). Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118.

40.

Liu

Yuan

Jiang

Hayashi

Neubig

(2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1–35. https://doi.org/10.1145/3560815

41.

Lucy

Bamman

(2021). Gender and representation bias in GPT-3 generated stories. In Proceedings of the third workshop on narrative understanding (pp. 48–55). Association for Computational Linguistics.

42.

Lynch

Tompkins

J. E.

Van Driel

I. I.

Fritz

(2016). Sexy, strong, and secondary: A content analysis of female characters in video games across 31 years. Journal of Communication, 66(4), 564–584. https://doi.org/10.1111/jcom.12237

43.

Zhao

Wang

(2023). Stories that engage the audience: An investigation of popular breast cancer narratives on social media. Telematics and Informatics, 85, 102048. https://doi.org/10.1016/j.tele.2023.102048

44.

Matthes

Kohring

(2008). The content analysis of media frames: Toward improving reliability and validity. Journal of Communication, 58(2), 258–279. https://doi.org/10.1111/j.1460-2466.2008.00384.x

45.

Mosqueira-Rey

Hernández-Pereira

Alonso-Ríos

Bobes-Bascarán

Fernández-Leal

Á.

(2023). Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review, 56(4), 3005–3054. https://doi.org/10.1007/s10462-022-10246-w

46.

Neuendorf

K. A.

(2017). The content analysis guidebook. Sage.

47.

Nori

King

McKinney

S. M.

Carignan

Horvitz

(2023). Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.

48.

Park

J. S.

O’Brien

Cai

C. J.

Morris

M. R.

Liang

Bernstein

M. S.

(2023). Generative agents: Interactive simulacra of human behavior. In 36th annual ACM symposium on user interface software and technology (pp. 1–22). Association for Computing Machinery.

49.

Patrício

Franco

(2022). A systematic literature review about team diversity and team performance: Future lines of investigation. Administrative Sciences, 12(1), 31. https://doi.org/10.3390/admsci12010031

50.

Piatti

Jin

Kleiman-Weiner

Schölkopf

Sachan

Mihalcea

(2024). Cooperate or collapse: Emergence of sustainability behaviors in a society of LLM agents. arXiv preprint arXiv:2404.16698.

51.

Renner

(2020). Designing for the human in the loop: Transparency and control in interactive machine learning, PhD thesis. University of Maryland.

52.

Riffe

Lacy

Watson

B. R.

Lovejoy

(2023). Analyzing media messages: Using quantitative content analysis in research. Routledge.

53.

Saghafian

Idan

(2024). Effective generative AI: The human-algorithm centaur. Harvard Data Science Review. https://hdsr.mitpress.mit.edu/pub/3rvlzjtw/release/4.

54.

Shoshitaishvili

Weissbacher

Dresel

Salls

Wang

Kruegel

Vigna

(2017). Rise of the HaCRS: Augmenting autonomous cyber reasoning systems with human assistance. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security (pp. 347–362). Association for Computing Machinery.

55.

Shugars

(2021). Good decisions or bad outcomes? A model for group deliberation on value-laden topics. Communication Methods and Measures, 15(4), 273–291. https://doi.org/10.1080/19312458.2020.1768521

56.

Sohn

Geidner

(2016). Collective dynamics of the spiral of silence: The role of ego-network size. International Journal of Public Opinion Research, 28(1), 25–45. https://doi.org/10.1093/ijpor/edv005

57.

Sundar

S. S.

(2020). Rise of machine agency: A framework for studying the psychology of human–AI interaction (HAII). Journal of Computer-Mediated Communication, 25(1), 74–88. https://doi.org/10.1093/jcmc/zmz026

58.

Tankard

J. , J.

(2001). The empirical approach to the study of media framing. Framing public life: Perspective on media and our understanding of the social world. Lawrence Erlbaum.

59.

Törnberg

(2023). How to use llms for text analysis. arXiv preprint arXiv:2307.13106.

60.

Tshetshema

C. T.

Chan

K.-Y.

(2020). A systematic literature review of the relationship between demographic diversity and innovation performance at team-level. Technology Analysis & Strategic Management, 32(8), 955–967. https://doi.org/10.1080/09537325.2020.1730783

61.

Vaccaro

Almaatouq

Malone

T. W.

(2024). When combinations of humans and ai are useful. Nature Human Behaviour.

62.

Van Atteveldt

Peng

T.-Q.

(2018). When communication meets computation: Opportunities, challenges, and pitfalls in computational communication science. Communication Methods and Measures, 12(2-3), 81–92. https://doi.org/10.1080/19312458.2018.1458084

63.

Van Atteveldt

Velden

M. A. V.

Boukes

(2021). The validity of sentiment analysis: Comparing manual annotation, crowd-coding, dictionary approaches, and machine learning algorithms. Communication Methods and Measures, 15(2), 121–140. https://doi.org/10.1080/19312458.2020.1869198

64.

Van Knippenberg

Nishii

L. H.

Dwertmann

D. J.

(2020). Synergy from diversity: Managing team diversity to enhance performance. Behavioral Science & Policy, 6(1), 75–92. https://doi.org/10.1177/237946152000600108

65.

Waldherr

Hilbert

González-Bailón

(2021). Worlds of agents: Prospects of agent-based modeling for communication research. Communication Methods and Measures, 15(4), 243–254. https://doi.org/10.1080/19312458.2021.1986478

66.

Weber

Mangus

J. M.

Huskey

Hopp

F. R.

Amir

Swanson

Gordon

Khooshabeh

Hahn

Tamborini

(2021). Extracting latent moral information from text narratives: Relevance, challenges, and solutions. In Computational methods for communication science (pp. 39–59). Routledge.

67.

Wei

Wang

Schuurmans

Bosma

Xia

Chi

Q. V.

Zhou

(2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS), 35, 24824–24837.

68.

Wooldridge

(2009). An introduction to multiagent systems. John Wiley & Sons.

69.

Wong

C.-W.

Zhao

Liu

(2021). Toward effective automated content analysis via crowdsourcing. In IEEE international conference on multimedia and expo (ICME), (pp. 1–6). IEEE.

70.

Dainoff

M. J.

Gao

(2023). Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI. International Journal of Human–Computer Interaction, 39(3), 494–518. https://doi.org/10.1080/10447318.2022.2041900

71.

Zhan

M. M.

Hample

(2023). Reconciling conflicting results of cultural diversity’s effect on project team performance: A quasi-experimental examination. Communication Research, 50(8), 1019–1043. https://doi.org/10.1177/00936502221097040

72.

Zhao

Sun

Liu

Wong

C.-W.

(2025). Tailoring generative AI chatbots for multiethnic communities in disaster preparedness communication: extending the CASA paradigm. Journal of Computer-Mediated Communication, 30(1), 1–12. https://doi.org/10.1093/jcmc/zmae022

73.

Zhao, X., Ma, Z., & Ma, R. (2026). Analyzing narrative contagion through digital storytelling in social media conversations: An AI-powered computational approach. New Media & Society, 28(1), 99–118 . https://doi.org/10.1177/14614448241285445

74.

Zhao

Tan

Wong

C.-W.

Zhao

Chen

Liu

(2025). SCALE: Towards collaborative content analysis in social science with large language model agents and human intervention. In Annual meeting of the association for computational linguistics (ACL) (pp. 1–31). Association for Computational Linguistics.

75.

Zhao

Wong

C.-W.

(2024). Automated measures of sentiment via transformer-and lexicon-based sentiment analysis (tlsa). Journal of Computational Social Science, 7(1), 145–170. https://doi.org/10.1007/s42001-023-00233-8

76.

Ziems

Held

Shaikh

Chen

Zhang

Yang

(2024). Can large language models transform computational social science? Computational Linguistics, 50(1), 237–291. https://doi.org/10.1162/coli_a_00502

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.18 MB