Abstract
Studying collective intelligence in teams poses key challenges because individual contributions to team outcomes—such as ideas—can be abstract and difficult to measure. Although many studies have examined how team members shape collective outputs, they often utilize measures of influence focused on perceptions, participation, or performance, which can differ from objective measures of how team member contributions shape team outcomes. In this study, we develop an objective influence measure within the context of an open-ended design task. We operationalized influence as the objective similarity between designs created in an independent Brainstorming Phase and subsequent collaborative (or independent) Design Phase. We demonstrate our measure’s validity by establishing its ability to reflect similarities between designs beyond those expected due to chance and by examining its correlation with predictors and proxies of influence. Key takeaways are that team members’ initial ideas influenced team-level designs greater than expected, especially for team members with greater expertise, more work contributions, and ideas that were more similar to the mean. Our approach demonstrates our measure’s utility in enabling a mechanistic understanding of idea integration within teams, and we discuss how it can foster new insights within the field of collective intelligence.
Significance statement
Teams are increasingly prevalent within the workplace, serving as a key organizational strategy for enhancing problem-solving, innovation, and performance. However, there is limited understanding of how individual ideas are (and should be) integrated into team outcomes during open-ended problem-solving tasks. We introduce an objective measure of influence within the framework of a virtual open-ended design game. By reflecting measurable shifts in designs between an individual and team phase, this measure avoids the pitfalls associated with relying upon more subjective or indirect measures of influence, which may not reflect how team members shape team outputs. We establish the validity of our influence measure by demonstrating how it captures meaningful similarities between individual and team designs and by establishing its correlation with common correlates and proxies of influence, including expertise, work contributions, and an idea’s similarity to the mean. Overall, our study highlights the utility of our objective measure of influence in clarifying the behavioral mechanisms underlying idea integration, influence, and team performance, and we discuss how this approach can be leveraged to test critical hypotheses related to the emergence of collective intelligence.
Introduction
The enhanced performance of groups relative to individuals is a critical emergent property of collective behavior. This phenomenon has been documented in diverse contexts, ranging from animal groups (Seeley and Buhrman, 1999), to human organizations (Riedl et al., 2021), to artificial swarms (Bonabeau et al., 1999). A central question in the study of this emergent phenomenon is: How do the contributions of individual group members combine to influence the emergence of intelligent group outcomes (De Dreu et al., 2008)? While exciting progress has been made in the study of intelligent collective behaviors, especially in animal groups, studying this topic in human teams has been more challenging (O’Bryan et al., 2020). One reason for this discrepancy is that team members’ contributions often take the form of abstract knowledge, information, or ideas, making their influence over team outcomes difficult to measure.
To address this challenge, researchers often rely upon indirect measures, or proxies, of influence, such as those derived from self-report surveys and behavioral measures. For example, self-report surveys may assess team member perceptions of influence, voice, and expertise utilization (attributed influence; Bottger, 1984; Sherf et al., 2018), and behavioral measures can capture influence attempts or proxies of influence attempts, such as speaking time and work contributions (measures of participation; Engel et al., 2014; Mast, 2002; Woolley et al., 2010). Although these measures provide valuable insights into team members’ perceptions and behaviors, they do not directly reflect whether or how team member contributions influence team outcomes.
March (1955) defines influence as a process that causes behavior at time t1 to differ from that which might be predicted at time t0. This definition suggests that the most direct measures of influence track changes displayed by individuals or groups between two time points (shifts in opinion or behavior; March, 1955, 1956). Such measures provide an objective assessment of changes that occur over time (e.g., in opinions, performance, and ideas), presumably as the result of influence processes. Measures focused on shifts in opinion or behavior have low levels of correlation with influence metrics based upon attributed influence or measures of participation (March, 1956) and often yield conflicting results (Bottger, 1984), suggesting that different influence measures capture distinct processes. Unfortunately, team decision-making studies utilizing measures of shifts in opinion or behavior have largely been restricted to highly structured tasks in which contributions take the form of ranked lists. As a result, our understanding of how team member contributions are transformed by influence processes remains narrow.
Our study addresses these limitations and contributes to the literature by expanding upon measures of influence based upon shifts in opinion or behavior within the context of an open-ended design task. Open-ended tasks reflect a key form of real-world team problem-solving and provide opportunities for flexible and creative idea generation and integration. We use our measure to calculate the level of influence individuals’ ideas have over their subsequent designs in both an Individual and Team Condition. We leverage the quantitative nature of our data to demonstrate how the levels of influence observed in our study differ from those expected due to chance alone, and we further validate our measure by establishing its relationship with common predictors (and proxies) of influence. Finally, we use our measure to explore which characteristics of teams’ most influential team members correlate with absolute team performance and relative team performance (i.e., compared to the initial median individual-level performance of their team members. By quantifying the objective influence individuals’ ideas have over team decision-making within the context of an open-ended task, our approach can provide a more mechanistic understanding of team performance and collective intelligence—the term typically used to describe the phenomenon of enhanced team performance compared to individuals (O’Bryan et al., 2020). In the Discussion, we consider how this approach can be used to advance research within the fields of collective decision-making and intelligence.
Previous studies of idea integration
Teams frequently display higher mean performance compared to individuals (Almaatouq et al., 2021; Goldman, 1965; Lorge et al., 1958). One explanation for this finding is that team members combine their contributions (e.g., through equal levels of influence) into new solutions that are more innovative or higher-performing than team members’ initial contributions (Kohn et al., 2011; Paulus and Dzindolet, 2008). Indeed, some argue that how team members combine their contributions is key to greater team performance and collective intelligence, even surpassing the role of team member intelligence (Engel et al., 2014; Haan et al., 2021). In line with these arguments, studies indicate that when influence is centralized around a given individual (or individuals), team performance declines due to the underutilization of other team members’ expertise (Engel et al., 2014; Haan et al., 2021; Woolley et al., 2015).
An alternative explanation for why teams often perform better than individuals is that team members with greater expertise have outsized influence, pulling team outcomes towards their own high-quality ideas (Bottger, 1984). For example, Goldman (1965) found that low-performing individuals paired with high-performers received the greatest boost in performance, suggesting that the higher-performing partner had more influence over decision-making for the pair. In addition, studies have found that higher-quality individual ideas more closely match the final team-level solution (Bottger, 1984; Mayo et al., 2020).
One difficulty in ameliorating these conflicting views of idea integration is that studies often differ in how they measure influence. For example, some studies that have found negative effects of centralized influence have measured this construct through peer ratings of voice (e.g., expression of work-related ideas) (Sherf et al., 2018) or measures of participation that focus on number of speaking turns (Woolley et al., 2010). On the other hand, some studies that have found positive effects of unequal influence (when such influence positively correlates with expertise) have utilized measures of influence that focus on measurable changes in team member contributions following periods of interaction (Bottger, 1984; Mayo et al., 2020). The use of these different measures makes it difficult to compare results across studies, thereby clouding understanding of the relationships between influence, idea integration, and collective intelligence.
Measures of influence
The three most common categories of influence measures are attributed influence, measures of participation, and shifts in opinion or behavior (March, 1955, 1956). Measures of attributed influence are often collected through self-report or peer ratings and are thus relatively easy to implement. Furthermore, they can capture internal cognitions and attitudes which are less readily measured by more objective measures (March, 1955). However, a shortcoming of measures of attributed influence is that they do not directly measure whether influence occurs, but rather how it is perceived. Furthermore, they are subject to the same limitations as any self-report measure, such as bias, inconsistency, and the limitations of memory (Kihlstrom et al., 1999; Kozlowski and Chao, 2018).
Measures of participation focus on quantifying influence attempts, or proxies of influence attempts, which often take the form of communication acts. Indeed, prior research suggests that higher participation levels are associated with greater perceptions of influence, expertise, and leadership (Bales, 1953; Riecken, 1958). However, shortcomings of this measurement approach are that measures of participation fail to capture whether acts of participation actually result in influence (Mayo et al., 2020), and not all units of participation contribute equally (or at all) towards influence, as their impact may depend upon their content and who is participating (March, 1955).
A third approach focuses on measures of shifts in opinion or behavior. We argue that measures falling under this category offer a more direct and objective measure of the influence team member contributions have over team outcomes by reflecting changes in these contributions over time. This approach requires obtaining measures (e.g., of opinion, behavior, performance, and ideas) during at least two time points and recording the change that occurs. Researchers are not interested in the absolute value of a given measure, but rather the change from t0 to t1, as this change suggests the presence of influence processes that took place between time points. Thus, this measure most closely aligns with the influence construct defined by March (1956) and is what we examine in the current study.
March (1956) compared eight different measures of influence across the three major categories: attributed influence, measures of participation, and shifts in opinion or behavior. His findings revealed low correlations between measures, especially across categories, leading him to conclude that these measures “do not represent universally interchangeable indices of an underlying variable” (p. 264). This conclusion suggests that the choice of measurement has important implications for understanding influence processes within teams, and is supported by studies that have found conflicting results when using different influence measures. For example, Bottger (1984) found that although speaking time was more highly correlated with perceived influence, expertise was more highly correlated with an objective measure of influence over team outcomes (which reflected changes in team member responses between an independent phase and subsequent team phase).
Previous approaches to measuring shifts in opinion or behavior
Measuring shifts in opinion or behavior can entail leveraging a variety of data sources, including both self-report and objective measures. For example, Bernstein et al. (2018) leveraged solutions to the traveling salesman problem, which involves finding the shortest route between a set of cities. This task enabled the measurement of changes in individuals’ routes after they were exposed to others’ ideas (in the absence of any interaction) and the degree to which individuals integrated components of others’ routes into their own. Studies of team decision-making have the added complexity of examining how individuals’ contributions influence team-level outcomes. Some studies approach this challenge by comparing performance between t0 and t1, with individuals working independently at t0 and working together at t1 (Goldman, 1965). This approach can highlight performance changes that occur when individuals work together. However, a limitation of this approach is that features of the individual and team outputs (e.g., their design or composition) are not considered, preventing examination of whether or how team members’ ideas, or other contributions, influenced the team output.
A subset of studies has solved this challenge by directly measuring changes in the features of contributions between an individual (t0) and a team (t1) phase. Most studies using this approach have utilized tasks that involve ranking items in a list, including the NASA moon landing task (Bottger, 1984), the Dessert Survival task (Littlepage et al., 1995; Mayo et al., 2020), and other experimenter-derived rankings (March, 1956). Using these tasks, researchers can compare how closely individuals’ initial rankings correspond with their final team-level ranking to calculate an objective measure of the level of influence each individual had over the team outcome. For some tasks, individual and team rankings can also be compared to the rankings provided by an expert to obtain an objective measure of ranking quality (Bottger, 1984; Littlepage et al., 1995). The benefits of this approach are that it is possible to directly link features of individual ideas to features of group outcomes, which can facilitate a more precise understanding of how individuals’ contributions influence team outcomes (Bottger, 1984; March, 1956).
Despite these advantages, rank-based tasks represent a relatively narrow subset of problem-solving tasks in which team member contributions represent a highly structured form. Thus, it is important to expand upon the set of influence measures focused on changes in opinion or behavior by adapting them to a wider range of tasks. Our study expands upon this approach by developing an objective measure of influence within the context of an open-ended creative design challenge. This expansion can improve understanding of the generalizability of influence processes involved in team decision-making and collective intelligence, as well as how the features or constraints of a given task impact these processes.
Our approach
Description of task
Our study leverages a virtual design game that allows for a diversity of potential solutions that can vary in quality relative to a set of design criteria. This context, whereby the best solutions cannot be readily ascertained by participants, facilitated our measurement of how ideas influence decision-making (Jayles et al., 2017; Laughlin and Ellis, 1986). Our task is particularly relevant to the study of decision-making, prototyping, creativity, and performance (Han et al., 2022; Starkey et al., 2016; Zheng et al., 2018). Our study design is also relevant to temporary teams (i.e., groups of unfamiliar people with diverse skills who come together for a short duration to accomplish a complex task and then disband once the project is completed; Lv and Feng, 2021) and virtual teams, which have become more prevalent in the workplace over time (Bell and Kozlowski, 2002; Turesky et al., 2020).
The task utilized the online program Line Rider (https://www.linerider.com/), wherein a virtual sled rider rides down lines drawn by the user in a manner that approximates the rules of physics (see Methods). Participants were assigned the challenge of designing a simple track (i.e., consisting of a single continuous and non-overlapping line) that could enable the sled rider to achieve a goal defined by the experimenters. To better understand the effect of working on a team, we leveraged this platform to examine idea integration and performance across both teams (Team Condition) and individuals working independently (Individual Condition) (Han et al., 2022). Participants in both conditions experienced two distinct phases (Figure 1). In the Brainstorming Phase (Phase 1, 10 min), all participants designed a solution to the challenge independently. In the Design Phase (Phase 2, 15 min), participants produced a design that would be judged by the experimenters. Participants in the Team Condition worked in teams of three to come up with a single collaborative team design, and participants in the Individual Condition worked independently to produce their individual designs. Note. The study design displays the Brainstorming and Design Phases across the Individual and Team Conditions. All participants completed a personality survey prior to the study and all designs were tested following the Design Phase. The Team Condition comprised 162 individuals nested within 54 3-person teams.
Because tracks were composed of simple lines, the area between tracks represents a straightforward, quantitative measure of design similarity (Figure 2). We used this measure to calculate changes in track design that took place between the Brainstorming Phase and Design Phase. This approach enabled us to infer the level of influence each participant’s initial idea (i.e., the track they produced in the Brainstorming Phase) had over their subsequent design (i.e., the track they produced in the Design Phase), with smaller values indicating more influence (Bottger, 1984) (see Methods for more details). The advantages of our influence measure are that it is not impacted by team member perceptions or behaviors but rather reflects the quantifiable level of similarity between individuals’ initial ideas and subsequent outputs. We did not manipulate participants’ characteristics or behaviors as our study focuses on the natural associations between these variables and our influence measure (Bottger, 1984). Note. Figure (a) displays the background course (green) upon which participants designed their tracks. Stylized examples of team members’ initial Brainstorming Phase tracks are displayed in blue, yellow, and red as well as a Design Phase (team) track in black. Figures (b–d) show how the area between each Brainstorming Phase track and the Design Phase track can be calculated to obtain an objective measure of its influence over the team design (smaller areas mean greater influence).
Implementation
Using our research approach, we first tested how teams performed compared to individuals—both those who worked independently throughout the study and the individual members of each team when they worked independently during the Brainstorming Phase. Comparing team with individual performance permits us to measure whether the mechanisms of idea integration we identify are associated with process gains or losses. We then conducted exploratory analyses to (1) validate whether our measure captures meaningful influence processes beyond those that might be due to chance alone, while also gaining insight into the distribution of influence within teams, (2) establish which common predictors and proxies of influence correlate with our objective influence measure, and (3) determine how these characteristics relate to team performance by way of the team’s most influential team member (Humphrey et al., 2009; Sherf et al., 2018). Based upon previous research focused on objective measures of influence (Bottger, 1984; Goldman, 1965; Mayo et al., 2020), we expected that team performance would be associated with influential team members who had greater expertise, either through the effect of expertise alone or in combination with characteristics that heighten their influence.
The individual characteristics—or predictors—we considered were speaking time, dominance level, and expertise. Speaking time is a measure of participation which is commonly used as a proxy for influence (Bottger, 1984; Mayo et al., 2020). Speaking time is also frequently associated with positive perceptions of team member abilities, such as expertise, and may thus also correlate with measures of attributed influence (Littlepage et al., 1995; March, 1955). There are mixed results regarding the degree to which speaking time correlates with objective measures of influence (Bottger, 1984; Littlepage et al., 1995; Mayo et al., 2020). In our study, speaking time reflects the total time an individual spent speaking in the Design Phase, as measured by automated transcripts of the team’s conversation. Dominance has been tied to both perceived and objective measures of influence and may also correlate with speaking time (Cheng et al., 2013; Mast, 2002). In our study, dominance was measured using a pre-study self-report survey. Expertise (both perceived and objective) has been found to correlate with perceived and objective measures of influence (Bottger, 1984; Bunderson, 2003; Littlepage et al., 1995; Mayo et al., 2020). In our study, expertise was measured as the objective performance of an individual’s Brainstorming Phase track (which was unknown to participants but may be estimated based upon track characteristics). As previous studies suggest that the interaction between communication behaviors and dominance (Sherf et al., 2018) or expertise (Bottger, 1984; Gintner and Lindskold, 1975) can heighten influence, we also examined interactions between (1) speaking time and dominance and (2) speaking time and expertise.
In addition to the above variables, we controlled for an individual’s work contributions (a measure of participation reflecting the total time spent controlling the shared screen during the Design Phase) given that participants could both communicate and produce their designs by drawing on a shared screen. It is important to control for this variable as work contributions are an additional means by which team members can contribute their expertise (regardless of its extent) towards team outcomes (Haan et al., 2021). In addition, we controlled for how closely each team member’s design conformed to the mean of all team members’ ideas. We controlled for this variable because brainstorming ideas that happen to be more similar to the mean solution may have greater influence due to consensus processes, rather than individual influence (De Dreu and West, 2001; Gigone and Hastie, 1997; Hackman and Morris, 1975; March, 1956).
Materials and methods
Participants
This research complied with the American Psychological Association Code of Ethics and was approved by the institution’s Institutional Review Board. During the Summer of 2020 and Spring of 2021, the university where the study took place (located in the Southern United States) moved all research online due to the COVID-19 pandemic. Thus, this study took place within virtual meeting rooms (i.e., Zoom). Participants were compensated either with course credit or $10. We collected data from 84 individuals and 59 3-person virtual teams. We only extracted data from tracks that followed all study guidelines, and we only included data from individuals and teams for whom we had data for all tracks produced across both phases (N = 71 individuals and 162 individuals nested within N = 54 3-person teams).
The line rider task
Following a training video, participants were given 5 minutes to engage in a structured training exercise during which they were familiarized with the Line Rider platform (https://www.linerider.com/) and how the sled rider responds to various tracks. Training was followed by a 10-minute independent Brainstorming Phase and a 15-minute independent (Individual Condition) or collaborative (Team Condition) Design Phase (Figure 1). Participants began from a blank slate in the Design Phase and were not allowed to access their brainstorming tracks. Participants were also not allowed to test their designs themselves in the Brainstorming or Design Phases, although participants learned the performance of their Design Phase track upon completion of the Design Phase.
The design challenge involved designing a track on top of a background “course” (Figure 2). The sled rider could not interact with the background course but could interact with lines drawn by the participants. The challenge was to design a track that enabled the sled rider to reach a target at the bottom right-hand corner of the course as fast as possible while passing through as many checkpoints as possible and while avoiding crashing. Checkpoints were circles drawn on the course that had different point values written inside them. To gain the points assigned to a given checkpoint, the track had to pass through it. Tracks could not pass through the shaded squares. We designed the course to represent an open-ended task without one clear optimal solution. We incentivized participants by offering a $20 award per participant for the individual and team that produced the best-performing designs in the Design Phase across the Individual and Team Conditions, respectively. Tracks were collected from individuals and teams following the completion of each phase. All study sessions were video and audio recorded using the Zoom platform with automatic transcript generation enabled.
Track data processing
All tracks were visually assessed to determine whether they met the requirements laid out by the experimenters at the start of the study. The requirements included designing a track that was a continuous line, avoiding drawing a line where any section of the line was above or below another section of the line (e.g., no loops), and ensuring that the line did not pass through the shaded squares. All tracks were visually examined to determine whether there were any extraneous marks on the screen that were not part of the track that the rider interacted with. If present, these marks were manually removed. X-Y coordinates of all line segments comprising each track were imported into R (Version 2022.07.2) (R Core Team, 2022) using the jsonlite package (Ooms, 2014). We subsampled tracks at each integer x-coordinate, and we calculated the median y-coordinate for each x-coordinate to account for thick or irregular lines which may have more than one y-coordinate positioned close to one another.
Measures
Track performance/expertise
The track performance score that participants were instructed to maximize during the study was based on the value of the checkpoints the track passed through as well as the sled rider’s completion speed. During the task, scores were calculated using the following formula: Score = (10 − Finish Time) + Points, so that tracks that were both faster and obtained more points performed better. To promote the study of how ideas influence decision-making and to prevent participants from simply choosing the highest-performing track (Jayles et al., 2017; Laughlin and Ellis, 1986), participants were not allowed to test their tracks during the study. As a result, 62.2% of designs (227 out of 358 designs across all phases and conditions) resulted in the rider crashing before reaching the end of the track. In these cases, the track received a score of 0 since a finish time could not be ascertained. To avoid losing discriminatory information from a large subset of track designs, we adapted our performance measure to enable differentiation between tracks based upon their characteristics before the point of failure. This adjusted measure considers the number of points the track successfully passed through, the speed of the track, and the percentage of the course that was successfully completed. We counted points only for the checkpoints that the track passed through that were located to the left of either the finish line or the point where the rider crashed, whichever came first. If the rider crashed before reaching the end of the course, we marked the point where the rider crashed on the course and extracted the x-coordinates. We used this value to calculate the percentage of the course that the rider completed. Finally, we calculated the time it took the rider to reach the end of the course or crash, whichever came first. We compared this time to a default time, which we calculated as the time it took a rider to reach that same x-coordinate while riding down a straight diagonal line. Our measure of speed is the default time divided by the actual time so that faster tracks resulted in more points. We calculated the final score by multiplying the number of successful points, the speed ratio, and the percentage of the course completed. For completed tracks, there was a high correlation between the score that was provided during the study and our updated measure (r = 0.98, p < .001). Note that Brainstorming Phase track performance (which the experimenters calculated after the study was completed and which participants were unaware of) is referred to as team member expertise when examined as a predictor of influence and team performance.
Level of influence
We calculated the level of influence that an individual’s track in the Brainstorming Phase had over its associated track in the Design Phase by calculating the area between these two lines (rgeos package (Bivand and Rundel, 2023); Figure 2). We divided a given area by the maximum possible area between lines (i.e., forming a rectangle encompassing the entire design environment) resulting in a measure of the normalized difference between tracks. We interpret normalized difference values close to 0 to indicate Brainstorming Phase designs that had a high level of influence on the Design Phase track and normalized difference values closer to 1 to indicate lower levels of influence.
Speaking time
We calculated the speaking time of participants in the Team Condition by extracting data from Zoom transcripts. Our Zoom account was set up to automatically record data to the cloud, including meeting audio and video (with timestamps) and an audio transcript. Transcripts record speaker identities, the start and end times of speaking bouts (herein referred to as speaking turns), and transcribed text. We derived our speaking time measure from transcripts by extracting speaker names and their speaking turns’ start and end times using an R script (R Core Team, 2022; O’Bryan et al., 2024). We divided each individual’s speaking time by the duration of the team’s interaction period in the Design Phase, which we calculated by subtracting the start time of the team’s first speaking turn from the end time of their team’s last speaking turn in the Design Phase. Due to data recording errors, we did not obtain automated speaking time data from three teams. Thus, analyses involving speaking time include data from N = 51 teams.
Screen control time
We calculated screen control time by coding the start and end time of the period in which each team member controlled the screen during the Design Phase and summing these durations for each team member (see Supplemental Materials). These measures were extracted by coding a Zoom-recorded video of the team’s shared screen using the program ELAN (version 6.3). Assistants were trained to code both the first and last time a given team member moved the mouse on the shared screen after requesting access. If a participant controlled the screen during multiple bouts between which another team member controlled the screen, a start and end time was determined for each separate bout. The screen control time for a given team member represents the sum of the duration of all their screen control bouts. Like speaking time, we divided screen control time by the duration of the team’s screen control period during the Design Phase, calculated as the end of the last screen control bout minus the beginning of the first screen control bout. Interrater reliability scores (ICC) for two pairs of coders who double-coded 17% of the data were 0.75 and 0.88, respectively.
Dominance
Before participating in the study, participants completed a questionnaire that assessed their personality and demographic characteristics. Dominance was measured via the 11-item dominance scale from the International Personality Item Pool (α = 0.82, Goldberg, 1999). Items were rated on a scale of 1 (very inaccurate) to 5 (very accurate). Example items include “Try to surpass others’ accomplishments” and “Try to outdo others.”
Idea similarity to mean
We calculated the team-level track that would be expected if team members converged to the mean of their three tracks. To do so, we found the mean y value for each x value present in the three tracks. We then calculated the normalized difference (see Level of Influence section above) between a given team member’s Brainstorming Phase track and this theoretical mean track. As it is common for teams to converge towards the mean, this measure controls for influence that may occur simply due to consensus processes.
Statistical analysis
Comparing performance across teams and individuals
All statistical analyses were executed in R. Due to the non-normality of the performance data, we used Wilcoxon Rank Sum tests to compare track performance between the Individual and Team Conditions in the Brainstorming Phase and the Design Phase. We used paired Wilcoxon Rank Sum tests to compare team performance in the Design Phase to their team’s worst-, median-, and best-performing brainstorming track scores. Effect sizes were calculated by dividing the Z statistic by the square root of the sample size using the wilcox_effsize function within the rstatix package (Kassambara, 2023).
Distribution of influence within teams
Due to the constraints of the experiment, it is possible that track designs could share similar properties, even if they were created entirely independently (i.e., in the absence of team member influence). Thus, we expected some similarity between tracks due to chance alone. To determine the distribution of normalized difference values that should be expected due to chance, we conducted a permutation analysis (Puga-Gonzalez et al., 2021) using the designs produced in our study. By comparing participants’ Brainstorming and Design Phase tracks, we calculated their levels of normalized difference, as described in the Measures section. We then compared these observed values to null distributions representing the absence of influence between Brainstorming and Design Phase tracks. We generated these null distributions by permutating Design Phase tracks across teams (in the Team Condition) and individuals (in the Individual Condition) and recalculating the normalized differences (see Supplemental Materials for more information). Within teams, we then compared the median observed normalized difference values displayed by the most, intermediate, and least influential ideas of each team to the distribution of median values expected due to chance alone within each influence category. For individuals, we compared median observed levels of normalized difference across all individuals to the distribution of values expected due to chance.
In addition to the above analysis, we used Wilcoxon Rank Sum tests to examine how observed levels of normalized difference values displayed by team member ideas (divided into influence categories) compare to the observed normalized difference values displayed by individuals’ ideas in the Individual Condition. Effect sizes were calculated as described above. These analyses enabled us to compare the influence displayed by team member ideas to the influence of ideas created by individuals working independently.
Characteristics associated with influence
We used generalized linear mixed-effects models to examine the characteristics associated with the influence team members’ ideas had over team outcomes. The predictor variables we considered were an individual’s expertise, speaking time, and dominance level, as well as the interactions between (1) speaking time and dominance and (2) speaking time and expertise. In addition, we controlled for a team member’s work contributions and their idea’s similarity to the mean. Due to differences in scale across model variables, all variables were normalized by subtracting the sample mean and dividing by the standard deviation using the scale function in R (R Core Team, 2022). Although all predictor variables in this analysis were measured at the individual level (Level 1), these individuals were grouped into teams. Thus, we use multilevel modeling with a random effect for Team to account for variation of the intercept between teams (i.e., “random intercept model”). Because the response variable was between 0 and 1, we used the glmmTMB function in R (Brooks et al., 2017) with a beta family and logit link function to fit the model. We verified model fit using the DHARMa package (Hartig, 2022).
Variation in performance across teams
We tested how the characteristics of central team members (the most influential team member of each team) impacted the performance of teams’ Design Phase tracks. Team tracks tended to closely match their most influential track, with a median [IQR] normalized difference value of 0.049 [0.026–0.092] (with 0 indicating a perfect match between designs and 1 indicating the maximum possible difference). However, the distribution of normalized difference values for these tracks ranged from 0.0091 to 0.32 (Supplemental Figure 1a). Because small differences in track design could result in big differences in performance, particularly if a change resulted in the rider crashing before reaching the end of the course, a high-quality influential brainstorming track is not guaranteed to lead to a high-quality team design.
Track performance in the design phase remained zero-inflated (4 out of 51, or 7.8%, of tracks) despite our revised performance score because a small subset of tracks failed before the rider passed through any checkpoints. Thus, we rounded performance scores to the nearest whole number (which generated three additional zeros) and used a zero-inflated linear model to model our performance data. This model separates the underlying process that generates extra zeros (using logistic regression) from that which produces non-zero values (using multiple regression with a Poisson error structure and log link function). Tracks could fail very early due to a variety of reasons, including poor track design or the presence of small bumps or gaps in the track. Thus, we used an intercept-only model to model excess zeros, meaning that the probability of observing a zero was modeled as a constant. The predictors we included for the multiple regression component of our model included all (scaled) characteristics of a team’s most influential team member (described above). This analysis enables us to determine how central team members’ characteristics related to variation in performance across teams. In addition to analyzing variation in overall performance, we examined variation in team performance relative to each team’s median-performing brainstorming score. For this analysis, the response variable was team performance in the Design Phase minus the performance of the team’s median-performing Brainstorming Phase track. As this measure was normally distributed, we used linear regression for this analysis. In addition, we group-mean centered the characteristics of teams’ most influential team members to reflect the value of the central individuals’ characteristics (e.g., expertise, speaking time, and dominance) relative to their team members. This analysis enabled us to test, for example, whether influential team members who had greater relative expertise compared to their team members were associated with teams that performed better than their median team member.
Results
Comparing performance across individuals and teams
There was no significant difference between the performance of Team and Individual Condition tracks in the Brainstorming Phase, when all participants worked independently (r = 0.016, W = 5867, p = .81, Figure 3(a)). However, Design Phase tracks performed significantly better in the Team Condition, when team members worked together, compared to the Individual Condition when individuals worked independently (r = 0.21, W = 2378.5, p = .022, Figure 3(a)). Figure 3(b) displays the performance of teams’ worst, median, and best-performing Brainstorming Phase tracks. Comparing these values to the performance of Design Phase tracks in the Team Condition, we found that teams’ Design Phase tracks performed significantly better than their worst-performing Brainstorming Phase track (r = 0.68, W = 1279, p < .001), no differently than their median-performing Brainstorming Phase track (r = 0.21, W = 918, p = .13), and significantly worse than their best-performing Brainstorming Phase track (r = 0.45, W = 355, p < .001; Figure 3(b)). Thus, teams performed better than individuals when team members worked together but did not match the performance of their best idea. Note. (a) Comparison of track performance across individuals and teams during the Brainstorming and Design Phases. Participants worked independently in the Brainstorming Phase across the Team and Individual Conditions. Participants worked together in teams of 3 in the Team Condition during the Design Phase. (b) Breakdown of the worst-, median-, and best-performing brainstorming phase tracks per team. Values at the top of the graphs represent sample sizes.
Distribution of influence within teams
In the Team Condition, the most influential idea of each team and the idea of intermediate influence had lower normalized difference values than expected due to chance alone, and thus greater influence (most influence: p < .001, intermediate influence: p < .001; Figure 4(a)). Levels of influence displayed by the least influential idea of each team did not significantly differ from chance (p = .086). Note. Error bars represent the 95% confidence intervals for the levels of normalized difference expected due to chance alone. These distributions were generated by permutating Design Phase tracks across teams (Team Condition) or individuals (Individual Condition). The black points represent the median values observed across participants. Team members are divided into whether they had the most, intermediate, or least influential ideas in the team, corresponding to the lowest, intermediate, and highest levels of normalized difference. *** indicates p < .001.
In the Individual Condition, individuals’ ideas had lower normalized difference values than expected due to chance (p < .001; Figure 4(b)) indicating that individuals’ Brainstorming Phase tracks had greater influence on their Design Phase tracks than expected. On average, the influence an individual’s initial idea had on their own subsequent track design was not significantly different from the influence a team’s most influential idea had on their team’s design (r = 0.17, W = 1529, p = .053). However, individuals’ ideas had significantly greater influence than team member ideas with intermediate influence (r = 0.31, W = 2603, p < .001) and lowest influence (r = 0.55, W = 3153, p < .001).
These results validate our influence measure by indicating that it reflects meaningful similarities between participants’ designs beyond those expected due to chance alone. Even more, they indicate that team designs were influenced by one team member’s idea to the same degree that individuals’ designs were influenced by their own initial idea in the Individual Condition.
Characteristics associated with influence
Means, Standard Deviations, and Correlations Among Study Varibles.
Note. N = 153 team members within 51 teams. M and SD represent mean and standard deviation, respectively. Values in square brackets indicate the 95% confidence interval for each correlation. Influence and similarity to mean are reverse coded, with smaller values indicating higher levels of influence and similarity, respectively. *p < .05. **p < .01.
Regression Coefficients for Models of Influence and Performance.
Note. Regression coefficients (with SE in parentheses) for models of team member influence, team performance, and team performance relative to the performance of the team’s median Brainstorming Phase track score.
†p < .10; *p < .05; **p < .01; ***p < .001.
aN = 153 individuals nested within 51 groups. All variables were scaled before being entered into the model. We included Team as a random effect due to the nested nature of the data.
bN = 51 teams. Variables reflect the characteristics of the team’s most influential team member. All variables were scaled before being entered into the model.
cN = 51 teams. Variables reflect the characteristics of the team’s most influential team member. All variables were scaled and group-mean centered before being entered into the model.
Central individuals and performance
As none of the interaction terms in our models of team performance were significant, we removed them from the final models. The results of these reduced models are reported below and all regression model results can be found in the Team Performance and Relative Team Performance columns in Table 2. We found that the central (i.e., most influential) team member’s expertise (B = 0.19, SE = 0.058, p = .0013) and their work contributions (B = 0.17, SE = 0.066, p = .012) were positively correlated with team performance, with no effect of their speaking time (B = −0.11, SE = 0.060, p = .074), dominance level (B = −0.047, SE = 0.072, p = .51), or the similarity of their idea to the mean (B = −0.037, SE = 0.053, p = .48). Thus, teams performed best when influential team members had high-quality ideas and when they had greater control over work contributions. None of the group-mean-centered predictors were significantly correlated with relative team performance (i.e., how a team performed relative to their median-performing brainstorming track score; Table 2).
Discussion
This study’s unique contribution is the expansion of methods that can be used to examine mechanisms by which team members’ ideas influence team outcomes. To this end, we developed and validated an objective measure of influence that focuses on the measurement of shifts in ideas over time. Expanding upon previous studies that have focused on objective measures of influence in rank-based tasks (Bottger, 1984; Littlepage et al., 1995; March, 1956), we develop an objective measure of influence within the context of an open-ended creative design task. We demonstrate that our influence measure reflects meaningful similarities between designs and correlates with key predictors and proxies of influence, including expertise, work contributions, and where an idea falls relative to the mean of all team members’ ideas. We interpret these findings within the context of teams that performed better than independent individuals but no better than their median team member. Furthermore, we use our measure to identify influential team members and explore which of their characteristics are associated with greater team performance. Overall, our study demonstrates the validity of our measure and how it can clarify the mechanisms through which teams perform better than individuals.
In line with previous studies (Lorge et al., 1958; Salas et al., 2018), the teams in our study performed better, on average, than individuals working independently even though there was no difference in brainstorming track performance between participants in the Team and Individual Conditions. Thus, the teams in our study displayed a collective advantage compared to individuals who continued to work alone. However, since teams did not perform any differently than the median performance of their team members’ brainstorming ideas and underperformed compared to their highest-performing idea, teams did not display process gains that enabled them to perform better than the individual members of their team. Although these findings contradict some reports of teams performing better than their average team member (Lorge et al., 1958) and even their best team member (Michaelsen and Watson, 1989; Nemiroff and King, 1975), they are in line with other studies that have found that teams often fall short of these markers of a collective advantage (Hackman, 2002; O’Neill and Salas, 2018). Comparing team to individual performance (both individuals working independently and individuals within a given team) is an important step for identifying whether mechanisms of idea integration are associated with process gains or losses (Han et al., 2022).
A key contribution of our study is that we combine our assessment of teams’ collective advantage with analyses of how ideas were integrated within these teams. First, we validated our influence measure by demonstrating that it reflects levels of similarity between designs that are greater than those expected due to chance alone. This demonstration leverages the quantitative nature of our measure which reflects changes in individuals’ contributions over time and cannot be performed using measures of attributed influence or participation. In addition to validating our measure, we also used this analysis to identify the degree to which the three initial ideas within each team influenced the final team outcome. We found that the most influential idea within teams had high levels of influence (comparable to the level of influence individuals’ ideas had over their own subsequent designs) and that team member ideas of intermediate influence also had greater influence than expected. Thus, the teams in our study gravitated towards some ideas over others, rather than integrating them more equally. This finding may be due to some ideas having incompatible characteristics, such as a track following a route towards the top of the course and another towards the bottom. This is because the simulated rules of physics within the game limited how the rider moved along tracks. Indeed, studies of engineering design teams have found that teams tend to select feasible designs over original designs to reduce uncertainty (Rietzschel et al., 2010; Starkey et al., 2016). Thus, the constraints of our task may have led participants to avoid more creative ideas in favor of ones they anticipated to perform effectively. Our approach to determining how ideas are integrated into team outcomes can facilitate the identification of which methods of idea integration are most effective and under which contexts. By building upon our approach, future studies could test both the task characteristics (i.e., how well full or partial ideas can be merged) and team interaction processes that promote more or less equal methods of idea integration. For example, Woolley et al. (2015) found that teams that display more equal speaking turns across team members perform better. Using our approach, it could be possible to test whether teams that display more equal speaking turn patterns tend to integrate their ideas more equally and how this relationship depends upon both task characteristics and the distribution of expertise within the team (Haan et al., 2021).
An additional advantage of our influence measure is that it can be applied to individuals working independently (i.e., the influence individuals’ ideas have on their subsequent designs) in addition to those working as part of a team. Doing so can provide insight into how the presence or absence of social interaction impacts influence and idea evolution. We found that both individuals and teams strongly converged towards a single previous idea. This finding contrasts with previous studies indicating that individuals working in the absence of social influence tend to explore the decision space more thoroughly (i.e., producing a wider range of ideas) while individuals who are exposed to others’ ideas tend to converge towards those ideas (Bernstein et al., 2018; Lorenz et al., 2011). By using our approach to compare the levels of influence across team members and individuals, future studies can test the conditions under which teams explore the decision space more or less thoroughly than individuals working independently. For example, studies could explore how teams’ tendencies to explore the decision space are impacted by the use of different concept selection tools (i.e., methods for evaluating, choosing, and synthesizing design alternatives) (Zheng et al., 2018) or by training that emphasizes convergent versus divergent thinking (Hirshfield and Koretsky, 2021).
We found that expertise was positively correlated with our influence measure, which is in line with previous studies focused on other objective measures of influence (Bottger, 1984; Mayo et al., 2020). Furthermore, our measure correlated with team member work contributions, which represent a measure of participation and proxy of influence. This result is consistent with previous findings indicating that greater participation can boost the influence one’s ideas have over team outcomes (Bottger, 1984; Haan et al., 2021; Mayo et al., 2020). Team members’ work contributions were significantly correlated with speaking time and dominance even though these variables were not directly associated with our influence measure themselves. Although verbal communication is commonly thought to impact influence (Bottger, 1984; Sherf et al., 2018), the dual communication modalities available in our study (verbal communication and drawing on the shared screen) may have reduced the effect of speaking time on influence. On the other hand, some studies have found that even though speaking time is positively associated with perceived influence and leadership, it does not correspond with objective influence (Bottger, 1984; MacLaren et al., 2020). Thus, our study’s focus on objective measures of influence may be another reason why we did not find an effect of speaking time (though see Mayo et al., 2020). Expanding the use of objective influence measures, such as that developed in our study, can help to clarify the relationships between influence measures focused on shifts in opinion or behavior and those focused on measures of participation. In future studies, our approach could be used to test the conditions under which participation measures correlate with objective influence, such as by manipulating the number or type of modalities that are available for communication and/or the importance of expertise to a given task (Engel et al., 2014). For example, the communication modality in the Line Rider task could be restricted to only verbal communication by having a separate period of discussion and decision-making prior to any design creation.
Finally, influence was correlated with an idea’s similarity to the mean of all team members’ ideas, which is consistent with many studies emphasizing the impact of consensus processes on influence (De Dreu and West, 2001; Gigone and Hastie, 1997; Hackman and Morris, 1975; March, 1956). Furthermore, this may explain why the two most influential members of each team both had greater influence than expected. If two team members’ ideas happened to be similar to one another, they would pull the group mean (and the team’s decision-making) towards their designs. Unfortunately, team members with greater expertise tended to have ideas that were further away from the mean. Thus, the tendency to choose ideas that were more similar to the mean could have pulled teams away more unique, but better, ideas and could have led some teams to underperform (De Dreu and West, 2001). Nevertheless, we found that teams were influenced by expertise even after controlling for the tendency to converge towards the mean.
Although the final designs of both teams and individuals closely matched one initial idea, teams likely benefited by having access to a greater diversity of ideas to select from (i.e., one from each team member). Overall, teams displayed higher performance when the most influential idea within the team displayed greater expertise and when its creator made more work contributions. This finding is in line with previous studies that have found that teams benefit when good ideas align with higher levels of participation (Bottger, 1984; Haan et al., 2021). Thus, rather than prioritize equal contributions by all team members, teams may benefit from environments in which the best ideas are free to come forward and influence team outcomes. Nevertheless, studies have found that more equal levels of participation may benefit performance by facilitating the identification of those with greater expertise (Haan et al., 2021). Thus, participation processes may play a distinct, but complimentary, role from influence processes (Haan et al., 2021; Mayo et al., 2020). Objective measures of influence, such as the one developed in our study, can contribute towards our understanding of this relationship by disentangling which measures of participation are directly related to influence and which are associated with more faciliatory roles.
One limitation of our approach is that our measure of team member expertise reflected the objective performance of their design in the Brainstorming Phase (which the participants could not test during the study). Since small errors in the track (e.g., bumps and gaps) had the potential to impede performance, our measure may have underestimated the quality of some tracks and may not have captured a track’s future potential (Girotra et al., 2010). An additional limitation of our influence measure is that it only takes into account an individual’s initial idea. Although individuals may come up with additional ideas as they work with their team members, our approach currently does not take these additional inputs into account. However, as team designs tended to closely match the initial idea of one team member, capturing additional ideas may not be as important as focusing on team members’ initial ideas. Another limitation of our influence measure is that while it can identify influential ideas, it does not necessarily identify influential individuals because the creator of a given idea may not be the one who recognizes its value and promotes it within the team (Toh and Miller, 2014). To investigate the importance of this facet, our measure could be combined with peer ratings of perceived influence and/or participation measures generated through content analysis which could clarify which team members support which ideas during verbal interactions. Thus, combining our objective measure of influence with additional measures (i.e., attributed influence and measures of participation) may provide a more complete picture of influence processes.
By developing an objective measure of influence within the context of an open-ended design task, our study provides a novel approach to studying how team members’ ideas, behaviors, and traits impact influence processes during team decision-making. By improving our ability to study these influence processes in an objective and quantitative manner, we can not only better understand the influence processes associated with the emergence of collective intelligence but also gain a more precise understanding of how to improve these processes. Finally, by moving away from more subjective measures of influence, such as self-report, and towards more objective measures, we may enhance our ability to compare the influence processes underlying collective intelligence and other forms of enhanced group-level abilities observed across biological, social, and artificial systems (O’Bryan et al., 2020). For example, as animal trajectories are also typically represented as lines along an x- and y-axis, our measure could be used to quantify how group members are influenced by others’ preferred trajectories when traveling together (Biro et al., 2006). By further developing and utilizing such objective influence measures, we can gain greater insight into similarities and differences in the influence processes underlying collective decision-making and intelligence across social groups.
Supplemental Material
Supplemental Material - A novel approach to studying the role influence plays in team collective intelligence
Supplemental Material for A novel approach to studying the role influence plays in team collective intelligence by Lisa R O’Bryan, Timothy Oxendahl, Simon Garnier, Santiago Segarra, Matthew Wettergreen, Ashutosh Sabharwal and Margaret E Beier in Journal of Collective Intelligence.
Footnotes
Acknowledgments
We would like to thank the undergraduate research assistants in the Adult Skills and Knowledge Lab who aided the development of the study task, the facilitation of experiments, and the coding of experimental data.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this project was provided by a Microsoft Productivity Research Grant and the National Science Foundation (Award Number: 1910117).
Data Availability Statement
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
