The Computer-Mediated Online Round Robin (CMORR): An Online Method for Studying Impressions and Social Interactions

Abstract

Research on impressions and social interactions has predominately examined perceptions of artificial stimuli or those made by convenience samples of undergraduates. In the present work, we introduce and validate a new experimental method, the Computer-Mediated Online Round Robin (CMORR), with the aim of providing researchers a tool to extend the study of interpersonal phenomena to more diverse populations. We describe the method and provide guidance for future CMORR studies. We collected CMORR data from an undergraduate sample (N = 171), and compared the structure and accuracy of impressions of Big Five personality trait to two in-person studies; one with group interactions (N = 225), one with dyadic interactions (N = 511), and meta-analytic estimates from the literature. The results showed a general correspondence between impressions formed in online interactions and in in-person contexts. The findings support using CMORR to study general questions about impressions and social interactions.

Keywords

person perception research methods personality social interaction impression formation social cognition individual differences

The COVID-19 pandemic forced many in-person interactions to be moved online, making an important and lasting impact on the nature of human social interactions. Even people who were initially unaccustomed to online video interactions have adapted to this new reality (McClain et al., 2021). The widespread adoption of videoconference interactions as a substitute for in-person ones presents a new opportunity to expand the study of social interactions and impression formation, and to study these phenomena in interactions among people from diverse backgrounds. To advance that goal, we adapted round-robin methods from interpersonal perception to online, video-mediated interactions. In this article, we describe the method, and we compare key features of data from online and in-person settings using a well-established model of interpersonal perception, the social relations model (SRM; Kenny & La Voie, 1984).

The Interpersonal Perception Paradigm: Opportunities and Barriers

Understanding the impressions people form of others, especially first impressions, has long been a goal of psychological research. For example, early work by Asch (1946) examined how individuals incorporate a multitude of distal characteristics into the impression of a single person. More recently, psychological research on impressions and impression formation has taken two distinct methodological approaches.

Social-Cognitive Approaches

One major paradigm is social cognition, which prioritizes internal validity and experimental control. In studies of social cognition, participants are generally asked to make judgments about static or hypothetical stimuli (e.g., pictures or vignettes). The goal of these studies is often to isolate a single feature of targets (e.g., group membership, gender) to determine how this specific feature changes perceptions of the other characteristics of the target (Fiske & Taylor, 1991) or alter decisions made about the target. Because the social cognition experimental paradigm does not require social interactions, it has been relatively straightforward to adapt to online settings.

The Interpersonal Perception Approach

The other primary approach to studying impressions, interpersonal perception (Kenny, 1994, 2019), was developed to study impressions formed during social interactions between participants. Interpersonal perception research relinquishes experimental control of a target’s attributes, favored in social cognition, and instead uses multiple-rating designs (e.g., round-robin, full-block) and advanced statistical models, such as the SRM (Kenny & La Voie, 1984), the social accuracy model (SAM; Biesanz, 2010; Human & Biesanz, 2011), and others, to make precise estimates of interpersonal effects from noisy social interaction data.

A significant barrier to realizing the full potential of the interpersonal perception approach is that to date, interpersonal perception studies have relied heavily on convenience samples, which are often unrepresentative and homogeneous. This is especially true of research on first impressions, which has been conducted almost exclusively with samples of undergraduate students. Indeed, we reviewed 10 years (2010–2020) of entries into the SRM reference list, maintained by David Kenny (available at http://davidakenny.net/srm/srm.htm), and found that only three of the 125 empirical articles featured unacquainted, non-student participants. All used data from the same speed dating study (Asendorpf et al., 2011; Back, Penke, Schmukle, & Asendorpf, 2011; Back, Penke, Schmukle, Sachse, et al., 2011).

This raises important questions about the generalizability of previous work outside of this population. College samples at research universities are less diverse than the general population in important ways, including age, race/ethnicity, and socioeconomic status. This lack of diversity can have several effects. One broadly applicable one, which has been widely discussed, is that of external validity—unrepresentative samples may not generalize to broader populations (Henrich et al., 2010; Syed, 2021). A second problem of special relevance for interpersonal perception research, is the homogeneity itself. Many of the key parameters estimated in the SRM, SAM, and other models are variances and covariances. A lack of within-sample variability will lead to systematic biases in estimates of consensus, accuracy, and other important phenomena. This might help explain why Borkenau and Liebler (1992) found much higher levels of target variance in a community sample than is typical in undergraduate samples.

Interpersonal Perception Research and Computer-Mediated Communication

Research in computer-mediated communication (CMC) suggests that modern videoconferencing can approximate face-to-face interactions much better than sparser channels like text messaging (Antheunis et al., 2020). Interpersonal perception work supports this, showing some differences in impressions formed during text-based CMC, to those formed during in-person interactions (Liao et al., 2018; Mohr, 1997). More recent work has extended this approach to video-mediated interactions, testing for differences in meta-impressions between in-person and Zoom interactions (Tissera et al., 2023). This work found that meta-perceptions were very similar across the communication medium, suggesting a similar impression formation process occurs in both types of interactions.

A New Approach: The Computer-Mediated Online Round-Robin

To advance interpersonal perception research to more diverse and harder-to-reach populations, we developed an interactive online data collection paradigm called the computer-mediated online round-robin (CMORR; “see-more”). CMORR leverages the widespread availability of computers, the internet, and videoconferencing software to study interactions between participants from anywhere in the world.

The availability of videoconferencing and broad adoption of online meetings makes it appear straightforward to move the study of social interactions and impressions online. But, as we discovered when we tried to run a round-robin study online, there are important considerations and challenges that need to be addressed to limit attrition, collect high-quality data, and provide participants a seamless experience.

General Considerations in Round-Robin Studies

When conducting a round-robin study, there are several general considerations: (1) should participants interact as a group or dyadically? (2) how long should participants spend interacting? (3) how many ratings or judgments should participants make of one another? and (4) how big should the groups and sample be? Decisions about study design are typically based on a mix of domain knowledge, statistical considerations, and practical limitations. With proper planning and support, CMORR can be used to study both group and dyadic interactions among people with different relationships (e.g., strangers, coworkers, friends) and in different social contexts.

Online-Specific Challenges and Solutions

CMORR sessions are complex. A six-person dyadic CMORR study session features five interaction rounds, and each round has three simultaneous dyadic interactions (see Figure 1). Pilot sessions showed that even with clear instructions and dedicated support, participants found it challenging to navigate between different software and browsing windows for the interaction and survey. To address this, we eliminated the need to move back and forth by embedding videoconference rooms directly into the survey software. To increase the accessibility of participating in a CMORR study, we chose videoconferencing software that did not require a software download. We found that Jitsi Meet (https://meet.jit.si) provided an easy way to embed the virtual rooms into a Qualtrics survey using html code. Jitsi Meet is open source, and features end-to-end encryption, making it ideal for scientific use, but there are a growing number of software options for researchers to choose from to study video-mediated interactions online (e.g., Brodsky et al., 2022; Molnar, 2019). Other challenges included: how to ensure participants follow instructions and how to record the interactions. These were addressed by having a well-trained team of trained research assistants (RAs) to monitor the study in real time. Each six-person CMORR session requires a minimum of three RAs, one in each virtual room to supervise and record the interactions.

Figure 1

Interaction Flow for a Six-Person CMORR Session

Comparing CMORR and In-Person Impressions

Before applying CMORR to study novel populations or ask new questions, it is important to evaluate whether current videoconferencing technology has enough rich behavioral information that personality impressions from CMORR interactions are similar to those from in-person interactions. Because the majority of interpersonal perception research has been conducted with undergraduates who are able to come to a research lab, we chose to conduct an initial CMORR in that population as well. This enabled us to make comparisons across media (CMORR vs. in-person) without changes in population. If CMORR produces comparable results, it would support the validity of future CMORR studies in new populations.

To assess similarity in impressions across mediums, we focused our evaluation on several key parameters of the SRM: the relative size of variance components, the cross-trait covariances among the perceiver effects and among the target effects, and self-other agreement. These parameters are often either the focus of SRM studies or are examined as a preliminary step to more complex analyses that depend on them. There is an extensive body of research on them with in-person designs, making them a useful and important basis for comparison.

Variance Components

An SRM variance decomposition shows how much variance in ratings can be attributed to perceivers, targets, and the unique relationships between perceivers and targets. Each of these components is associated with a fundamental question in interpersonal perception. Perceiver variance is an indicator of assimilation, the extent to which the same perceiver views different targets in the same way (Kenny, 1994, 2019). Target variance is an indicator of consensus, the extent to which different perceivers view the same target in the same way. Relationship variance cannot be differentiated from error variance in an SRM with a single indicator of the perception. Therefore, in our comparison, we focused on perceiver and target variance.

If video-mediated and in-person interactions provide a similar opportunity for the expression, availability, detection, and utilization of behavioral information (Funder, 1995), we would expect perceiver and target variance to be in the range of previous research. If the medium of interaction impacted the expression and detection of relevant behavioral cues, we would expect to see differences in these variance components. For example, if behavioral cues were degraded in CMORR interactions, we might see an increase in perceiver variance. Perceivers would have to rely on heuristics or global judgments instead of behavioral observation.

Correlations Among Perceiver and Target Effects

In addition to estimating sample-level variances, the SRM provides individual-level effect estimates for each component. Perceiver effects indicate how each participant tends to rate others, and target effects indicate how each participant tends to be rated by others. Correlations among perceiver effects of Big Five traits show whether the tendency to rate others as high or low in one trait is associated with rating tendencies in other traits. For example, do people who tend to rate others high in extraversion also tend to rate others high in conscientiousness. Similarly, correlations among target effects show whether a person perceived as high or low in a trait is also perceived as high or low in other traits. Differences between CMORR and in-person studies in the direction or strength of these correlations would suggest that people are using a different process or information to inform their trait judgments. For example, if participants cannot discern trait-specific information about targets, they might fall back on more holistic judgments based on global positive or negative evaluation. This would lead to strong correlations among perceiver effects that are in a consistent evaluative direction.

Self-Other Agreement

Self-other agreement, a common indicator of accuracy, is indexed by correlations between the target effects and self-reports of the same trait (Connelly & Ones, 2010; Kenny, 1994). Some traits, such as extraversion, are typically judged more accurately than others, such as neuroticism, an effect that has previously been attributed to how easy it is for others to observe these traits (Funder & Dobroth, 1987; Vazire, 2010). Much like structure, if the medium of interaction impacts the expression, availability, detection, or utilization of behavioral cues of personality (Funder, 1995; Gosling et al., 2008), we would expect accuracy in the perception of these traits to be affected.

The Present Study

To assess the similarities and differences between CMORR and in-person impressions, we conducted a CMORR study with college undergraduates. We compare the results from this study to three data sources. We calculated the same statistics for two in-person studies, one featuring group interactions and one with dyadic interactions. The in-person group study was conducted with participants from the same population and used the same measure of Big Five traits (Soto & John, 2017), but differed in that participants interacted in groups and worked on a different task. The in-person dyadic study was conducted with undergraduates from a different university (Mignault et al., 2022), but like CMORR featured dyadic get-to-know-you interactions. These studies provide useful points of comparison, but neither should be interpreted as a pure control condition. To understand how CMORR impressions relate to the broader literature, we also compare CMORR results to data from meta-analyses of SRM studies. Meta-analytic data reflects heterogeneity in research questions, rating instruments, and other details. Thus, rather than using the comparison to pinpoint a single “correct” answer to compare CMORR results to, we use it to compare the CMORR results with a range of outcomes common in interpersonal perception research.

Method

The methods for the two studies collected in our lab, including procedures and materials, were preregistered on the Open Science Framework (OSF) prior to data collection: https://osf.io/y2rke/. The hypotheses included in these preregistrations are reported elsewhere and we collected additional measures, not reported in this article, to test those hypotheses. Further information is posted to OSF about the in-person group study: https://osf.io/95amb/ and the in-person dyadic study: https://osf.io/qrw3j/. None of the analyses in this article were preregistered.

We developed a CMORR Handbook with extensive documentation about building and conducting a CMORR study and made it publicly available online (https://osf.io/q9f6x). The handbook includes detailed instructions on creating surveys with example links to embed virtual rooms, recommendations about best practices for conducting a CMORR study with a community sample, and example protocols for RAs.

A round-robin study with a sample size of N = 139 has 92.5% power to detect a relative variance component of 10% (Lashley & Kenny, 1998; Salazar Kämpf et al., 2018). All three studies surpass this minimum sample size and therefore are adequately powered to reliably estimate the variance decomposition and accuracy (indexed as self-other agreement) of impressions of big five traits.

CMORR Study

Participants

CMORR data come from a sample of 187 undergraduates participated in groups of five or six, who received partial course credit for participating. Three groups were missing perception data for multiple targets and were removed from the final analysis, excluding 16 participants. The final sample of N = 171 participated in 30 groups, and self-reported the following demographics: M_age = 20, SD_age = 2; 74% women (three participants identified as gender non-conforming); 63% White, 13% Asian; 6% Latino/Latina, 2% Black, and the other 16% of participants selected multiple responses or “other.”

Procedure

Participants were scheduled in groups of six. Due to no-shows, sessions were run with groups of five or six participants. Upon arrival, participants were escorted to a private room, consented, and an RA provided a brief overview of the CMORR procedure. Participants entered the survey and self-reported Big Five personality traits and provided demographic information. Then, they followed a link to a virtual room where they were met by an RA and another participant. The RA provided the instructions for the get-to-know-you task, which consisted of taking turns answering five personality relevant questions for 5 min. After answering any questions, the RA posted the discussion questions to the chat in the virtual room, and began recording the interaction. The RA then “left” the virtual room by muting their audio and video (though participants were aware that the experimenter was able to see and hear them and that the interaction was being recorded).

During each round of interactions, three dyads interacted in three virtual rooms. After 5 min, the RA unmuted their audio and video, and informed participants to exit the virtual room and return to the survey to provide impressions of their interaction partner. Then, participants followed a link to a different virtual room where they met and interacted with another participant. This process was repeated until the participants had interacted with and rated each other member of the group.

In-Person Group Study

Participants

In-person group data come from a sample of 247 undergraduate students who participated in 48 groups of four to six in exchange for partial course credit. In accordance with the preregistration, 22 participants in eight groups were excluded from the analysis. The final sample consisted of N = 225 participants who participated in 44 groups: M_age = 19, SD_age = 2; 68% women; 64% White, 7% Asian, 7% Hispanic, 2% Black, 0.8% Native American, and 20% selected multiple responses or “other.”

Procedure

Groups of six participants were scheduled. Due to no-shows, sessions were run with groups of four to six participants. Upon arrival, participants were each put into a private room and consented, self-reported personality, and responded to demographic items, including age, sex, and race and ethnicity. Next, participants were brought together, seated at a round table, and provided instructions for the Leaderless Group Discussion (LGD; adapted from DesJardins et al., 2015) task. In this task, participants assumed the roles of a scholarship committee and spent 20 min working as a group to allocate scholarships. After completing the group task, participants returned to private rooms and rated each other member of the group.

In-Person Dyadic Study

Participants

In-person dyadic data come from a sample of 557 undergraduate students who participated in 88 groups of four to eight in exchange for partial course credit (Mignault et al., 2022). We excluded perceptions made between previously acquainted participants and then excluded groups with missing data. In total, 46 participants in nine groups were excluded. The final sample consisted of N = 511 participants who interacted in 79 groups: M_age = 20, SD_age = 2; 84% women (four did not respond); 73% White, 7% Asian, 8% Black, 1% Middle Eastern, and 20% selected multiple responses or “other.”

Procedure

Participants arrived in the lab in groups of four to eight and were greeted by an RA who provided an overview of what they would be doing during the study and informed consented. Participants were asked to volunteer to wear physiological measurement devices during the study and up to three were fitted with the equipment prior to the round-robin interactions. Participants self-reported personality traits before being paired with another participant for the first interaction. For the dyadic interactions, participants were instructed to: “just introduce yourself, and try to get to know one another.” After each interaction participants rated the personality traits of their interaction partner before being paired with another participant for the next interaction. Each interactions lasted between 2 and 3 min. After interacting with and rating each other member of the group, participants self-reported demographics.

Measures

CMORR Study and In-Person Group Study

Self-Report

Big Five Inventory 2-XS (Soto & John, 2017). The “extra-short” Big Five Inventory 2 (BFI-2-XS) has 15 items, three for each domain. Respondents indicated agreement to each item with a 5-point scale anchored at 1—strongly disagree and 5—strongly agree. Scores for BFI-2-XS traits were calculated by reverse scoring and averaging the associated items.

Perceptions

Big Five Inventory 2-XS other report (Soto & John, 2017). A modified version of the BFI-2-XS modified for other reports. Respondents provided ratings of their interaction partner by indicating agreement to the items on a 5-point scale anchored at 1—strongly disagree and 5—strongly agree. Scores for BFI-2-XS traits were calculated by reverse scoring and averaging the associated items.

In-Person Dyadic

Self-Report

Big Five Inventory (John & Srivastava, 1999). The BFI has 44 items. Respondents indicated agreement to each item with a 7-point scale anchored at 1—strongly disagree and 7—strongly agree. Scores for the BFI traits were calculated by reverse scoring and averaging the associated items.

Perceptions

A 21-item modified version of the BFI modified for other reports (John & Srivastava, 1999). Respondents provided ratings of their interaction partner by indicating agreement to the items on a 7-point scale anchored at 1—strongly disagree and 7—strongly agree. Scores for the BFI traits were calculated by reverse scoring and averaging the associated items.

Results

Descriptive statistics and reliability for self-report and perceived Big Five traits are presented in Tables 1 and 2, respectively. For each Big Five trait, we conducted an SRM analysis (Kenny, 1994; Kenny & La Voie, 1984) using the TripleR package (Version 1.5.4; Schönbrodt et al., 2012, 2016) in R (Version 4.3.2; R Core Team, 2023).

Table 1

Descriptive Statistics for Self-Report Big Five Traits

Trait	CMORR^a (N = 171)			In-person group^a (N = 225)			In-person dyadic^b (N = 510)
	M	SD	α	M	SD	α	M	SD	α
Extraversion	3.11	0.91	.70	3.16	0.88	.67	4.42	1.01	.87
Agreeableness	3.88	0.76	.59	3.82	0.70	.46	5.24	0.76	.77
Conscientiousness	3.51	0.77	.47	3.43	0.75	.54	4.86	0.82	.80
Neuroticism	3.27	1.00	.76	3.19	0.88	.67	4.23	1.07	.85
Openness	3.78	0.77	.62	3.76	0.74	.58	4.91	0.86	.81

Measured on 5-point scale. ^bMeasured on 7-point scale.

Table 2

Descriptive Statistics for Perceptions of Big Five Traits

Trait	CMORR^a (N = 802)			In-person group^a (N = 932)			In-person dyadic^b (N = 2764)
	M	SD	α	M	SD	α	M	SD	α
Extraversion	3.13	0.87	.73	3.21	0.96	.80	4.59	1.10	.90
Agreeableness	3.93	0.70	.70	3.79	0.64	.65	5.08	0.83	.77
Conscientiousness	3.47	0.77	.58	3.60	0.70	.67	5.10	0.79	.77
Neuroticism	2.83	0.69	.56	2.62	0.66	.62	3.13	0.81	.64
Openness	3.55	0.66	.58	3.43	0.62	.57	4.81	0.87	.52

Measured on 5-point scale. ^bMeasured on 7-point scale.

Variance Components

The standardized SRM variance components for the CMORR study are presented in Table 3.

Table 3

SRM Standardized Variance Components

	Perceiver			Target			Relationship/error
Trait	CMORR	In-person dyadic	In-person group	CMORR	In-person dyadic	In-person group	CMORR	In-person dyadic	In-person group
Extraversion	.07	.07	.05	.40	.48	.57	.53	.45	.38
Agreeableness	.25	.40	.32	.19	.11	.17	.57	.50	.51
Conscientiousness	.15	.39	.31	.30	.11	.16	.55	.50	.54
Neuroticism	.21	.31	.38	.19	.09	.09	.60	.59	.53
Openness	.21	.33	.23	.19	.08	.15	.59	.58	.62

Note. All variance components significantly different from zero (p < .01).

Perceiver Variance

Perceiver variance has been estimated in a large number of studies, providing a range of expected values for CMORR perceiver variance if it is comparable with those made in-person. To provide point-estimate comparisons, we also report the variance components from the in-person studies in Table 3. We estimated the range of expected values as plus or minus two standard deviations from the meta-analytic mean of standardized perceiver variance (from Table 2 of Rau et al., 2021). Because the CMORR study is of initial impressions formed during social interactions, we included initial interaction studies and the first wave of longitudinal studies, and excluded two studies examining impressions of videos. The anticipated range of perceiver variance and the point-estimates from both the CMORR study and in-person studies are plotted in Figure 2. The figure shows that the amount of perceiver variance in impressions formed during CMORR are in the expected range.

Figure 2

Perceiver Variance: Assimilation

Target Variance

For target variance, we estimated the anticipated range as plus or minus two standard deviations from the meta-analytic of standardized target variance (Table 1; Kenny, 2004). We again included the first wave of longitudinal studies. The anticipated range of target variance and the point-estimates from the CMORR study and the in-person studies are plotted in Figure 3. The figure shows the estimates of consensus, as indicated by target variance, in the CMORR study are in line with estimates from previous in-person studies.

Figure 3

Target Variance: Consensus

Correlations Among Perceiver Effects and Target Effects

The correlations among perceiver effects for the Big Five are presented in Figure 4. There is notable similarity between the CMORR and in-person studies. The between trait correlations for perceiver effects are all in the same direction. The CMORR correlations are generally smaller in magnitude than in either of the in-person studies, which suggest that people differentiated between the personality characteristics of others better in the CMORR interactions. This is the opposite of what we would expect if the interaction medium was causing CMORR participants to make low-effort ratings or basing them on global evaluation rather than considering different traits separately. Similarly, seven of 10 correlations among target effects (Figure 5) are in the same direction. For two of the exceptions, the CMORR ratings are in the same direction as one of the other studies (extraversion and neuroticism, conscientiousness and openness), and for the other (agreeableness and openness), the correlations for CMORR and the in-person group study are very near zero. There is no consistent pattern of one study having stronger or weaker correlations between target effects than the other. The correspondence between the perceiver and target effect correlations among the studies supports that impressions formed in CMORR and those formed in-person share a similar structure.

Figure 4

Correlations Among Perceiver Effects in CMORR and In-Person Perceptions of Big Five Traits

Figure 5

Correlations Among Target Effects in CMORR and In-Person Perceptions of Big Five Traits

Self-Other Agreement

We calculated self-other agreement as the correlation between self-reports and target effects of the same Big Five trait, controlling for group (Table 4), and tested for differences between these correlations using the test of the difference between two independent correlation coefficients (Table 5; Cohen & Cohen, 1983; Preacher, 2002). There were significant differences in self-other agreement between CMORR and both in-person studies in perceptions of conscientiousness and neuroticism.

Table 4

Self-Other Agreement for the CMORR Study and In-person Study

	CMORR		In-person dyads		In-person group
Trait	r	p	r	p	r	p
Extraversion	.44	< .001	.46	< .001	.39	< .001
Agreeableness	.30	< .001	.17	< .001	.18	.016
Conscientiousness	.38	< .001	.12	.010	.07	.341
Neuroticism	.36	< .001	.18	< .001	.04	.562
Openness	.30	< .001	.18	< .001	.19	.010

Table 5

Differences in Accuracy by Interaction Medium

	CMORR/in-person dyad		CMORR/in-person group		In-person dyad/in-person group
Trait	z	p	z	p	z	p
Extraversion	–0.28	.778	0.59	.555	1.06	.288
Agreeableness	1.55	.121	1.25	.212	0.13	.898
Conscientiousness	3.14	.002	3.23	.001	0.63	.530
Neuroticism	2.19	.029	3.29	< .001	1.76	.078
Openness	1.43	.152	1.15	.252	–0.13	.898

Note. The test for difference between two independent correlations was conducted with an online interactive calculator (http://quantpsy.org; Preacher, 2002) that converts the correlations to a z-score using Fisher’s r-to-z transformation and then compares z-scores with a two-tailed test.

p < .05 are in bold.

To examine whether self-other agreement in CMORR was in line with previous work, we estimated the anticipated range of self-other agreement as plus or minus two standard deviations from the mean of a meta-analysis of the accuracy of big five trait perceptions (Table 5 of Connelly & Ones, 2010). The anticipated range of self-other agreement from the meta-analysis, and the point-estimates from the CMORR study, the in-person group study, and in-person dyadic study are plotted in Figure 6. The figure shows that self-other agreement in a CMORR study are in line with estimates from previous in-person studies and supports that personality is perceived with similar accuracy in initial online and in-person interactions.

Figure 6

Self-Other Agreement in Perceptions of Big Five Traits

Discussion

Research on impression formation and social interactions would benefit from studying these phenomena in more diverse populations online. We presented a method for doing so, CMORR, and used it to collect impressions of Big Five personality domains. Pervasive similarities across the results support the conclusion that current videoconferencing technology provides enough rich behavioral information to make CMORR suitable to test hypotheses and answer general questions about impressions and the impression formation process. The results support that similar relevant behavioral cues are available, detected, and utilized (Funder, 1995) by perceivers in initial video-mediated and in-person social interactions. There were not substantial differences in assimilation (perceiver variance), which could have meant that online perceivers are more reliant on personal heuristics, such as response style; or in consensus (target variance), which indicates that relevant cues were available and detected in CMORR. Estimates of self-other agreement for the CMORR study were near the meta-analytic mean for all traits further supporting that modern video interactions provides similar trait-relevant information to in-person ones.

The correlations among the perceiver effects and the correlations among the target effects were similar for the CMORR study and the two in-person studies. This rules out two other potential issues: a general factor and random responding. If the correlations among Big Five perceiver or target effects in the CMORR study were too high, it would suggest that people are rating or being rated on a general factor. If the correlations were too low, it would suggest that participants in CMORR were providing random responses. Neither appears more of a concern for a CMORR study than an in-person one.

Strengths of the CMORR Paradigm

Many psychological studies now use some form of online data collection. Survey software has long replaced paper and pencils, and cognitive experiments, even those conducted in the lab, are now often run on online servers. Moving single-participant studies online made it possible to recruit participants from a much broader range of populations. Similarly, CMORR provides an online data collection paradigm for the study of social interactions that can be used to study a wide range of populations, offering promising new directions for interpersonal perception research.

Recruitment

Data collection from online participants took off in social psychology in the early 2000s (Gosling et al., 2004), and the proportion of studies collecting data online has increased over the last decade (Sassenberg & Ditrich, 2019). Access to these online samples can provide researchers of interpersonal perception, social interactions, and relationships a way to address concerns about the overreliance on college undergraduates and homogeneity of samples (Gosling et al., 2004), and also increase sample sizes (Sassenberg & Ditrich, 2019). Online samples from participant recruitment platforms (e.g., Prolific, Qualtrics Panels, mTurk) are not necessarily representative but they tend to be more diverse than college samples.

Almost all Americans (Mobile Fact Sheet, 2021) and the vast majority of people across the globe (Turner, 2018) have access to a smartphone or computer that can run videoconferencing software. Researchers can leverage this technology and its widespread availability to recruit participants from anywhere in the world and bring them together online to interact and complete measures about their impressions of one another, experiences during the interaction, and much more. This means that CMORR can also be used with truly representative samples, as well as with purposive samples, such as harder-to-each populations or members of organizations, support groups, patient populations, and so on. CMORR can be used in conjunction with nearly any recruitment method and any population that has access to devices capable of videoconferencing. Future work can potentially further extend the reach of CMORR studies by using smartphones to facilitate interactions and collect data. There will be some limitations caused by limited access to, or unfamiliarity with, the required technology, or a reluctance to meet strangers online, but CMORR provides researchers access to larger and more diverse samples.

Efficiency and Logistics

A traditional in-person round-robin study requires that at least four participants and multiple researchers show up to the lab at the same time. In a CMORR study, participants and researchers still need to participate at the same time, but they can do so from their own computer or smart phone, wherever they are. This eliminates the need to bring people into a lab and can simplify logistical issues, even for studies that recruit local participants. CMORR can also increase the speed and efficiency of data collection by enabling researchers to schedule sessions closer together or run more sessions per week.

Data Quality

Many concerns about running surveys and single-participant experiments with online samples center on bots and inattentive responses. These factors threaten the validity of online survey data (Chmielewski & Kucker, 2020), but are less of an issue with CMORR studies. Participants must be visible on a webcam and interact with others, which reduces the possibility of bots and suggests that CMORR studies should not experience higher inattentive responding rates than in-person studies.

One major concern when developing CMORR was the potential of participants to cause social disruptions. We preemptively addressed potential disruptions (e.g., aggressive, demeaning, or racist behavior) by having an RA monitor the interactions in each CMORR room. We also developed a plan on how to react to any antisocial behavior. RAs were trained to intervene if a participant caused a social disruption and provide a single opportunity for behavior change before removing them from the videoconference room and survey. The presence of RAs seems to have had a strong preventive effect. We now have extensive experience collecting CMORR data from students and an online sample, and to date, we have not had to remove anyone for disruptive behavior.

Limitations

The overarching goal of this work was to examine whether impressions formed in online virtual rooms are in general comparable with those formed during in-person interactions. The evidence supports that they are. However, the procedure used in the CMORR study were not identical to either of the in-person studies, and differed from many of the meta-analytic comparison studies in ways beyond being computer-mediated. For example, CMORR used low behavioral overlap interactions (dyadic) and many of the comparison studies used high overlap interactions (group). Previous work suggests that the difference is negligible for in-person studies (Kenny, 2004), but additional research will be needed to examine how overlap impacts online interactions.

Other differences in procedure might have contributed to an interesting difference between this CMORR study and the in-person studies we compared it to—CMORR impressions were higher in self-other agreement than the in-person studies for neuroticism and conscientiousness. Most likely, this is a task difference rather than an interaction medium difference. In this CMORR study, participants discussed personality-relevant questions, which might have provided perceivers with additional trait-relevant information. This is not an inherent feature of the CMORR method, but it suggests that the substance of an interaction or topic of a discussion is pivotal to the formation of impressions.

The focus of the present work on impressions of Big Five traits might raise concerns about how other impressions and judgments made during CMORR interactions compare with those made during in-person interactions. For example, people could show less interest in affiliating with others who they meet online because they see little chance of pursuing a relationship outside of the study. Findings from the present work cannot directly address these concerns, but future work can use the CMORR method to test the limits of studying social interactions online.

Future Applications of CMORR: New Populations and New Questions

Initial interactions and online samples are only one type of interaction and population that can be studied using CMORR. It can also be used to study interactions among friends, coworkers, and within social and professional organizations. For example, I/O psychologists can use CMORR to study how impressions impact team productivity, UX and marketing researchers to study what people think about new products or services, and relationship researchers to facilitate speed-dating studies. Furthermore, clinical researchers could use a CMORR approach by embedding items after teletherapy sessions to better understand how a client’s impressions of a therapist impact the effectiveness of treatment, or to study group therapy sessions.

Interpersonal perception researchers can use CMORR to extend the study of impressions to diverse populations and test if social-group stereotypes manifest in impressions of individuals. Furthermore, researchers can combine CMORR with the intergroup SRM (Kenny et al., 2015) to study cross-group interactions, test hypotheses about how intergroup biases impact everyday social interactions, and to complement field work examining the contact hypothesis (Mousa, 2020; Scacco & Warren, 2018).

In addition, the video-recordings of each CMORR interaction offer a myriad of possibilities for the study of human social interactions. Unlike an in-person study, in a CMORR study researchers have a complete record of all the information available to participants about one another. Multiple first-person videos of each participants can be coded and analyzed to better understand the features of people and contexts that contribute to impressions and social decisions, and be used as new stimuli for lab-based studies.

Conclusion

Moving the study of social interactions and the impressions online will provide researchers an opportunity to study these phenomena in larger and more diverse samples. This, in turn, will extend the study of interpersonal perception and social cognition to new populations and enable researchers to answer new questions.

Footnotes

Acknowledgements

The authors would like to thank Jennifer L. Heyman, Lauren J. Human, Marie-Catherine Mignault, and Hasagani Tissera for providing the in-person dyadic data, and Shifa Hamid and Joshua Pearman for collecting the in-person group data. They would also like to thank David Kenny for his helpful suggestions on an early version of this work.

Handling Editor: André Mata

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially supported by a University of Oregon Doctoral Dissertation Research Fellowship awarded to Bradley T. Hughes.

ORCID iD

Bradley T. Hughes

Data Availability

The preregistration for the methods, R code for analysis, and data for the CMORR study and in-person dyadic study are available on OSF: .

Author Biographies

Bradley T. Hughes earned his PhD at the University of Oregon, and is currently a postdoctoral fellow at the University of Toronto. He studies SES and impression formation with a special interest in understanding how SES is perceived and experienced.

Sanjay Srivastava is an emerit professor of psychology at the University of Oregon. He studies how personal attributes are expressed and perceived in various social spaces, both virtual and in-person.

References

Antheunis

M. L.

Schouten

A. P.

Walther

J. B.

(2020). The hyperpersonal effect in online dating: Effects of text-based CMC vs. videoconferencing before meeting face-to-face. Media Psychology, 23(6), 820–839.

Asch

S. E.

(1946). Forming impressions of personality. The Journal of Abnormal and Social Psychology, 41(3), 258–290. https://doi.org/10.1037/h0055756

Asendorpf

J. B.

Penke

Back

M. D.

(2011). From dating to mating and relating: Predictors of initial and long–term outcomes of speed–dating in a community sample. European Journal of Personality, 25(1), 16–30.

Back

M. D.

Penke

Schmukle

S. C.

Asendorpf

J. B.

(2011). Knowing your own mate value: Sex-specific personality effects on the accuracy of expected mate choices. Psychological Science, 22(8), 984–989.

Back

M. D.

Penke

Schmukle

S. C.

Sachse

Borkenau

Asendorpf

J. B.

(2011). Why mate choices are not as reciprocal as we assume: The role of personality, flirting and physical attractiveness. European Journal of Personality, 25(2), 120–132.

Biesanz

J. C.

(2010). The social accuracy model of interpersonal perception: Assessing individual differences in perceptive and expressive accuracy. Multivariate Behavioral Research, 45(5), 853–885.

Borkenau

Liebler

(1992). Trait inferences: Sources of validity at zero acquaintance. Journal of Personality and Social Psychology, 62(4), 645–657. https://doi.org/10.1037/0022-3514.62.4.645

Brodsky

Lee

M. J.

Leonard

(2022). Discovering new frontiers for dyadic and team interaction studies: Current challenges and an open-source solution—SurvConf—for increasing the quantity and richness of interactional data. Academy of Management Discoveries, 8(3), 337–340. https://doi.org/10.5465/amd.2021.0257

Chmielewski

Kucker

S. C.

(2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464–473.

10.

Cohen

(1983). Applied multiple regression/correlation analysis for the behavioral sciences. Erlbaum.

11.

Connelly

B. S.

Ones

D. S.

(2010). An other perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136(6), 1092–1122. https://doi.org/10.1037/a0021212

12.

DesJardins

N. M. L.

Srivastava

Küfner

A. C.

Back

M. D.

(2015). Who attains status? Similarities and differences across social contexts. Social Psychological and Personality Science, 6(6), 692–700.

13.

Fiske

S. T.

Taylor

S. E.

(1991). Social cognition. McGraw-Hill Book Company.

14.

Funder

D. C.

(1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102(4), 652–670. https://doi.org/10.1037/0033-295X.102.4.652

15.

Funder

D. C.

Dobroth

K. M.

(1987). Differences between traits: Properties associated with interjudge agreement. Journal of Personality and Social Psychology, 52(2), 409–418. https://doi.org/10.1037/0022-3514.52.2.409

16.

Gosling

S. D.

Gaddis

Vazire

(2008). First impressions based on the environments we create and inhabit. In Ambady

Skowronski

J. J.

(Eds.), First impressions (pp. 334–356). Guilford Press.

17.

Gosling

S. D.

Vazire

Srivastava

John

O. P.

(2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104. https://doi.org/10.1037/0003-066X.59.2.93

18.

Henrich

Heine

Norenzayan

(2010). Most people are not WEIRD. Nature, 466, 29. https://doi.org/10.1038/466029a

19.

Human

L. J.

Biesanz

J. C.

(2011). Through the looking glass clearly: Accuracy and assumed similarity in well-adjusted individuals’ first impressions. Journal of Personality and Social Psychology, 100(2), 349–364. https://doi.org/10.1037/a0021850

20.

John

O. P.

Srivastava

(1999). The Big-Five trait taxonomy: History, measurement, and theoretical perspectives. In Pervin

L. A.

John

O. P.

(Eds.), Handbook of personality: Theory and research (Vol. 2, pp. 102–138). Guilford Press.

21.

Kenny

D. A.

(1994). Interpersonal perception: A social relations analysis. Guilford Press.

22.

Kenny

D. A.

(2004). PERSON: A general model of interpersonal perception. Personality and Social Psychology Review, 8(3), 265–280.

23.

Kenny

D. A.

(2019). Interpersonal perception: The foundation of social relationships. Guilford Publications.

24.

Kenny

D. A.

Gomes

S. B.

Kowal

(2015). The Intergroup Social Relations Model: ISRM. Group Dynamics: Theory, Research, and Practice, 19(3), 152–165.

25.

Kenny

D. A.

La Voie

(1984). The social relations model. In Berkowitz

(Ed.), Advances in experimental social psychology (Vol. 18, pp. 141–182). Academic Press.

26.

Lashley

B. R.

Kenny

D. A.

(1998). Power estimation in social relations analyses. Psychological Methods, 3(3), 328–338. https://doi.org/10.1037/1082-989X.3.3.328

27.

Liao

Bazarova

N. M.

Yuan

Y. C.

(2018). Unpacking medium effects on social psychological processes in computer-mediated communication using the Social Relations Model. Journal of Computer-Mediated Communication, 23, 90–106.

28.

McClain

Vogels

E. A.

Perrin

Sechopoulos

Raine

(2021). How the internet and technology shaped Americans’ personal experiences amid COVID-19. Pew Research Center. https://www.pewresearch.org/internet/2021/09/01/how-the-internet-and-technology-shaped-americans-personal-experiences-amid-covid-19/

29.

Mignault

M. C.

Heyman

J. L.

Biesanz

J. C.

Human

L. J.

(2022). On being and feeling transparent: Examining expressive accuracy awareness in first impressions of personality and links to well-being. Journal of Research in Personality, 96, 104185. https://doi.org/10.1016/j.jrp.2021.104185

30.

Mobile Fact Sheet. (2021, April 7). Pew Research Center: Internet, science & tech. https://www.pewresearch.org/internet/fact-sheet/mobile/

31.

Mohr

C. D.

(1997). Stereotypes, communication context, and accuracy in impression formation [Master’s thesis]. University of Connecticut.

32.

Molnar

(2019). SMARTRIQS: A simple method allowing real-time respondent interaction in Qualtrics surveys. Journal of Behavioral and Experimental Finance, 22, 161–169. https://doi.org/10.1016/j.jbef.2019.03.005

33.

Mousa

(2020). Building social cohesion between Christians and Muslims through soccer in post-ISIS Iraq. Science, 369(6505), 866–870.

34.

Preacher

K. J.

(2002, May). Calculation for the test of the difference between two independent correlation coefficients [Computer software]. http://quantpsy.org

35.

Rau

Carlson

E. N.

Back

M. D.

Barranti

Gebauer

J. E.

Human

L. J.

Leising

Nestler

(2021). What is the structure of perceiver effects? On the importance of global positivity and trait-specificity across personality domains and judgment contexts. Journal of Personality and Social Psychology, 120(3), 745–764.

36.

R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

37.

Salazar Kämpf

Liebermann

Kerschreiter

Krause

Nestler

Schmukle

S. C.

(2018). Disentangling the sources of mimicry: Social relations analyses of the link between mimicry and liking. Psychological Science, 29(1), 131–138. https://doi.org/10.1177/0956797617727121

38.

Sassenberg

Ditrich

(2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 2(2), 107–114.

39.

Scacco

Warren

S. S.

(2018). Can social contact reduce prejudice and discrimination? Evidence from a field experiment in Nigeria. American Political Science Review, 112(3), 654–677.

40.

Schloerke

Cook

Larmarange

Briatte

Marbach

Thoen

Elberg

Crowley

(2024). GGally: Extension to “ggplot2.” (R Package Version 2.2.1). https://github.com/ggobi/ggally

41.

Schönbrodt

F. D.

Back

M. D.

Schmukle

S. C.

(2012). TripleR: An R package for social relations analyses based on round-robin designs. Behavior Research Methods, 44, 455–470. https://doi.org/10.3758/s13428-011-0150-4

42.

Schönbrodt

F. D.

Back

M. D.

Schmukle

S. C.

(2016). TripleR: Social Relation Model (SRM) analyses for single or multiple groups (R Package Version 1.5.3). http://cran.r-project.org/package=TripleR

43.

Soto

C. J.

John

O. P.

(2017). Short and extra-short forms of the Big Five Inventory–2: The BFI-2-S and BFI-2-XS. Journal of Research in Personality, 68, 69–81. https://doi.org/10.1016/j.jrp.2017.02.004

44.

Syed

(2021, December 3). Reproducibility, diversity, and the crisis of inference in psychology. https://doi.org/10.31234/osf.io/89buj

45.

Tissera

Mignault

M.-C.

Human

L. J.

(2023). “Zooming” in on positive and accurate metaperceptions in first impressions: Examining the links with social anxiety and liking in online video interactions. Journal of Personality and Social Psychology, 125(4), 852–873. https://doi.org/10.1037/pspp0000457

46.

Turner

B. A.

(2018, July 10). How many people have smartphones worldwide (Jan 2023). BankMyCell. https://www.bankmycell.com/blog/how-many-phones-are-in-the-world

47.

Vazire

(2010). Who knows what about a person? The self–other knowledge asymmetry (SOKA) model. Journal of Personality and Social Psychology, 98(2), 281–300.

48.

Wickham

(2016). ggplot2: Elegant graphics for data analysis. Springer. https://ggplot2.tidyverse.org