Gender Effects in Gaming Research: A Case for Regression Residuals?

Introduction

The last decade has seen a proliferation of studies examining various implications of video gaming, ranging from negative effects such as increased aggression ¹ to profound positive effects such as increased spatial skills^2,3 and selective attention. ⁴ Most of these studies reported gender differences in many of the investigated variables.^5–7 Also, a second line of research separately targeted the actual usage of computer games by male and female players of different age groups.^8–11 Here, male players were reliably shown to spend more time playing video games than female players.^12,13

These findings point at a possible confound: as prolonged experience and training is a major determinant of both performance ¹⁴ and affective reactions, ¹⁵ gender differences in any measured variable related to video gaming might also partly reflect differences in video game experience. An example for this speculation is a recent study by Bourgonjon and colleagues, ¹⁶ who used path modeling to identify predictors for secondary-school students' preference for using video games as an educational tool in the classroom. Even though gender differences in the assessed variable were present, path modeling showed them to be mediated by gaming experience, rendering conclusions about gender differences per se difficult.

Whenever gender effects are investigated in video game studies, differential gaming experience should thus be taken into consideration. This can be achieved by multiple regression analysis ¹ or analysis of covariance (ANCOVA).^17,18 However, because the assumptions of ANCOVA are routinely violated by datasets concerning gender and video game performance, authors often report standard analysis of variance (ANOVA) for their data. ¹⁹ The purpose of the present study was to exemplify the alternative strategy of analyzing the residuals of a pooled linear regression to control for the influence of gaming experience. Following the terminology of Maxwell and colleagues, ¹⁸ this approach will be referred to in the present study as analysis of regression residuals (ANORES).

The underlying rationale of ANORES can be depicted in Venn diagrams of unique and shared variance proportions of the variables of interest: the participants' gender, gaming experience, and the dependent variable in question (Fig. 1). The overall variance of the dependent variable will always be caused by several factors and only a portion of this variance will be due to the participants' gender. This shared variance, in turn, can be dissected in parts that are uniquely caused by the participants' gender and other parts that are confounded with other variables such as gaming experience. ANORES removes this confounded part of the variance to analyze only those proportions of the shared variance that are uniquely caused by the participants' gender.

OPEN IN VIEWER

FIG. 1.

Variance components in an analysis targeting the impact of gender on the dependent variable of interest. Left panel: The Venn diagram shows a modest correlation of gender and gaming experience. In this case, both gender and experience share a considerable unique proportion of the dependent variable's variance (a and c) whereas the confounded portion of the variance (b) is comparatively small. Thus, controlling for the correlation of gender and gaming experience will only have limited impact on the results of the statistical analysis. Right panel: Although gender explains the same portion of the dependent variable's variance as in the left setting, most of this shared variance is confounded with the variance due to gaming experience (b). Controlling for the correlation of gender and experience will have a profound effect on the results of any statistical test and might even show the unique impact of gender (a) to be nonsignificant. Color images available online at www.liebertonline.com/cyber.

In a nutshell, the basic procedure of ANORES follows two simple steps. First, a (linear) regression identifies the impact of the confounding variable on the dependent variable. Regression residuals can then be computed by subtracting the predictions of the regression equation from the participants' actual scores. These residuals resemble scores where the (linear) impact of the confounding variable is entirely removed. In a second step, the regression residuals enter a typical ANOVA with those factors that are of interest to the analysis, including the participants' gender.

The impact of gaming experience or any other confounding variable will vary from experiment to experiment and its actual impact depends on several factors. It is thus impossible to predict whether ANORES will produce qualitatively different results than the standard ANOVA approach for an individual experiment and researchers have to judge whether their data are best analyzed with the former or latter test. For this reason, the present study relies on two simulated datasets to provide prototypical examples for situations showing the potential value as well as the limitations of ANORES. A step-by-step description of how to perform ANORES with the SPSS software package (IBM Corp.) is available as Supplementary Data (Supplementary Data are available online at www.liebertonline.com/cyber).

Methods

Each simulated dataset comprises 20 men and 20 women that we assume to have played Solitaire and another 20 men and 20 women that we assume to have played Tetris right before a mental rotation test (Fig. 2A; higher scores indicate better performance; arbitrary scaling). Each participant provided a rough estimation of his gaming experience in hours per week (mimicking typical questionnaire data). Assume that we were interested in the impact of gender and game type on the participants' ability to perform mental rotation.

OPEN IN VIEWER

FIG. 2.

(A) Male and female participants of a fictive experiment played either Solitaire or Tetris and completed a mental rotation test afterward (high scores indicate better performance; errors bars represent 95% confidence intervals). The two simulated datasets yield identical results when analyzed with standard analysis of variance (ANOVA). This ANOVA on the mental rotation scores suggests a profound main effect of gender, with males outperforming females. (B) The impact of gaming experience is controlled for by linear regression as a first step in an analysis of regression residuals. (C) The second step of analysis of regression residuals—an ANOVA on the obtained regression residuals—shows a reliable gender effect for dataset 1 only (left panel) whereas the participants' gender no longer influences mental rotation scores for dataset 2 (right panel). Color images available online at www.liebertonline.com/cyber.

Dataset 1 was created to show only a moderate correlation of gender and gaming experience (cf. Fig. 1, left panel), an assumption that might hold true for most studies targeting gender effects in video gaming. Dataset 2 was explicitly created to show a manifest impact of gaming experience and clearly exaggerates its impact in empirical studies (cf. Fig. 1, right panel). Most importantly, both datasets feature the same data except for a different assignment of years of experience to the participants—creating identical situations when analyzed with ANOVA (Fig. 2A) but showing more subtle differences when analyzed with ANORES (Fig. 2B, C).

Results

Discussion: Benefits and Limitations of ANORES

The present analysis aimed at applying ANORES to data of video gaming experiments, which are typically analyzed with ANOVA. Two simulated datasets served this purpose. Both datasets yielded the same ANOVA results but featured a different pattern of intercorrelations of the participants' gender, gaming experiments, and mental rotation scores. For dataset 1, a moderate correlation of gender and gaming experience yielded comparable results for ANOVA and ANORES. However, even though the qualitative pattern looked identical, the effect size of the participants' gender was reduced by about 30%, indicating that even a medium-sized correlation can distort the statistical analysis. Dataset 2 exemplified a situation wherein a standard ANOVA would indicate a massive impact of gender on the dependent variable that was, in fact, driven by the male player's pronounced experience in video gaming. When the residuals of a linear regression were analyzed instead (ANORES), the main effect of gender turned out to be nonsignificant.

The striking difference between standard ANOVA and ANORES for dataset 2 of course resulted from the artificial nature of the dataset, and in most studies on gender effects in video gaming, gender does explain a unique portion of the criterion variance that is not due to differences in experience.^1,15 However, considering the close relation of gender and video game experience, on the one hand, ¹⁶ and video game experience and performance and affective reactions, on the other hand,^2–4 the present study might indeed overestimate gender effects. The results of the two datasets point at a continuum of settings wherein ANORES will have a different impact on the statistical analysis—ranging from a mere adjustment of statistical parameters such as effect sizes (dataset 1) to profound changes in the pattern of results (dataset 2). The fact that the present analysis relied on simulated rather than empirical data points out that any actual experiment might show a different pattern of intercorrelations and it has to be evaluated whether ANORES provides a more accurate picture than standard ANOVA does individually.

Taken together, the potential danger of experience-related confounds in video game research could easily be avoided, if authors reported the bivariate correlation of gaming experience and the respective dependent variable. If this correlation indicates a medium or stronger association of both variables (r≥0.30 in Cohen's terminology ²⁰ ), it might be worth reporting ANORES instead of standard ANOVA—at least if an ANCOVA is inappropriate for the data in question.^17,18 Further, ANORES as a data analysis strategy is of course not limited to gender effects or video gaming research. It can be easily employed to other settings, including other types of experimental design such as within-subject designs or different types of regression analysis. ¹⁸ ANORES thus provides a flexible statistical tool with valuable applications in the social and behavioral sciences.

Disclosure Statement

No competing financial interests exist.