Refinements to Effect Sizes for Tests of Categorical Moderation and Differential Prediction

Modifications to Reduce Bias From Modest Violations of Normality

A subtlety of the d_Mod effect-size measures that makes them particularly well-suited for use in operational settings is that one can integrate differences in prediction within a finite range of scores (i.e., one need not integrate the infinite normal distribution if one is studying predictor scores that span a bounded operational range). However, when d_Mod requires integration involving a finite minimum value and/or a finite maximum value, the cumulative density within that range may fall short of unity by a nontrivial amount due to departures from normality. A cumulative density smaller than 1 indicates that the d_Mod effect size one has computed may differ from what one would have obtained if one’s operational distribution of scores were indeed normally distributed. We propose a simple adjustment to d_{Mod_Signed} and d_{Mod_Unsigned} to offset this bias; we describe an even more effective adjustment later in the article, but the logic of the initial refinement described below is necessary to lay the groundwork for the more advanced method.

Integrating over a finite distribution can result in a biased d_Mod effect size unless one rescales the effect size by the sum of the weights generated by the normal-density function (i.e., the cumulative density). The problem of integrating over a finite distribution of scores is analogous to algebraically computing a weighted average (i.e., x ¯ = ∑ i n w i x i ∑ i n w i , where w is a weight and x is a score) in which the sum of weights in the denominator exceeds the sum of the weights used in the numerator. Of course, cumulative densities appreciably smaller than unity may indicate that the assumption of a normal distribution is inappropriate. When normality cannot be reasonably assumed of one’s data, one should use the nonparametric versions of d_Mod described later in this article. For our present discussion, we assume that departures from normality are modest and that using a normal-density function to compute d_Mod is a reasonable choice.

When integrating a d_Mod function over a finite range of predictor scores, an option for addressing a cumulative density smaller than one is to simply divide the d_Mod effect size by the cumulative density of scores within the operational range. For example, if the operational range of scores includes 95% of the theoretical normal distribution, dividing the d_Mod effect size by .95 will rescale the weights to account for the fact that scores outside of the operational range are impossible to achieve and that 95% of the theoretical distribution represents 100% of the distribution of possible scores. An adjustment factor that implements this correction is incorporated in Equations 1b and 2b and we recommend using these updated equations over Equations 1a and 2a. The gains in precision from using our corrections are likely to be small in most settings, but the corrections will consistently attenuate the bias from minor violations of normality. Unless otherwise noted, all subsequent mentions of d_{Mod_Signed} and d_{Mod_Unsigned} in this article will refer to Equations 1b and 2b, respectively.

In formulating our updated version of d_{Mod_Unsigned} in Equation 2b, we have further reduced bias by modifying the way in which the sign is removed from differences in prediction. Although Nye and Sackett’s d_{Mod_Unsigned} formula (Equation 2a) accomplishes the advertised goal of removing the sign from differences in prediction, it does so in a way that slightly alters the meaning of the resulting d_{Mod_Unsigned} effect size relative to the meaning of the d_{Mod_Signed} effect size (Equation 1a). Under the radical in Equation 2a, one only squares the differences between regression lines and does not square the density weights. Due to the fact that only the differences between regression predictions are squared (rather than squaring the products of the differences and densities), the square root of the product under the radical does not result in an absolute difference like one might expect. Over the range of predictor scores, taking the square root of the unsquared densities in Equation 2a alters the proportional weight given to each predictor score: Scores toward the middle of the distribution receive too little weight and extreme scores receive too much weight. See Figure 1 for comparisons of the distributions of raw and square-root densities, as well as the corresponding distributions of proportional weights. As absolute differences are the intuitive metric for unsigned differences, d_{Mod_Unsigned} should be computed using Equation 2b rather than Equation 2a.

OPEN IN VIEWER

Figure 1.

Comparisons of the distributions of densities and square roots of densities for the normal distributions used in Equations 2a and 2b, respectively. Distributions of proportional weights were computed by dividing the density associated with each predictor score by the corresponding cumulative density. Comparison of the figure panels reveals that taking the square roots of densities distorts the weight given to each predictor score, whereas using raw densities does not.

A limitation of Equations 1b and 2b is that they assume that violations of normality are symmetric, with similar effects on the low and high ends of a score distribution, but this not likely to be the case in all settings. We present methods that account for asymmetric violations of assumptions after first introducing our directional d_Mod effect sizes.

Separate Effect Sizes for Positive and Negative Differences in Predicted Criterion Scores

The d_{Mod_Signed} and d_{Mod_Unsigned} effect sizes are useful for quantifying effects that are moderated by a categorical variable. However, some categorically moderated effects occur in domains in which the regions of negative and positive differences in predicted criterion values have substantive importance. An example of this is predictive bias from the industrial and organizational psychology literature. Predictive bias occurs when assessment scores predict performance differently as a function of one’s protected-class status (e.g., one’s sex or race). For example, a biased test might predict lower performance for Black job applicants when the White (referent group) regression model is used to make predictions than when the Black (focal group) regression model is used. If predicted criterion scores are Y ^ 1 = b 0 1 + X b 1 1 using the referent model (where X is a vector of focal predictor scores) and Y ^ 2 = b 0 1 + X b 1 1 using the focal model, “underprediction” for the focal group occurs when Y ^ 1 < Y ^ 2 and “overprediction” occurs when Y ^ 1 > Y ^ 2 . We suggest that quantifying these directional differences can have interpretive value.

The d_{Mod_Signed} and d_{Mod_Unsigned} effect sizes both combine directional differences in prediction into a single effect size, which can occasionally make it difficult to interpret the precise form of an interaction. Nye and Sackett (2017) suggested that d_Mod effect sizes could be computed over any meaningful range of predictor scores, which means that one could compute separate d_Mod effect sizes within any segments of a distribution that are of interest to one’s research question. In our use of d_Mod effect sizes to quantify predictive bias, we have found it informative to break d_{Mod_Signed} into two directional effect sizes. We propose an effect size called d_{Mod_Under} that only quantifies differences in prediction in the score range where negative differences in prediction occur (d_{Mod_Under} is the standardized average of differences in prediction for all Y ^ 1 < Y ^ 2 ; see Equation 5) and an effect size called d_{Mod_Over} that only quantifies differences in prediction in the score range where positive differences in prediction occur (d_{Mod_Over} is the standardized average of differences in prediction for all Y ^ 1 > Y ^ 2 ; see Equation 6). These directional effect sizes facilitate the interpretation of moderated effects, especially when groups’ regression lines cross within an operational score range. If subgroup regression lines do not cross and there are no Y ^ 1 < Y ^ 2 differences in prediction within an operational score range, d_{Mod_Under} will be zero and d_{Mod_Over} will be equal to d_{Mod_Signed}. Similarly, if subgroup regression lines do not cross and there are only Y ^ 1 < Y ^ 2 differences in prediction within an operational score range, d_{Mod_Over} will be zero and d_{Mod_Under} will be equal to d_{Mod_Signed}.

The d_{Mod_Under} and d_{Mod_Over} directional effect sizes are useful for isolating the magnitudes of negative (i.e., Y ^ 1 < Y ^ 2 ) and positive (i.e., Y ^ 1 > Y ^ 2 ) differences in prediction, respectively. However, provided that the cumulative density of the operational predictor distribution equals unity, d_{Mod_Under} and d_{Mod_Over} can also be used to compute d_{Mod_Signed} and d_{Mod_Unsigned} via addition. When d_{Mod_Under} and d_{Mod_Over} effect sizes computed from a complete normal distribution are added together, they equal d_{Mod_Signed} and, when their absolute values are added together, they equal d_{Mod_Unsigned}. The two directional effect sizes are therefore quite versatile and, because of this, we view d_{Mod_Under} and d_{Mod_Over} as elemental equations for quantifying moderated effects. The results from d_{Mod_Under} and d_{Mod_Over} capture all of the information necessary to make sense of the magnitude of a moderated effect: Separately, they communicate information about negative and positive differences in prediction and, by addition, they can result in d_{Mod_Signed} and d_{Mod_Unsigned}. The information gained from d_{Mod_Under} and d_{Mod_Over} is complementary to the information found by comparing d_Min and d_Max and these directional effect sizes can be helpful for making sense of d_{Mod_Signed} and d_{Mod_Unsigned}.

Based on our recommendations for rescaling d_{Mod_Signed} and d_{Mod_Unsigned} by the cumulative density of predictor scores, Equations 5 and 6 include adjustment factors that rescale the effect sizes by the ratio of the cumulative density from integrating the infinite normal distribution to the cumulative density from integrating over operational scores (i.e., the ratio of the sum of the theoretical weights to the sum of the actual weights). In general, the adjustment to the d_{Mod_Under} and d_{Mod_Over} effect sizes is more appropriate than making a global correction to the d_{Mod_Signed} and d_{Mod_Unsigned} effect sizes as we did in Equations 1b and 2b. This is because the separate corrections made to d_{Mod_Under} and d_{Mod_Over} capture the possibility that violations of normality are asymmetric and differentially impact regions where Y ^ 1 < Y ^ 2 and where Y ^ 1 > Y ^ 2 . We recommend computing d_{Mod_Under} and d_{Mod_Over} effect sizes separately (ensuring that the raw directional effect sizes are appropriately rescaled, as shown in Equations 5 and 6) and then adding them together to obtain d M o d _ S i g n e d * and d M o d _ U n s i g n e d * (see Equations 7 and 8). Although the adjustment factors included in Equations 1b and 2b account for the overall cumulative density’s deviation from unity, we recommend using d M o d _ S i g n e d * and d M o d _ U n s i g n e d * instead of d_{Mod_Signed} and d_{Mod_Unsigned} when possible because the asterisked additive effect sizes are more robust to asymmetric violations of normality than are the directly computed effect sizes.

We note that our corrections for nonunity cumulative densities do not perfectly correct for abnormalities in the distribution of actual predictor scores. However, they offer a relatively simple way to approximate the effect size of interest. If deviations from the assumed normal distribution are substantial, use of normal-density weights will result in effect sizes of questionable validity. When deviations from normality are great and one has access to the actual distribution of predictor scores, we recommend computing d_Mod using nonparametric equations.

Nonparametric Methods for Computing d_Mod Effect Sizes

The d_Mod equations discussed thus far have assumed that there is an underlying parametric function that describes the distribution of predictor scores. This assumption is not always reasonable and there may be settings in which researchers wish to use observed frequencies as weights when they compute d_Mod. To compute nonparametric versions of all of the effect-size measures described above, we recommend using Equations 9 through 14 in Table 1. These equations provide standardized weighted averages of the differences in predicted criterion scores between referent and focal models using frequencies of observed focal-group scores as weights. These equations are direct algebraic analogs of their parametric counterparts listed in Table 1; if the observed data are normally distributed, the results from corresponding parametric and nonparametric procedures should agree within a reasonable margin of error.

Our notation for the nonparametric formulas implies that one is using a frequency distribution as weights, but it is also possible to perform this procedure with raw data directly. If X is a vector of the focal group’s observed scores in which each entry represents a case, one can simply compute the average of X ( b 1 1 − b 1 2 ) + b 0 1 − b 0 2 and divide that average by the referent-group criterion standard deviation to obtain a nonparametric version of d_{Mod_Signed}. The same logic applies to computing the other nonparametric d_Mod effect sizes from observed scores.

These nonparametric versions of d_Mod will be appropriate only if one can reasonably assume that the population to which one seeks to generalize will follow approximately the same distribution as the observed frequencies. If the sample used to derive the weights is atypical of the population of interest, these nonparametric equations will only be suitable for describing differences in prediction in the sample and will not be useful for making generalizations.