Response Surface Analysis in Personality and Social Psychology: Checklist and Clarifications for the Case of Congruence Hypotheses

Abstract

Response surface analysis (RSA) enables researchers to test complex psychological effects, for example, whether the congruence of two psychological constructs is associated with higher values in an outcome variable. RSA is increasingly applied in the personality and social psychological literature, but the validity of published results has been challenged by some persistent oversimplifications and misconceptions. Here, we describe the mathematical fundamentals required to interpret RSA results, and we provide a checklist for correctly identifying congruence effects. We clarify two prominent fallacies by showing that the test of a single RSA parameter cannot indicate a congruence effect, and when there is a congruence effect, RSA cannot indicate whether a predictor mismatch in one direction (e.g., overestimation of one’s intelligence) is better or worse than a mismatch in the other direction (underestimation). We hope that this contribution will further enhance the validity and strength of empirical studies that apply this powerful approach.

Keywords

polynomial regression congruence similarity discrepancy quadratic regression

Congruence hypotheses state that the agreement (i.e., congruence) between two psychological constructs should positively (or negatively) affect some outcome variable. Such hypotheses play a central role in many psychological disciplines, for example, in research on the benefits of person–environment fit, similarity effects on social relationships, or the consequences of judgment accuracy (see Table 1 for further examples; see also table 1 in Barranti, Carlson, & Côté, 2017, for a broad overview of potential applications).

Table 1.

Examples of Congruence Hypotheses in Personality and Social Psychology.

Example questions
What are the consequences of (perceived) dyadic similarity?
Is parent–offspring personality similarity linked to externalizing problems in adolescence? (Franken, Laceulle, Van Aken, & Ormel, 2017)
Are people more likely to vote for politicians whom they believe resemble themselves in terms of personality traits? (Koppensteiner & Stephan, 2014)
Are children who are similar in terms of personality more likely to become friends? (Ilmarinen, Vainikainen, Verkasalo, & Lönnqvist, 2017)
When people meet others for the first time, are they more romantically interested in those who have levels of mate value, physical attractiveness, or personality traits that are similar to their own? (Olderbak, Malter, Sofio, Wolf, & Jones, 2017)
Are couples more satisfied with their relationship when the two partners resemble each other in terms of their personality, goals, or sexual desire? (e.g., Derrick et al., 2016; Rosen, Bailey, & Muise, in press; Weidmann, Schönbrodt, Ledermann, & Grob, 2017)
Are members of a couple better intrapersonally adjusted or more satisfied with their relationship on days when the two partners provide similarly strong social support for each other? (Bar-Kalifa, Pshedetzky-Shochat, Rafaeli, & Gleason, in press)
Do partners feel closest to their mate when they have similar evaluations of a stimulus (e.g., a film, music, a book)?
What are the consequences of person–group similarity?
Does (in)congruence between people’s ideological orientation and their community’s ideological orientation affect their tendency to take the perspective of others? (Chopik & Motyl, 2016)
Is person–group dissimilarity in personality linked to peer victimization? (Boele, Sijtsema, Klimstra, Denissen, & Meeus, 2017)
What are the consequences of person–environment fit?
Do individuals earn a higher income if their personality traits fit the personality demands of their jobs? (Denissen et al., in press)
Does the fit between a person’s personality traits and the prevalence of these traits in the inhabitants of the person’s home city predict this person’s self-esteem? (Bleidorn et al., 2016)
What are the consequences of self-other agreement?
Do people have stronger goals to change a personality trait the more they agree with knowledgeable others about their level on the respective trait? (Quintus, Egloff, & Wrzus, 2017)
Is disagreement about moral character associated with interpersonal costs? (Barranti, Carlson, & Furr, 2016)
What are the consequences of meta-accuracy?
Does the meta-accuracy of trust (i.e., an accurate perception of how much one is trusted by others) affect trust development? (Brion, Lount, & Doyle, 2015)
What are the consequences of intrapersonal consistency?
Are people who show consistent behaviors or opinions across two time points evaluated more positively than those who are less consistent?
Is congruence between belongingness needs and relationship satisfaction linked to higher well-being? (Verhagen, Lodder, & Baumeister, in press)

Congruence hypotheses have traditionally been investigated by correlating difference scores (e.g., absolute or squared differences or residuals) with the outcome variable. This approach is, however, biased toward falsely claiming support for the hypothesis (Cronbach & Furby, 1970; Edwards, 1994, 2001; Edwards & Parry, 1993). Response surface analysis (RSA; e.g., Edwards, 2002, 2007; Nestler, Grimm, & Schönbrodt, 2015; Schönbrodt, 2016b) provides a powerful alternative approach that overcomes this limitation and also enables researchers to test more elaborate effects. The basic element of RSA is the estimation of a polynomial regression model and the graphical and statistical interpretation of its coefficients.

Several tutorial introductions have meanwhile been published on the application and interpretation of RSA for psychological research questions, aimed at making this method comprehensible and applicable to social and personality psychologists doing empirical work (Barranti et al., 2017; Schönbrodt, 2016b; Shanock, Baran, Gentry, Pattison, & Heggestad, 2010). Moreover, the RSA package (Schönbrodt, 2016a) for the R environment provides a convenient tool for analyzing congruence (and many other) effects. These articles and tools have led to an increase in RSA applications in our field in recent years. In many of these applications, however, the validity of results is impaired by some persistent misconceptions, which were contained in an early introduction to RSA (Shanock et al., 2010) and which have recently been repeated in Barranti, Carlson, and Côté’s (2017) tutorial. These misconceptions involve the assumptions that (a) it is sufficient to consider a single RSA parameter to test the congruence hypothesis and that (b) RSA¹ can test whether, in addition to a congruence effect, a mismatch in one direction (e.g., overestimation of one’s intelligence) affects an outcome differently than a mismatch in the other direction (e.g., underestimation; e.g., Barranti et al., 2017; Shanock et al., 2010). We will show that both of these claims are mathematically unwarranted.

This article is aimed at providing the reader with an accessible summary of the mathematical foundations required to critically evaluate the RSA advice provided in the literature and to apply and correctly interpret RSA, especially (but not only) when investigating congruence effects.

The Basics of RSA

Imagine that we are interested in whether it is beneficial for people to hold self-views that are in line with others’ views of them. We might, for example, hypothesize that there is a congruence effect in the sense that people are happier the closer their self-views of a specific trait (e.g., some personality trait, motive, or intellectual ability) are to their reputation for embodying this trait.² We would thus have to assess three variables: the two predictors X (e.g., self-view) and Y (e.g., reputation), whose comparison is of interest, and the outcome variable Z (e.g., happiness). During data assessment and preparation, we must assess the two predictor variables on commensurable scales (Edwards & Shipp, 2007), center them on a meaningful common point (e.g., their grand mean or the midpoint of their shared scale; e.g., Aiken & West, 1991; Edwards & Parry, 1993), make sure the data contain discrepant predictor pairs for both directions of incongruence (i.e., people whose self-view is higher and people whose self-view is lower than their reputation; see Shanock et al., 2010), ensure that multicollinearity between the predictors is sufficiently low (e.g., variance inflation factor [VIF] smaller than 5; see Fox, 2016, for a discussion of VIFs and their cutoffs), and apply reliable measurements so that quadratic and interaction effects of the predictor variables can be detected (e.g., see MacCallum & Mar, 1995). Finally, the data should have high power, and this should ideally be accomplished by determining the necessary sample size (depending on assumptions about effect sizes, correlations between the predictors, etc.) in a respective simulation study (Nestler et al., 2015), but there should be at least 2 to 3 times as many participants as would be needed to detect linear main effects (Aiken & West, 1991).

Having assessed and prepared the data in these ways, RSA consists of two steps: First, a polynomial regression model is fitted to the data:

Z = b_{0} + b_{1} X + b_{2} Y + b_{3} X^{2} + b_{4} X Y + b_{5} Y^{2} .

Figures 1a and 2a depict the estimated regression models for two different example data sets (see the first and the fourth row of Table 2; see Open Science Framework [OSF] Material A at osf.io/yvw93 for the respective p values and 95% confidence intervals, and see the OSF materials at osf.io/yvw93 for the example data set and R code). In each figure, the raw data points are depicted as black dots. In the second step of the RSA, the graph of the estimated regression model is used as a guide to interpret the estimated regression coefficients in Equation 1 in terms of the associations of X, Y, and Z. The graph of Equation 1 is a surface in the three-dimensional coordinate system (see Figures 1 and 2). It can be shaped like a dome (Figures 1a–c and 2a–d), a saddle (Figure 2e), or a bowl (Figure 2f). Also, the surface can be a plane, namely, when the quadratic and interaction terms in Equation 1 are 0. Response surface methodology (Box & Draper, 1987; Box & Wilson, 1951; see also Edwards & Parry, 1993; Hill & Hunter, 1966; Myers, Khuri, & Carter, 1989) provides tools for simplifying the interpretation of a response surface. Here, we focus on the tools that are needed to detect congruence effects: the first principal axis, the line of congruence (LOC), and the line of incongruence (LOIC). The information that we present on these RSA elements were originally introduced by Box and Draper (1987) and Edwards and Parry (1993; see also Edwards, 2002). By considering the first principal axis, the LOC, and the LOIC, one can identify clear conditions that are necessary to conclude that the data support a congruence hypothesis (four conditions to test congruence effects in a broad sense and six conditions to test congruence effects in a strict sense).

Figure 1.

Response surfaces for simulated example data. The surface in panel (a) indicates a strict congruence effect with flat ridge. The other surfaces indicate congruence effects in a broad sense, that is, congruence effects combined with linear (b) or curvilinear (c) common main effects of the predictor variables.

Figure 2.

Response surfaces for simulated example data. None of the surfaces (a–e) are in line with the congruence hypothesis. Surface (f) indicates a “reverse” congruence effect, such that the outcome variable is lower for more congruent predictor combinations.

Table 2.

Response Surface Results for Simulated Data.

	Estimated Regression Model						Position of First Principal Axis		Shape of Surface Along Lines				Final Conclusion
	Z = b ₀ + b ₁ X + b ₂ Y + b ₃ X ² +b ₄ XY + b ₅ Y ²						Position of First Principal Axis		LOC		LOIC
Surface	b ₀	b ₁	b ₂	b ₃	b ₄	b ₅	p ₁₀	p ₁₁	a ₁	a ₂	a ₃	a ₄
Figure 1a	1.8*	0	0	−0.05*	0.1*	−0.05*	−0.02	0.96	0	0	0	−0.19*	Strict congr. eff.
Figure 1b	1.61*	0.11*	0.1*	−0.05*	0.1*	−0.05*	−0.08	1.01	0.21*	0	0.01	−0.2*	Congr. eff. with linear common main effect of X and Y
Figure 1c	1.6*	0.1*	0.1*	−0.1*	0.11*	−0.1*	0.02	0.99	0.2*	−0.09*	0	−0.3*	Congr. eff. with curvilinear common main effect of X and Y
Figure 2a	1.79*	−0.18*	0.17*	−0.11*	0.19*	−0.09*	0.94*	1.09	0	0	−0.35*	−0.39*	No congr. eff. (p ₁₀ ≠ 0, a ₃ ≠ 0)
Figure 2b	1.8*	0	0	−0.1*	0.12*	−0.04*	0.02	1.63^a	0	−0.02*	0	−0.25*	No congr. eff. (p ₁₁ ≠ 1)
Figure 2c	15.16*	0.11*	1.55*	−0.26*	0.8*	−0.55*	0.89*	0.7^a	1.66*	−0.01	−1.44*	−1.61*	No congr. eff. (p ₁₀ ≠ 0, p ₁₁ ≠ 1, a ₃ ≠ 0)
Figure 2d	1.6*	0.29*	−0.01	−0.07*	0.04	0.01	−8.81	4.39	0.28*	−0.02	0.3*	−0.1*	No congr. eff. (a ₃ ≠ 0)
Figure 2e	1.8*	0	0	0	0.12*	−0.11*	0.01	0.45^a	0	0.01	0	−0.23*	No congr. eff. (p ₁₁ ≠ 1)
Figure 2f	1*	0	0	0.02*	−0.05*	0.02*	^b	^b	0	0	0	0.1*	No congr. eff. (a ₄ > 0)

Note. The position of the first principal axis in the XY plane is given by Y = p ₁₀ + p ₁₁ X. The shape of the surface above the LOC is described by Z = b ₀ + a ₁ X + a ₂ X ², and the shape above the LOIC is Z = b ₀ + a ₃ X + a ₄ X ². Congr. eff. = congruence effect; LOC = line of congruence; LOIC = line of incongruence.

^aThe 95% confidence interval of P ₁₁ excludes 1.

^bFor bowl-shaped surfaces, the first principal axis is of no interest when considering congruence effects.

*p < .05.

The First Principal Axis

When the graph of the estimated regression model is shaped like a dome or a saddle, a crucial feature of the respective surface is its first principal axis (see Figure 2a and the dotted black lines in Figures 1a–c and 2b–e).³ One could colloquially describe the first principal axis as the “ridge” of the surface,⁴ and this ridge must thus be considered if we are interested in determining which predictor values lead to the highest outcome according to the results of the regression model (Edwards, 2002). The respective combinations of the predictors can be revealed by projecting the first principal axis onto the XY plane (see Figure 2a). The XY plane can be understood as a (two dimensional) coordinate system, lying on the floor of the three-dimensional cube, with self-view as the x-axis and reputation as the y-axis. Therefore, we can express the projection of the first principal axis as a linear equation that relates Y to X:

Y = p_{10} + p_{11} X .

The values of p ₁₀ and p ₁₁ can be computed from the estimated coefficients b ₁ to b ₅ in Equation 1 (see Edwards, 2002, 2007). When the R package RSA (version 0.9.11; Schönbrodt, 2016a) is used to conduct the analysis, the default output provides estimates of p ₁₀ and p ₁₁.

The LOC

When we are interested in whether it is the congruence between self-views and reputations that leads to the highest happiness, we need to compare the position of the ridge to the line in the XY plane that contains all congruent predictor combinations Y = X. This LOC is depicted as a blue (black) line running from the front corner to the back corner of the cube in Figures 1 and 2. The LOC can be written as Y = X = 0 + 1X, that is, as a linear equation with an intercept of 0 and a slope of 1.

A necessary condition for a congruence effect is that the first principal axis does not significantly differ from the LOC (Edwards, 2002) because only then the surface can predict the highest happiness for people with congruent predictors. In Figure 1a, this is the case: The first principal axis of this figure is given by Y = p ₁₀ + p ₁₁ X = −0.02 + 0.96X (see also Table 2). Its intercept p ₁₀ = −0.02 does not significantly differ from 0, that is, from the intercept of the LOC. The confidence interval of p ₁₁ = 0.96 includes 1, which means that the slope of the first principal axis does not significantly differ from the slope of the LOC. Overall, the first principal axis in Figure 1a does not differ significantly from the LOC.

In Figure 2a, by contrast, the first principal axis is parallel to the LOC but shifted away from it because its intercept p ₁₀ = 0.94 is significantly different from 0 (see also Table 2). This surface contradicts a congruence effect because it systematically predicts the highest happiness for incongruent combinations of self-views and reputations. Consider, for example, Kim (X _Kim = 0, Y _Kim = 0) and Mia (X _Mia = −0.45, Y _Mia = 0.45; see Figure 2a): Whereas Kim’s self-view is perfectly in line with her reputation, the model predicts that Mia is happier, even though her reputation is higher than her self-view. In other words, because the first principal axis is significantly shifted away from the LOC, Kim, despite having perfectly congruent predictor values, is not located at the ridge line but on the falling side of the surface; thus, she has a nonoptimal combination of self-view and reputation.

In Figure 2b, the first principal axis is rotated away from the LOC because its slope p ₁₁ is significantly different from the slope of the LOC (i.e., from 1), and in Figure 2c, the first principal axis is both shifted and rotated away from the LOC. Both of these figures also contradict a congruence effect.

To summarize, when the intercept p ₁₀ or the slope p ₁₁ of the first principal axis differs significantly from 0 or 1, respectively, the first principal axis differs from the LOC, and the surface contradicts a congruence effect (see also Edwards, 2002). The properties p ₁₀ ≈ 0 and p ₁₁ ≈ 1 thus provide the first two necessary conditions for a congruence effect.⁵

The LOIC

To test the congruence hypothesis, it is not sufficient to know that its ridge line does not significantly differ from the LOC. When this is the case, we still need to determine whether people with more and more incongruent predictor combinations have significantly lower outcome values. To this aim, we can consider the LOIC, which is the line of predictor combinations where X and Y are equal in magnitude but opposite in sign (i.e., Y = −X; see Edwards & Parry, 1993). In Figures 1 and 2, the LOIC is depicted as a blue (black) line that is perpendicular to the LOC, ranging from incongruent (but equal in magnitude) low X–high Y predictor combinations (the left corner of the cube) to the X = Y = 0 combination (the origin) to incongruent high X–low Y combinations (right corner of the cube).

To mathematically describe the LOIC, Y is set equal to −X in Equation 1 (Edwards & Parry, 1993):

\begin{array}{l} Z = b_{0} + b_{1} X + b_{2} (- X) + b_{3} X^{2} + b_{4} X * (- X) + b_{5} {(- X)}^{2} \\ = b_{0} + (b_{1} - b_{2}) X + (b_{3} - b_{4} + b_{5}) X^{2} \\ = b_{0} + a_{3} X + a_{4} X^{2}, \end{array}

where we set a ₃ = b ₁ − b ₂ and a ₄ = b ₃ − b ₄ + b ₅ for reasons of brevity. That is, the surface above the LOIC can be expressed as a quadratic equation that relates Z to X (see OSF Material B at osf.io/yvw93 for a recap of the interpretation of quadratic equations).

For a congruence effect to occur, two conditions that must be met are that the surface above the LOIC must have an inverted U-shape and the results must not contradict the assumption that this inverted U is maximized at the congruent predictor combination (0,0). This means that, first, the quadratic term coefficient a ₄ must be significantly negative because, in this case, the surface above the LOIC will be an inverted U-shaped parabola (e.g., as in Figures 1a and 2a). This also means that, second, a ₃ must not be significantly different from 0 (e.g., as in Figure 1a but not in Figure 2a). As a mathematical fact, a ₃ equals the slope of the LOIC at the point (0,0), and a parabola is maximized at a certain point only if it is constant at this point so that the parabola can fall to both sides of the point (see Edwards, 2007; see also OSF Material B at osf.io/yvw93). An example surface where a ₄ is significantly negative and a ₃ is nonsignificant is presented in Figure 1a (see also Table 2). Here, the surface falls significantly in the direction of incongruence when moving along the LOIC.

Figure 2f shows a surface where the conditions on the LOIC are not satisfied. Here, a ₄ is not significantly negative, which means that the surface does not predict that people will be happier the more congruent their self-view and reputation are. When a ₃ is significantly different from 0, the surface also contradicts a congruence effect. In Figure 2a, for example, the surface above the LOIC is given by Z = 1.79 − 0.35X − 0.39X ² (see Table 2). It is an inverted U-shaped parabola (because a ₄ < 0), which falls at (0,0) because its slope a ₃ = −0.35 is significantly negative at this point. Thus, even when only people on the LOIC are considered, we must reject the assumption that the people whose self-view equals their reputation are the ones with the highest predictions of happiness; the RSA results contradict a congruence effect. In this example, the highest outcome above the LOIC instead occurs for X _vertex = −0.45 (see OSF Material B at osf.io/yvw93 for details on the computation of the vertex position), that is, for people with the predictor combination (X,Y) = (−0.45,0.45), for example, Mia. In sum, the third and fourth necessary conditions for a congruence effect are the properties a ₄ < 0 and a ₃ ≈ 0.

Conditions for a Congruence Effect in a Broad and Strict Sense

When the RSA parameters satisfy the four conditions p ₁₀ ≈ 0, p ₁₁ ≈ 1, a ₄ < 0, and a ₃ ≈ 0, the data support a congruence effect in a broad sense (which is the case for all surfaces in Figure 1 but for none of the surfaces in Figure 2): The four conditions imply that congruence has a positive effect on the outcome, whereas they allow for the possibility that, in addition to this effect, the predictor variables can have common main effects. For example, the surface in Figure 1b indicates a congruence effect combined with positive main effects of self-view and reputation, which are reflected in a positive slope of the surface above the LOC: For two people with equal discrepancies between their self-views and their respective reputations (e.g., both located on the LOC), the model predicts greater happiness for the person with higher predictor levels (e.g., Kim) than for the person with lower predictor levels (e.g., Tom). Similarly, the surface in Figure 1c indicates a congruence effect combined with a curvilinear common main effect of the predictors, reflected by a curvilinear shape of the surface above the LOC. Such (linear or curvilinear) main effects can be theoretically justified in research domains in which the predictor variables (e.g., self-estimated and actual intelligence) are expected to be per se related to the outcome variable (e.g., self-esteem), so that not only congruence in a person’s predictor values should affect the person’s outcome value but also the person’s levels on the predictors.

One important property of RSA models that indicate a congruence effect combined with main effects is that such models systematically predict that some people with incongruent predictors are happier than other people with congruent predictors. For the model in Figure 1b, for example, this situation can occur when the incongruent person (e.g., Mia) has sufficiently higher self-view and reputation levels than the congruent person (e.g., Tom) so that her outcome prediction is higher due to the positive main effects. Similarly, the surface in Figure 1c with curvilinear main effects systematically predicts that some people with incongruent predictors (e.g., Mia) are happier than other people whose self-view equals their reputation (e.g., Tom). If our theory justifies additional main effects despite these observations, we can conclude that the data support the “broad” version of a congruence hypothesis when all of the four introduced conditions are satisfied.

By contrast, it might also be the case that our theory does not justify additional main effects. This is true if we expect that out of any two people, the person with more congruent predictor values should be happier than the more incongruent person (even if, e.g., the former person is congruent at low predictor levels, whereas the latter person is incongruent at high predictor levels). To test this “strict” version of a congruence hypothesis, two additional conditions that restrict the surface above the LOC to a constant shape have to be met, thereby preventing additional main effects (Figure 1a; see also Edwards, 2002). The shape of the surface above the LOC can be traced mathematically by setting X = Y in Equation 1 (Edwards & Parry, 1993):

\begin{array}{l} Z = b_{0} + b_{1} X + b_{2} X + b_{3} X^{2} + b_{4} X * X + b_{5} X^{2} \\ = b_{0} + (b_{1} + b_{2}) X + (b_{3} + b_{4} + b_{5}) X^{2} \\ = b_{0} + a_{1} X + a_{2} X^{2} . \end{array}

Here, a ₁ = b ₁ + b ₂ and a ₂ = b ₃ + b ₄ + b ₅. Thus, as for the LOIC, the surface above the LOC can be described by a quadratic equation.

For a strict congruence effect to occur, the surface above the LOC must not differ significantly from a constant shape; that is, neither a ₂ nor a ₁ should be significantly different from 0 (Edwards, 2002). This is the case for the surface in Figure 1a, as its shape above the LOC is given by Z = 1.8 + 0X + 0X ² (see Table 2); all people whose self-view is in line with their reputation (e.g., Tom, Kim, and Sam) are predicted to be equally happy (with a happiness value of 1.8). In Figure 1b, by contrast, the LOC has a linear but rising shape, which is reflected in a significantly positive a ₁ parameter, while a ₂ is nonsignificant. The LOC in Figure 1c has an inverted U-shape due to a significantly negative parameter a ₂, and its vertex is positioned at the backward part of the LOC due to the significantly positive parameter a ₁ (see also Table 2; see OSF Material B at osf.io/yvw93 for details on interpreting quadratic equations).

The properties a ₂ ≈ 0 and a ₁ ≈ 0 thus provide the fifth and sixth necessary conditions when the aim is to test the strict version of the congruence hypothesis. For Figure 1a, we found that all six conditions were satisfied, which implies that people are predicted to be happier the closer their self-view is to their reputation (see OSF Material C at osf.io/yvw93 for the proof); the data support the strict congruence hypothesis.

A Checklist for Testing Congruence Effects With RSA

As outlined above, a response surface must satisfy four conditions (six, if no additional main effects of the predictors are allowed) to reflect a congruence effect (see Figure 3; see also Edwards, 2002). The first principal axis Y = p ₁₀ + p ₁₁ X must not differ significantly from the LOC, so p ₁₀ must not be significantly different from 0 (Condition 1), and p ₁₁ must not be significantly different from 1 (i.e., the confidence interval of p ₁₁ should include 1; Condition 2). Moreover, the surface above the LOIC, given by Z = b ₀ + a ₃ X + a ₄ X ², must be an inverted U-shape, and it must have a nonsignificant slope above the origin (0,0). That is, a ₄ must be significantly negative (Condition 3), and a ₃ must not be significantly different from 0 (Condition 4).⁶ When it is not theoretically justified to allow additional main effects of the predictors, the surface above the LOC, given by the quadratic equation Z = b ₀ + a ₁ X + a ₂ X ², must not differ significantly from a constant shape, which means that the coefficients a ₂ and a ₁ must not be significantly different from 0 (Conditions 5 and 6).

Figure 3.

A checklist to test the congruence hypothesis, stating that a person’s outcome variable is higher, the closer the person’s two predictor variables are to one another. Note that to test the “reverse” congruence hypothesis, namely, whether the outcome variable is lower for more congruent predictors, the first four conditions must be replaced by (1) p ₂₀ ≈ 0, (2) p ₂₁ ≈ 1, (3) a ₄ > 0, and (4) a ₃ ≈ 0, where p ₂₀ and p ₂₁ denote the intercept and slope of the second principal axis, respectively (see, e.g., Figure 2f; Edwards & Parry, 1993).

If any of the four (or six, respectively) conditions is violated, the congruence hypothesis must be rejected (see also Figure 3). In the OSF, we provide R syntax that follows the strategy outlined in Figure 3 (see osf.io/yvw93).

Two Common Misconceptions in RSA Interpretation

In contrast to the procedure outlined above (see also Edwards, 2002), a “simpler” procedure for testing congruence hypotheses has recently become increasingly prominent in social and personality psychology articles applying RSA. It differs from the strategy described in Figure 3 in two major ways, which, as we will now outline, include mathematical misconceptions. Both of these misconceptions have already led to flawed interpretations of RSA results in published empirical articles. Note that we are not going to mention specific articles that include RSA misinterpretations. Because the two misconceptions are popular fallacies in personality and social psychological articles involving RSA, every attempt to quantify their prevalence would be arbitrary and would needlessly expose specific authors. Instead, our explicit aim is to prevent researchers from falling for the outlined misconceptions in their future empirical work.

Misconception 1: RSA Parameters Can Be Interpreted in Isolation

It has sometimes been suggested that each of the RSA parameters a ₁ to a ₄ can be interpreted in isolation, answering “unique questions about how (mis)matches matter” (Barranti et al., 2017, p. 468; see also Shanock et al., 2010). For example, it was indicated that it is sufficient to find a significantly negative a ₄ parameter to identify a congruence effect (e.g., see the example analyses in Barranti et al., 2017, and in Shanock et al., 2010). However, when we find that a ₄ is significantly negative, all that we know is that the surface above the LOIC, which is given by Z = b ₀ + a ₃ X + a ₄ X ², is an inverted U-shaped parabola. We do not know, for example, whether the first principal axis is significantly shifted or rotated away from the LOC (Figure 2a–e). This information would, however, be crucial because we would have to reject the (broad and the strict) congruence hypothesis in this case.

When we test only for a ₄ < 0, we would falsely claim support for a congruence hypothesis in many cases in which our data in fact contradict a congruence effect.⁷ For example, as a consequence of this misconception, published research has claimed support for congruence effects when the line of the highest outcome was in fact shifted or rotated away from the LOC (Figure 2a–c and e), and even when results indicated a simple curvilinear main effect of one predictor while the outcome was unrelated to the other predictor and to the level of congruence (Figure 2d).

Analogously, none of the other RSA parameters (e.g., a ₁ to a ₃) can, when considered in isolation, determine what effect a response surface reflects because they each provide only one coefficient from one line on the surface. As explained above, at least four RSA parameters need to be tested for the respective conditions before the data can be concluded to support the congruence hypothesis (see also Figure 3).

Misconception 2: RSA Can Indicate Congruence Effects Where the Direction of Mismatch Matters

Another common misconception is that the polynomial model in Equation 1 can test whether, in addition to a congruence effect, the direction of mismatch matters for the outcome (e.g., see Figure 2 and the example analysis in Barranti et al., 2017; see also the example analysis in Shanock et al., 2010). For example, it was suggested that when a ₄ < 0 and a ₃ > 0, then “matches tend to be better than mismatches, but underestimates [e.g., self-view lower than reputation] are worse than overestimates [self-view higher than reputation]” (Figure 2 in Barranti et al., 2017). However, this and similar assumptions are problematic for two reasons.

First, when a ₄ < 0 and a ₃ > 0, the respective surface contradicts a congruence effect instead of supporting it. In this situation, the LOIC Z = b ₀ + a ₃ X + a ₄ X ² is an inverted U-shape, but the significantly positive slope (a ₃) at the point (0,0) contradicts the assumption that the LOIC is maximal at this point. That is, we must reject the hypothesis that the surface is highest when the predictors agree because this assumption does not hold even when only predictor combinations that are on the LOIC are considered; the surface contradicts a congruence effect (see also Figure 2a, c, and d; this is similarly true for the surfaces in figures 2B, C and 3 in Barranti et al., 2017; and in figure 1 in Shanock et al., 2010).

Second, a combination of a congruence effect with a “direction of mismatch matters” effect cannot, for mathematical reasons, be reflected by the RSA model currently applied in social and personality psychology (see Equation 1). This would require the surface to be highest when self-view and reputation agree (i.e., above the LOC, see Figure 1a), whereas in addition, people whose self-view falls behind their reputation to some degree (e.g., Mia) would need to be predicted as less happy than people whose self-view exceeds their reputation to the same degree (e.g., Gil). For example, this implies that the LOIC must have its vertex at (0,0) and fall more quickly to the “left” side of this point than to the “right.” This is, however, mathematically impossible. The LOIC is a parabola and is therefore, as a mathematical fact, symmetric around the vertical axis through its vertex. As long as two people on the LOIC are equally “far away” from the vertex (e.g., Mia and Gil in Figure 1a), they are predicted to be equally happy (see OSF Material B at osf.io/yvw93 for further details on the symmetry of a parabola).

As a consequence of Misconception 2, empirical studies were conducted to test a congruence hypothesis where the “direction of mismatch mattered,” and the results were interpreted to support this hypothesis due to significant a ₄ and a ₃ parameters, whereas in fact, (a) these parameters contradicted a congruence effect and (b) the statistical approach did not enable a test of the hypothesis in the first place. To indeed test the suggested effect, one would need to loosen the symmetry restriction on the surface by applying piecewise defined models (Edwards, 2002) or spline regression (Edwards & Parry, 2017) or to extend Equation 1 by adding cubic terms (Humberg, Nestler, Schönbrodt, & Back, 2017; see also OSF Material D at osf.io/yvw93).

RSA Variants and More Advanced Response Surface Methodology

The aim of this article was to clarify how to correctly identify congruence effects with RSA. Beyond this specific application, response surface methodology can address a wide range of further questions and challenges (see OSF Material D at osf.io/yvw93 for details on the following variants and extensions): Not only can the mathematical fundamentals that we just described be used to test the congruence hypothesis, but they are also obligatory for interpreting additional main effects or for testing hypotheses that go beyond congruence effects (see also Cohen, Nahum-Shani, & Doveh, 2010; Edwards, 2002, 2007; Humberg et al., in press). For example, common curvilinear main effects of self-views and reputations (e.g., Figure 1c) can be understood by considering the shape of the surface above the LOC reflected by its coefficients a ₁ and a ₂ (see Edwards, 2002). Here, researchers should be sure to interpret the surface only for ranges of the predictors that actually occurred in the data (e.g., see Schönbrodt, 2016b). As an example of a hypothesis that goes beyond congruence effects, a surface as in Figure 2a might indicate an “optimal margin” effect (Baumeister, 1989), where the first principal axis is shifted away from the LOC such that happiness is highest for people whose reputation exceeds their self-view by a certain amount (see Edwards, 2002, for information on bootstrapping tests for this lateral shift; see also Edwards & Parry, 1993, for more information on how to interpret complex surfaces). Furthermore, one can test the conditions for a congruence effect simultaneously instead of one by one by putting them all in a single model test (Edwards, 2002; Schönbrodt, 2016b). RSA can also be used for testing several competing hypotheses against each other (Burnham & Anderson, 2002; Humberg, Dufner, et al., 2017), and RSA can be adapted to fit complex data structures such as multilevel (e.g., Nestler, Humberg, & Schönbrodt, 2017) or dyadic data (Nestler et al., 2015; Schönbrodt, Humberg, & Nestler, 2017).

Conclusion

RSA is increasingly applied in personality and social psychology. Unfortunately, some persistent misconceptions undermine the validity of many conclusions drawn from RSA results. Here, we provided the mathematical fundamentals for understanding RSA and using it to test congruence hypotheses. We supplied a user-friendly checklist and R syntax that can guide readers through the application of RSA. Moreover, we reasoned that RSA parameters cannot be interpreted in isolation and that standard RSA cannot detect congruence effects where the direction of mismatch matters. We believe that, when applied carefully, RSA will essentially improve statistical inferences in congruence research and beyond, and we hope that this article will help researchers reach this goal.

Footnotes

Authors’ Note

Additional materials for this article can be found in the OSF at . The OSF materials include a recap of the interpretation of quadratic equations, the proofs of the central mathematical statements in this article, example data, and R code that guides the user through the test of congruence effects with RSA and that will also enable users to reproduce all example analyses reported in this article.

Acknowledgments

We thank Simon Breil, Jennifer Deventer, and Natalie Förster for their valuable comments on a previous version of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

The supplemental material for this article is available at .

Notes

References

Aiken

L. A.

West

S. G.

(1991). Multiple regression: Testing and interpreting interactions. Thousand Oaks, CA: Sage.

Bar-Kalifa

Pshedetzky-Shochat

Rafaeli

Gleason

M. E. J.

(in press). Daily support equity in romantic couples: Response surface analyses of monadic and dyadic data. Social Psychological and Personality Science.

Barranti

Carlson

E. N.

Côté

(2017). How to test questions about similarity in personality and social psychology research: Description and empirical demonstration of response surface analysis. Social Psychological and Personality Science, 8, 465–475.

Barranti

Carlson

E. N.

Furr

R. M.

(2016). Disagreement about moral character is linked to interpersonal costs. Social Psychological and Personality Science, 7, 806–817.

Baumeister

R. F.

(1989). The optimal margin of illusion. Journal of Social and Clinical Psychology, 8, 176–189.

Bleidorn

Schönbrodt

Gebauer

J. E.

Rentfrow

P. J.

Potter

Gosling

S. D.

(2016). To live among like-minded others: Exploring the links between person-city personality fit and self-esteem. Psychological Science, 27, 419–427.

Boele

Sijtsema

J. J.

Klimstra

T. A.

Denissen

J. J. A.

Meeus

W. H. J.

(2017). Person-group dissimilarity in personality and peer victimization. European Journal of Personality, 31, 220–233.

Box

G. E. P.

Draper

N. R.

(1987). Empirical model-building and response surfaces. Oxford, England: John Wiley & Sons.

Box

G. E. P.

Wilson

K. B.

(1951). On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society, Series B, 13, 1–45.

10.

Brion

Lount

R. B.

Jr. Doyle

S. P.

(2015). Knowing if you are trusted: Does meta-accuracy promote trust development? Social Psychological and Personality Science, 6, 823–830.

11.

Burnham

K. P.

Anderson

D. R.

(2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York, NY: Springer.

12.

Chopik

W. J.

Motyl

(2016). Ideological fit enhances interpersonal orientations. Social Psychological and Personality Science, 7, 759–768.

13.

Cohen

Nahum-Shani

Doveh

(2010). Further insight and additional inference methods for polynomial regression applied to the analysis of congruence. Multivariate Behavioral Research, 45, 828–852.

14.

Cronbach

L. J.

Furby

(1970). How we should measure “change”: Or should we? Psychological Bulletin, 74, 68–80.

15.

Denissen

J. J. A.

Bleidorn

Hennecke

Luhmann

Orth

Specht

Zimmermann

(2018). Uncovering the power of personality to shape income. Psychological Science, 29, 3–13.

16.

Derrick

J. L.

Houston

R. J.

Quigley

B. M.

Testa

Kubiak

Levitt

… Leonard

K. E.

(2016). (Dis)similarity in impulsivity and marital satisfaction: A comparison of volatility, compatibility, and incompatibility hypotheses. Journal of Research in Personality, 61, 35–49.

17.

Edwards

J. R.

(1994). The study of congruence in organizational behavior research: Critique and a proposed alternative. Organizational Behavior and Human Decision Processes, 58, 51–100.

18.

Edwards

J. R.

(2001). Ten difference score myths. Organizational Research Methods, 4, 265–287.

19.

Edwards

J. R.

(2002). Alternatives to difference scores: Polynomial regression analysis and response surface methodology. In Drasgow

Schmitt

N. W.

(Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 350–400). San Francisco, CA: Jossey-Bass.

20.

Edwards

J. R.

(2007). Polynomial regression and response surface methodology. In Ostroff

Judge

T. A.

(Eds.), Perspectives on organizational fit (pp. 361–372). San Francisco, CA: Jossey-Bass.

21.

Edwards

J. R.

Parry

M. E.

(1993). On the use of polynomial regression equations as an alternative to difference scores in organizational research. Academy of Management Journal, 36, 1577–1613.

22.

Edwards

J. R.

Parry

M. E.

(2017). On the use of spline regression in the study of congruence in organizational research. Organizational Research Methods. Advance online publication.

23.

Edwards

J. R.

Shipp

A. J.

(2007). Perspectives on organizational fit. In Ostroff

Judge

T. A.

(Eds.), Perspectives on organizational fit (pp. 209–258). San Francisco, CA: Jossey-Bass.

24.

Fox

(2016). Applied regression analysis and generalized linear models. Thousand Oaks, CA: Sage.

25.

Franken

Laceulle

O. M.

Van Aken

M. A. G.

Ormel

(2017). Using response surface analysis to interpret the impact of parent–offspring personality similarity on adolescent externalizing problems. European Journal of Personality, 31, 104–117.

26.

Hill

W. J.

Hunter

W. G.

(1966). A review of response surface methodology: A literature survey. Technometrics, 8, 571–590.

27.

Humberg

Dufner

Schönbrodt

F. D.

Geukes

Hutteman

Küfner

A. C. P.

… Back

M. D.

(2017). Is accurate, positive, or inflated self-perception most advantageous for psychological adjustment? A competitive test of key hypotheses. Retrieved from osf.io/9w3bh

28.

Humberg

Dufner

Schönbrodt

F. D.

Geukes

Hutteman

van Zalk

M. H. W.

… Back

M. D.

(in press). Enhanced versus simply positive: A new condition-based regression analysis to disentangle effects of self-enhancement from effects of positivity of self-view. Journal of Personality and Social Psychology.

29.

Humberg

Nestler

Schönbrodt

F. D.

Back

M. D.

(2017). Testing advanced congruence effects with response surface analysis of third degree. Manuscript in preparation.

30.

Ilmarinen

V. J.

Vainikainen

M. P.

Verkasalo

M. J.

Lönnqvist

J. E.

(2017). Homophilous friendship assortment based on personality traits and cognitive ability in middle childhood: The moderating effect of peer network size. European Journal of Personality, 31, 208–219.

31.

Koppensteiner

Stephan

(2014). Voting for a personality: Do first impressions and self-evaluations affect voting decisions? Journal of Research in Personality, 51, 62–68.

32.

MacCallum

R. C.

Mar

C. M.

(1995). Distinguishing between moderator and quadratic effects in multiple regression. Psychological Bulletin, 118, 405–421.

33.

Myers

Khuri

Carter

(1989). Response surface methodology: 1966-1988. Technometrics, 31, 137–157.

34.

Nestler

Grimm

K. J.

Schönbrodt

F. D.

(2015). The social consequences and mechanisms of personality: How to analyse longitudinal data from individual, dyadic, round-robin and network designs. European Journal of Personality, 29, 272–295.

35.

Nestler

Humberg

Schönbrodt

F. D.

(2017). The study of congruence effects with response surface analysis in case of multilevel data. Manuscript submitted for publication.

36.

Olderbak

S. G.

Malter

Sofio

Wolf

Jones

D. N.

(2017). Predicting romantic interest at zero acquaintance: Evidence of sex differences in trait perception but not in predictors of interest. European Journal of Personality, 31, 42–62.

37.

Quintus

Egloff

Wrzus

(2017). Predictors of volitional personality change in younger and older adults: Response surface analyses signify the complementary perspectives of the self and knowledgeable others. Journal of Research in Personality, 70, 214–228.

38.

Rosen

N. O.

Bailey

Muise

(2018). Degree and direction of sexual desire discrepancy are linked to sexual and relationship satisfaction in couples transitioning to parenthood. Journal of Sex Research, 55, 214–225.

39.

Schönbrodt

F. D.

(2016a). RSA: An R package for response surface analysis (Version 0.9.10). Retrieved from https://cran.r-project.org/package=RSA

40.

Schönbrodt

F. D.

(2016b). Testing fit patterns with polynomial regression models. Retrieved from osf.io/3889z

41.

Schönbrodt

F. D.

Humberg

Nestler

. (2017). Testing similarity effects with dyadic response surface analysis. Manuscript submitted for publication.

42.

Shanock

L. R.

Baran

B. E.

Gentry

W. A.

Pattison

S. C.

Heggestad

E. D.

(2010). Polynomial regression with response surface analysis: A powerful approach for examining moderation and overcoming limitations of difference scores. Journal of Business and Psychology, 25, 543–554.

43.

Verhagen

Lodder

G. M. A.

Baumeister

R. F.

(in press). Unmet belongingness needs but not high belongingness needs alone predict adverse well-being: A response surface modeling approach. Journal of Personality.

44.

Weidmann

Schönbrodt

F. D.

Ledermann

Grob

(2017). Concurrent and longitudinal dyadic polynomial regression analyses of Big Five traits and relationship satisfaction: Does similarity matter? Journal of Research in Personality, 70, 6–15.