Abstract
The purpose of this study was to develop and evaluate the Relationship Red Flags Scale, a measure of ability to recognize warning signs of dating violence. The factor structure and psychometric properties of this measure were tested with independent samples of college women and men, and test–retest reliability was assessed with college women. Specifically, our first study examined the factor structure, validity, and reliability of the Relationship Red Flags measure with a sample of 433 undergraduate women. Confirmatory factor analyses provided support for a 25-item five-factor structure (i.e., Monitoring Behaviors, Controlling Behaviors, Demeaning Behaviors, Threatening and Aggressive Behaviors, and Jealous and Possessive Behaviors). Internal consistency reliability was adequate and the validity of the measure was supported for use with college women (negative correlations were found among measures of warning signs and acceptance of dating violence). Our second study replicated these findings with an independent sample of 330 college men. Finally, a third investigation found that the measure demonstrated adequate test–retest reliability with an independent sample of 47 college women.
Risk recognition, or the ability to detect personal danger, was established as important in the prevention of sexual assault (Marx & Soler-Baillo, 2005). Risk recognition has also been linked positively to changes in preventive health behaviors (e.g., obtaining vaccinations and mammograms; Brewer et al., 2007; Orom, Kiviniemi, Shavers, Ross, & Underwood, 2013). The ability to recognize risks associated with dating violence, that is, to identify warning signs of abuse in romantic relationships, may reduce the prevalence of dating violence (defined as the use or threat of physical force, verbal denigration, coercion into sexual activities, and social isolation within a dating relationship). Occurring often on college campuses (Knowledge Networks, 2011), between 16% and 50% of college women reported experiences of dating violence prior to graduation (Murray & Kardatzke, 2007). Relatedly, between 20% and 30% of college men reported experiencing physical aggression from an intimate partner and 70% and 90% of college men were victims of psychological aggression from an intimate partner (Shorey, Cornelius, & Bell, 2008). Thus, the purpose of this study was to develop and investigate the psychometric properties of a measure of warning signs of dating violence for use with college students.
Young women in the United States between the ages of 18 and 24 are at the greatest risk of experiencing dating violence (Bureau of Justice Statistics, 2012). However, regardless of gender, dating violence can have detrimental effects on a victim’s mental health, physical health, and academic performance (e.g., victims may experience anxiety, depression, posttraumatic stress disorder, and alcohol and drug abuse; Haynie et al., 2013; Sabina & Straus, 2008; Wolitzky-Taylor et al., 2008). Moreover, the effects of psychological aggression include poor academic outcomes (i.e., low grades) for college women (LeBlanc, Barling, & Turner, 2014).
Protection Motivation Theory (Maddux & Rogers, 1983) provides insight into how risk recognition can help young women and men identify warning signs of dating violence and potentially leave unhealthy relationships. Dating violence may occur partially from the misinterpretation of risk cues. Many college students confuse warning signs of dating violence with romance or love (e.g., insisting the partner not see friends and only spend time together) and may remain in relationships where they are at risk of abuse. In fact, 59% of college women and 54% of college men indicated that dating violence is difficult to identify (Knowledge Networks, 2011). Protection Motivation Theory states that a person’s willingness to perform a behavior is related to the seriousness of risk that a person perceives when confronted with an unsafe situation. Threat perception mediates the relationship between contact with a risky event and willingness to perform a behavior to increase the likelihood of safety (Singh, Orwat, & Grossman, 2011).
Although Protection Motivation Theory was not developed to address partner violence, it can be useful in understanding how risk recognition can contribute to the reduction of dating violence. When applied to dating violence, a person’s ability to recognize warning signs of dating violence (i.e., threat perception) may mediate the relationship between experiencing a warning sign of dating violence (i.e., risky event) and leaving the potentially abusive relationship (i.e., willingness to perform a behavior). For example, one meta-analysis found that enhancing elements of risk appraisal (i.e., risk perception and perceived severity) positively changed intentions and safety-seeking behaviors such as using sun protection, reducing drug or alcohol intake, and changing dietary behaviors (Sheeran, Harris, & Epton, 2014). Thus, Protection Motivation Theory would suggest that increasing risk perception of dating violence warning signs could enable college students to exit potentially dangerous relationships.
Relatedly, the concept of risk recognition has been studied in relation to sexual assault. Several studies have measured recognition of warning signs in sexual assault scenarios using audio or video vignettes to assess the ability to recognize danger (Loh, Orchowski, Gidycz, & Elizaga, 2007; Marx & Soler-Baillo, 2005; Witte & Kendra, 2010). Specifically, the ability to detect danger was determined by response latencies from when the vignette began until when participants indicated the scenarios became problematic. Although this method was successful at determining when participants felt that danger was imminent, it did not detect the ability to recognize the additive impact of subtle signals that may be present at the beginning of violent dating relationships as participants were asked to indicate when the scenario had “gone too far” rather than when warning signs of potential violence were detected.
In addition, while ther are several scales that measure attitudes toward overt violent behaviors in intimate relationships (Fincham, Cui, Braithwaite, & Pasley, 2008; Price & Byers, 1999), no instrument measures knowledge of warning signs associated with dating violence. This may, in part, be a function of the limited empirical research that has been conducted regarding what constitutes a warning sign of dating violence. The identification of warning signs poses challenges to our field as some behaviors, in isolation and occurring rarely, are not indicators of dating violence (e.g., feeling jealous or wanting the partner to not see family to spend time together). However, the occurrence and intensity of a particular warning sign or the presence of multiple warning signs may be predictive of dating violence. Another challenge to our field is conducting longitudinal research to determine whether the measure of warning signs of dating violence can identify relationships that ultimately become abusive (test sensitivity) and those that do not (test specificity). An important first step, therefore, is creating an instrument that measures potential warning signs of dating violence that can later be evaluated for both sensitivity and specificity.
Therefore, this study will report on the development and initial evaluation of a measure of ability to recognize warning signs of dating violence, the Relationship Red Flags Scale. Specifically, the factor structure and psychometric properties of this measure were tested with independent samples of college women and men, and test–retest reliability was assessed with college women. Creating a measure of warning signs of dating violence in romantic relationships will enable future interventions to be developed to increase risk perception among college students and to test the sensitivity and specificity of the instrument. While many factors are involved when determining whether or not to leave a dating relationship, increased risk perception may aid college students in leaving potentially abusive relationships.
Study 1: Assessing the Factor Structure and Psychometric Properties of the Relationship Red Flags Scale With College Women
First, the construct of interest was defined as behaviors or indicators that are warning signs of dating violence. This construct differs from attitudes toward dating violence in that attitudes reflect the degree to which violent behaviors within an intimate relationship are deemed acceptable or unacceptable. Our measure focused on the ability to recognize the degree to which a behavior is potentially problematic or indicative of future violent behaviors. Items were generated by the primary researcher after reviewing and compiling a list of warning signs of dating violence from Internet sources designed to provide the general public with information regarding unhealthy and healthy relationships, including the National Center of Domestic and Sexual Violence, breakthecycle.org, and stoprelationshipabuse.org.
The list was presented to a research team consisting of the primary researcher, a licensed psychologist, seven graduate students, and five undergraduate students to generate additional items overlooked by the sources. Subsequently, the authors collaborated to identify themes for the measure based on the generated items. The domains included Isolate, Monitor, Control, Demean, Physical Aggression, Jealous, Anger, Minimize, Intimidate, Relationship Characteristics, and Other. Based on the 11 generated themes, the primary researcher and the licensed psychologist who were involved in the item generation process, as well as an independent additional psychologist and a graduate student familiar with the field of intimate partner violence, independently sorted the items into their respective domains and reviewed items to ensure that they were clear and representative of the themes.
The authors reviewed the items sorted by the reviewers. Items that three or more reviewers sorted into the same theme were grouped into the domain with the most votes. Items that three or more reviewers did not categorize under the same domain were not included in the measure (i.e., Asks dating partner to miss class to spend time with him or her, makes negative comments about dating partner’s friends, steals from partner, owns weapons, makes negative comments about dating partner’s family, makes partner feel guilty, foster’s partner dependence on him or her, buys partner gifts after arguments, betrays partner’s confidence, and has mood swings). Many of these items were very similar to items that remained in the proposed measure.
Based on the suggestions from the independent licensed psychologist and graduate student, the domains were revised by the primary author and licensed psychologist involved in the item generation process. The “Other” domain was eliminated from the measure and the domain “Partner History” was added. “Healthy Relationship” items were added to the scale to prevent participants from answering all items in the same way. Ten items were created from a compiled list of healthy relationship characteristics from seven Internet sources including Findyouthinfo.gov, University of Washington’s Sexual Assault and Relationship Violence Information Service (SARIS), and Loveisrespect.org. The 10 most frequently cited items were selected, checked for reading level, and revised accordingly.
The initial measure consisted of 59 items that were written on third grade reading level as determined by readability-score.com. Fifteen experts in the field of violence against women (including professors, shelter staff members, advocates, and graduate students) were contacted via email to provide feedback regarding whether the items were warning signs of dating violence. Based on the experts’ review, nine items that were determined not to be indicative of future violent behaviors were removed from the measure and the measure was reduced to 50 items. The researchers then reduced the number of hypothesized themes into six domains (Monitoring Behaviors [i.e., checking on a dating partner’s location and communication with others], Controlling Behaviors [i.e., limiting a dating partner’s autonomy], Demeaning Behaviors [i.e., putting down a dating partner, trying to decrease a dating partner’s self-esteem], Threatening and Aggressive Behaviors [i.e., making a dating partner afraid of being harmed], Jealous and Possessive Behaviors [i.e., demanding all of a dating partners affection and an unwillingness to share a dating partner’s attention], and Healthy Relationship Behaviors [i.e., enhancing trust, communication, and affection between dating partners]). Two items that did not reflect any of the six domains were removed (i.e., witnessed abuse as a child and gets serious about the relationship quickly), and the analyses were conducted using the remaining 48 items.
The first study examined the factor structure, validity, and reliability of the Relationship Red Flags Scale. We hypothesized that the measure would consist of five dimensions of warning signs of dating abuse plus the Healthy Relationship Behaviors Subscale.
Convergent validity was assessed by examining correlations between the Relationship Red Flags subscales and measures of attitudes toward dating violence. Individuals who scored high on the Relationship Red Flags subscales were hypothesized to score low on acceptance of dating violence, as those who were able to recognize warning signs of dating violence were expected to have disapproving attitudes toward dating violence.
Method
Procedure
After receiving approval from the University’s Institutional Review Board, several methods were used to recruit undergraduate participants who were between the ages of 18 and 22 years and heterosexual (as the measure was developed for individuals in heterosexual relationships and did not include specific items that might characterize abuse in lesbian or gay relationships, that is, “threatening to tell others that their partner is gay”). First, participants were recruited from Introduction to Psychology courses at a large mid-eastern university and offered course credit for completing an online survey. The survey was also posted on an online database for research being conducted at the university. At the end of the survey, participants accessed another survey where they provided an identification number to receive course credit. No identifiable information was collected on the main survey.
Second, participants were recruited using fliers, emails, personal contacts, and invitations in undergraduate courses and to student groups on campus. The researcher or research assistants provided the link to the online survey. Participants recruited using these methods were offered the opportunity to enter a raffle to win one of 10 cash prizes of US$50 for their participation.
Participants
The survey was completed by 527 women. Participants were eliminated if they were missing more than 15% of the items (n = 43) or if they did not answer two validity check questions correctly (i.e., Please select “agree” for this item; n = 51). Thus, the final sample comprised 433 heterosexual, unmarried, undergraduate women between the ages of 18 and 22 (M = 19.6, SD = 1.23). In addition, 25% were in their first year of their undergraduate education, 25% were sophomores, 25% were juniors, and 25% were seniors. Furthermore, 61% identified as White, 14% Asian, 13% Black/African American, 7% Hispanic/Latino, and 5% Biracial/Multiracial. The majority were single (54%), or in a relationship (46%), with fewer than 1% being engaged. In addition, the majority reported no relationship violence in their home growing up (77%), with 19% reporting some violence, 3% quite a bit, and 1% an extreme amount.
Measures
Recognition of warning signs
Participants were instructed to rate the 48 items on the Relationship Red Flags Scale on a 4-point Likert-type scale ranging from 1 (not at all a warning sign [of dating violence]) to 4 (very much a warning sign [of dating violence]).
Dating violence acceptance
The 76-item Attitudes Toward Dating Violence Scale (Price & Byers, 1999) was used to measure attitudes about dating violence toward women (AFDV; 39 items) and attitudes about dating violence toward men (AMDV; 37 items). Each dimension consists of three subscales: psychological dating violence, physical dating violence, and sexual dating violence. Participants responded on a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). A sample item is “A girl should ask her boyfriend first before going out with her friends.” High scores on the subscales indicated acceptance of violent behavior in dating relationships. Price and Byers (1999) indicated adequate reliability, with the alphas of each subscale ranging from .75 to .87, and provided support for construct validity, as females and males with more traditional attitudes toward gender roles were more accepting of violent dating behaviors. For this study, the total scales of AFDV and AMDV showed adequate reliability (α = .92 and α = .92).
Demographic questionnaire
Participants provided their age, race, undergraduate year classification, current relationship status, and experiences with family violence.
Results
A confirmatory factor analysis was conducted to test the hypothesized five-factor model of the 48-item Relationship Red Flags Scale (the healthy relationship items were removed from the analyses). Given that the authors developed the proposed model from a conceptual and theoretical foundation, a confirmatory factor analysis was deemed appropriate to test the proposed model (Brown, 2015). The root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), and comparative fit index (CFI) were chosen a priori to evaluate model fit. Using a single-index strategy, RMSEA values less than .06, SRMR values less than or equal to 0.08, and CFI values greater than or equal to .90 would be indicative of model fit (Hu & Bentler, 1999).
The five-factor model exhibited mixed results in terms of model fit: Satorra–Bentler (SB) χ2(390, N = 433) = 854.493, p < .001, CFI = .821, SRMR = 0.059, and RMSEA = .056, 90% confidence interval (CI) = [.053, .059]. To improve model fit, items that did not load at least .40 on each factor were removed: three items from the Controlling Behaviors dimension (i.e., blames dating partner unfairly, embarrasses dating partner, and never admits mistakes) and two items from the Threatening and Aggressive Behaviors dimension (i.e., abuses alcohol and threatens to harm himself or herself if dating partner wants to break up). An additional item was removed from the Threatening and Aggressive Behaviors dimension as the item loaded on more than one dimension (i.e., charming at the beginning of the relationship and then changes; model modification index >100; Brown, 2015). Then, one to five additional items with the lowest factor loadings on each dimension were removed simultaneously to create a scale with five items on each dimension. The items that were deleted on each dimension were the following: Monitoring Behaviors: Constantly checks up on dating partner; Controlling Behaviors: Tries to control dating partner, apologizes for actions but continues to repeat them, buys dating partner gifts after an argument, and interferes with dating partner’s ability to work; Demeaning Behaviors: Cheats on dating partner, pretends like nothing is wrong after an argument, and wants dating partner to apologize all the time; Threatening and Aggressive Behaviors: Throws things during an argument, grabs dating partner during an argument, cruel to animals, gets angry about small things, and abuses drugs; Jealous and Possessive Behaviors: Will not allow dating partner to leave during an argument.
An additional confirmatory factor analysis was run on the 25-item five-factor structure. The model exhibited adequate fit: SB χ2(974, N = 433) = 2,284.103, p < .001, CFI = .908, SRMR = 0.046, and RMSEA = .052, 90% CI = [.048, .057]. All factor loadings were significant (see Table 1), and the correlations among the five factors ranged from .71 to .89 (see Table 2). The data were not normally distributed as expected from items rated on a 4-point Likert-type scale (Norman, 2010). On average, participants scored in the moderate range on the subscales, indicating that the items were “quite a bit” warning signs. For this sample, the total scale demonstrated adequate internal consistency reliability (α = .93) as did the subscales: Monitoring Behaviors (α = .81), Controlling Behaviors (α = .79), Demeaning Behaviors (α = .78), Threatening and Aggressive Behaviors (α = .69), and Jealous and Possessive Behaviors (α = .80). Although not the focus of the study, adequate internal consistency reliability was found for the Healthy Behaviors subscale as well (α = .74). As hypothesized, scores on the measures assessing accepting attitudes regarding dating violence were correlated negatively with scores on all of the Relationship Red Flags subscales.
Final Items Retained on the Relationship Red Flags Scale and Standardized Factor Loadings for Women (N = 433).
Means, Standard Deviations, Ranges, Reliabilities, and Correlations for the Relationship Red Flags Scale and Attitudes Toward Dating Violence Scales for Women (N = 433).
p < .01.
To further test the fit of the five-factor model, a comparison model consisting of one factor was examined. The one-factor solution did not exhibit adequate fit: SB χ2(405, N = 433) = 1,435.131, p < .001, CFI = .758, SRMR = 0.068, and RMSEA = .077, 90% CI = [.072, .081]. In addition, the ∆χ2 was significant (p < .001), suggesting that the five-factor model was the superior model.
A multivariate analysis of variance (MANOVA) was conducted to explore whether there were differences in responses on the five subscales between participants who witnessed relationship violence in their homes growing up and those who did not witness relationship violence in their families. No differences were found on the scores on the five subscales of the measure, F(6, 449) = .37, p = .897, Pillai’s V = .01,
Study 2: Analyzing the Factor Structure and Psychometric Properties of the Relationship Red Flags Scale With College Men
The previous study was replicated with an independent sample of college men. It was hypothesized that the five-factor (25 item) final model would be replicated, and that scores on the subscales of Relationship Red Flags Scale would be correlated negatively with scores on the AMDV and AFDV Scales.
Method
Procedure
After approval from the institutional review board, participants who were between the ages of 18 and 22, unmarried, and identified as heterosexual were recruited through an online database for research being conducted at the university, where interested participants could access the online survey and participate for course credit. At the end of the survey, participants were prompted to access another survey where they provided their identification number to receive course or extra credit. No identifiable information was collected on the main survey. Participants were also recruited using fliers, personal contacts, and in-person invitations to student organizations. Participants recruited using these methods were offered the opportunity to enter a raffle to win one of 10 cash prizes of US$50 for their participation.
Participants
The survey was accessed and completed by 373 men. Participants were eliminated if they were missing more than 15% of the items (n = 9) or if they did not answer the validity checks correctly (n = 34), leaving a final sample 330 heterosexual, unmarried, undergraduate men between 18 and 22 years old (M = 19.40, SD = 1.29). Of these men, 108 were part of a larger study and completed measures for convergent validity. Regarding undergraduate classification, 34% were in their first year of their undergraduate education, 22% were sophomores, 25% juniors, and 19% seniors. In addition, 58% identified as White, 24% Asian, 8% Black/African American, 7% Hispanic/Latino, and 3% Biracial/Multiracial. The majority of participants were single (69%), or in a relationship (31%), with fewer than 1% engaged. In addition, the majority reported no relationship violence in their home growing up (80%), with 18% reporting some, 1% reporting quite a bit, and 1% reporting an extreme amount of family violence.
Measures
Descriptions of the measures (i.e., Relationship Red Flags Scale, AFDV, AMDV, and demographic questionnaire) are provided in the previous Method section.
Results
A confirmatory factors analysis was conducted to test the five-factor model of the Relationship Red Flags Scale. The model exhibited adequate model fit: SB χ2(390, N = 330) = 858.843, p < .001, CFI = .879, SRMR = 0.060, and RMSEA = .060, 90% CI = [.055, .066]. Although the CFI value was less than .90, all factor loadings were significant (see Table 3), and no items loaded less than .40 on any given factor. The relationships among the dimensions ranged from .76 to .88.
Standardized Factor Loadings on the Relationship Red Flags Scale for Men (N = 330).
To test whether the five-factor model was the best fit, a comparison model with one factor was examined. The one-factor solution demonstrated poor fit: SB χ2(405, N = 330) = 1,521.421, p < .001, CFI = .712, SRMR = 0.087, and RMSEA = .091, 90% CI = [.087, .096]. In addition, the ∆χ2 was significant (p < .001), suggesting differences between the models, with the five-factor solution being a better fit.
To summarize, the five-factor model had adequate fit for college women and men, suggesting configural invariance or similarity in the subscales comprising the measure across gender. However, the model fit indices were less adequate for men, suggesting that the proposed model may be functioning differently for men and women. Thus, further analyses of factorial invariance were conducted.
Specifically, metric invariance was examined to determine whether factor loadings were similar across groups (college women and men). The factor loadings were constrained to be equal across groups. As can be seen in Table 4, the constrained model did not demonstrate adequate fit: (SB) χ2(810) = 6,226.329, p < .001, CFI = .334, SRMR = 3.097, and RMSEA = .132, 90% CI = [.129, .135]. The adjustment to Δχ2 from the baseline model to the constrained model was significant (p < .001), suggesting that the factor loadings differ for women and men on our measure. Given the measure did not demonstrate metric invariance across the gendered groups, analysis of scalar invariance was not conducted.
Invariance Testing Across Gender.
Note. Values in brackets represent 90% confidence intervals. CFI = comparative fit index; SRMR = standardized root mean square residual; RMSEA = root mean square error of approximation; CI = confidence interval.
p < .001.
The means, standard deviations, ranges, reliability estimates, and correlations among the scales can be found in Table 5. On average, participants scored in the moderate range on the subscales, indicating that the items were “quite a bit” warning signs. The total scale demonstrated adequate internal consistency reliability (α = .93), and four of the five subscales demonstrated adequate internal consistency reliability: Monitoring Behaviors (α = .83), Controlling Behaviors (α = .79), Demeaning Behaviors (α = .75), and Jealous and Possessive Behaviors (α = .79). For college men, the Threatening and Aggressive Behaviors subscale demonstrated lower internal consistency reliability (α = .63). The Healthy Relationship Behaviors subscale demonstrated adequate internal reliability consistency as well (α = .76).
Means, Standard Deviations, Ranges, Reliabilities, and Correlations for the Relationship Red Flags Scale and Attitudes Toward Dating Violence Scale for Men (N = 108).
p < .01.
Correlations were calculated to assess convergent validity. There were no significant correlations between scores on the AFDV or AMDV Scales and scores on the five subscales of the Relationship Red Flags Scale for this sample of men.
A MANOVA was calculated to assess differences in subscale scores among participants who witnessed relationship violence in their homes versus those who were not exposed to relationship violence in their families. There were no differences in responses between the two groups, F(5, 325) = 1.80, p = .113, Pillai’s V = .03,
Study 3: Assessing Test–Retest Reliability of the Relationship Red Flags Scale With College Women
Test–retest reliability of the Relationship Red Flags Scale was assessed using an independent sample of college women.
Method
Procedure
This study was approved by the institutional review board. Participants were recruited via in-person classroom invitations by the first author who provided a link to the online survey. Upon completion of the survey, participants who were willing to be contacted to complete the online survey in 2 weeks provided their primary email address on a separate survey. No identifying information was collected on the main survey. An automatic email notification was sent in 14 days and participants received extra credit in their Psychology course for completion of the measures.
Participants
The study was accessed and completed by 47 women. Participants were eliminated if more than 15% of the items were missing (n = 2) or if they did not answer one validity check question correctly (i.e., Please select “Slightly a warning sign” for this item; n = 4). Thus, 41 heterosexual, unmarried, undergraduate women between the ages of 18 and 22 (M = 19.7, SD = 1.35) participated in this study (15% were in their first year, 32% were sophomores, 7% juniors, and 46% seniors). Regarding race, 56% identified as White, 27% Asian, 5% Black/African American, 10% Hispanic/Latino, and 2% Biracial/Multiracial. The majority of participants were in a relationship (61%), and 39% were single. In addition, the majority reported no relationship violence in their home growing up (76%), with 22% reporting some violence and 2% quite a bit of family violence.
Measures
The measures (i.e., the Relationship Red Flags Scale and demographics questionnaire) are described earlier.
Results
Data were analyzed using SPSS 22, and test–retest reliability was assessed using an intraclass correlation coefficient (ICC); an average ICC greater than .75 reflects adequate test–retest reliability (Roach, 2006). Correlations among scores from the first and second administration of the Relationship Red Flags Scale supported test–retest reliability for the five dimensions of the measure (ICC ranged from .81 to .90, p < .001; see Table 6).
Study 3: Test–Retest Reliability Correlations, Mean, and Standard Deviations for Women (N = 47).
Note. ICC = intraclass correlation coefficient.
p < .001.
Discussion
This study provided preliminary support for the use of the Relationship Red Flags Scale, a measure of one’s ability to recognize warning signs of dating violence, with college students. The five-factor model demonstrated adequate model fit for college women and men; items on the measure represented five domains of warning signs, including Monitoring Behaviors, Controlling Behaviors, Demeaning Behaviors, Threatening and Aggressive Behaviors, and Jealous and Possessive Behaviors. In addition, the measure demonstrated good internal consistency and test–rest reliability with one sample of college women. Finally, the validity of the measure was supported for college women in this study by the negative correlations among measures of warning signs and acceptance of dating violence.
On average, women and men scored in the moderate range on the Relationship Red Flags subscales, indicating that the items were “quite a bit” warning signs when compared with the Healthy Relationship Behavior items, which they tended to rate as “not at all a warning sign.” This suggested that college students have some indication that the behaviors on the scale were problematic. However, the women and men did not endorse these behaviors as clear indicators of potential abuse. This finding is consistent with prior research that found that over half of college students reported difficulty identifying warning signs of dating violence (Knowledge Networks, 2011). Framed in protection motivation theory, college students in the sample did not perceive the warning signs of dating violence as serious health threats, and thus, they may view their risk of being harmed in a relationship where these behaviors are present as low. According to protection motivation theory, it is likely that viewing these relationship behaviors as low risk would prevent college students from safety-seeking behaviors, such as leaving a dating relationship when one or more of these warning signs of dating violence is occurring (Singh et al., 2011). Thus, it is imperative to increase knowledge of warning signs of dating violence among college students.
As expected, scores on the five subscales of the Relationship Red Flags Scale were correlated negatively with scores on measures of attitudes toward female and male dating violence for a sample of college women, indicating that recognition of warning signs of dating violence is associated with less acceptance of violent dating behaviors. The magnitude of the relationships among scores on measures assessing ability to recognize warning signs of dating violence and attitudes toward dating violence suggested that while the Relationship Red Flags Scale is similar to measures of attitudes toward dating violence, it is measuring a unique and distinct construct. Thus, increasing risk recognition of warning signs of dating violence in combination with assessing and changing maladaptive attitudes toward the use of violence within dating relationships may be an innovative approach to preventing dating violence and increasing the likelihood that young people will leave potentially dangerous relationships sooner.
Interestingly, scores on the five subscales of the Relationship Red Flags Scale did not correlate with scores on the measures of attitudes toward male or female dating violence for the sample of college men. For the undergraduate women in our study, having less accepting attitudes toward dating violence corresponded with the ability to recognize warning signs of dating violence. However, the college men in our study who indicated that dating violence was wrong were no more able to recognize warning signs of dating violence than those who had more accepting attitudes. It is possible that for undergraduate men, the ability to recognize warning signs of dating violence may be distinct or unrelated to attitudes toward dating violence. This is important, as the majority of research on preventing dating violence focuses on identifying and changing maladaptive attitudes toward using violence within relationships (Vézina & Hébert, 2007). Thus, increasing recognition of warning signs of dating violence may be particularly important when working with college men to prevent dating violence, as targeting maladaptive attitudes toward using violence in intimate relationships alone may leave college men vulnerable to not knowing or recognizing harmful behaviors in their intimate relationships.
The measure did not demonstrate metric invariance, suggesting that the factor loadings on the subscales differed for women and men, and they may attribute different meanings to the latent constructs. This may be due in part to differences in how women and men view being at risk of experiencing and/or perpetrating dating violence. Compared with men, women are more likely to be socialized to believe they are at risk of experiencing dating violence. However, men are more likely to be socialized to accept violence and may even view violence as a way of being strong and fulfilling gender role expectations (Jakupcak, Lisak, & Roemer, 2002; Pence & Paymar, 1993) and may feel as though they are not at risk of being harmed in dating relationships. These findings suggested that comparisons should not be made between the genders and any differences in scores on the measure cannot be meaningfully interpreted.
The internal consistency reliability for the Threatening and Aggressive Behaviors domain was lower than the other five subscales of the Relationship Red Flags Scale for the samples of women and men (α = .69 and α = .63). This is less than ideal and may be related to the variance of the items within the subscale. It is possible that the items on this subscale are measuring different aspects of the construct. Threatening and aggressive behaviors can manifest themselves diversely, ranging from looks that convey danger to actual physical or verbal hostility. Thus, the items within the Threatening and Aggressive Behaviors domain may be less similar to one another than the items on the other five domains. Researchers should use this subscale with caution until the findings regarding the internal consistency reliability estimates are replicated in future studies.
Moreover, one prominent measure of intimate partner violence has been criticized for the inability to determine the intention or context behind the behaviors assessed on the scale (i.e., Were the intentions underlying the behaviors “playful” or “aggressive”?—Krahé, Bieneck, & Möller, 2005). This is a salient issue for measurement of dating violence warning signs—Are there times when individual items on the measure are not problematic and thus are not warning signs? Many items on our scale would not relate to this concern as they assess behaviors that are problematic, regardless of the intent or context underlying their occurrence (e.g., “yells at dating partner” and “constantly insults dating partner”). However, some items might be more amenable to consideration of context when determining whether the behavior is a dating violence warning sign (e.g., “often uses the Internet to check where dating partner is”). Our colleagues are examining how context affects attitudes toward cyber dating violence (Benner, Picciani, & O’Brien, 2018)—in the aforementioned example, college students may often check the location of their partner to determine where to meet after class or work. It is important to note, however, that our measure assesses knowledge of warning signs, and the items included on the measure were determined to be of concern by experts in the field of intimate partner violence. Education about warning signs would include caveats that some of the items, if occurring rarely, might not be indicative of dating violence. Also possible is that some college students might excuse actual warning signs as being teasing, playful, or harmless. Future research might include focus groups with college students to assess the degree to which intent is being considered when responding to measures of warning signs of dating violence and when assessing potentially problematic behaviors in their relationships.
Although one of the strengths of this study was the racial and undergraduate class rank diversity of participants (consistent with the population of the university from which the data were collected), the majority of participants were White college women and men, and thus, generalizability is limited. In addition, given that only 30 items were retained so that the length of the measure would be appropriate for clinical and research use, the instrument may not represent the entire domain of dating violence warning signs; other dimensions of warning signs of dating violence may include partner family and romantic history, minimizing, denying, and blaming, and isolation from social support and economic resources. Furthermore, only one index of convergent validity (attitudes toward dating violence) was tested. Future research should determine whether recognition of warning signs is distinct from other constructs, including dating violence bystander behaviors, use of violence within an intimate relationship, and acceptance of bullying.
The Relationship Red Flags Scale must be administered to more diverse samples to assess the stability of the factor structure and psychometric properties of the measure. This measure was normed using a sample of highly educated college students who chose to participate in the research studies. Self-selection may have influenced the research, as the college students who chose to participate may be well informed about dating violence or related subjects. Thus, results may not be generalizable to more diverse populations with less access to education. In addition, the study may not have captured students who were not interested or motivated to contribute to dating violence research, such as active perpetrators of dating violence, which also may have influenced the results of the studies. In addition, future research should examine the predictive utility of the measure to determine whether people who recognize warning signs of dating violence are less likely to experience violent romantic relationships or more likely to leave potentially violent relationships.
Moreover, the sample consisted of very few people who witnessed relationship violence in their homes, and thus, meaningful interpretations of the differences between participants who witnessed violence and those who did not observe domestic violence could not be made. Future research is needed to determine how this measure functions with individuals who have witnessed or experienced intimate partner violence. Given the literature on risk recognition abilities with sexual assault survivors (Wilson, Calhoun, & Bernat, 1999), we hypothesize that people who experienced dating violence would have less ability to recognize warning signs of dating violence than those individuals without a history of dating violence. Researchers studying this topic should consider administering the initial scale (prior to item deletion), as meaningful items for those who experienced dating violence may have been removed in our instrument development process. After further validation of this measure with dating violence victims, this measure could be used to understand how dating violence victims assess warning signs of dating violence, which could lead to more focused and useful interventions to reduce dating violence.
Although healthy behaviors items were included on the scale to prevent participants from assuming all items were warning signs of dating violence, the inclusion of these items does not detect whether one is able to differentiate between warning signs of dating violence and general conflict within romantic relationships. It would be beneficial to incorporate normal relationship conflict items within the scale.
After further development and validation, this measure has the potential to be used in a variety of contexts with college women and men. For example, the Relationship Red Flags Scale may be used by researchers studying dating violence to determine whether ability to recognize warning signs is correlated with other important behaviors such as intention to be an active bystander or important health outcomes such as likelihood of experiencing or perpetrating dating violence. Furthermore, psychologists can use this measure to assess initial knowledge, growth, and effectiveness for prevention and intervention program targeted at ending dating violence. In addition, upon further validation, the measure may have clinical utility in prevention programs for middle school and high school students. Early intervention with young women and men is key in the prevention of dating violence. If the factor structure were to be replicated with younger students, the measure could serve as a tool for assessing student’s abilities to recognize warning signs of dating violence upon entering and completing prevention programs.
Psychologists also may use this measure with clients who are experiencing psychological distress in romantic relationships. Clients may not consider the dating behaviors that they or their romantic partners are presenting in therapy to be indicative of a potentially dangerous relationship. This measure can be used as a tool for psychologists to determine which dating behaviors clients see as problematic and which dating behaviors their clients do not endorse as warning signs of dating violence. This could be a useful starting point in exploration of these behaviors and psychoeducation about dating violence.
In conclusion, risk recognition of warning signs of dating violence could be an important factor in preventing dating violence. This study provided initial support for a measure that assesses college students’ abilities to recognize warning signs of dating violence. Psychologists may use the Relationship Red Flags Scale to assess the degree to which their clients identify warning signs of dating violence as such and provide psychoeducation about the manifestation of abusive behaviors in dating relationships. We hope that the use of the measure facilitates the prevention of dating violence for college women and men.
Footnotes
Acknowledgements
The authors express their appreciation to the Counseling Psychology Research Center for assistance in the development of items for the measure.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this research was provided by the University of Maryland College of Behavioral and Social Sciences Recruitment Supplement Award.