Identifying Invalid Responders in a Campus Climate Survey: Types,Impact on Data,and Best Indicators

Abstract

Self-report surveys that are online, lengthy, and contain sensitive material greatly increase the probability of invalid responding (IR) on the instrument. Most research to inform our identification of invalid responders have not been able to test their methodologies where all these conditions are present. This study systematically adopted 10 IR indicators based on direct, archival, and statistic strategies to identify IR providing answers on a lengthy survey collecting campus climate/violence information that college students (N = 6,995) accessed online. Exploratory factor analysis indicated two internal factors (i.e., careless and extreme responding) underlying these IR indicators. Latent class analysis identified 4.8% of the sample as being invalid responders. Compared with honest responders, invalid responders were significantly more likely to report forms of victimization and a greater negative impact from physical abuse or sexual assault. Of importance, mean scores on victimization scales were significantly higher for invalid responders, illustrating the potential for IR data to skew prevalence rates. IR indicators differentially identified honest and invalid responders. The findings of this study contribute to the systematic investigation of IR with college students completing online and lengthy surveys that address sensitive material.

Keywords

invalid responding survey measurement college surveys statistical strategies for invalid responding

Self-administered questionnaires or self-report survey data are important modes for measuring participants’ experiences regarding sensitive topics, such as disability status; victimization; and lesbian, gay, bisexual, transgender, and queer or questioning (LGBTQ) identity (e.g., Saewyc et al., 2004; Schwarz, 1999; Tourangeau & Yan, 2007; Turner et al., 1998). However, because self-administered surveys are typically confidential and/or anonymous to encourage truthful responses, and researchers cannot monitor individual responses, especially for online surveys, it is likely that some responses may be invalid (Cornell et al., 2012; Fan et al., 2006; Huang et al., 2015; Robinson & Espelage, 2011; Savin-Williams & Joyner, 2014). To add to the knowledge regarding invalid responding (IR) to online surveys, the purpose of this project was to investigate IR in a lengthy data set which collected sensitive campus climate/violence information online.

Survey Characteristics That Increase the Probability of Invalid Responses

Survey literature has demonstrated that not all surveys are at equal risk for invalid responses, but rather surveys that are online, lengthy, and contain sensitive information are more likely to produce such responses (Meade & Craig, 2012). Online surveys, even so, continue to be utilized, because they offer advantages of easy dispersal, automated data collection and coding, and responder anonymity. Invalid responses may occur with online surveys because there is no social accountability, that is, the survey taker has no knowledge of or contact with the survey administrator, so they feel no social obligation of truthfulness or carefulness in that anonymous setting. In addition, because online survey takers cannot be monitored as to the conditions under a survey is completed, there is no accounting for possible environmental distractions (Douglas & McGarty, 2001; Lee, 2006; Meade & Craig, 2012) that may be present during the administration (e.g., a student listening to music while answering survey items). A lengthy survey may be vulnerable to IR because it has the potential to produce fatigue that may result in careless or random responses, particularly toward the middle or end of it (Baer et al., 1997; Berry et al., 1992). And surveys which ask participants to reveal potentially sensitive information, even when instructions inform respondents that their answers are anonymous and/or confidential, have a greater potential to collect inaccurate information (e.g., Turner et al., 1998). For example, the Centers for Disease Control and Prevention (1997) conducted a more nationally representative survey with college students, the National College Health Risk Behavior Survey in 1997, which included sensitive victimization questions, but there was no attempt to identify invalid responders in this data set.

Characteristics of Research Investigating IR Methodology

Investigations seeking the best methodology for detecting IR in surveys often could not or did not conduct their research to maximize the conditions discussed above that are more likely to produce IR (i.e., lengthy, online, and collecting sensitive information). Studies investigating invalid responses among college students or adults (usually recruited volunteers), typically provided incentives for participation and/or provided survey content regarding personality or attitude variables (thus removing the need for participants to avoid responding to sensitive content). For instance, Meade and Craig (2012) utilized 438 college students enrolled in an introductory psychology course who were required to participate in at least 3 hr of research studies. Although they found that different IR indicators could identify different types of IR, the students essentially “volunteered” for this particular study, because they chose it from among listed experiments and they had an incentive for participation, albeit in the form of course credit. Other studies that included college students have also not employed research designs that tested most of the conditions known to increase IR. Huang et al. (2012) recruited a sample of 725 undergraduates from a large Midwestern university who received credit for participation in completing Goldberg’s (1999) 300-item personality inventory. Although they found that psychometric antonyms (i.e., answering items opposite in content dissimilarly), and self-report of individual reliability were effective at detecting insufficient effort responses, the only characteristic of their survey likely to increase IR was that it was lengthy. More recently, Huang et al. (2015), seeking to test the efficacy of an infrequency detection scale with items that were counterfactual or impossible, used another undergraduate psychology pool from a large Midwestern university and incentivized their participation with extra course credit. The researchers concluded that using an infrequency approach may be a useful tool for identification of invalid responders for low-stakes surveys (Huang et al., 2015), but their sample still consisted of an incentivized sample which engaged in a project that was neither lengthy nor contained sensitive content. To what degree do college students view the task of completing a survey seriously and/or responsibly is not clear; thus, further study to determine whether a subset of college respondents respond invalidly needs to occur. Therefore, providing data on IR in a college population completing a lengthy online survey related to the sensitive topic of victimization is expected to advance our knowledge regarding indicators and patterns predictive of IR in college students.

Methods for Detecting Invalid Responses in Surveys

To enhance confidence in collected data, scholarship has emerged dedicated to detecting invalid responses and/or responders. One model for classifying indicators for identifying invalid responses, proposed by DeSimone et al. (2015), categorizes these strategies as direct methods, archival methods, and statistical methods.

Direct methods involve preplanning by the survey team to include items, ranging in intent and function that allow for evaluating response validity. These include the following: self-report of careful or accurate test taking, items that give instructions to the test-taker, and bogus items stating impossibilities that respondents should not endorse. Self-report of carefulness or accuracy has been used as a screening mechanism to remove surveys from a database when a respondent directly reports not paying attention while responding, not being careful completing the survey, and even not telling the truth (Curran, 2016; DeSimone et al., 2015; Maniaci & Rogge, 2014). Instructed items have also served as screening devices because respondents not following the specific instructions posed by these items are determined to be careless and/or inaccurate (DeSimone et al., 2015; Maniaci & Rogge, 2014). Bogus items ascertaining impossible or highly unlikely realities (Curran, 2016; DeSimone et al., 2015; Robinson-Cimpian, 2014) can identify respondents who intentionally or unintentionally (i.e., carelessly) endorse these impossibilities.

Archival methods utilize survey takers’ responses over the course of the survey to identify invalid responders based on response time and invariant responding. The length of time a respondent takes to complete a survey, easily recorded with many online survey platforms electronically recording the time a survey is open, can signal potential invalidity if test takers finish below a minimum amount of time for responding reasonably carefully on a survey. Huang et al. (2012) pointed out here is no standard or existing cutoff time to use because surveys vary widely in question length and complexity; thus, it is improbable that an appropriate universal cutoff time will ever be established. However, an example of how to systematically calculate the minimum amount of time necessary to consider a survey validity was Huang et al.’s (2012) cutoff based on an “educated guess” while erring on the conservative side, which was less than 52 seconds per page, or less than 2 seconds per item (for surveys with personality/attitude items).

Another common archival method identifies invariability in a test-taker’s responses that is signaled by the endorsement of the same response option to survey items in a row. Invariant responding may be especially noticeable when surveys mix positively and negatively valenced items (i.e., some reversed worded items in a scale), such that a person having particular attitudes would not respond similarly to the reverse-worded items (Curran, 2016; DeSimone et al., 2015). In addition, where repeated items measure frequency of behaviors or events, it is highly unlikely that all stated items would have occurred at the exact same frequency.

Statistical screening methods constitute the third category for identifying invalid responders. The first type, and probably the simplest statistical approach, is the identification of possible outliers using descriptive statistics (DeSimone et al., 2015). For example, a formula has been derived using the interquartile range (IQR) of participant responses to identify outliers (e.g., Rousseeuw & Croux, 1992; Upton & Cook, 1996; Zwillinger & Kokoska, 2000). The IQR between the first (Q1) and third quartile (Q3) is used in the calculation to identify respondents whose scores fall substantially below the first quartile or substantially above the third quartile, that is, observations below Q1 − 1.5 IQR or above Q3 + 1.5 IQR. Another way of detecting outliers is by computing Mahalanobis D, wherein a respondent is flagged when individual scores are compared with the sample mean scores across an entire survey to produce an estimate of the multivariate distance between an individual’s score and sample mean scores (DeSimone et al., 2015). Any score with a p value less than .05 indicates IR.

The second type of statistical screening methods utilizes psychometric synonyms and psychometric antonyms. These two methods assume respondents’ attitudes stay the same across a survey. In particular, participants are assumed to respond consistently with similar items and respond differently with dissimilar items. By examining the inter-item correlation, item pairs with high positive correlations are identified as psychometric synonyms (e.g., Meade and Craig [2012] used a criterion of .60 as the benchmark of “high” correlation), and item pairs with high negative correlations are identified as psychometric antonyms. However, using a statistical strategy involving psychometric synonyms and antonyms is not always predictable as to whether they will be useable for individual data sets. On one hand, researchers cannot predefine the item pairs before screening for IR, so this limitation adds to the statistical objectivity of the strategy. On the other hand, the existence of such item pairs is largely dependent on the data, so it is possible that psychometric synonyms or psychometric antonyms will not be found through inter-item correlation and thus would not be available for screening IR (DeSimone et al., 2015).

Current Research Questions

The detection of IR in this study was motivated by possession of a survey data set with the three conditions described above that increase the likelihood of IR, thus raising concerns about the quality of responses. First, the survey was conducted online. The survey was also lengthy due to the intent to capture victimization across a range of interpersonal behaviors as well as to assess perceptions of campus climate. In addition, the survey assessed potentially sensitive topics such as sexual assault, sexual harassment, and other forms of victimization, which respondents may be hesitant to report. Thus, the three factors stated above render this survey data more prone to IR. A major goal, therefore, was to determine whether invalid responders affect reporting of prevalence data for different forms of victimization.

The following questions were posed for this study:

Research Question 1 (RQ1): What is the relationship among IR indicators that are developed based on direct, archival, and statistical strategies for detecting IR?

Research Question 2 (RQ2): Is it possible to identify common factors that explain the associations among IR indicators?

Research Question 3 (RQ3): Can latent classes (i.e., unknown classes that cannot be directly measured) of participants be identified based on the IR indicators?

Research Question 4 (RQ4): If so, does latent class membership influence how students report victimization?

Research Question 5 (RQ5): Do invalid responders differ from the rest of the sample on demographic variables and victimization?

Research Question 6 (RQ6): Which of the suggested IR indicators were most useful for identifying invalid responders?

By answering these questions, we could disentangle the relationships among the IR indicators (RQ1 and RQ2), determine whether latent classes exist and, if yes, the proportion of each class (RQ3), detect the influence of the invalid responders (RQ4 and RQ5), and finally decide which IR indicators were the most valuable for detecting invalid responders (RQ6).

Method

Participants

The data for these analyses came from a larger survey project conducted in 2017 which examined campus climate and interpersonal violence/harassment victimization. Of the original data set, 6,995 attendees at a large Southern university consented for their data to be used in research projects which they understood would protect their anonymity and confidentiality and would only report on aggregate data to professional audiences. Of the students eligible to participate, 31 students responded “choose not to answer” (CNTA) for every question and, thus, were removed due to providing no data. The final data set consisted of 6,964 participants.

Demographic characteristics of the data set included the following: 62% female, 37% male, and 1% who preferred not to report their gender; 25% first-year students, 49% combined sophomores, juniors and seniors, and 23% graduate/professional students; 78%White students, 7% African American students, 4% East Asian students, 3% Hispanic students, and 8% Other students; and 94% of the students were domestic and 6% were international students. Thirteen percent of students were first generation college students.

Survey Information

The original campus climate survey consisted of two main sections: (a) assessment of students’ attitudes, perceptions, and behaviors about safety, university response to victimization, and social risk factors, and (b) assessment of adverse experiences/victimization. The only sections from the “climate” part of the survey that were used for this project were students’ perceptions about their safety at the university and their attitudes toward affirmative elements of consent for sexual activity, which were the source for identifying items that were similarly answered (i.e., Psychometric Synonyms and Psychometric Antonyms [see below]). The victimization section of the survey consisted of seven forms of violence/harassment that students may experience while being in college. Bullying was assessed by measuring direct verbal bullying, bullying via social media, and physical bulling with follow up questions regarding the perceived reason the bullying occurred (Hamburger et al., 2011). Sexual harassment (six items; Campbell, 2014) and stalking (five items; Dye & Davis, 2003) used general items to cover the major concepts of behaviors within those forms of victimization, such that specific behaviors (e.g., leaving unwanted flowers on your doorstep) were subsumed under general concepts (e.g., unwanted contact) to cover a wide range of behaviors with streamlined items. Three categories of intimate partner victimization were reported by students in relationships during the prior year: reproductive coercion (two items; Miller & McCauley, 2013), serious forms of psychological abuse (five items; Follingstad, 2011), and physical violence (nine items; adapted from Banyard et al., 2013). The most-specific follow-up data were collected from students reporting sexual assault resulting from one of five situations: physical force, threats of harm, incapacitation due to involuntary substance use, incapacitation due to voluntary substance use, or escape from an attempted sexual assault (adapted from Banyard et al., 2013; Campbell, 2014; Krebs et al., 2007). Finally, the survey included demographic information (e.g., gender), and items hypothesized to be indicators of IR (e.g., bogus items, self-report of carelessness).

Measures of Invalid Responding

Direct screening variables

Two types of direct screening variables were constructed. First, to gauge truthfulness (Not Truthful) and/or carelessness (Careless) of respondents, two self-report items asked respondents regarding lying and low effort on the survey (e.g., Cornell et al., 2012; DeSimone et al., 2015; Maniaci & Rogge, 2014). Respondents selected True (0), False (1), or CNTA (missing) in response to the statement: “I have told the truth on this survey.” Respondents reported on their carefulness by selecting True (0), False (1), or CNTA (missing) in response to the statement: “I didn’t pay attention to how I answered this survey.”

Bogus items were the second direct strategy for detecting whether respondents paid attention or deliberately answered dishonestly (DeSimone et al., 2015). The first bogus indicator was derived from questions assessing the presence of extreme physical disabilities (i.e., blindness or impaired vision that cannot be corrected; deafness or severe hearing loss; paralysis in the form of paraplegia or quadriplegia) for which endorsement of more than one would be extremely unlikely (Robinson-Cimpian, 2014). Consultation with the institution’s Disability Resource Center determined that no student was registered as having two or more of the specified physical disabilities. Thus, this bogus indicator (2+ Disabilities) was coded 1 = two or more physical disabilities endorsed and 0 = either one or no physical disability was endorsed.

The second bogus indicator assessed an impossibility regarding student life—“I attend Student Government meetings 20 or more times a month”—that was impossible to achieve because student government meetings never occur that often. This indicator (StuGov) was coded yes = 1, no = 0, or CNTA = missing.

Archival screening variables

The first archival approach for this study used the amount of time for respondents to complete the survey because the survey platform automatically collected response times. Based on Huang et al.’s (2012) strategies when there is no established or standard cutoff, we first measured response time per item because the survey pages were variable rather than the same format. Because many of the survey items were somewhat involved, the survey format varied, and instructions were included throughout the survey, we adopted a criterion of 3 s/item, viewing Huang et al.’s (2012) criterion of 2 s/item to be too conservative for flagging possible invalid responses. Because students who had not experienced any victimization would answer a minimum number of 94 core survey questions, the 3-s criterion to flag respondents who were unlikely to have reasonably considered the items was calculated at 282 s. This indicator (Time) was coded 1 = response time ≤282 s and 0 = response time >282 s.

The second archival indicator of IR used invariant or longstring responding. Respondents’ answers on two victimization scales were flagged if a respondent chose the same frequency option (except for 0) for each of the items within that scale. One of the scales assessed sexual harassment for the prior year with six questions: (a) “Said sexual things to you that you did not want to hear”; (b) “Sent sexual messages or pictures that you did not want (including porn)”; (c) “Asked or pressured you for a date, hook up, or sexual favors even though you had already said no”; (d) “Made unwanted sexual gestures or imitated sexual motions when you did not want them to”; (e) “Touched you sexually (breasts, buttocks or genitals) when you did not want them to”; and (f) “Exposed themselves to you (breasts, buttocks, or genitals) when you did not want them to.” Response options were 0 = Never (0 times), 1 = Once (one time), 2 = Sometimes (two to five times); 3 = Often (six+ times), 4 = CNTA (missing). Respondents were coded as invariant = 1 on this scale if they selected all 1 s, 2 s, or 3 s. They were coded as not invariant = 0 if they selected any combination of responses (thereby indicating variability) or reported all items as never happening because many college students have not experienced these behaviors, thus accurately reporting no incidents. Data from respondents choosing CNTA for more than two items were coded as missing data.

The second scale used to assess invariant responding was the stalking scale that is comprised of five items and has the same response options and scoring as the sexual harassment scale. Items on the stalking scale included the following: (a) “Followed or spied on you in ways that made you afraid,” (b) “Repeatedly tried to communicate with you in ways that made you afraid,” (c) “Repeatedly showed up where you didn’t want them to in ways that made you afraid,” (d) “Invaded your privacy in ways that made you afraid,” and (e) “Did other threatening or damaging things in ways that made you afraid.” Data from respondents who chose CNTA for more than one item on the stalking scale were coded as missing data. The two invariant indicators derived in this way were labeled SexHarInv and StalkInv.

It should be noted that while there were other victimization scales which could yield invariant responding, they were not used because either not all students received the scale or the scale contained fewer items than the suggested responses in a row recommended for screening (e.g., Huang et al., 2012). Specifically, scales assessing physical violence, psychological abuse, or reproductive coercion in an intimate relationship were only administered to students reporting an intimate relationship in the past year. The three-item bullying scale was not considered because it did not have enough items to appropriately assess invariant responding.

Statistical screening variables

Two IR indicators were derived from determining extreme outliers on the sexual harassment scale (SexHar99%) and the stalking scale (Stalk99%). The most conservative option (i.e., 99th percentile) was chosen to indicate outlier status when calculating mean scores of respondents for these victimization scales. The more common method in the literature for determining outlier status, which calculates outliers based on their deviation below the first and above the third quartiles, was not workable for this data set due to the strong skew in the data from individuals who experienced no victimization of a particular type (score = 0). Because of this skew, the typical method for identifying outliers placed almost every person experiencing any victimization into the outlier category. Thus, we established a threshold requiring respondents’ mean scores to fall into the 99th percentile before they were assigned as an “outlier” for this IR indicator (1 = mean score at 99%; 0 = mean score < 99%). We did not employ Mahalanobis D as a screening method for the same reason that our victimization data do not follow a normal distribution (see Meade & Craig, 2012). Requiring scores to reach the threshold of the 99th percentile before indicating IR was expected to result in fewer misclassifications of individuals’ whose high reported rates may be valid and truthful.

To develop a statistical IR indicator based on response consistency (PsySyn), Meade and Craig’s (2012) and DeSimone et al.’s (2015) Psychometric Synonyms index was used as a model. The minimum correlation coefficient defining psychometric synonyms was predefined as .60 for similar concept item pairs identified from two of the survey’s climate scales—Perception of Safety (PS) scale and Attitudes toward Affirmative Elements of Consensual Sex (AECS) scale. Correlations of all item pairs between the two scales revealed three pairs of items correlating above .60. Correlations across these item pairs were conducted for each student to form a binary Psychometric Synonyms index. Any student whose within-person correlation was negative was coded as PsySyn = 1 (IR) for this variable, whereas PsySyn = 0 indicated valid responding.

Data Analysis Plan

Because all the IR indicators were binary, tetrachoric correlations were used to examine the relationships among the IR indicators. Exploratory factor analysis (EFA) with weighted least square of mean and variance (WLSMV) was used to examine the construct validity of the indicators and the relationship of the indicators with other variables in the study. Eigenvalues greater than 1 for the factor analysis were examined (Guttman, 1954; Kaiser, 1960) and scree plots were used to determine where the discontinuity of the eigenvalues occurred (Gorsuch, 1983; Tabachnick & Fidell, 2013). Factor loadings were then evaluated and reported (Pett et al., 2003). To answer whether classes of participants can be identified based on the IR indicators, latent class analysis (LCA) was used to determine if any latent class of IR exists and, if yes, the proportion of each class. Logistic regression analyses examined whether the responses from the invalid responder group(s) differed from responses of the honest group regarding victimization scales. Chi-square tests and independent t-tests were used to test the difference between invalid respondents and the rest of the sample for demographics and victimization. Poisson log-linear regressions were conducted to investigate the strongest IR predictor of each class when conducting this survey. Chi-square tests, independent t-tests, logistic regressions, and Poisson log-linear regressions were conducted in SPSS (IBM Corp, 2016). Tetrachoric correlations, EFA, and LCA were conducted using Mplus 7.4 (Muthén & Muthén, 1998–2017).

Results

Correlational Assessment

Nine of 10 IR indicators had a low to moderate positive correlation with each other, r ranging from .21 to .77 (see Table 1). However, psychometric synonyms (PsySyn) was uncorrelated with all other IR indicators except for response time (r = .228), indicating PsySyn measured a different type of IR, compared with other indicators.

Table 1.

Tetrachoric Correlations Among the Ten Invalid Response Indicators (N = 6,964).

Variables	1	2	3	4	5	6	7	8	9
1. Not Truthful
2. Careless	.765***
3. 2+ Diabilities	.308***	.206*
4. StuGov	.559***	.459***	.534**
5. Time	.684***	.642***	.292***	.718***
6. SexHarInv	.378***	.372***	.357**	.400**	.314***
7. StalkInv	.437***	.412***	.371**	.466**	.371***	.670***
8. SexHar99%	.344***	.312***	.357**	.492***	.314***	.523***	.518***
9. Stalk99%	.461***	.344***	.431**	.542***	.298***	.567***	.772***	.745***
10. PsySyn	−.136	−.028	−.010	.165	.228***	.133	.084	.133	−.002

Note. PsySyn = psychometric synonyms.*p < .05. **p < .01. ***p < .001.

Factorial Evidence

Given that PsySyn was not associated with most of the IR indicators, it was excluded from EFA analyses through which we intended to find common factors of those IR indicators. Results from the EFA suggested two factors to extract as evidenced by two eigenvalues greater than 1 and the scree plot indicating two factors. Table 2 shows the oblimin rotated standardized factor pattern loadings. A further examination of factor loadings showed the two factors can be interpreted as a carelessness factor (including Not Truthful, Careless, and Time) and an extreme responding factor (including SexHarInv, StalkInv, SexHar99%, and Stalk99%) with substantive loadings (>.30). The correlation between the factors was moderate, r = .51.

Table 2.

Standardized Factor Loading Results of the Exploratory Factor Analysis Among the 10 Invalid Response Indicators (N = 6,964).

Variables	Factor 1 (Carelessness)	Factor 2 (Extremeness)
1. Not Truthful	.878	.036
2. Careless	.855	−.036
3. 2+ Disabilities	.124	.432
4. StuGov	.442	.404
5. Time	.822	−.023
6. SexHarInv	.086	.647
7. StalkInv	.071	.791
8. SexHar99%	−.016	.774
9. Stalk99%	−.072	.970

Note. Factor 1 (Carelessness) correlated with Factor 2 (Extremeness) at .51, p < .001. Standardized factor loadings that are greater than .30 and do not cross-load on two factors are in bold.

Latent Group Results

LCA results

To answer whether participants can be identified based on different styles of responding, underlying latent classes using the 10 identified IR indicators were examined through LCA. We included the PsySyn variable in our LCA analysis given that internal correlations among the latent class membership indicators was not assumed for LCA models. Three latent classes were specified, mimicking strategies of Meade and Craig (2012). LCA analyses suggested three latent classes were significantly better than two latent classes, p_{(Vuong–Lo–Mendell–Rubin test)} < .001 and Bootstrapped parametric Likelihood ratio test = 185.589, p < .001. Table 3 summarizes the probability of class membership by each indicator. Students in Class 1 (n = 6,632; 95.2%) have a close to zero probability of being an invalid responder based on the indicators (p ranges from .002 to .031). Students in Class 2 (n = 269; 3.9%) have a high probability of being classified into that group on the basis of indicators of Time (p = .91), Not Truthful (p = 1.0), and Student Government Attendance (p = .354). Students in Class 3 (n = 63; 0.9%) have high probabilities of being classified into that group based on their use of invariant responding to sexual harassment and/or stalking scales (i.e., SexHarInv and StalkInv), and/or their extremely high mean scores on the sexual harassment and/or stalking scales (i.e., SexHar99% and Stalk99%; p ranges from .388 to .678). Students in Class 3 who selected extreme responses are similar to Class 2 in that they have very short completion times (Time) and report they were not truthful (Not Truthful; p ranges from .384 to .378). The final count indicates that out of the 6,964 respondents, approximately 4.8% responded in an invalid manner. Thus, labels for the final determination of classes are as follows: Class 1 = Honest Class; Class 2 and Class 3 = Invalid Class.

Table 3.

Probability of Categorizing the Invalid Responder Based on the Latent Class Analysis (N = 6,964).

Variables	Honest Class	Invalid Class
Variables	Class 1 (n = 6,632; 95.2%)	Class 2 (n = 269; 3.9%)	Class 3 (n = 63; 0.9%)
1. Not Truthful	.031***	1.000***	.378***
2. Careless	.009***	<0.001	.223***
3. 2+ Disabilities	.002**	0.050*	.131**
4. StuGov	.006**	0.354***	.196
5. Time	.015***	0.912***	.384***
6. SexHarInv	.006***	0.033	.388***
7. StalkInv	.002**	0.021	.455***
8. SexHar99%	.005***	<0.001	.415***
9. Stalk99%	.005***	<0.001	.678***
10. PsySyn	.017***	<0.001	.026

Note. Entropy for the latent class analysis with three classes was .954. Classes 2 and 3 were combined into the Invalid Group given that both classes included students who are not truthful and careless. PsySyn = psychometric synonyms.

p < .05. **p < .01. ***p < .001.

Logistic regression analyses

Logistic regression analysis showed that, in the whole sample, class membership is a significant predictor of reporting seven types of victimization, including sexual assault, bullying, sexual harassment, stalking, intimate partner physical abuse, intimate partner psychological abuse, and reproductive coercion (Table 4). The odds of reporting these forms of victimization were much higher in the Invalid Class than in the Honest Class, odds ratio ranging from 2.84 to 9.98. Also, class membership as an invalid responder was a significant predictor of reporting one’s gender as transgender or non-gender conforming (odds ratio (OR) = 8.29) and reporting greater negative impact from physical abuse (OR = 4.32) or from sexual assault (OR = 6.31). In general, students in the Invalid Class are more likely to report victimization, negative impacts from victimization, and a nontraditional gender identification.

Table 4.

Binary Logistic Regression Results for the Class Membership Predicting Victimization Variables (N = 6,964).

Variable	n	Chi-Square Test	Honest = 0 vs. Invalid = 1
Variable	n	χ²(1)	β	SE	Odds Ratio
Sexual assault	233	51.15	1.84	0.22	6.28
Bullying	1,151	47.34	1.14	0.15	3.05
Sexual harassment	1,740	32.78	1.04	0.18	2.84
Stalking	570	125.45	2.27	0.18	9.64
Physical abuse	210	63.26	2.30	0.25	9.98
Psychological abuse	573	42.93	1.61	0.23	5.02
Reproductive coercion	212	81.24	1.93	0.25	6.92
Gender not conforming	62	27.05	2.12	0.33	8.29
Negative impact from sexual assault	63	11.43	1.84	0.60	6.31
Negative impact from physical abuse	152	5.31	1.46	0.76	4.32

Note. All dependent variables were binary. All p values were less than .001. SE = standard error of the coefficients.

Differences between the Honest Class and the Invalid Class

Chi-square tests showed that the distributions of academic classification, ethnicity, first generation student status, and number of organizations to which students belong were similar between the Honest and Invalid Classes (see Table 5). Although the distribution of gender differed between the two classes, χ²(2) = 61.30, p < .001, the effect size was small, Cramer’s V = .10. Thus, differences between the Honest and Invalid Classes do not appear to be a function of demographic data.

Table 5.

Descriptive Statistics and the Chi-Square Test Results of the Demographic Variables (N = 6,964).

Variable	Whole Sample (N = 6,964)		Honest (n = 6,632)		Invalid (n = 332)
Variable	n	%	N	%	N	%
Gender	χ²(df) = 61.30(2), p < .001, Cramer’s V = .10
Male	2,332	35.5	2,255	35.4	77	39.5
Female	4,171	63.5	4,065	61.3	106	54.4
Transgender/non-gender conforming	62	1.0	50	0.8	12	6.1
Classification	χ²(df) = 7.88(3), p = .048, Cramer’s V = .03
Freshmen	1,723	26.7	1,661	25.9	62	19.7
Undergraduates (not freshmen)	1,679	50.6	3,227	50.3	179	57.0
Graduate	1,158	17.1	1,101	17.1	49	15.6
Professional	455	6.8	431	6.7	24	7.6
Ethnicity	χ²(df) = 2.92(4), p = .57, Cramer’s V = .02
Caucasian	5,354	78.0	5,113	78.1	241	76.5
African American	446	6.5	422	6.4	24	7.6
Hispanic	236	3.4	227	3.5	9	2.9
Asian	290	4.2	272	4.2	18	5.7
Other	539	7.9	516	7.9	23	7.3
First generation students	χ²(df) = 0.19(1), p = .66, Cramer’s V = .01
Yes	894	12.8	854	12.9	40	12.0
No	6,070	87.2	5,778	87.1	292	88.0
Student organization	χ²(df) = 2.59(2), p = .27, Cramer’s V = .02
None	4	0.1	4	0.1	—	—
One	3,291	75.3	3,188	75.3	103	74.6
Two	876	20.1	851	20.1	25	18.1
Three or more	198	4.5	188	4.4	10	7.2

Note. Sensitivity analysis for the student organization variable excluded the cells that had less than five observations.

Table 6 summarizes mean scores of the Honest and Invalid Classes for the victimization scales (i.e., bullying, sexual harassment, stalking, physical abuse, impact of physical abuse, psychological abuse, and reproductive coercion). The Invalid Class had more reports of victimization compared with the Honest Class, t values ranging from −4.58 to −8.89. Further examination of the effect sizes showed that victimization differences between the Honest Class and the Invalid Class were large, with the majority of the Cohen’s ds above .50 (Cohen, 1988), suggesting the Invalid Class could have a significant impact on reported rates of victimization.

Table 6.

Means, Standard Deviations, and the Independent T-Test Results (Honest vs. Invalid) of the Victimization Scales.

Variable	Whole Sample (N = 6,964)		Honest (n = 6,632)		Invalid (n = 332)		Independent T-Test
Variable	M	SD	M	SD	M	SD	t ^a	Cohen’s d
Sexual assault	0.02	0.13	0.02	0.12	0.12	0.32	−4.23	0.55
Bullying	0.14	0.37	0.12	0.34	0.53	0.82	−6.81	0.65
Sexual harassment	0.21	0.46	0.19	0.42	0.98	1.13	−7.86	0.93
Stalking	0.06	0.28	0.05	0.21	0.85	1.01	−8.89	1.10
Physical abuse	0.04	0.22	0.02	0.17	0.53	0.87	−5.12	0.81
Psychological abuse	0.12	0.38	0.11	0.34	0.75	1.00	−5.76	0.86
Reproductive coercion	0.06	0.32	0.05	0.29	0.48	0.92	−4.58	0.63
Negative impact from sexual assault	0.23	0.32	0.19	0.29	0.53	0.37	−4.38	0.28
Negative impact from physical abuse	0.58	0.69	0.46	0.57	1.47	0.86	−5.60	1.38

All p values were less than .001.

Poisson log-linear regressions

Tables 7 and 8 summarize the level of each IR indicator predicting class membership. Time was shown to be the strongest indicator of honest responders, β = −2.65, OR = 0.07, indicating the odds of being an honest responder was 14.29 times higher among those who did not finish the survey quickly compared with those whose completion times were very short (Table 7). Not Truthful was shown to be the strongest indicator distinguishing whether a person was an invalid responder, β = 4.71, OR = 110.67 (Table 8). Thus, the odds of being an invalid responder was 110.67 times higher among respondents who reported they were not truthful, compared with those who responded that they answered honestly. The second strongest indicator of Invalid Class was Careless, β = 4.25, OR = 70.14, followed by StalkInv, Stalk99%, Time, SexHar99%, SexHarInv, Student Government, and 2+ Disabilities (β = 2.07–3.80, OR = 7.93–44.87). The only insignificant IR predictor was PsySyn, p = .745.

Table 7.

Single Poisson Log-Linear Regression Results of Each Invalid Respondent Indicator Predicting the Honest Class (N = 6,964).

Variable	Wald χ²	β	SE	Odds Ratio	95% Wald Confidence Interval for Odds Ratio		1/Odds Ratio
Not Truthful	100.97	−1.09	0.11	0.34	0.27	0.42	2.94
Careless	69.12	−0.57	0.07	0.57	0.50	0.65	1.75
2+ Disabilities	2.05	−0.19	0.13	0.83	0.64	1.07	1.20
Stugov	7.39	−0.79	0.29	0.46	0.26	0.80	2.17
Time	70.22	−2.65	0.32	0.07	0.04	0.13	14.29
SexHarInv	10.68	−0.53	0.16	0.59	0.43	0.81	1.69
StalkInv	18.54	−1.15	0.27	0.32	0.19	0.53	3.13
SexHar99%	11.49	−0.61	0.18	0.54	0.38	0.77	1.85
Stalk99%	22.89	−0.88	0.18	0.42	0.29	0.60	2.38
PsySyn^a	0.01	0.01	0.10	1.01	0.83	1.22	0.99

Note. Degrees of freedom for the Wald Chi-square tests = 1. All p values for the Wald χ² test of model effects except for PsySyn variable were less than .001. PsySyn = psychometric synonyms.

Test of model effect was not significant, p = .952.

Table 8.

Single Poisson Log-linear Regression Results of Each Invalid Respondent Indicator Predicting the Invalid Class (N = 6,964).

Variable	Wald χ²	β	SE	Odds Ratio	95% Wald Confidence Interval for Odds Ratio
Not truthful	674.51	4.71	0.18	110.67	77.59	157.87
Careless	562.31	4.25	0.18	70.14	49.36	99.67
2+ Disabilities	55.01	2.07	0.28	7.93	4.59	13.72
Stugov	132.12	3.11	0.27	22.30	13.14	37.87
Time	971.36	3.48	0.11	32.40	26.04	40.33
SexHarInv	231.11	3.25	0.21	25.89	17.02	39.39
StalkInv	338.18	3.80	0.21	44.87	29.92	67.30
SexHar99%	240.34	3.37	0.22	29.07	18.98	44.50
Stalk99%	403.93	3.78	0.19	43.65	30.21	63.09
PsySyn^a	0.11	−0.19	0.58	0.83	0.27	2.59

Note. Degrees of freedom for the Wald chi-square tests equals to 1. All p values for the Wald χ² test of model effects except for the PsySyn variable were less than .001. PsySyn = psychometric synonyms.

Test of model effect was not significant, p = .745.

Discussion

The contribution of this study lies in the systematic identification of invalid responders completing a campus climate/victimization survey with characteristics previously demonstrated to increase the likelihood of IR. Specifically, the large data set was generated from a survey that was online (i.e., students were provided with their own survey platform link), lengthy (i.e., minimum of 94 items with the potential of 164 items), and assessed sensitive topics (i.e., seven forms of victimization). We are unaware of a project like ours investigating IR while using a survey with all of these characteristics; thus, we expect the findings to be an important contribution to this literature.

The percentage of invalid responders identified in this project was 4.8%. This finding seems in line with other studies investigating problematic response sets in college students, a national sample, or high school students completing surveys. For a national sample, Maniaci and Rogge (2014) cited rates of 3%–9% across their studies, whereas Johnson (2005) identified 3.5% of invalid responders based on an index of invariant responding in his sample that appeared to represent a broad base of respondents. Beach (1989), utilizing a very small sample of undergraduates, reported that 4% of respondents to a paper–pencil questionnaire self-reported inattention compared with 10% of respondents completing the same questionnaire online who also reported inattention to the survey content. Finally, Cornell et al. (2012) reported 11.8% of a sample of 7,801 high school students completing a national student health survey to be invalid responders. It is possible that individuals beyond high school age are less likely to respond with invalid answers, but further investigation comparing age groups under similar conditions is required before any definitive statement can be made.

Because we consulted the IR literature when developing our survey, this project included a range of strategies defined in the literature as direct, archival, and statistical (DeSimone et al., 2015). This inclusive approach to identifying invalid responders through multiple indicators allowed us to determine the relationship of these indicators to each other. In addition, the results indicated that different indicators differentially predicted membership into two classes of invalid responders distinct from an “honest” group, suggests that utilizing just one strategy for invalid responder identification may not capture the range of those survey-takers misreporting data, and that indicators representing different strategies for identification may be necessary. Even though extreme responders maybe easier to identify (i.e., choosing highest scores, responding invariantly), individuals intentionally providing random responses (i.e., careless) may require other indicators to identify them. Supporting this, the factor analysis of the IR indicators bolstered the identification of two classes of invalid responders by demonstrating that the indicators formed two factors with similar themes to the two classes formed by the LCA (i.e., careless and extreme).

Analyses were conducted to suggest the best predictors for identifying survey takers as invalid responders to indicate whether future investigators need to apply the range of IR detection strategies used in this project. Although many of the included indicators using these data performed well, the indicator of self-reported lack of truthfulness best predicted “invalid” group membership, followed by the self-reported indicator of carelessness, and both the invariant and 99th percentile indicators of stalking. Indicating that you have not told the truth on a survey may best predict IR because, rather than making a judgment whether one was or was not careless (i.e., “I didn’t pay attention to how I answered this survey”), willingness to admit intentional misleading suggests a stronger and clearer sense that one did not take the survey as it was meant to be completed. Scoring at the 99th percentile for the stalking scale may have been one of the strongest predictors for IR because stalking behaviors are quite clearly defined actions not likely requiring interpretation compared with some other forms of victimization (e.g., sexual harassment, psychological abuse). Thus, reporting that all five categories of stalking behavior (which include extreme actions) occurred at the highest frequency seems to clearly indicate aberrant responding. In addition, claiming that all five categories of another person’s stalking actions toward you occurred at the exact same frequency also suggests impossible happenings and therefore a choice to deliberately use a mode of responding (i.e., invariant) to speed one’s progress through a survey. In contrast, the recorded time for completion of a survey online appears to best classify honest responders, possibly because thoughtful answers, or even taking the time to fully read items, likely require more time than the minimum number of seconds calculated for finishing a survey with little or random effort.

What may be the most significant finding from this series of analyses was that the data on reported victimization were significantly different between the honest and invalid responders. Of concern, is that higher reported rates of victimization by invalid responders, whether done intentionally as mischievous responding or indirectly by not reading items or answering invariantly, could affect overall prevalence rates of victimization, in this case on college campuses. Response rates, victimization status, and mean scale scores on victimization scales were all significantly different between the honest and invalid responders, with very high odds ratios and effect sizes. When comparing data between the two classes of responders on their claims of victimization for every type of interpersonal violence/harassment, higher rates were recorded by the invalid group. In addition, there were several other reported data points for which invalid responders claimed much greater representation, namely, claiming transgender or non-gender conforming as their gender identity, claiming negative impacts from sexual assault, and claiming negative impacts from intimate partner physical violence. These findings suggest that even with large data sets, identifying invalid responders through various indicators of invalidity to determine whether their data are significantly affected by careless and extreme responders seems warranted. Because invalid responders were not identified by demographic information, removing them from the sample would not change the distribution of demographic information. However, if researchers are interested in reported information regarding interpersonal victimization, then the rates are likely to be significantly lower if invalid responders are removed from the sample, potentially providing more accurate prevalence information. We retained the full sample if the purpose was to display the distribution of demographic information or report descriptive statistics of the items. However, when the report established associations between certain variables and interpersonal victimization, data from the invalid responders were excluded. Our finding that invalid respondents altered data in meaningful ways supports the growing literature that identifying invalid responders is important to reduce problematic data generated in experimental studies or for correlational analyses due to the increased error variance (Johnson, 2005; Maniaci & Rogge, 2014; Meade & Craig, 2012).

Limitations

The data for this study were obtained from one university’s campus climate/victimization survey, and thus, the findings would benefit from replication with universities representing a variety of regions in the United States. In addition, the sample of respondents were primarily Caucasian, although somewhat lower than race/ethnicity statistics for the state in which the survey was conducted (88%; U.S. Census Bureau, 2016). However, the indicators which identified invalid responders in this study are in line with indicators found with other college samples and a national sample. Generalizability of the findings may also be limited due to sampling techniques. Further investigations need to determine whether particular demographic characteristics are associated with IR.

Conclusion

These findings contribute to the investigation of IR in college samples for online surveys, especially lengthy and sensitive ones. Our analyses found that invalid responders are able to be detected from indicators assessing both careless (random) approaches to survey-taking as well as extreme responding. The invalid responders produced very high odds ratios for reporting interpersonal victimization and significantly higher means than honest responders. Different validity indicators were better at predicting classification as an honest respondent compared with invalid respondents. This exploratory study emphasizes the need for continued investigation of IR when collecting sensitive material as well as for online surveys using frequency data in addition to Likert-type responses.

Footnotes

Acknowledgments

The authors wish to thank the Office of the President at the University of Kentucky for financial support of the project as well for institutional support.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Caihong R. Li

Margaret I. Campe

Jaspreet K. Chahal

Author Biographies

Caihong R. Li, PhD, is the Statistician working at the Center for Research on Violence Against Women at the University of Kentucky. Her major research interests include quantitative methods and psychometrics. Her research focuses on the application of latent variable modeling, structural equation modeling, and multilevel models to complex data sets. She is also interested in the development and validation of measurement scales in the field of violence against women.

Diane R. Follingstad, PhD, ABPP, is the Executive Director and Women’s Circle Endowed Chair of the Center for Research on Violence Against Women and a Professor in the Department of Psychiatry at the University of Kentucky. Her quantitative research as a psychologist over 40 years in the area of violence against women has covered issues related to intimate partner violence, physical dating violence, and factors impacting jury verdicts in cases where battered women killed a partner. Her research has led to more sophisticated measurement of psychological aggression and abuse, and she has published critiques regarding the problems of measurement in this field.

Margaret I. Campe, PhD, is the Director of the University of Nevada, Las Vegas Jean Nidetch Women’s Center. Her primary research interests include college campus violence against women and social networks, and intimate partner violence in the context of criminal and family court systems. Margaret received her PhD in Sociology from the University of Kentucky and her MA in Criminology from the University of Minnesota.

Jaspreet K. Chahal, MS, is the research Project Manager for the Center for Research on Violence Against Women at the University of Kentucky. Her research interests include violence and victimization on college campuses, cultural factors and vulnerability to sexual violence, and the adverse health outcomes of violence and victimization. She received her MS in criminal justice from Eastern Kentucky University and is currently finishing her doctoral degree in Criminology, Law, and Society from George Mason University.

References

Baer

R. A.

Ballenger

Berry

D. T. R.

Wetter

M. W.

(1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139–151.

Banyard

Cohn

Edwards

K. M.

Moynihan

M. M.

Walsh

Ward

(2013). Unwanted sexual experiences at UNH: 2012 Study and changes over time. University of New Hampshire. https://cola.unh.edu/sites/default/files/media/2018/10/unwanted-sexual-experiences-unh.pdf

Beach

D. A.

(1989). Identifying the random responder. The Journal of Psychology, 123(1), 101–103.

Berry

D. T. R.

Wetter

M. W.

Baer

R. A.

Larsen

Clark

Monroe

(1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.

Campbell

(2014). Johns Hopkins University campus sexual assault project. Johns Hopkins University.

Centers for Disease Control and Prevention. (1997). Youth risk behavior surveillance: National college health risk behavior survey—United States, 1995 (DHHS Publication No. SS-6). Epidemiology Program Office.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

Cornell

Klein

Konold

Huang

(2012). Effects of validity screening items on adolescent survey data. Psychological Assessment, 24, 21–35. https://doi.org/10.1037/a0024824

Curran

P. G.

(2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. doi.org/10.1016/j.jesp.2015.07.006

10.

DeSimone

J. A.

Harms

P. D.

DeSimone

A. J.

(2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171–181.

11.

Douglas

K. M.

McGarty

(2001). Identifiability and self-presentation: Computer-mediated communication and intergroup interaction. British Journal of Social Psychology, 40, 399–416.

12.

Dye

M. L.

Davis

K. E.

(2003). Stalking and psychological abuse: Common factors and relationship-specific characteristics. Violence and Victims, 18, 163–180.

13.

Fan

Miller

B. C.

Park

Winward

Christensen

Grotevant

Tai

(2006). An exploratory study about inaccuracy and invalidity in adolescent self-report surveys. Field Methods, 18(3), 223–244.

14.

Follingstad

D. R.

(2011). A measure of severe psychological abuse normed on a nationally representative sample of adults. Journal of Interpersonal Violence, 26(6), 1194–1214. https://doi.org/10.1177/0886260510368157

15.

Goldberg

L. R.

(1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In Mervielde

Dreary

Fruyt

F. D.

Ostendorf

(Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilberg University Press.

16.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Lawrence Erlbaum.

17.

Guttman

(1954). Some necessary and sufficient conditions for common factor analysis. Psychometrika, 19, 149–161.

18.

Hamburger

M. E.

Basile

K. C.

Vivolo

A. M.

(2011). Measuring bullying victimization, perpetration, and bystander experiences: A compendium of assessment tools. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control.

19.

Huang

Bowling

Liu

(2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business & Psychology, 30(2), 299–311.

20.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business Psychology, 27, 99–114. https://doi.org/10.1007/s10869-011-9231-8

21.

IBM Corp. (2016). IBM SPSS statistics for Windows (Version 24.0).

22.

Johnson

J. A.

(2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.

23.

Kaiser

H. F.

(1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151.

24.

Krebs

C. P.

Lindquist

C. H.

Warner

T. D.

Fisher

B. S.

Martin

S. L.

(2007). Campus sexual assault (CSA) study, Final report. National Institute of Justice. https://www.ncjrs.gov/pdffiles1/nij/grants/221153.pdf

25.

Lee

(2006). Privacy, publicity, and accountability of self-presentation in an on-line discussion group. Sociological Inquiry, 76, 1–22.

26.

Maniaci

M. R.

Rogge

R. D.

(2014). Caring about carelessness: Participant inattention and its effect on research. Journal of Research in Personality, 48, 61–83.

27.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085.

28.

Miller

McCauley

H. L.

(2013). Adolescent relationship abuse and reproductive and sexual coercion among teens. Current Opinion in Obstetrics & Gynecology, 25(5), 364–369.

29.

Muthén

L. K.

Muthén

B. O.

(1998–2017). Mplus user’s guide: Statistical analysis with Latent variables (7th ed.).

30.

Pett

M. A.

Lackey

N. R.

Sullivan

J. J.

(2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. SAGE.

31.

Robinson

J. P.

Espelage

D. L.

(2011). Inequities in educational and psychological outcomes between LGBTQ and straight students in middle and high school. Educational Researcher, 41, 309–319. https://doi.org/10.3102/0013189X11422112

32.

Robinson-Cimpian

J. P.

(2014). Inaccurate estimation of disparities due to mischievous responders: Several suggestions to assess conclusions. Educational Researcher, 43(4), 171–185. https://doi.org/10.3102/0013189X14534297

33.

Rousseeuw

P. J.

Croux

(1992). Explicit scale estimators with high breakdown point. In Dodge

(Ed.), L1-statistical analysis and related methods (pp. 77–92). North-Holland.

34.

Saewyc

E. M.

Bauer

G. R.

Skay

C. L.

Bearinger

L. H.

Resnick

M. D.

Reis

Murphy

(2004). Measuring sexual orientation in adolescent health surveys: Evaluation of eight school-based surveys. Journal of Adolescent Health, 35(4), 345.E1–345.E15. https://doi.org/10.1016/j.jadohealth.2004.06.002

35.

Savin-Williams

Joyner

(2014). The dubious assessment of gay, lesbian, and bisexual adolescents of Add Health. Archives of Sexual Behavior, 43(3), 413–422.

36.

Schwarz

(1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105.

37.

Tabachnick

B. G.

Fidell

L. S.

(2013). Using multivariate statistics (6th ed.). Pearson.

38.

Tourangeau

Yan

(2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883. https://doi.org/10.1037/0033-2909.133.5.859

39.

Turner

C. F.

Rogers

S. M.

Lindberg

L. D.

Pleck

J. H.

(1998). Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science, 280(5365), 867–873.

40.

Upton

Cook

(1996). Understanding statistics (p. 55). Oxford University Press.

41.

U.S. Census Bureau. (2016). https://www.census.gov/quickfacts/fact/table/KY#viewtop

42.

Zwillinger

Kokoska

(2000). CRC standard probability and statistics tables and formulae (p. 15). CRC Press.