Cheater Detection Using the Unrelated Question Model

Abstract

Randomized response techniques (RRTs) are useful survey tools for estimating the prevalence of sensitive issues, such as the prevalence of doping in elite sports. One type of RRT, the unrelated question model (UQM), has become widely used because of its psychological acceptability for study participants and its favorable statistical properties. One drawback of this model, however, is that it does not allow for detecting cheaters—individuals who disobey the survey instructions and instead give self-protecting responses. In this article, we present refined versions of the UQM designed to detect the prevalence of cheating responses. We provide explicit formulas to calculate the parameters of these refined UQM versions and show how the empirical adequacy of these versions can be tested. The Appendices contain R-code for all necessary calculations.

Keywords

sensitive questions randomized response technique unrelated question model cheater detection instruction nonadherence

Throughout the social sciences, many findings are based on surveys of various groups of individuals. Most such surveys rely on the assumption that respondents will provide honest answers to survey questions. However, this assumption falters when asking respondents sensitive questions (see Tourangeau and Yan 2007)—questions that are perceived as intrusive, stigmatizing, socially undesirable, or even legally incriminating (Tourangeau, Rips, and Rasinski 2000). Faced with sensitive questions, respondents may refuse to participate in the survey or may simply answer dishonestly (Tourangeau et al. 2000), especially if they are carriers of the sensitive attribute being assessed. Thus, direct questioning has frequently been found to underestimate the true prevalence of sensitive attributes, such as having received an abortion (Fu et al. 1998), having been convicted of driving while intoxicated, having engaged in doping in athletics, and many other issues.

To address this problem, several indirect questioning techniques have been developed throughout the last half-century (see Chaudhuri and Christofides 2013). One of these methods, the Randomized Response Technique (RRT), developed by Warner (1965), introduced the idea of creating anonymity by employing random encryption of the respondents’ answers. In Warner’s model, the respondent receives one of two questions about a sensitive issue. For example, the survey instrument might be designed so that respondents will receive the question S: Have you ever used illicit drugs? with probability p (where $p \neq .5$ ), or they will receive the negative of this question $\neg S$ : Have you never used illicit drugs? with the complementary probability $1 - p$ . A random element (e.g., the throw of a die) determines which of the two questions the respondent receives. The survey is designed so that only the respondent knows the outcome of the randomization (e.g., the respondent is asked to throw the die out of the sight of the investigator). Since only the respondent knows which question he or she has answered the investigator cannot infer the respondent’s status when the respondent answers “yes” or “no” to the survey instrument. However, even though the investigators cannot infer the status of any individual respondent, they can nevertheless estimate the prevalence of the sensitive attribute in a large survey population because the probability p underlying the randomization is known, and hence the estimated prevalence of the sensitive attribute can be derived from the proportion of “yes” answers.

Several revisions and modifications of Warner’s (1965) model have been proposed over the years (e.g., Kuk 1990; Mangat 1994). One of these is the well-established unrelated question model (UQM; Greenberg et al. 1969; see Figure 1). In the UQM, as in the original Warner model, a randomization procedure determines whether the respondent is instructed to answer the sensitive question S. The alternative question, however, is not the reversed sensitive question $\neg$ S, but instead an unrelated innocuous question, the neutral question N (e.g., “Think of someone close to you whose birthdate you know, and answer “yes” if that individual was born on an odd-numbered day”). Thus, the UQM is potentially more psychologically acceptable to survey respondents than the original Warner method because question N is obviously not related to the sensitive attribute and is therefore clearly not incriminating.

Figure 1.

Probability tree of the unrelated question model. The sensitive question S and the neutral question N are randomly received by respondents with probability p and $1 - p$ , respectively. The probabilities of responding “yes” and “no” to the neutral question N are q and $1 - q$ , and the probabilities of responding “yes” and “no” to the sensitive question S are $π$ and $1 - π$ .

With the UQM, as with Warner’s original method, the investigator cannot determine any individual respondent’s status on the sensitive attribute. However, given a large sample of respondents, the investigator can still estimate the prevalence $π$ of the sensitive attribute, provided that the randomization probability p and the prevalence of the neutral attribute q are known. Specifically, the prevalence $π$ can be estimated from the observed proportion $\hat{λ}$ of “yes” responses by the formula:

\hat{π} = \frac{\hat{λ} - (1 - p) \cdot q}{p} .

In several studies, the UQM has elicited prevalence estimates substantially exceeding estimates derived from direct questioning (see Lensvelt-Mulders et al. 2005), such as the prevalence of induced abortion (Abernathy, Greenberg, and Horvitz 1970) and doping in elite athletics (e.g., Ulrich et al. 2018).

However, by introducing an unrelated question N, the UQM opens the possibility that some respondents (“cheaters”) will be tempted to answer a self-protective “no” to either of the two alternative questions on the survey regardless of the true answer to the question. Even though a “yes” response does not necessarily imply having the sensitive attribute, a “no” response greatly reduces the possibility of that conclusion. Specifically, under the standard version of the UQM, the conditional probability $P (A | “yes”)$ of being a carrier given a “yes” response is generally larger than the conditional probability $P (A | “no”)$ of being a carrier given a “no” response, when $π$ is less than one. For example, for $p = 0.75$ , $q = 0.5$ , and $π = 0.2$ one computes $P (A | “yes”) = 0.636$ and $P (A | “no”) = 0.034$ using Bayes’s theorem. Correspondingly, the odds that one is a carrier of the attribute would be 49 times greater given a “yes” response than given a “no” response. Interestingly, this conclusion does not depend on $π$ . As a consequence, this difference in conditional probabilities may encourage cheating behavior in the form of answering “no” under all circumstances.

Another modification of the RRT, the cheater detection model (CDM; Clark and Desharnais 1998), addresses this drawback by dividing respondents into three mutually exclusive categories: (a) honest respondents who are carriers of the sensitive attribute, who will respond “yes” if they receive the sensitive question, (b) honest respondents who are noncarriers of the sensitive attribute, who will respond “no” if they receive the sensitive question, and (c) cheaters who choose the safe option by always responding “no” to any question regardless of whether they are carriers or noncarriers. For illustration, let A be a carrier and $\neg A$ be a noncarrier. Furthermore, let H be an honest respondent and $\neg H$ be a cheater. Then, the probabilities of the three subgroups can be expressed as compound probabilities. These probabilities are for subgroup (a)

P (A \cap H) = P (A | H) \cdot P (H) = ∊ \cdot (1 - γ),

for subgroup (b)

P (\neg A \cap H) = P (\neg A | H) \cdot P (H) = (1 - ∊) \cdot (1 - γ),

and for subgroup (c)

P (\neg H) = P (C) = γ .

Note that these three probabilities add to one.

The CDM is based on another RRT variant, the forced response model (Boruch 1971). This model modifies Warner’s model by replacing the inverted question $\neg$ S by the instruction to simply say “yes.” In other words, the forced instruction to say “yes” simply replaces the neutral question N in the UQM. Hence, if no cheating is assumed and one is therefore not attempting to assess for cheating, the forced response model is mathematically equivalent to a special case of the UQM, namely when the prevalence q of the neutral attribute equals 1. This situation is depicted in the upper part of Figure 2 starting at node H, representing honest respondents only.

Figure 2.

Probability tree of the cheater detection model. Respondents are either cheaters C with probability $γ$ or honest respondents H with probability $1 - γ$ . All respondents randomly receive either the sensitive question S or the instruction F to respond “yes” with probability p_i and $1 - p_{i}$ , respectively. Cheaters C always answer “no” regardless of their carrier status and regardless of whether they receive question S or instruction F. Honest respondents H respond honestly under all conditions. Specifically, if instructed to say “yes,” honest participants always answer “yes.” If instructed to answer the sensitive question S, honest participants answer “yes” with probability $∊$ and “no” with probability $1 - ∊$ . Thus, participants can be divided into three groups: (a) carriers of the sensitive attribute who will honestly respond “yes” with probability $(1 - γ) \cdot ∊ = π$ when receiving S; (b) noncarriers of this attribute who will honestly respond “no” with probability $(1 - γ) \cdot (1 - ∊)$ when receiving S; and (c) cheaters who will respond “no” with probability $γ$ regardless of receiving S or the instruction F to respond “yes.”

However, note that the temptation to cheat may be especially pronounced in the forced response model because the respondent can completely eliminate any suggestion of being a carrier of the sensitive attribute by simply answering “no.” Expressed more formally, in the forced response model, the conditional probability $P (A | “yes”)$ must be always larger than the conditional probability $P (A | “no”)$ because $P (A | “no”) = 0$ (except in the implausible case where $P (A | “yes”)$ is also 0). For example, for $p = 0.75$ and $π = 0.2$ , one computes $P (A | “yes”) = 0.5$ and $P (A | “no”) = 0$ . Correspondingly, the odds that the respondent is a carrier of the attribute would be infinitely greater given a “yes” response than given a “no” response. In other words, answering with “no” is a completely safe option.

Therefore, the CDM includes a parameter to assess the extent of cheating. This is depicted in the lower part of Figure 2 starting at node C and representing cheaters. In this diagram of the CDM, the proportion of cheaters is $γ$ , whereas the proportion of honest respondents is $1 - γ$ . The proportion of respondents carrying the sensitive attribute cannot be estimated because only the proportion $π$ of honest carriers in the overall respondent population, but not the proportion of carriers who are cheaters in the overall population, can be identified by the model. Importantly, $π$ in the CDM is, therefore, not equivalent to $π$ in the UQM because in the former it is defined as the proportion of honest carriers and in the latter as the total proportion of carriers. Nevertheless, in the CDM, the total proportion of carriers in the population must lie within the range that is defined by the lower bound $π = (1 - γ) \cdot ∊$ and the upper bound $π + γ$ . The proportions $π$ and $γ$ thus represent two of the above introduced categories, namely (a) honest carriers and (c) cheaters, respectively. Therefore, the proportion of respondents in the remaining category (b)—the honest noncarriers—is simply given by $1 - (π + γ)$ . In order to estimate the parameters $π$ and $γ$ for computing the two bounds, two probabilities $λ_{1}$ and $λ_{2}$ of responding “yes” are required. They can be estimated by the observed proportion of “yes” responses in two independent samples with $p_{1} \neq p_{2}$ . The resulting equation system can then be solved for $π$ and $γ$ .

Several empirical implementations of the CDM (e.g., Elbe and Pitsch 2018; Moshagen et al. 2010; Ostapczuk 2011; Ostapczuk et al. 2009; Pitsch, Emrich, and Klein 2007; Schröter et al. 2016) have provided evidence of cheating behavior—showing the importance of including a cheating parameter in RRTs. However, studies utilizing the forced response model (e.g., Höglinger, Jann, and Diekmann 2016; Kirchner 2015; Wolter and Preisendörfer 2013) have raised doubts about the validity of this particular method. Specifically, it has been shown (Coutts and Jann 2011; Höglinger et al. 2016) to elicit lower estimates than other indirect questioning techniques, and respondents have reported greater difficulties in understanding this technique. Respondents also seem to express less trust that the technique guarantees anonymity. For example, Lensvelt-Mulders and Boeije (2007) reported that respondents perceived being forced to give a “yes” response as being “forced to be dishonest” (p. 600), which seemingly triggered reluctance.

Ostapczuk et al. (2009) proposed a method to reduce this problem by adding a forced “no” response to the forced “yes” response. In this symmetric design, none of the response options is conclusive of the respondents’ status. Specifically, it is not only possible to be forced to respond “yes” even though one is a noncarrier but also to be forced to respond “no” even though one is in fact a carrier. This should increase compliance with the instructions, and indeed, the authors found cheating to be reduced in an empirical comparison to the original design. Still, it is plausible that a forced response can feel like an implicit response to the sensitive question, something that even this approach does not address.

In summary, although it appears important to account for possible cheating when using RRTs, a technique based on the forced response model may not be ideal. By contrast, the UQM is conceptually and mathematically similar without potentially triggering reluctance by forcing responses. Here, responses to the neutral question are clearly not responses to the sensitive question because the neutral question has content of its own. Thus, in the next section, we propose a model combining the greater psychological acceptability of the UQM’s design with the CDM’s concept of cheating.

Unrelated Question Model—Cheating Extension (UQMC)

Below, we introduce the UQMC, a model combining the basic idea of the CDM (Clark and Desharnais 1998) with the standard version of the UQM (Greenberg et al. 1969). The setup of the UQMC resembles that of the UQM, in that respondents receive the sensitive question S with probability p and the neutral question N with probability $1 - p$ . As in the CDM, participants are categorized as being either honest respondents or cheaters. Figure 3 depicts the resulting probabilities. The same parameters generated in the CDM can be estimated using this model. Specifically, $γ$ corresponds to the probability of being a cheater, and $π = (1 - γ) \cdot ∊$ depicts the probability of being an honest carrier of the sensitive attribute. As in the CDM, the prevalence of the sensitive attribute cannot be inferred because the proportion of carriers can only be estimated among honest respondents and not among cheaters. However, it is still possible to compute an estimated range for the prevalence, which is defined by the bounds $π$ and $π + γ$ .

Figure 3.

Probability tree of the unrelated question model—cheating extension. The prevalence of cheaters C is $γ$ and the prevalence of honest participants H is $1 - γ$ . In both cases, the sensitive question S and the neutral question N are received by participants with probability p_i and $1 - p_{i}$ , respectively. Cheaters always say “no” regardless of the question received. Honest participants respond “yes” with probability q and “no” with probability $1 - q$ if instructed to answer the neutral question N. They answer “yes” with probability $∊$ and “no” with probability $1 - ∊$ , if instructed to answer the sensitive question S. Thus, there are three groups of participants: (a) honest participants who are carriers of the sensitive attribute, who will respond “yes” with probability $(1 - γ) \cdot ∊ = π$ if they receive S; (b) honest noncarriers of this attribute who will respond “no” with probability $(1 - γ) \cdot (1 - ∊)$ if they receive S; and (c) cheaters who will respond “no” with probability $γ$ regardless of whether they receive S or N.

As in the CDM, two independent samples of respondents are required to estimate $π$ and $γ$ . Again, different values of p_i must be used with the two samples, $i = 1, 2$ . Thus, the probability of responding “yes” in sample i is given by

λ_{i} = p_{i} \cdot π + (1 - p_{i}) \cdot (1 - γ) \cdot q .

As $λ_{1}$ and $λ_{2}$ can be estimated from the corresponding observed proportion ${\hat{λ}}_{1}$ and ${\hat{λ}}_{2}$ of “yes” responses in each sample, the resulting equation system can be solved for $π$ and $γ$ ,

\hat{π} = \frac{{\hat{λ}}_{2} \cdot (1 - p_{1}) - {\hat{λ}}_{1} \cdot (1 - p_{2})}{p_{2} - p_{1}},

and

\hat{γ} = 1 - \frac{{\hat{λ}}_{2} \cdot p_{1} - {\hat{λ}}_{1} \cdot p_{2}}{q \cdot (p_{1} - p_{2})} .

The corresponding sampling variances of the two estimates are

V a r (\hat{π}) = \frac{1}{{(p_{2} - p_{1})}^{2}} [{(1 - p_{1})}^{2} \cdot \frac{λ_{2} (1 - λ_{2})}{n_{2}} + {(1 - p_{2})}^{2} \cdot \frac{λ_{1} (1 - λ_{1})}{n_{1}}],

and

V a r (\hat{γ}) = \frac{1}{q^{2} \cdot {(p_{1} - p_{2})}^{2}} [p_{2}^{2} \cdot \frac{λ_{1} (1 - λ_{1})}{n_{1}} + p_{1}^{2} \cdot \frac{λ_{2} (1 - λ_{2})}{n_{2}}] .

The covariance of these estimators is

C o v (\hat{π}, \hat{γ}) = \frac{1}{q \cdot (2 p_{1} p_{2} - p_{1}^{2} - p_{2}^{2})} [(p_{1}^{2} - p_{1}) \cdot \frac{λ_{2} (1 - λ_{2})}{n_{2}} + (p_{2}^{2} - p_{2}) \cdot \frac{λ_{1} (1 - λ_{1})}{n_{1}}] .

Table 1 provides a numerical example to illustrate the UQMC. This example assumes that the estimates ${\hat{λ}}_{1}$ and ${\hat{λ}}_{2}$ of $“yes”$ responses were obtained from two independent samples. The observed proportions of $“yes”$ responses in this table were simulated with $π = 0.2$ , and $γ = 0.3$ . Inserting the values of Table 1 into equations (6 –9) yields parameter estimates $\hat{π}$ and $\hat{γ}$ with their standard errors, which are depicted in Table 2. These estimates can be used to generate the possible range of the prevalence of the sensitive attribute. The lower bound of this range (i.e., the lowest possible estimate of the prevalence) is $\hat{π} = 0.190$ , with a 95 percent confidence interval of $0.149$ to $0.231$ . The upper bound is $\hat{π} + \hat{γ} = 0.190 + 0.305 = 0.495$ . The sampling variance of this upper bound is given by

V a r (\hat{π} + \hat{γ}) = V a r (\hat{π}) + V a r (\hat{γ}) + 2 \cdot C o v (\hat{π}, \hat{γ}),

Table 1.

Numerical Example Illustrating the Unrelated Question Model—Cheating Extension.

Sample	n_i	p_i	q	$o_{y i}$	$o_{n i}$	${\hat{λ}}_{i}$
1	1,000	.75	.5	229	771	.229
2	1,000	.25	.5	308	692	.308

Note. n_i = size of sample i; p_i = probability of being assigned to the sensitive question in sample i; q = prevalence of the neutral attribute; $o_{y i}$ = observed frequency of “yes” responses in sample i; $o_{n i}$ = observed frequency of “no” responses in sample i; ${\hat{λ}}_{i}$ = proportion of “yes” responses in sample i.

Table 2.

Numerical Example Illustrating the Unrelated Question Model—Cheating Extension (Continued).

Parameter	Prevalence	Estimate	SE	CI
$π$	.200	.190	.021	[.149, .231]
$γ$	.300	.305	.046	[.215, .395]
$π + γ$	.500	.495	.079	[.341, .648]

Note. SE = standard error of parameter estimate; CI = 95 percent confidence interval of parameter estimate.

using equations (8 –10). Therefore, the 95 percent confidence interval of the upper bound ranges from $0.341$ to $0.648$ . Hence, even though the prevalence of carriers among cheaters remains unknown, one can conclude from this model that the estimated total proportion of carriers is at least $0.190$ and at most $0.495$ with 95 percent confidence intervals ranging from $0.149$ to $0.648$ .

It is important to note that the size of this range is in large part due to the true cheating proportion, which is 0.3 in this example, and not merely due to random sampling error. A model that does not take cheating into account, such as the original UQM, would therefore yield an estimate with a smaller confidence interval. On first sight, this may look preferable. However, this estimate would be biased, as it disregards the true prevalence of cheating. As such, there is uncertainty in both cases, but only the UQMC makes the degree of this uncertainty explicit by taking cheating into account. If on the other hand, there is in fact no cheating, the UQMC can capture this as well (with $\hat{γ}$ approximating 0), and the confidence interval of the prevalence estimate range will decrease correspondingly. By way of illustration, if one changes the true cheating prevalence in the above example to $γ = 0.1$ , the estimates resulting from simulation are $\hat{π} = 0.239$ with 95 percent confidence interval ranging from $0.193$ to $0.284$ and $\hat{π} + \hat{γ} = 0.239 + 0.082 = 0.321$ with 95 percent confidence interval ranging from $0.193$ to $0.449$ . As can be seen from this example positing a lower rate of cheating, the 95 percent confidence interval for the estimated range of the carrier proportion is much smaller, namely $0.193$ to $0.449$ .

In addition to estimating the above parameters, the UQMC can test whether a substantial amount of cheating is present. Indeed, Clark and Desharnais (1998) introduced a likelihood ratio test for this purpose in their initial presentation of the CDM. This test utilizes the ratio of the maximum likelihood of a model setting cheating to $γ = 0$ and the maximum likelihood of a model allowing for cheating. It can be applied to the UQMC in a similar manner, where it is formalized as

χ^{2} (1) = 2 \cdot [log L (\hat{π}, \hat{γ}) - log L ({\hat{π}}^{*}, γ = 0)] .

In the above example, this likelihood ratio test supports the hypothesis that cheating is present, with $χ^{2} (1) = 41.119$ , $p < .001$ . Appendix A (which can be found at http://smr.sagepub.com/supplemental/) contains R-code that can be used for applying the calculations to one’s own data.

As is true for all indirect questioning techniques, the sampling variance of the estimates is quite high. Due to the additional estimation of the cheating parameter, this variance becomes even higher than in one-parameter RRMs, such as the original UQM. An optimized choice of p_i and q, and an optimized division of the sample into the two subsamples can minimize this drawback. Appendix B (which can be found at http://smr.sagepub.com/supplemental/) illustrates the influence each of these parameters has on the sum of standard errors and power of the model estimates. In short, more extreme values of p_i and larger values of q make the sum of standard errors smaller and the relative size of the two subsamples within the overall sample has only a small impact, as long as the difference is not too extreme. Thus, a division of the sample into two equal subsamples is desirable. However, minimizing the standard error cannot be the only consideration when choosing the values for p_i and q because in case of values for p_i and q close to 0 or 1, the responses become more indicative of the respondents’ status and thus anonymity protection decreases. Therefore, the applied values must be chosen to represent a compromise between efficiency and anonymity protection. Recommended values would therefore be 0.75 and 0.70 for p ₁ and q, respectively.

Different parameter combinations might be advantageous if the focus of the study is mainly on prevalence estimation or mainly on cheater estimation. In the former case p_i should be more extreme, q should be smaller, and the larger part of the sample should be allocated to the subsample with higher p_i . In the latter case, p_i should be closer to 0.5, q should be higher, and the larger part of the sample should be allocated to the subsample with lower p_i .

The above recommendations are based on the influence that the design parameters have on the standard error and statistical power, together with an intuitive evaluation of the influence that these parameters have on perceived privacy protection. In specific applications, the parameters should be informed by the specific sensitive question at hand and the implementation of the questioning design. In doing so, one can refer to theoretical as well as empirical work on the optimal choice of design parameters in RRMs with respect to efficiency and perceived privacy protection (e.g., Greenberg et al. 1977; Lanke 1975; Leysieffer and Warner 1976; Ljungqvist 1993; Soeken and Macready 1982). An overview on this topic is given by Fox (2016).

Partial Cheating

As explained above, the UQMC utilizes the cheating concept as initially defined in the CDM, where “cheaters” are assumed to always choose the safe option of a “no” response, regardless of the question presented. However, this may be an unduly restrictive assumption, as there might be respondents who would cheat when confronted with the sensitive question but would answer the neutral question truthfully, since they do not feel threatened by this latter question. Allowing for cheating in this broader and probably more realistic sense implies that the original categories (completely honest respondents and complete cheaters) should be extended by the category “partial cheaters” (i.e., cheating only if presented with the sensitive question). In the following, we refer to the original group of cheaters, who always respond “no,” as “complete cheaters.”

Figure 4 depicts how partial cheating affects the probabilities for “yes” and “no” responses. Honest respondents still answer honestly to whichever question they are assigned. Complete cheaters, as before, respond “no” to whichever question they are assigned. In this figure, we add partial cheaters, who answer honestly if assigned to the neutral question, but always respond “no” to the sensitive question, regardless of whether they are carriers of the sensitive attribute. Thus, there is a new branch of the probability tree leading to a “yes” response, $γ_{p} \cdot (1 - p_{i}) \cdot q$ . The resulting total probability for answering “yes” if there is partial cheating can be reduced to

Figure 4.

Probability tree of the unrelated question model—cheating extension including partial cheating. Participants are (a) honest H with probability $1 - γ_{c} - γ_{p}$ , (b) partial cheaters P with probability $γ_{p}$ , or (c) complete cheaters C with probability $γ_{c}$ . All types of participants receive the sensitive question S and the neutral question N with probability p_i and $1 - p_{i}$ , respectively. (a) Honest participants respond “yes” with probability q and “no” with probability $1 - q$ , if instructed to answer the neutral question N. They answer “yes” with probability $∊$ and “no” with probability $1 - ∊$ , if instructed to answer the sensitive question S. (b) Partial cheaters always say “no” if they are instructed to answer the sensitive question S, regardless whether or not they are carriers, but if instructed to answer the neutral question N, they answer honestly by saying “yes” with probability q and “no” with probability $1 - q$ . (c) Complete cheaters always answer “no” regardless of the question that they receive and regardless of whether or not they carry the attribute.

λ_{i} = (1 - γ_{c} - γ_{p}) \cdot ∊ \cdot p_{i} + (1 - p_{i}) \cdot (1 - γ_{c}) \cdot q .

It should be stressed that not all three parameters $γ_{c}$ , $γ_{p}$ , and $∊$ can be estimated from empirical data. In other words, the same value of $π$ can be achieved by an infinite number of combinations of $γ_{p}$ and $∊$ , which would give rise to the same probability $λ_{i}$ . Therefore, this extension can be only partially solved for parameters $π = (1 - γ_{c} - γ_{p}) \cdot ∊$ and $γ_{c}$ . As such, $π$ can be inserted into equation (13) resulting in

λ_{i} = π \cdot p_{i} + (1 - p_{i}) \cdot (1 - γ_{c}) \cdot q .

It is clear that equation (14) is equivalent to equation (5), except that $γ$ is replaced by $γ_{c}$ . Thus, the lower bound for the estimated prevalence of the sensitive attribute is still defined by $π$ when allowing for partial cheaters. However, the upper bound of the estimated prevalence, which was formerly given by $π + γ$ , may no longer be given by $π + γ_{c}$ after allowing for partial cheaters because the remaining category now comprises not only the proportion of honest noncarriers but additionally $γ_{p}$ . Since partial cheaters can be carriers of the attribute, $γ_{p}$ should be added to the possible prevalence range. This results in an increased upper bound of $π + γ_{c} + γ_{p}$ , which cannot be determined because $γ_{p}$ is not identifiable.

For the above numerical example, this would mean that the estimate for the lower bound of the prevalence range would remain at $\hat{π} = 0.188$ . The estimate for the upper bound, however, would potentially exceed $\hat{π} + {\hat{γ}}_{c} = 0.188 + 0.304 = 0.492$ because there could be an additional unknown proportion of partial cheaters. In other words, if one computes the prevalence of the sensitive attribute using the UQMC, which formally assumes only the possibility of complete cheating, the estimate of the lower bound of carrier prevalence is not affected by the presence of partial cheaters, but the upper bound of this range may be underestimated if partial cheaters are present. This consideration should be kept in mind when interpreting the results of a study using the UQMC. In other words, if one wants to address partial cheating within the UQMC framework, the same estimates can be calculated but need to be interpreted differently concerning the upper bound of the prevalence estimate.

It is worth mentioning that the same line of reasoning would apply to the CDM. That is, the possibility of partial cheating would involve a reinterpretation of the parameters estimated by the CDM. Specifically, as before, in the presence of partial cheating, the lower bound of the prevalence would remain at $π$ . However, the upper bound could exceed $π + γ$ if partial cheaters are present.

A Survey Design for Testing the UQMC

A limitation of RRTs in general is that their empirical adequacy cannot be tested because the number of unknown parameters usually equals the number of independent samples, and therefore, there are no degrees of freedom left for testing empirical adequacy. Thus, empirical adequacy must simply be assumed. Fortunately, this drawback can be resolved in the UQMC by varying the prevalence of the neutral attribute q. In the basic UQMC, p ₁ and p ₂ are applied to two independent samples in order to generate two independent equations for $λ_{1}$ and $λ_{2}$ , allowing for two parameters to be identified. However, if q_j is varied orthogonally to p_i , four independent samples can be drawn, each with a unique combination of these design parameters, $(p_{1}, q_{1})$ , $(p_{1}, q_{2})$ , $(p_{2}, q_{1})$ , and $(p_{2}, q_{2})$ . The resulting model with four independent equations for $λ_{i j}$ ( $λ_{11}$ , $λ_{12}$ , $λ_{21}$ , and $λ_{22}$ ) provides two degrees of freedom, allowing for an empirical test of adequacy.

Table 3 illustrates what the setup of the UQMC with four samples could look like, including exemplary estimates ${\hat{λ}}_{i j}$ . Like in the first example, the observed proportions of $“yes”$ responses in this table were simulated with $π = 0.2$ and $γ = 0.3$ . In this case, there is no explicit solution for the estimation of the model parameters. Parameter estimates $\hat{π}$ and $\hat{γ}$ can be obtained by numerical maximum likelihood estimation. Furthermore, the standard errors of the estimated parameters can be numerically evaluated using the observed Fisher information. For the example in Table 3, these estimates are depicted in Table 4. The likelihood ratio test can also be conducted in the four-sample extension. In the numerical example here, the results are in favor of the hypothesis that cheating is present, with $χ^{2} (1) = 55.029$ , $p < .001$ . The exemplary results shown so far are equivalent to those obtainable by the UQMC with two samples. However, the four-sample extension additionally enables testing of the model’s adequacy using Pearson’s $χ^{2}$ goodness-of-fit test. In the UQMC, this is formalized as

χ^{2} (2) = \sum_{i = 1}^{2} \sum_{j = 1}^{2} [\frac{{(o_{y i j} - e_{y i j})}^{2}}{e_{y i j}} + \frac{{(o_{n i j} - e_{n i j})}^{2}}{e_{n i j}}],

Table 3.

Numerical Example Illustrating the Unrelated Question Model—Cheating Extension with Four Samples.

Sample	$n_{i j}$	p_i	q_j	$o_{y i j}$	$o_{n i j}$	${\hat{λ}}_{i j}$
$11$	500	.75	.7	129	371	.258
$12$	500	.75	.3	96	404	.192
$21$	500	.25	.7	204	296	.408
$22$	500	.25	.3	98	402	.196

Note. $n_{i j}$ = size of sample $i j$ ; p_i = probability to be assigned to the sensitive question in samples i; q_j = prevalence of the neutral attribute in samples j; $o_{y i j}$ = observed frequency of “yes” responses in sample $i j$ ; $o_{n i j}$ = observed frequency of “no” responses in sample $i j$ ; ${\hat{λ}}_{i j}$ = proportion of “yes” responses in sample $i j$ .

Table 4.

Numerical Example Illustrating the Unrelated Question Model—Cheating Extension with Four Samples (Continued).

Parameter	Prevalence	Estimate	SE	CI
$π$	.200	.186	.020	[.146, .225]
$γ$	.300	.317	.042	[.234, .400]
$π + γ$	.500	.502	.056	[.393, .612]

Note. SE = standard error of parameter estimate; CI = 95 percent confidence interval of parameter estimate.

where $o_{y i j}$ and $o_{n i j}$ are the observed frequencies of “yes” responses and “no” responses, respectively, in each sample with p_i and q_j . Likewise, $e_{y i j}$ and $e_{n i j}$ are the corresponding expected frequencies. The test supports the fit of the UQMC in the numerical example, $χ^{2} (2) = 0.080$ , $p = .961$ . Appendix C (which can be found at http://smr.sagepub.com/supplemental/) contains R-code for parameter estimation and the goodness-of-fit test that can be applied to one’s own data.

Discussion

The present article extends the UQM to allow it to assess cheating while still ensuring respondents’ anonymity. This extension incorporates the basic idea of the CDM (Clark and Desharnais 1998) while preserving the more psychologically acceptable design of the UQM. Such an extension seems appropriate because there is ample evidence that many respondents cheat by always answering “no” in randomized response surveys (e.g., Elbe and Pitsch 2018; Moshagen et al. 2010; Ostapczuk 2011; Ostapczuk et al. 2009; Pitsch et al. 2007; Schröter et al. 2016), probably because a “no” response reduces the fear of embarrassment or other negative consequences. In particular, when a respondent is administered the UQM, such cheating would greatly diminish the conditional probability of being deemed a carrier of the sensitive attribute. For example, as noted earlier, Bayesian analysis reveals that for the design parameters $p = 0.75$ and $q = 0.50$ , the odds of carrying the sensitive attribute would be 49 times higher in the presence of a “yes” response as opposed to a “no” response, if respondents were to obey the UQM’s instructions. Therefore, disobeying these instructions by cheating with uniform “no” responses is potentially attractive as a self-protecting strategy.

In the present article, we have first introduced an extension of the UQM utilizing the standard assumptions of the CDM—namely the assumption that cheaters will always respond “no” regardless of whether they are directed to the sensitive or to the neutral question. For this extension of the UQM, which we have termed the UQMC, we provide explicit formulae to compute the lower and upper bound of the prevalence estimate range, together with a likelihood ratio test to statistically assess the presence of cheating.

Second, we have discussed in this article the possibility of partial cheating in addition to complete cheating—a perhaps more realistic assumption. Partial cheaters answer honestly if directed to the neutral question but always respond “no” if directed to the sensitive question, even if they are in fact carriers of the sensitive attribute. The parameters of a model including partial cheating are only partially identifiable. Currently, we are not aware of a mathematical or experimental solution for this limitation. However, we have shown that even if partial cheating is disregarded, as in the UQMC, the lower prevalence limit is not affected if partial cheaters are present, although the upper limit may be higher than that estimated by the UQMC if partial cheaters are present. Importantly, such a lower bound provides relevant information like, for example, in a study on the prevalence of doping in elite athletics using the UQM (Ulrich et al. 2018). The UQM estimates of more than 30 percent were clearly much higher than the prevalence estimates from physical doping tests, which indicated a prevalence of about 2 percent at the time (World Anti-Doping Agency 2012). Consequently, even if this only represents a lower bound to the prevalence, the implications are considerable. In addition, the UQMC can account for a very likely type of nonadherence, namely complete cheating. Thus, even if one wants to avoid overconfident conclusions and regards partial cheating, UQMC estimates can have important implications.

Third, we have also shown how the adequacy of the UQMC can be empirically tested. Finally, we have performed power analyses to show that reliable parameter estimates can be obtained even with modest total sample sizes.

The described RRT cheating models assume the presence of “no” cheating for self-protective reasons. Nevertheless, it is at least conceivable that some respondents could cheat with a false “yes” response. For example, a clean athlete might be tempted to cheat with “yes” in order to inflate the prevalence estimate of doping in the hope that this would lead to stricter anti-doping policies (Elbe and Pitsch 2018). In light of this possibility, Feth et al. (2017) extended the CDM to address not only “no” cheating but also “yes” cheating. These authors regard the idea of the CDM in the context of a more general variant of the forced response method, in which there is a forced “no” response in addition to the forced “yes” response. The authors provide an in-depth discussion of the estimation of “yes” and “no” cheating within this framework and also mention the possibility of transferring this idea to the UQM. This CDM extension was recently applied to estimate the prevalence of doping among elite Danish athletes (Elbe and Pitsch 2018). Although the model revealed a high proportion of “no” cheaters, the proportion of “yes” cheaters was virtually nil. A similar conclusion was reached in a recent experimental individual-level validation study (Höglinger and Jann 2018), which examined whether cheating in a dice game could be accurately assessed by several indirect questioning techniques—and, if not, in which direction respondents misreport on their actual behavior. In case of the UQM, these investigators found a substantial prevalence of false-negative responses (i.e., “no” cheating), but not of false-positive responses (i.e., “yes” cheating). These findings are consistent with several lines of evidence indicating that misreporting usually occurs in the socially desirable direction (see Tourangeau and Yan 2007). In the present article, we have extended the standard UQM only for “no” cheating, but future extensions of the UQM could include the possibility of “yes” cheating (including, at least in theory, the possibilities of both complete and partial “yes” cheating). However, assessing for “yes” cheating would likely be useful only in rare situations where social desirability plays a subordinate role, or where there might be a plausible motivation for “yes” cheating.

In the UQMC, the estimation of two parameters requires independent subsamples. A possible limitation of this approach is that it relies on the assumption that these subsamples do not differ with respect to the true parameter values. In case of the cheating parameter, this assumption could be violated because different probabilities of receiving the sensitive question might induce different levels of trust and hence different levels of cheating. There are alternative approaches to estimate nonadherence parameters that do not rely on independent subsamples (e.g., Böckenholt and van der Heijden 2007; Böckenholt, Barlas, and van der Heijden 2009; Cruyff, Böckenholt, and van der Heijden 2016). However, these approaches usually involve the assessment of multiple RRM questions instead of using independent subsamples. Thus, these alternative approaches are not equally suited to the same research questions as approaches using subsamples. When applying the UQMC, this risk of violating the above-mentioned assumption can be minimized by defining the design parameters such that the motivation to cheat would not be expected to strongly differ between subsamples. Additionally, and most crucially, the model test proposed in this article allows one to assess the adequacy of these assumptions.

In this article, we have focused on the UQM and CDM. The Crosswise Model (Yu, Tian, and Tang 2008) provides an alternative to these two models. An advantage of this model is that it does not necessitate a randomization device, nor does it require a “yes”/“no” response. Thus, a response cannot be interpreted as a direct response to the sensitive question, which seems to increase perceived anonymity (Hoffmann et al. 2017). Despite these advantages, this model also has drawbacks. First, the sampling variance of this model’s prevalence estimate is relatively high and thus samples much larger than those typically used in the original UQM are required (Ulrich et al. 2012). Second, the Crosswise Model has been shown to be susceptible to other types of instruction nonadherence, which may distort the prevalence estimate (e.g., Höglinger and Diekmann 2017; Höglinger and Jann 2018).

In summary, the present article attempts to enrich the RRT toolbox by extending one of the most common RRT models, the UQM, to allow for the estimation of cheaters. This extended model is relatively easy to implement in surveys. Therefore, we recommend that cheating and model adequacy should be routinely taken into account in future RRT surveys that will employ the UQM.

Supplemental Material

Supplemental Material, Appendix - Cheater Detection Using the Unrelated Question Model

Supplemental Material, Appendix for Cheater Detection Using the Unrelated Question Model by Fabiola Reiber, Harrison Pope and Rolf Ulrich in Sociological Methods & Research

Supplemental Material

Supplemental Material, onlineSupp - Cheater Detection Using the Unrelated Question Model

Supplemental Material, onlineSupp for Cheater Detection Using the Unrelated Question Model by Fabiola Reiber, Harrison Pope and Rolf Ulrich in Sociological Methods & Research

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Deutsche Forschungsgemeinschaft (DFG), grant 2277, Research Training Group “Statistical Modeling in Psychology” (SMiP).

ORCID iD

Fabiola Reiber

Supplemental Material

Supplemental material for this article is available online.

References

Abernathy

James R.

Greenberg

Bernard G.

Horvitz

Daniel G.

. 1970. “Estimates of Induced Abortion in Urban North Carolina.” Demography 7:19–29.

Böckenholt

Ulf

Heijden

Peter G. M. van der

. 2007. “Item Randomized-Response Models for Measuring Noncompliance: Risk-return Perceptions, Social Influences, and Self-protective Responses.” Psychometrica 72:245–62. doi:10.1007/s11336-005-1495-y.

Böckenholt

Ulf

Barlas

Sema

van der Heijden

P. G. M.

. 2009. “Do Randomized-Response Designs Eliminate Response Biases? An Empirical Study of Non-compliance Behavior.” Journal of Applied Economics 24:377–92. doi:10.1002/jae.1052.

Boruch

Robert F.

1971. “Assuring Confidentiality of Responses in Social Research: A Note on Strategies.” The American Sociologist 6:308–11.

Chaudhuri

Arijit

Christofides

Tasos C.

. 2013. Indirect Questioning in Sample Surveys. Berlin, Germany: Springer.

Clark

S. J.

Desharnais

R. A.

. 1998. “Honest Answers to Embarrassing Questions: Detecting Cheating in the Randomized Response Model.” Psychological Methods 3:160–68.

Coutts

Elisabeth

Jann

Ben

. 2011. “Sensitive Questions in Online Surveys: Experimental Results for the Randomized Response Technique (RRT) and the Unmatched Count Technique (UCT).” Sociological Methods and Research 40:169–93. doi:10.1177/0049124110390768.

Cruyff

Maarten J. L. F.

Böckenholt

Ulf

Heijden

Peter G. M. van der

. 2016. “The Multidimensional Randomized Response Design: Estimating Different Aspects of the Same Sensitive Behavior.” Behavior Research 48:390–99. doi:10.3758/s13428-015-0583-2.

Elbe

Anne-Marie

Pitsch

Werner

. 2018. “Doping Prevalence Among Danish Elite Athletes.” Performance Enhancement and Health 6:28–32. doi:10.1016/j.peh.2018.01.001.

10.

Feth

Sascha

Frenger

Monika

Pitsch

Werner

Schmelzeisen

Patrick

. 2017. Cheater Detection for Randomized Response-Techniques: Derivation, Analyses and Application. Saarbrücken, Germany: Saarland University Press.

11.

Fox

James Alan

. 2016. Randomized Response and Related Methods: Surveying Sensitive Data. Quantitative Applications in the Social Sciences, 2nd ed. Thousand Oaks, CA: Sage.

12.

Haishan

Darroch

Jacqueline E.

Henshaw

Stanley K.

Kolb

Elizabeth

. 1998. “Measuring the Extent of Abortion Underreporting in the 1995 National Survey of Family.” Family Planning Perspectives 30:128–133.

13.

Greenberg

B.G.

Kuebler

R.R.

Abernathy

J.R.

Horvitz

D.G.

. 1977. “Respondent Hazards in the Unrelated Question Randomized Response Model.” Journal of Statistical Planning and Inference 1:53–60. doi:10.1016/0378-3758(77)90005-2.

14.

Greenberg

Bernard G.

Abul-Ela

Abdel-Latif A.

Simmons

Walt R.

Horvitz

Daniel G.

. 1969. “The Unrelated Question Randomized Response Model: Theoretical Framework.” Journal of the American Statistical Association 64:520–39.

15.

Hoffmann

Adrian

Puiseau

Berenike Waubert De

Schmidt

Alexander F.

Musch

Jochen

. 2017. “On the Comprehensibility and Perceived Privacy Protection of Indirect Questioning Techniques.” Behavior Research Methods 49:1470–1483. doi:10.3758/s13428-016-0804-3.

16.

Höglinger

Marc

Diekmann

Andreas

. 2017. “Uncovering a Blind Spot in Sensitive Question Research: False Positives Undermine the Crosswise-Model RRT.” Political Analysis 25:131–37. doi:10.1017/pan.2016.5.

17.

Höglinger

Marc

Jann

Ben

. 2018. “More Is Not Always Better: An Experimental Individual-level Validation of the Randomized Response Technique and the Crosswise Model.” PLoS ONE 13:e0201770. doi:10.1371/journal.pone.0201770.

18.

Höglinger

Marc

Jann

Ben

Diekmann

Andreas

. 2016. “Sensitive Questions in Online Surveys: An Experimental Evaluation of Different Implementations of the Randomized Response Technique and the Crosswise Model.” Survey Research Methods 10:171–87. doi:10.18148/srm/2016.v10i3.6703.

19.

Kirchner

Antje

. 2015. “Validating Sensitive Questions: A Comparison of Survey and Register Data.” Journal of Official Statistics 31:31–59. doi:10.1515/JOS-2015-0002.

20.

Kuk

Anthony Y. C.

1990. “Asking Sensitive Questions Indirectly.” Biometrica 77:436–38.

21.

Lanke

Jan

. 1975. “On the Choice of the Unrelated Question in Simmons’ Version of Randomized Response.” Journal of the American Statistical Association 70:80–83. doi:10.1080/01621459.1975.10480265.

22.

Lensvelt-Mulders

G. J. L. M.

Hox

J. J.

van der Heijden

P. G. M.

Maas

C. J. M.

. 2005. “Meta-analysis of Randomized Response Research: Thirty-five Years of Validation.” Sociological Methods and Research 33:319–48. doi:10.1177/0049124104268664.

23.

Lensvelt-Mulders

Gerty J. L. M.

Boeije

Hennie R.

. 2007. “Evaluating Compliance with a Computer Assisted Randomized Response Technique: A Qualitative Study into the Origins of Lying and Cheating.”Computers in Human Behavior 23:591–608. doi:10.1016/j.chb.2004.11.001.

24.

Leysieffer

Frederick W.

Warner

Stanley L.

. 1976. “Respondent Jeopardy and Optimal Designs in Randomized Response Models.” Journal of the American Statistical Association 71:649–56. doi:10.1080/01621459.1976.10481541.

25.

Ljungqvist

Lars

. 1993. “A Unified Approach to Measures of Privacy in Randomized Response Models: A Utilitarian Perspective.” Journal of the American Statistical Association 88:97–103.

26.

Mangat

N. S.

1994. “An Improved Randomized Response Strategy.” Journal of the Royal Statistical Society. Series B (Methodological) 56:93–95.

27.

Moshagen

Morten

Musch

Jochen

Ostapczuk

Martin

Zhao

Zengmei

. 2010. “Reducing Socially Desirable Responses in Epidemiologic Surveys: An Extension of the Randomized-Response Technique.” Epidemiology 21:379–82. doi:10.1097/EDE.0b013e3181d61dbc.

28.

Ostapczuk

Martin

Moshagen

Morten

Zhao

Zengmei

Musch

Jochen

. 2009. “Assessing Sensitive Attributes Using the Randomized Response Technique: Evidence for the Importance of Response Symmetry.” Journal of Educational and Behavioral Statistics 34:267–87. doi:10.3102/1076998609332747.

29.

Ostapczuk

Martin

. 2011. “Improving Self-report Measures of Medication Non-adherence Using a Cheating Detection Extension of the Randomised-Response-Technique.” Statistical Methods in Medical Research 20:489–503. doi:10.1177/0962280210372843.

30.

Pitsch

Werner

Emrich

Eike

Klein

Markus

. 2007. “Doping in Elite Sports in Germany: Results of a www Survey.” European Journal for Sport and Society 4:89–102. doi:10.1080/16138171.2007.11687797.

31.

Schröter

Hannes

Studzinski

Beatrix

Dietz

Pavel

Ulrich

Rolf

Striegel

Heiko

Simon

Perikles

. 2016. “A Comparison of the Cheater Detection and the Unrelated Question Models: A Randomized Response Survey on Physical and Cognitive Doping in Recreational Triathletes.” PLoS One 11:e0155765. doi:10.1371/journal.pone.0155765.

32.

Soeken

Karen L

Macready

George B.

. 1982. “Respondents’ Perceived Protection When Using Randomized Response.” Psychological Bulletin 92:487–89.

33.

Tourangeau

Rips

L. J.

Rasinski

. 2000. The Psychology of Survey Response. Cambridge, England: Cambridge University Press.

34.

Tourangeau

Roger

Yan

Ting

. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin 133:859–83. doi:10.1037/0033-2909.133.5.859.

35.

Ulrich

Rolf

Schröter

Hannes

Striegel

Heiko

Simon

Perikles

. 2012. “Asking Sensitive Questions: A Statistical Power Analysis of Randomized Response Models.” Psychological Methods 17:623–41. doi:10.1037/a0029314.

36.

Ulrich

Rolf

Pope

Harrison G.

Cléret

Léa

Petróczi

Andrea

Nepusz

Tamás

Schaffer

Jay

Kanayama

Gen

Dawn Comstock

Simon

Perikles

. 2018. “Doping in Two Elite Athletics Competitions Assessed by Randomized-Response Surveys.” Sports Medicine 48:211–19. doi:10.1007/s40279-017-0765-4.

37.

Warner

S. L.

1965. “Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias.” Journal of the American Statistical Association 60:63–66.

38.

Wolter

Felix

Preisendörfer

Peter

. 2013. “Asking Sensitive Questions: An Evaluation of the Randomized Response Technique versus Direct Questioning Using Individual Validation Data.” Sociological Methods and Research 42:321–53. doi:10.1177/0049124113500474.

39.

World Anti-Doping Agency. 2012. 2011 Laboratory Testing Figures. Montreal.

40.

Jun-Wu

Tian

Guo-Liang

Tang

Man-Lai

. 2008. “Two New Models for Survey Sampling with Sensitive Characteristics: Design and Analysis.” Metrika 67:251–63. doi:10.1007/s00184-007-0131-x.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB

0.34 MB