It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks

Study 1: Trap Questions Improve Performance on the Cognitive Reflection Test (CRT)

The CRT (Frederick, 2005) is an individual difference measure based on three numeric reasoning problems. For each problem, an intuitively appealing answer is likely to come to mind quickly but is incorrect. Instead, arriving at the correct answer requires applying the appropriate formula, performing calculations, and checking one’s solution. The CRT is assumed to be a measure of a person’s stable disposition to engage in reflective, systematic thought, predicting performance on heuristics-and-biases tasks (Toplak, West, & Stanovich, 2011), delay of gratification, and Wunderlic IQ scores (Frederick, 2005). Furthermore, numerous studies have shown that task order does not affect CRT scores. For instance, CRT scores are similar regardless of whether the CRT comes before or after responding to various moral dilemmas (Paxton, Bruni, & Greene, 2014; Paxton, Ungar, & Greene, 2012; Pinillos, Smith, Nair, Marchetto, & Mun, 2011). Thus, it is often assumed that contextual cues in a survey have no impact on CRT scores. If the IMC acts as a mere measure of attention, then CRT scores should be similar regardless of whether the IMC occurs before or after the CRT.

However, note that correctly answering CRT problems requires checking and overriding an intuitive answer, much as passing an IMC requires checking the instructions and overriding the lure response. If the IMC teaches participants that there is more than meets the eye to the questions, then varying the order of the IMC and the CRT should affect participants’ CRT scores. We therefore predicted that participants who receive an IMC question prior to the CRT will score better on the CRT than participants who complete the tasks in the reverse order. If so, this effect would be especially powerful considering that order effects on CRT scores are unobserved in the literature on the CRT (Paxton et al., 2014; Paxton et al., 2012; Pinillos et al., 2011).

Results and Discussion

Three hundred fifty participants (92%) responded correctly to the IMC on their first try, while 30 participants did not. This IMC pass rate is consistent with those of recent research on MTurk (Nauts, Langner, Huijsmans, Vonk, & Wigboldus, 2014; Wolf et al., 2014; for related research, see Hauser & Schwarz, in press). Passing the IMC was not affected by IMC presentation order, χ² (1, N = 380) = 0.98, p = .321. Therefore, we limited our analysis to those 350 participants who passed the IMC because the group of participants who responded incorrectly was not large enough to draw firm conclusions. ¹ Furthermore, studies that use IMCs typically include only the participants who respond correctly to the IMC, and answering the IMC should only affect participants who recognize that the question is intended to trick inattentive participants (i.e., those who answered the IMC correctly). Thus, this group is the most theoretically relevant population for the current analyses.

Overall, CRT scores were comparable with those observed in prior research (West, Meserve, & Stanovich, 2012, Study 2). More important, IMC presentation order significantly affected CRT scores as shown in the first row of Table 1. Answering the IMC first raised CRT scores relative to answering the CRT first. This effect was neither moderated by participants having seen the IMC in prior MTurk studies (two-way interaction of IMC order and prior exposure to the IMC: F< 1) nor moderated by participants having seen the CRT in prior MTurk studies (two-way interaction of IMC order and prior exposure to the CRT: F< 1). ²

OPEN IN VIEWER

Table 1.

IMC Presentation Order on Mean Correct CRT Score, Mean Intuitive CRT Score, and Mean Time Spent on CRT.

	IMC presentation order
Measure	IMC first	IMC last	F	d
Correct CRT score	1.76 (1.16) [1.59, 1.93]	1.33 (1.10) [1.17, 1.49]	12.64***	.38
Intuitive CRT score	1.03 (1.10) [0.87, 1.19]	1.37 (1.06) [1.21, 1.53]	8.59**	.31
Time spent (in log-transformed seconds)	3.65 (0.84) [3.52, 3.78]	3.48 (0.77) [3.36, 3.59]	3.99*	.21

Note. Standard deviations in parentheses; 95% confidence intervals in brackets. IMC = instructional manipulation check; CRT = Cognitive Reflection Test.

*

p< .05. **p< .01. ***p< .001.

IMC presentation order also significantly affected intuitive responses to the CRT problems and the number of seconds spent answering the CRT questions. As shown in the second and third rows of Table 1, answering the IMC first lowered the number of intuitive responses and increased the number of seconds (log-transformed) participants spent completing the problems (relative to answering the CRT first).

In sum, prior exposure to an IMC increased CRT scores and increased the time spent working on the CRT. These effects were not moderated by familiarity with the CRT or the IMC. As the CRT is an individual difference measure with links to IQ (Frederick, 2005) and which is rarely affected by task order (Paxton et al., 2014; Paxton et al., 2012; Pinillos et al., 2011), IMCs appear to be powerful manipulations of attention and systematic thinking, highlighting that they operate as interventions rather than mere measures.

Study 2: Trap Questions Improve Performance in a Probabilistic Reasoning Task

To further test the impact of IMCs on cognitive performance, Study 2 employs a different task, namely, a probabilistic reasoning task (described below) on which performance is impaired by denominator neglect (Toplak et al., 2011). As observed in earlier research, probabilistic reasoning is often improved by systematic thinking (e.g., Denes-Raj & Epstein, 1994; Toplak et al., 2011). Therefore, if exposure to an IMC promotes systematic thinking, it should improve performance on a probabilistic reasoning task.

General Discussion

Our findings converge to show that an IMC is more than merely a measure of attention—It is an intervention that increases systematic thinking on subsequent tricky-seeming tasks. When highly accessible and intuitively appealing answers are incorrect, a trap question can improve performance by prompting participants to reconsider their spontaneous solution and adopt a more systematic reasoning strategy. In Study 1, this resulted in improved performance on the CRT (Frederick, 2005), an individual difference measure that is associated with rational problem solving. In Study 2, this resulted in increased optimal responses on a probabilistic reasoning task also associated with rational problem solving (Denes-Raj & Epstein, 1994; Toplak et al., 2011). In combination, these findings show that an IMC can signal a context in which questions may contain “more than meets the eye,” a conclusion that significantly alters participant’s reasoning strategies compared with participants not exposed to an IMC.

These studies have important implications for the use of IMCs in online tasks. IMCs are typically conceptualized as measures, not interventions. However, as demonstrated here, an IMC prompts more systematic reasoning on subsequent tricky-seeming tasks. Accordingly, IMC administration could have moderating effects on a study’s results. For instance, systematic thinking could attenuate or eliminate effects expected to be driven by intuitive reasoning and could amplify effects expected to be driven by reflective reasoning. One should therefore exercise caution in IMC use, as IMCs could eliminate effects one might otherwise observe or create effects that might otherwise not emerge. We assume that these complications are most likely under conditions where intuitive and systematic reasoning can lead to different answers, but future research may fruitfully explore whether the influence of IMCs extends beyond this type of task. In addition, different types of IMCs may cause different effects. For instance, more novel or more difficult IMCs may more strongly prompt systematic thinking, whereas commonplace IMCs may have little influence. The observed effects may also depend upon whether participants have the knowledge or skills required to complete the systematic thinking task. For example, an IMC may send the message to be on the lookout for trick questions, but if a participant does not have the mathematical skills to know that their intuitive answer of 10 cents to the bat ball problem is incorrect, then a prior IMC may do little to increase their accuracy on such tasks. These are interesting questions, which we will leave for future research to explore.

Our results implore a seemingly obvious suggestion: If IMC use is necessary, place the IMC after all crucial measures and manipulations in a study. Reserving IMCs for the end of a survey at least guarantees that the IMC will not affect how participants interpret and answer other questions. However, placing IMCs at the end does have drawbacks. Inattentiveness may not be merely a trait, measureable with only one IMC, but instead may uniquely ebb and flow for each participant throughout the tasks (Berinsky et al., 2014). Further, participants often become fatigued as they progress through long surveys, and survey fatigue predicts satisficing behaviors (Krosnick, 1999), which are expected to also positively predict IMC failure (Oppenheimer et al., 2009). For these reasons, an IMC placed at the end of a survey may not accurately assess a participant’s attentiveness to the previously administered questions and thus may misidentify who did or did not pay attention at an earlier point.