Behaviour versus performance: The veiled commitment of experimental psychology

Individual characteristics of agents

Averaging data enables experimental psychology to bypass discussing individuals and their differences (Billig, 2013). Nonetheless, two approaches can be adopted to distinguish among participants. First, we can use a performance measure to create categories of participants. In a study of the influence of memory on perception, for instance, we divided participants based on how well they performed the mental rotation component of the task (Gozli, Wilson, & Ferber, 2014). In another study examining the relationship between associative memory and attention, we divided participants based on their attention to the first of a series of events (Gozli, Moskowitz, & Pratt, 2014). In these cases, the target of investigation is not individual characteristics, but the relation between different features of task performance. The purpose of dividing the participants is so we could ask how mental rotation affects visual attention, or how attention affects the role of memory. Since performance is used to create categories of participants, the success of this approach consists of understanding the relationship between those very features of performance.

The second approach aims to pinpoint individual differences, determined outside the experimental task, in terms of performance features (e.g., Hommel, Colzato, Scorolli, Borghi, & van den Wildenberg, 2011; Paglieri, Borghi, Colzato, Hommel, & Scorolli, 2013). The motive here is to discover low-level features of performance that may be associated with high-level individual characteristics, or perhaps to “translate” the individual characteristics in terms of performance features. For example, Wilson, Lowe, Ruppel, Pratt, and Ferber (2016) asked whether the Openness personality trait can predict the breadth of visual attention in space. They found attentional inhibition over a wider space in participants who scored higher on Openness. Setting aside the fact that the variance accounted for is too small to warrant the notion of translating Openness to attentional inhibition, and setting aside the conceptual difficulty in relating Openness and inhibition, the researchers might defend their approach because a measurable feature of performance brings a kind of specificity that is lacking in a qualitative description of Openness. However, the measure employed by the authors does not possess clear and agreed-upon meaning. Indeed, Dukewich and Klein (2015) surveyed expert opinions about the attentional measure used by Wilson et al. (2016) and found little agreement on what the measure represents. Consequently, linking the measure to Openness does not add conceptual clarity (see also, Tafreshi, Slaney, & Neufeld, 2016). More fundamentally, the drive behind this approach to personality traits is seriously challenged if we consider the multiply realizable (logically independent) relationship between individual characteristics and performance attributes (Ossorio, 1973, 1985; see also, Angell, 1913, p. 264).

In another series of experiments, Chan, Rajsic, and Pratt (in press) compared participants who volunteered at the beginning of the academic term (“go-getters”) and those who volunteered toward the end of the term (“procrastinators”). The authors found negligible performance differences in a subset of tasks. Aside from reassuring researchers interested in performance, what do the findings say about the individual characteristics of “go-getters” and “procrastinators”? Does the negligible difference in performance correspond to a negligible difference in individual characteristics?

Let us assume that the so-called go-getters and procrastinators are distinguished in how they prioritize among their available goals, with procrastinators favouring short-term goals over long-term goals (Chen & Chang, 2016). Accordingly, placing both groups in a situation where they act on the basis of a fixed goal would be an effective way of minimizing or eliminating the between-group difference. This objection applies more generally to studies that aim to associate individual characteristics with performance attributes within an experiment. To the extent that those characteristics relate to participants’ tendencies in selecting goals, adjusting or abandoning goals, and violating rules, the experimental setup is more likely to conceal than reveal those characteristics (Brandtstädter, 1987; Smedslund, 1991).

In short, an experimental study of performance is an ineffective way of studying individual characteristics, because it either (a) restricts the scope of description to features of performance, (b) fails to bring clarity to already assumed individual characteristics, or (c) systematically suppresses a range of individual characteristics.

Goals

We assume the participants are sufficiently motivated to complete the experiment, although it is regularly acknowledged that this motivation is substantially weaker compared to many everyday situations (Blackman, 2014; Herwig, Beisert, & Prinz, 2013). It should be noted that when experimenters make reference to goals, they tend to refer to the task goals provided by the experimenter, not the goals that bring the participant to the experiment (Hommel, 2015). The distinction is akin to that between wanting to play a game (analogous to participation goal) and wanting to win the game (analogous task goal). The former is simply assumed, which is why we typically bypass the question of motive with regard to participating in experiments. After all, an experiment is informative of those who participate in it. The kind of goal that remains in discussion, therefore, is the one provided by the experimenter and presumed to be shared by all the participants.

Cognitive and sensorimotor capacities

There is typically an overwhelming gap between what participants are capable of doing and what they are asked to do in an experiment. The focus on a small set of factors is a necessary part of the experimental approach, and in and of itself is not a problem if our experiments are parts of a larger theoretical system. Such a system would guide division of labour among researchers and would anticipate their integration. Unfortunately, experimental psychology is currently not grounded in any such system, which means many clusters of empirical findings are isolated fragments with questionable relevance (Allport, 2008; Hommel & Colzato, 2015; Kingstone, Smilek, & Eastwood, 2008; Newell, 1973).

A consequence of this fragmentation is a neglect of what falls outside the immediate scope of the experimenter’s question, because many capacities that enable participating in the experiment can be ignored and lost in the theoretical void. Most relevant to the present argument is the capacity to recognize, follow, and violate rules (Schmidt, Butler, Heinz, & Tomasello, 2016; Tworek & Cimpian, 2016), and the capacity to maintain a social contract with the experimenter. Neglecting these capacities, and that they are exercised in an experimental setting, frequently results in incorrect interpretation of experimental findings (cf. Hacker, 2015; Robinson, 2016).

Possible divergences between how participants and researchers may describe the same behaviour, and the capacities involved, should also be noted. Participants’ own accounts of completing an experiment would include reference to their capacity to distinguish experimental participation from other types of behaviour. The capacity to distinguish types of behaviour is characteristic of deliberate and cognizant behaviour (Ossorio, 2013, pp. 57–58). Furthermore, participants’ own accounts would include the fact that, in an experimental setting, behaving as a participant is itself a goal, regardless of any other outcome. Including behaviour, recursively, as a goal attribute is characteristic of social practices, for example, pronouncement of marriage, making a promise, etc. (Ossorio, 2013, p. 59). What research participants do within an experiment can, therefore, be described as deliberate and cognizant (knowingly behaving as a research participant and according to a set of instructions) social practice (fulfilling a social contract through following a set of instructions), even though such descriptions must be suppressed when the focus is on performance. Such divergence in description highlights the neglected capacities of experimental participants.

Meaning

To ask about the meaning of a behaviour is to ask what the behaviour signifies within a sociocultural context (Bergner, 2010; Ossorio, 2013). What does a person do by participating in an experiment? A reasonable response is that the person enters and fulfills a social contract. Within this contract, the experimenter determines what constitutes appropriate conduct within the experimental setting, and the participant either follows the instructions or is excluded from analysis. Participants’ conformity orients the researchers’ attention toward features of performance that are of interest in the given research project (e.g., reaction time, accuracy, movement trajectory, probability of a given response, etc.), and away from the experiment as social contract (Blackman, 2014; Fitzgerald & Callard, 2015; Robinson, 2016).

Relying on participants who conform with the rules of the experiment is an effective means of concealing the social contract, the rules, and the capacity to resist and violate those rules (De Genova, 2009). As a result, experimental literature includes claims that social factors are absent in experimental research (Eysenck & Keane, 2015, p. 302) or that a process is being investigated in a “non-social” experimental situation (Wirth, Pfister, Foerster, Huestegge, & Kunde, 2016, p. 850). Our focus is drawn away from the social aspect of the experiment, because participants’ conformity renders this aspect unproblematic. Similarly, the constancy of rules for what can be included (and what must be excluded) leads to neglecting the very existence of those rules. What supports this neglect is a form of abstraction and suspension, which results from bringing one attribute (performance) to the foreground and relegating other attributes to the background (Morley, 2010; Stenner, 2008).

The question of meaning is relevant to the question of biological reductionism (e.g., Bergner, 2016; Martin, 2010; Pickersgill, Cunningham-Burley, & Martin, 2011; Slife, Burchfield, & Hedges, 2010). If we restrict our description to performance, it appears much easier to anticipate a biological redescription. Consider the difference between being in a room alone and being in a room with another person. Experiments have revealed how the presence of another person changes the pattern of our eye movements (Laidlaw, Foulsham, Kuhn, & Kingstone, 2011), our hand movements (Quesque, Lewkowicz, Delevoye-Turrell, & Coello, 2013), and our memory for transient visual events (Constable, Pratt, Gozli, & Welsh, 2015). These descriptions orient us away from the relational situation in which the presence of another person has meaning (Hibberd, 2014). Their openness to biological redescription is because they are already restricted to one human body.

In summary, an experimental setup does not target non-performative attributes of behaviour (individual characteristics, goals, capacities, and meaning). This claim does not mean performative and non-performative attributes of behaviour are causally independent. It means that when we rely on participants’ conformity to experimental rules, which requires keeping some attributes constant (goal, meaning), while neglecting others (e.g., individual characteristics, certain cognitive/sensorimotor capacities), we undermine our ability to investigate the relation among the attributes.

Rules are inessential

A reader familiar with experimental literature might think of counter-examples in which participants are apparently not conforming to any rules. For example, studies of imitative movement (e.g., Genschow & Brass, 2015), studies of the role of the presence of other persons in the situation (e.g., Nasiopoulos, Risko, Foulsham, & Kingstone, 2015; Risko & Kingstone, 2011), and studies of the subliminal sources of behaviour (e.g., Bargh & Chartrand, 1999) may require nothing beyond waiting in a room, or at most looking at some apparently irrelevant display. Although researchers in these studies have the opportunity to adopt a more descriptive approach, the participants can still violate some inclusion rule, for example, by closing their eyes, making a phone call, or walking around in the room.

Not conforming to the inclusion rules means behaving in such a way that prevents hypothesis testing. An example can demonstrate how failing the inclusion rule does not require explicit awareness of the rule. In a recent study of learning social norms, 3-year-old children first observed an adult who used a tool in a particular manner. The children were then given the chance to impose the adult’s manner of tool-use on a puppet that used the tool differently (Schmidt et al., 2016). Because interacting with the puppet was necessary for testing the hypothesis, the researchers excluded any child who did not interact with the puppet during an initial phase. In this example, interacting with the puppet was an inclusion rule, even though the children were not aware of this rule. For an experiment to be a valid test, in general, it is required that participants engage with the experimental setup in such a way that enables hypothesis testing, regardless of their explicit awareness of this requirement.

Experimental study of rule violation

I anticipate an objection based on a recent series of studies on rule following and rule violation (Pfister, Wirth, Schwarz, Steinhauser, & Kunde, 2016; Wirth et al., 2016). The possibility of studying rule violation experimentally would be seriously damaging to my argument. If participants can freely adjust and abandon goals without compromising their role as participants, if participants can exert such an authority over the rules of the experiment, then we would have evidence that experimental researchers can target attributes other than performance. It is, therefore, necessary to examine the structure of these experiments.

The experiments involved instructing participants in a rule-governed movement task, consisting of a starting location on the screen (bottom-middle) and two target locations (top-left and top-right). Participants were required to move an avatar to one of the two target locations depending on the shape of the avatar (pawn vs. king). The rule could, for example, require taking pawns and kings, respectively, to the left and right target locations. On a subset of trials, prior to seeing the avatar, participants received a signal that instructed them to violate the task rule. The researchers carefully measured several performance features, and found that performing “rule-violation” trials was less efficient and contained reliable trajectory deviation toward rule conformity. The authors stated that we are beginning to understand the “signature and underlying mechanism of deliberate rule violations” (Wirth et al., 2016, p. 838).

From what perspective is it possible to describe participants’ behaviour as rule violation? From the experimenters’ perspective, participants were following the instructions. Indeed, it is helpful to note that when participants did not follow instructions, either on “rule-following” or “rule-violation” trials, their performance was excluded from analysis (Pfister et al., 2016; Wirth et al., 2016). From the participants’ perspective, the experimenters’ instructions were also being followed. They were given a set of rules that were supposed to be violated on a subset of trials. Participants’ behaviour is, therefore, not representative of rule violation, except perhaps from the perspective of an observer who happens to hear only half of the instructions. Characterizing what the participants do as non-conformity is possible after neglecting the conformity that enables the experiment. Therefore, these experiments do not deviate from, but exemplify, the kind of conformity to experimental rules discussed above.

“It can be done!”

I anticipate another objection, in response to my review of Pfister et al.’s (2016) studies. One could argue that I have, at best, demonstrated that violating experimental rules has not yet been included within an experimental design, but that does not mean rule violation, as a matter of principle, has no place in experimental psychology. Could one envision a variant of the aforementioned experiment, in which participants could violate the rules of the experiment? Of course, we would still need to know whether participants are following or violating the rule, in order to distinguish violations from mistakes. Consider the following variant of the task. At the beginning of each trial, instead of instructing the participant to follow/violate the task rule, we let participants decide whether they will follow/violate the task rule. They would record their decision using one button press, and then execute their decision.

In this new variant of the task, there are still (at least) three ways that we would be forced to exclude participants for not meeting inclusion criteria: A participant will be excluded if they do not maintain consistency between their first response (prior intention to follow/violate) and their second response (follow/violate); a participant will be excluded if they always follow the task rule, because that would not enable the experimenter to compare their performance across the two kinds of trials; and, for the same reason, a participant will be excluded if they always violate the task rule. These necessary exclusions reveal the persistence of inclusion rules in the modified version of the experiment.

Refusal-to-participate as a response

In well-known studies of obedience to authority, the experimenter monitors participants’ decision to terminate the session (Milgram, 1963; for a review see, Blass, 1999; for a critical analysis, see Stam, Lubek, & Radtke, 1998). That is, terminating the experiment is brought to the attention of the researchers as an available option and, indeed, as the expected alternative in a two-choice response (follow vs. refuse). Consequently, it could be argued that these studies represent genuine investigation of non-conformity.

Once again, from what perspective does terminating the session count as violating the rules of the experiment? Milgram set up his design with the explicit intention to elicit refusal, at least in some participants. A refusal, therefore, fits within the pre-specified structure of the experiment as one of the two possible outcomes, i.e., refusing and following instructions both conform to the two-choice structure of the experimental design (Brandtstädter, 1987). From the participants’ perspectives, a refusal appears as genuine violation only because they are misinformed about the experiment. Their experience is akin to someone who is presented with two keys, requested to only press key #1 and never key #2. Throughout the session, the participant presses key#1 and every time receives negative feedback (e.g., “MISTAKE!”). He turns to the experimenter, who deceitfully tells him “it is essential that you continue to press only key #1.” He finally presses key #2 and is then informed that pressing key #2 was always a genuine choice (and not a violation).

In describing the studies of Pfister et al. (2016), I argued that only someone unaware of a portion of the instructions may consider participants’ behaviour as rule violation. The type of design used by Milgram (1963) makes it possible for participants to consider ending the session as genuine non-conformity, precisely by keeping them uninformed about the design. In addition, the choices are restricted to “conformity” and “non-conformity,” which stand in sharp distinction. The instructions ensure that no middle ground is available, for example, negotiating alternative ways of continuing the experiment. This is in part because the experimenter must easily and clearly identify instances of non-conformity (subtle forms of non-conformity might go unnoticed). Aside from its unpleasant and ethically questionable details, the study is representative of the experimental method in its restrictions. One either accepts the given task goal and the rules for fulfilling the goal or altogether refuses to participate.

The argument is trivially true

Another objection to my argument is that it is trivially true. “Yes,” the critic will say, “we all know that experimental psychology requires conformity in order to render performance interpretable. We all know that sometimes non-experimental methods are more appropriate.” Even if my position faces no disagreement, it nevertheless faces a widespread neglect. As a member of the community of experimental psychologists, it is clear to me that the nature and consequences of this neglect are far from arbitrary. By evading the question of the scope and limits of experimental psychology, we can implicitly assume that the method can eventually—given enough time and effort—address all psychological questions. The purview of each experiment is small, the experimenter says, but give us enough time and we will cover the entire field.

Evading the question, furthermore, keeps unchallenged the assumption that ultimately the language that explains behaviour will consist of references to movements, postures, neural functions, stimuli, or—in other words—the language of performance (Hacker, 2015; Robinson, 2016). The neglect, therefore, serves a meta-scientific theory. Given that explicit defenses of this theory have not fared well (e.g., Marken, 1988; Robinson, 1985, Chapter 4; Sampson, 1981; Smedslund, 2015, 2016; Wheeler, 1925), evading the issue is an effective strategy to preserve it. By evading the question, experimental psychology can continue to make promises that it cannot fulfill (Billig, 2013).

Neglecting the scope and limits of experimental psychology is also a neglect of the gap between operational definitions and everyday use of concepts. If an experimenter asks you to violate a task rule, there is no limit to how creatively you could behave. If, however, the experimenter says, “by violation, I mean moving your avatar to point A, instead of point B,” then you recognize that the meaning of the word “violation” is narrowly defined in the context of the experiment, which in turn narrows the interpretation of the findings. Another experimenter interested in volition will present you with a button-press task and the instruction, “You are free to press the button when you want.” We should note, however, the many decisions already made prior to the so-called free act — whether or not to participate, which button to press, and which hand/finger to use (Hommel, 2017, in press). The operational definition of volition has to be taken into account in interpreting findings. The decision to begin a game of chess cannot be examined by analyzing a game, any game, of chess. The decision precedes the game. Similarly, in an experiment, the participant’s decision to begin performing the task stands outside the experimental task. The meaning of experimental participation cannot be found by looking at the pattern of performance (Teo, 2006, Chapter 4).

Awareness of how experimental psychology relies on human performance, and the preconditions that enable the description of performance to be of any use (fixed goal, restriction of possible expressions, determination of how an expression is performed, and conformity), can increase awareness of the method and its limits. It can enable more accurate interpretation of experimental findings, prevent misuse of concepts, and orient attention toward meta-theoretical questions.

Generalizability

Experimental psychologists have discovered that neglecting the context in which performance occurs, i.e., equating behaviour with performance, is harmful even to the study of performance. This is due to neglecting the possibility that the findings might be by-products of the rules and structure of the experiment (e.g., Allport, 2008; Besner, Stolz, & Boutilier, 1997; Hommel, 2000, 2010, 2015; Kingstone et al., 2008; Proctor & Cho, 2006; Proctor & Xiong, 2015). For instance, the conclusion that the visual system is always sensitive to a unique item in a scene often comes from experiments that implicitly prepare the participants to watch out for unique items (Bacon & Egeth, 1994; Li & Yeh, 2007). The conclusion that processing object location is automatic comes from experiments that require some form of localization (Hommel, 2011). Similarly, the conclusion that semantic processing of words is automatic comes from experiments in which participants engage in some form of categorization (e.g., Besner et al., 1997; Hommel, 2000, 2004). In short, experimental findings are, in part, determined by the rules and structure of the experiment (Brandtstädter, 1987), which restricts their generalizability.

Input–output and closed-loop systems

An experimental setup involves similar treatment of participants, not only by giving them the same set of possible responses, but also by presenting them with the same set of events (stimuli) meant to trigger those responses. This setup frames behaviour as the final step (“output”) in a chain of events that begins with the presentation of a stimulus. Critics of this approach do not refute the fact that people are capable of responding to events, nor do they refute that the responses can follow predictable patterns. What they refute is, first, the primacy of stimulus in the causal sequence (Hommel, 2000, in press) and, second, the assumption that behaviour is part of an open causal sequence (with a beginning and an end), instead of a closed sequence that includes feedback (Marken, 2009). The fundamental difference between input–output and closed-loop systems is that the latter enables the system to control and maintain a desired state.

Consider two friends in conversation who are walking together on their college campus. An input–output perspective would ask how each person understands and responds to their friend’s utterances, why one laughs at another’s joke, or how the appearance of an obstacle in their way triggers a change in their walking direction. In addition to these questions, a closed-loop perspective asks what is continually controlled by their behaviour (Marken, 2009). We may notice, for instance, that the two friends speak more loudly as their paths temporarily diverge around an obstacle that increases their distance or as they enter a noisy area. We can claim, roughly speaking, that being audible is a state that the two friends try to maintain, despite their increased distance or environmental noise. Each friend regularly relies on perceptible signs of being heard by the other, particularly when a disturbance is introduced. In this sense, the stimulus (e.g., a perceptible sign that one is heard) is better characterized as the end, and not the starting point, of behaviour (Gozli & Ansorge, 2016; Hommel, in press).

How is the closed-loop approach relevant to the present discussion? The approach is absolutely key in understanding the social nature of an experiment. The possibility that participants’ behaviour might be driven, in part, by the goal of controlling and maintaining certain states in the experimenter (e.g., perceptible signs of positive affect) or their relation with an experimenter (e.g., perceptible signs of a cooperative relation) escapes the scope of the input–output perspective. Yet, we know that factors such as the presence of the experimenter, interactions with the experimenter, and the continuous social feedback, can be critical in the formation of experimental findings (e.g., Blass, 1999; Fetsinger & Carlsmith, 1959). The notion of control is also connected to Stam et al.’s (1998) analysis of the context in which experiments are introduced, particularly to the new members of the discipline. Just as participants’ conformity to the rules of an experiment depends on their understanding of the experiment, larger-scale conformity in groups and institutions can be controlled via perpetuating a particular understanding of the experiment.

Equality and discovery

The issue of inequality between researchers and research participants has also been previously raised (e.g., Orne, 1962). Among others, Amedeo Giorgi (2014) emphasized that, although researchers and participants differ in their roles, whatever we say about the researcher has to also apply to the participant “in terms of human characteristics and possibilities” (p. 235). How do we determine and protect such an equality? Based on the present argument, essential to the equality is the authority over, and awareness of, the rules of the shared situation (Stam et al., 1998). In the experimental situation, we try to preserve equality by reminding participants that they can end the experiment at any time. Exercising that right, however, terminates the researcher’s opportunity to learn from the participant. What Giorgi has in mind, I believe, is to ensure that participants’ goals and interpretations could diverge from those of the researcher without necessarily ending their relationship. This equality implies that the researcher could study behaviour without pre-determining the possible outcomes. It implies that the participant’s behaviour could have meanings or motives or demonstrate capacities that are not initially understood by the researcher.

Interesting psychological questions arise when people deviate from implicitly held rules of behaviour, when multiple rules clash, or when the existing rules are silent about how one should behave (Giorgi, 2013; see also, De Genova, 2009). The behaviour of the chess player who pours hot coffee on the board, and the behaviour of the student who draws caricatures of the professor on their exam sheet may not be of interest to the chess analyst or the person grading the exam, but they are of interest to a psychologist. By deviating from the existing rules, they pose questions about attributes of behaviour other than performance (individual characteristics, goal, capacities, and meaning). The ability to address an unexpected attribute means the researcher is not the sole authority, who restricts the domain of study to pre-specified expressions performed in pre-specified ways. This important form of equality is absent in an experimental setting.