Electrophysiological Examination of Embodiment in Vision and Action

Abstract

A wealth of behavioral data has shown that the visual properties of objects automatically potentiate motor actions linked with them, but how deeply are these affordances embedded in visual processing? In the study reported here, we used electrophysiological measures to examine the time course of affordance resulting from the leftward or rightward orientation of the handles of common objects. Participants were asked to categorize those objects using a left- or right-handed motor response. Lateralized readiness potentials showed rapid motor preparation in the hand congruent with the affordance provided by the object only 100 to 200 ms after stimulus presentation and up to 400 ms before the actual response. Examination of event-related potentials also revealed an effect of handle orientation and response-hand congruency on the visual P1 and N1 components. Both of these results suggest that activity in the early sensory pathways is modulated by the action associations of objects and the intentions of the viewer.

Keywords

cognitive neuroscience electrophysiology vision motor processes

Theories of embodied cognition (Wilson, 2002) posit an intimate link between perception and action, in which the visual system provides direct information on the behavioral possibilities afforded by objects. Data said to demonstrate the affordance provided by visual objects has been reported in studies using a stimulus-response compatibility as a paradigm (e.g., Michaels, 1988; Thorpe, Fize, & Marlot, 1996; Tucker & Ellis, 1998), in which participants classify objects faster when their motor response is congruent with that afforded by the judged object than when it is incongruent. For example, participants would classify an orange faster than a grape as a type of fruit if their response required a power grip, the pattern being reversed when performing a precision grip. These compatibility effects have been shown for the hand used (Tucker & Ellis, 1998), wrist rotation, and hand shape (Ellis & Tucker, 2000).

The primary question in the study reported here is at which level of processing are these affordances established? Does visual processing proceed independently, with links to the motor system occurring only at the point of response selection? Or do the visual and motor responses of the brain form a far more integrated and dynamic system, with exchanges at very early stages, as is commonly assumed in accounts of the behavioral data on the affordances provided by visual objects (Ellis, 2009)?

In the study reported here, we examined temporal characteristics of object-response compatibility by recording electrical brain activity during a standard stimulus-response-compatibility experiment. In this experiment, participants were required to use their left and right hands to make category judgments about pictures of household objects with left- or right-facing handles. In this case, affordance would lead to facilitation when the hand with which participants responded was congruent with the direction the handle of the object was facing, and affordance would lead to inhibition when the handle orientation and the response hand were incongruent. Measurement of the lateralized readiness potential (LRP) during the task provided an indication of when this affordance takes place.

The LRP is measured from electrodes situated over the primary motor cortex, and it reflects the lateralization that occurs as a result of left- or right-handed response preparation (e.g., Coles, 1989). The LRP provides a real-time measure of the transmission of information from perceptual to motor processes, and thus an estimate of when particular information is used in response preparation. If action selection is driven by the semantics of participants’ experience with an object (Chao & Martin, 2000; Grafton, Fadiga, Arbib, & Rizzolatti, 1997), we would expect that the lateralized effect of affordance would be reflected in the LRP around the same time as the response selection to the semantic categorization task. However, if action is integrated with earlier sensory activation (Handy, Grafton, Shroff, Ketay, & Gazzaniga, 2003), then response selection should take place within the time frame of early visual perceptual processes, usually before 200 ms of stimulus onset, which is well before response activation.

If this were the case, we would also expect that the action intentions of participants would be reflected in modulations of event-related-potential (ERP) components related to early visual processing, such as the posterior P1 and N1 components, which reflect activity from low-level extrastriate visual cortical areas. These components are most often associated with studies of spatial-attention cuing (Handy & Mangun, 2000; Luck & Hillyard, 1995; Mangun & Hillyard, 1988), but they have also been shown to be sensitive to nonspatial and potentially object-based features, such as orientation (Karayanidis & Michie, 1997; O’Donnell, Swearer, Smith, Hokama, & McCarley, 1997). Therefore, we would fully expect that these components would be modulated by the visual cue provided by the orientation of the object handles. However, if embodiment really is deeply embedded in early visual processing, we would also expect that the P1 and N1 components would in turn be modulated by the motor intentions of the participants.

Method

Participants

Seventy volunteers (26 male, 44 female; mean age = 20 years, SD = 1.60) from the University of Plymouth were paid £6 for participation in this experiment. All reported being right-handed and having normal or corrected-to-normal vision. The data of 5 participants were removed from analyses because of excessive electroencephalogram (EEG) artifacts (> 30% of segments contaminated in any one condition).

Stimuli

Eighty-four color pictures of 42 common household objects were used as stimuli in this experiment. Each object was photographed with its handle facing leftward and with its handle facing rightward. Half of the objects fit the semantic category of kitchen utensil, and half fit the category of tool (see Fig. 1a for examples).

Fig. 1.

Example stimuli and results illustrating object-based affordances. Stimuli (a) consisted of photographs of two categories of objects (tools and kitchen utensils) with their handles facing either left or right. Stimulus-locked lateralized readiness potentials (b) were calculated separately for the congruent condition (in which the direction the object’s handle was facing and the response hand of the participant were congruent) and the incongruent condition (in which the direction the object’s handle was facing and the response hand of the participant were incongruent). Event-related potentials (ERPs; c) pooled from the P7, P8, O1, and O2 electrodes show the posterior-occipital P1 and N1 components. Waveforms for the congruent and incongruent conditions are illustrated separately. The bar that shows the ERP magnitude is positioned at stimulus onset.

Procedure

Participants were asked to make a speeded judgment to categorize each stimulus as a kitchen utensil or a tool, which they did by pressing a button with their left or right hand. Participants were given no instructions regarding the orientation of the objects’ handles. Each trial began with a fixation cross presented in the center of the screen for 1,000 to 1,200 ms. This was immediately followed by the stimulus image, which remained on the screen until the participant made a response but not longer than 2,000 ms. Finally, between 400 and 600 ms after the offset of the stimulus image, a rest symbol was presented for 1,400 ms. Participants were asked to blink or move their eyes only when they saw the rest symbol and not during the other parts of the trial.

Each participant was presented with six blocks of stimuli, with each block consisting of the entire stimulus set of 84 pictures presented in a different random order. There were two trials types: On congruent trials, the response hand and the orientation of the stimulus’s handle were the same; on incongruent trials, the response hand and the orientation of the stimulus’s handle were opposite. In the first three blocks, the participants responded with one mapping of response hand to category, switching to the reverse mapping for the remaining blocks. The mapping order was counterbalanced across participants. To familiarize the participant with the procedure, we presented a short practice block before the main experiment.

EEG recording

EEG data were collected from 30 actively amplified Ag/AgCl electrodes (actiCAP, Brain Products, Gilching, Germany) mounted on an elastic cap. Electrodes were referenced to the left mastoid and re-referenced off-line to the average of left and right mastoid activity. Vertical eye movement was monitored by a suborbital electrode, and horizontal eye movement was monitored using left and right electrodes on the external canthi. Electrode impedances were kept below 20 kΩ. EEGs were amplified using a BrainAmp MR Plus amplifier (Brain Products), continuously sampled at 500 Hz, and filtered off-line with a band-pass filter from 0.1 to 40 Hz. ERPs were computed by averaging artifact-free EEGs associated with correct behavioral responses (> 93% of trials). These EEGs were time-locked to 200 ms before the stimulus onset to 800 ms afterward, and then they were baseline-corrected using the period prior to stimulus presentation.

Results

Behavioral data

Reaction times for category judgments were calculated for the 93.62% of responses that were correct, with further exclusion of responses outside of the time window between 200 and 1,200 ms after stimulus onset or 2.5 standard deviations of the mean cell value. Repeated measures analyses of variance (ANOVAs) with between-participants factors of object category (kitchen utensil vs. tool) and response-hand/handle- orientation congruency (congruent vs. incongruent) revealed a significant main effect of congruency, F(1, 64) = 6.53, p = .012, plus an interaction between category and congruency, F(1, 64) = 16.81, p < .001. Participants showed faster reaction times when the responding hand was congruent with the orientation of the object handle (542 ms) than when the response hand and handle orientation were incongruent (547 ms). The Category × Congruency interaction revealed a significant effect of congruency for the objects in the tools group (i.e., responses in congruent trials were 10.31 ms faster than responses in incongruent trials for the tools group), F(1, 64) = 20.38, p < .001, but not for the objects in the kitchen-utensils group (F < 1).

LRP data

Separate LRPs for congruent and incongruent trials of both object categories were calculated using the Coles (1989) derivation. That is, the LRP for congruent trials was calculated by subtracting the C3 potential from the C4 potential for objects with left-facing handles requiring a left-handed response, subtracting the C4 potential from the C3 potential for objects with right-facing handles requiring a right-handed response, and averaging these two values. The LRP for incongruent trials was calculated by subtracting the C3 potential from the C4 potential for objects with right-facing handles requiring a left-handed response, subtracting the C4 potential from the C3 potential for objects with left-facing handles requiring a right-handed response, and averaging these two values. The resulting LRPs, shown in Figure 1b, had a negative polarity for preparation of the correct response hand and a positive polarity for the preparation of an incorrect response.

Average LRP amplitudes were analyzed over the five contiguous 100-ms time windows from 0 to 500 ms after stimulus onset using ANOVAs with within-participants factors of object category and congruency. The only significant effect (p < .05) was that of congruency in the 100- to 200-ms time window, in which congruent trials led to the preparation of the correct response hand and incongruent trials led to the preparation of the incorrect hand, F(1, 64) = 24.84, p < .001, η² = .280.

ERP data

Figure 1c shows average ERPs pooled across posterior and occipital electrodes P7, P8, O1, and O2 for congruent and incongruent trials. Figure 2 shows waveforms for each of these electrodes separately, for each of the four trial types. Average ERP amplitudes for the visual P1 and N1 components evident at these electrodes were calculated, respectively, for the time windows between 70 and 100 ms and between 120 and 170 ms after stimulus onset. These values were analyzed using repeated measures ANOVAs with within-participants factors of response hand (left vs. right), handle orientation (left vs. right), electrode hemisphere (left vs. right), and electrode laterality (posterior vs. occipital).

Fig. 2.

Average event-related potentials (ERPs) for electrodes P7, P8, O1, and O2. For each handle orientation, separate waveforms are shown for trials requiring a left-hand response and trials requiring a right-hand response. The bars that show the ERP magnitude are positioned at stimulus onset.

An interaction between hemisphere and handle orientation was revealed in both time windows—P1: F(1, 64) = 5.98, p = .017, η² = .085; N1: F(1, 64) = 100.87, p < .001, η² = .611—with the voltages in the hemisphere contralateral to the orientation of the object’s handle more positive across the P1 and more negative across the N1. Handle orientation also interacted with the response hand used by the participant—P1: F(1, 64) = 6.91, p = .011, η² = .097; N1: F(1, 64) = 5.49, p = .022, η² = .079—resulting in more positive voltages for congruent than for incongruent trials across both the N1 and the P1 time windows (shown in Fig. 1c). An additional ANOVA showed that this interaction did not occur within the time period used for baseline calculation (0 to 200 ms before stimulus onset: F < 1). There were also indications that congruency was modulated by object category—P1: F(1, 64) = 2.73, p = .1, η² = .041; N1: F(1, 64) = 3.74, p = .057, η² = .055—as in the behavioral results. Indeed, further ANOVAs specific to each object category showed that the congruency effect between handle orientation and response hand was only significant for the tools category—P1: F(1, 64) = 10.99, p = .001, η² = .146; N1: F(1, 64) = 12.40, p < .001, η² = .162—and not for kitchen utensils—P1 and N1: F < 1.

Discussion

The results of the study reported here confirm the intimate link between vision and action posited by theories of embodied cognition. Our findings provide evidence that even low-level visual processes are modulated by the relation between the action associated with a depicted object and the action intentions of the observer. The behavioral results of the present study revealed the expected lateralized affordance effect seen in previous studies (e.g., Tucker & Ellis, 1998), with actions facilitated when the handle of the object is congruent with the intended response hand.

Analyses of LRPs revealed that this affordance effect was reflected within 100 ms of stimulus onset in the preparation of the correct response hand during congruent trials and the preparation of the incorrect hand during incongruent trials. Such an early effect on motor activation suggests interactions between the visual and motor systems prior to object categorization and the subsequent selection of the overt response. This is not to suggest that all affordance effects depend on these early influences. The finding that the names of objects potentiate the grips associated with them (Tucker & Ellis, 2004) clearly supports semantic sources of affordance. Rather, our intention here was to reveal the neural interactions that underpin behavioral effects that have been assumed to demonstrate automatic or direct links between the visual and motor systems, such as micro-affordance (Tucker & Ellis, 1998, 2004), but also many other effects that may be generally described as visual-to-motor priming (e.g., Craighero, Fadiga, Rizzolatti, & Umilta, 1998; Rumiati & Humphreys, 1998).

The observed modulation of the posterior visual P1 and N1 components also indicates that exchanges between the visual and motor systems occur at early stages. In the present study, we found that the orientation of the object handle evoked increased P1 and N1 amplitudes in the hemisphere contralateral to handle orientation. Although when participants are asked to select stimuli on the basis of their horizontal-vertical orientation, the amplitudes have been found to increase in the P1 component (Karayanidis & Michie, 1997) and the N1 component (O’Donnell et al., 1997), the hemispheric modulation of the effect observed here is similar to that seen with spatial cuing (Handy & Mangun, 2000; Luck & Hillyard, 1995; Mangun & Hillyard, 1988), as if the visual asymmetry of the objects were shifting attention laterally.

The relation between handle orientation and the response hand of the participant is of potentially greater theoretical interest: Congruent relations generated more positive amplitudes for the N1 and P1 components across both hemispheres than did incongruent relations. This broad posterior increase in ERP amplitude has similarities to the selection negativity (Czigler & Geczy, 1996; Eimer, 1997; Heslenfeld, Kenemans, Kok, & Molenaar, 1997), but it is distinguished by differences in polarity, early onset (70 ms vs. 150 ms), and, most important, the manner in which it is elicited.

This compatibility effect cannot, of course, be explained in terms of the visual properties of the stimuli, as the differences were found between presentations of the same visual stimuli and were modulated only by the expected motor response of the participant. It is also unlikely that the compatibility effect results from changes in spatial attention that may be a consequence of a decision to respond with a particular hand. Such decisions can be reached only after object categorization, which is surely too late to affect these early visual responses. The relation between observed behavioral affordances (reaction times) and these ERP findings is also highlighted by similar interactions between affordance and stimulus category in the two domains. That is, the affordance-related effects on reaction times and early visual ERPs were both stronger for the objects in the tools category than for the objects in the kitchen-utensils category.

We believe that our discovery of an action-compatibility effect on what have been thought of as purely visual responses may have considerable theoretical significance. This discovery shows that some of the brain’s earliest responses to an individual visual object are modulated by the relation between the action associated with the object and the action intentions of the observer. We suggest that these action effects on early visual responses may be a neural signature for object-based visual attention (Egly, Driver, & Rafal, 1994). Elsewhere (Ellis, 2009), we have argued that affordance is the outcome of object-based attention and that visual-object representation in the brain is achieved by the binding of visual and action-related responses. We believe that findings showing both the rapid onset of object-based affordances seen in LRPs and the modulation of early visual ERP components due to intended actions are entirely consistent with this view. Moreover, these findings suggest that vision and action binding occurs very early in the sensory pathways, which supports the concept of deep embodiment.

Footnotes

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

This research was supported by the Engineering and Physical Sciences Research Council under the Cognitive Systems Foresight program (EP/F028598/1).

References

Chao

L. L.

Martin

(2000). Representation of manipulable man-made objects in the dorsal stream. NeuroImage, 12, 478–484.

Coles

M. G. H.

(1989). Modern mind-brain reading: Psychophysiology, physiology, and cognition. Psychophysiology, 26, 251–269.

Craighero

Fadiga

Rizzolatti

Umilta

(1998). Visuomotor priming. Visual Cognition, 5, 109–125.

Czigler

Geczy

(1996). Event-related correlates of color selection and lexical decision: Hierarchical processing or late selection? International Journal of Psychophysiology, 22, 67–84.

Egly

Driver

Rafal

R. D.

(1994). Shifting visual attention between objects and locations: Evidence from normal and parietal lesion subjects. Journal of Experimental Psychology: General, 123, 161–177.

Eimer

(1997). An event-related potential (ERP) study of transient and sustained visual attention to color and form. Biological Psychology, 44, 143–160.

Ellis

(2009). Interactions between action and visual objects. In Morsella

Bargh

Gollwitzer

(Eds.), The Oxford handbook of human action (pp. 213–224). New York, NY: Oxford University Press.

Ellis

Tucker

(2000). Micro-affordance: The potentiation of components of action by seen objects. British Journal of Psychology, 91, 451–471.

Grafton

S. T.

Fadiga

Arbib

M. A.

Rizzolatti

(1997). Premotor cortex activation during observation and naming of familiar tools. NeuroImage, 6, 231–235.

10.

Handy

T. C.

Grafton

S. T.

Shroff

N. M.

Ketay

Gazzaniga

M. S.

(2003). Graspable objects grab attention when the potential for action is recognized. Nature Neuroscience, 6, 421–427.

11.

Handy

T. C.

Mangun

G. R.

(2000). Attention and spatial selection: Electrophysiological evidence for modulation by perceptual load. Perception & Psychophysics, 62, 175–186.

12.

Heslenfeld

D. J.

Kenemans

J. L.

Kok

Molenaar

P. C. M.

(1997). Feature processing and attention in the human visual system: An overview. Biological Psychology, 45, 183–215.

13.

Karayanidis

Michie

P. T.

(1997). Evidence of visual processing negativity with attention to orientation and color in central space. Electroencephalography & Clinical Neurophysiology, 103, 282–297.

14.

Luck

S. J.

Hillyard

S. A.

(1995). The role of attention in feature detection and conjunction discrimination: An electrophysiological analysis. International Journal of Neuroscience, 80, 281–297.

15.

Mangun

G. R.

Hillyard

S. A.

(1988). Spatial gradients of visual attention: Behavioral and electrophysiological evidence. Electroencephalography & Clinical Neurophysiology, 70, 417–428.

16.

Michaels

C. F.

(1988). SR compatibility between response position and destination of apparent motion: Evidence of the detection of affordances. Journal of Experimental Psychology: Human Perception and Performance, 14, 231–240.

17.

O’Donnell

B. F.

Swearer

J. M.

Smith

L. T.

Hokama

McCarley

R. W.

(1997). A topographic study of ERPs elicited by visual feature discrimination. Brain Topography, 10, 133–143.

18.

Rumiati

R. I.

Humphreys

G. W.

(1998). Recognition by action: Dissociating visual and semantic routes to action in normal observers. Journal of Experimental Psychology: Human Perception and Performance, 24, 631–647.

19.

Thorpe

Fize

Marlot

(1996). Speed of processing in the human visual system. Nature, 381, 520–522.

20.

Tucker

Ellis

(1998). On the relations between seen objects and components of potential actions. Journal of Experimental Psychology: Human Perception and Performance, 24, 830–846.

21.

Tucker

Ellis

(2004). Action priming by briefly presented objects. Acta Psychologica, 116, 185–203.

22.

Wilson

(2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9, 625–636.