Embodied Effects of Conceptual Knowledge Continuously Perturb the Hand in Flight

Abstract

Attending to a manipulable object evokes a mental representation of hand actions associated with the object’s form and function. In one view, these representations are sufficiently abstract that their competing influence on an unrelated action is confined to the planning stages of movement and does not affect its on-line control. Alternatively, an object may evoke action representations that affect the entire trajectory of an unrelated grasping action. We developed a new methodology to statistically analyze the forward motion and rotation of the hand and fingers under different task conditions. Using this novel approach, we established that a grasping action executed after seeing a photograph of an object is systematically perturbed even into the late stages of its trajectory by the competing influence of the grasping posture associated with the object. Our results show that embodied effects of conceptual knowledge continuously modulate the hand in flight.

Keywords

action representations embodied knowledge hand trajectory kinematics

It is well-established that attending to a picture of a manipulable object evokes motor cortical activity; this finding is consistent with the view that the depicted object automatically leads to a mental representation of hand actions associated with it (e.g., Chao & Martin, 2000; Grèzes, Tucker, Armony, Ellis, & Passingham, 2003; Handy, Grafton, Shroff, Ketay, & Gazzaniga, 2003). Behavioral evidence supports the inference that manipulable objects evoke motor representations consistent with the objects’ form and function (Bub, Masson, & Cree, 2008; Sumner & Husain, 2008). Of particular interest is the nature of these representations and their influence on the planning and execution of a subsequent action. In one view, the mental representations of actions induced by a depicted object remain sufficiently abstract that the influence of such representations should be confined to the planning stages rather than to the on-line control of movement (Glover, 2004; Liu, Chua, & Enns, 2008; Milner & Goodale, 2008). The alternative view is that the object evokes action representations coded in a form compatible with the actual execution of a motor program, which once active, may then compete with and affect the entire course of an unrelated grasping action (Cisek, 2007; Cisek & Kalaska, 2005).

Characterizing the influence of a previously attended object on a grasping action requires that the trajectory of the hand under the influence of a competing object be compared with the trajectory of the hand when no such competition is present. To this end, we applied a powerful and novel statistical methodology incorporating a branch of geometric algebra that represents normal hand movement in three-dimensional space using six degrees of freedom: three for rotation about any axis that can be defined relative to the three standard axes (forward, lateral, and vertical) and three for movement independent of rotation, which we refer to as translation, along a path defined relative to those axes. Our approach allowed the statistical analysis of dynamic changes in the motion of the hand against the background of substantial between- and within-subjects variation. As a result, we were able to capture very subtle perturbations of reach-to-grasp trajectories and characterize their dependence on experimental manipulations.

The method we developed has a number of advantages over previous approaches to analyzing kinematics of the hand (i.e., the detailed measurement of the positional and rotational changes in the fingers and the hand as a person reaches toward and grasps an object). In some studies, analyses have been restricted to the measurement of the aperture between the thumb and forefinger (Glover & Dixon, 2002; Glover, Rosenbaum, Graham, & Dixon, 2004; Jakobson & Goodale, 1991). Others have relied on measuring joint angles to represent the shape of the hand as it approaches a target object (Santello, Flanders, & Soechting, 1998, 2002; Santello & Soechting, 1998). These techniques, however, do not provide a means of statistically comparing the nature of the differences in hand trajectories between conditions. Hansen and colleagues (Hansen & Elliott, 2009; Hansen, Elliott, & Khan, 2008) developed a method to measure at various time points during a reach response the variability across trials of the location of a sensor attached to the hand. Their analysis depicted the momentary variability of location along each axis in three-dimensional space. Their statistical method, however, did not consider rotations at all; nor was their analysis used to compare positional differences in trajectories across conditions.

The methodology most closely resembling our approach used functional analysis of variance applied to the location of a single sensor during forward motion (Chapman et al., 2010a, 2010b). This analysis was confined to two-dimensional projections of a three-dimensional trajectory. In contrast, our method captured, for the first time, moment-by-moment differences in the rotation and translation of the articulations of the hand. To visualize experimental effects on the hand’s trajectory, we represented statistical differences as a pair of three-dimensional representations—one for translation and the other for rotation.

We used an experimental task to examine how a depiction of an object with a handle influences a horizontal or vertical power (clenched fist) grasp. Subjects viewed a briefly presented photograph of an object with a horizontal or vertically oriented handle (e.g., frying pan, beer mug) before being cued to grasp either a horizontal or a vertical response element (see Fig. 1). To ensure that the object was continuously attended, we required subjects to report the name of the object after completing the grasping action on randomly selected trials. The cued action either matched or mismatched the action associated with the object (e.g., a beer mug matched a vertical power grasp but did not match a horizontal one), which allowed us to measure the impact of this mental representation on the kinematics of the reach-and-grasp response. Any change in a rotation or translation trajectory of the hand induced by the object was termed a difference trajectory. In the absence of a difference, this trajectory was represented by a “trajectory” that hovered close to the origin of the three-dimensional space, whereas a difference was statistically significant whenever the difference trajectory moved at least 2 standard errors away from the origin.

Fig. 1.

Example trial sequence for each of the four object primes. In each trial sequence, a photo of one of the objects was shown. The object could be in its canonical orientation (shown here), rotated 90° clockwise, or rotated 90° counterclockwise. The object photos were followed by a cue indicating whether subjects should respond by reaching for and grasping a vertically oriented or a horizontally oriented handle (shown at the bottom right). On each trial, the orientation of the object’s handle in the photograph was congruent or incongruent with the cued action.

The trajectories of fingertips, hand, and wrist were monitored using magnetic sensors that delivered information on position and rotation over time. Because we were particularly interested in the effect of a competing action representation on the rotation of the hand, we emphasized conditions in which moving from the start to the end position of a cued grasp required a 90° rotation of the hand (e.g., from a horizontal start, with the palm facing downward, to a vertical grasp, such as that used to grasp a teapot handle).

Method

Subjects

Eighteen right-handed students at the University of Victoria participated for extra credit in an undergraduate psychology course. This sample size is consistent with those in previous studies of hand trajectories and was established before data collection began.

Materials and procedure

Gray-scale digital photographs of a hand posed in a right-handed vertical power grasp and in a right-handed horizontal power grasp (palm down) were used as cues to indicate the action to be performed on a trial. Gray-scale digital photographs of four objects oriented for use with the right hand served as priming stimuli. Two objects were congruent with a vertical grasp when presented in their canonical orientation (beer mug and teapot), and two were congruent with a horizontal grasp when in their canonical orientation (frying pan and flashlight).

Two additional versions of the object photographs were created. One was generated by rotating the object 90° (counterclockwise for the beer mug and teapot; clockwise for the frying pan and flashlight) to produce an image that was compatible with the alternate grasping action. An object in that orientation could be grasped, and a simple 90° rotation of the wrist to vertical or horizontal (palm down) would reorient the object so that it would be ready for use. The second version of the photographs was created by rotating the mirror-image view of the object 90° (clockwise for the beer mug and teapot; counterclockwise for the frying pan and flashlight). Again, these images afforded the alternate action, but now a grasp followed by a simple 90° rotation of the wrist to vertical or horizontal would result in the object being held in an upside-down position (the complete set of object primes is presented in Fig. S2 in the Supplemental Material available online; also see Experimental Setup and Stimuli in Supplemental Method and Results). The images of the priming stimuli and the hand cues were digitally projected onto a translucent rear-projection screen positioned 80 cm in front of the subject.

Five magnetic sensors were attached by surgical tape to subjects’ right hand, positioned on the nails of the thumb, index finger, and middle finger; on the back of the hand; and on the dorsal surface of the wrist. A cable ran from each sensor up the arm and down the back; the cables were held in place by a fingerless glove, an upper-arm cuff, and a shoulder harness, all made of lightweight fabric. In the first phase of the procedure, subjects practiced responding to the two hand cues by making a vertical or horizontal reach-and-grasp response. Half of the subjects started with a horizontal position (hand flat with open palm facing downward and resting on a response box), and the other subjects started with a vertical position (flat hand with the wrist and hand in a vertical orientation, with the edge of the hand resting on the response box).

Reach-and-grasp responses were made by lifting off from the starting position and reaching forward 30 cm to grasp one of two acrylic response elements mounted on a base. The elements were displaced slightly from the midline, one to the left and the other to the right. The left-side element was a tall, C-shaped form 14 cm in height that afforded a vertical grasp. The right-side element was a right-angled form with a horizontal arm extending 9 cm rightward from a vertical post (9 cm in height) that afforded a horizontal grasp (see Fig. 1; see Fig. S1 in the Supplemental Material for a photograph of the experimental setup). The response elements occupied the same position for all subjects, rather than being counterbalanced with respect to position, so that reach-and-grasp trajectories across subjects would be as similar as possible.

In the second phase of the testing session, subjects were familiarized with the priming objects to ensure that they could easily be named. The final phase of the experiment consisted of 24 practice and 288 critical trials. At the beginning of each trial, a fixation cross appeared on the display screen until the subject placed his or her right hand in the designated start position, resting on the response box. The fixation cross then disappeared, and 500 ms later, the image of the priming object was presented for 300 ms. The object’s image was then replaced by the image of the hand cue indicating the response that was to be made (see Fig. 1). The subject then executed the corresponding reach-and-grasp response as quickly as possible. The subject continued to hold the response element until an auditory signal indicated that the trial was over. Across the practice and critical trials, each of the two possible actions was performed equally often, and each was primed equally often by the 12 object images (4 objects × 3 versions). The trials were presented in a random order for each subject. To ensure that subjects attended to the priming object, we asked them to report the name of the object after completing the reach-and-grasp response on a randomly selected 20% of trials.

Data acquisition

We used an Innovative Sports Training (Chicago, IL) MotionMonitor integrated system equipped with 8-mm Ascension Technology (Burlington, VT) miniBird sensors that simultaneously measured position and orientation (to provide information about rotation). Data from the sensors were collected at a 60-Hz sampling rate using MotionMonitor software. The recording epoch for a trial extended from 1 s prior to the hand lifting off the response box until 1.5 s after liftoff. This time range was adequate to capture normal reach-and-grasp responses.

Data analysis

Our design included a number of conditions, only some of which are relevant to the question of how a canonically viewed object affects the trajectory of a reach-and-grasp action. Although rotated objects were included as primes, they present a special interpretive problem that the present experimental design cannot adequately address. Namely, an object such as a rotated beer mug invites a horizontal grasp on the basis of its visible form but a vertical grasp on the basis of its canonical properties. As a result, it is not possible to determine which of these influences affects the trajectory of the hand. Our interest was specifically in how canonically oriented objects influence the production of a reach-and-grasp response. Moreover, because our technique offered, for the first time, the possibility of assessing statistical differences in the rotation as well as the forward motion of the hand, we were particularly interested in actions that required a wrist rotation when moving from the starting position to the final grasp posture. Therefore, our primary analyses were restricted to trials on which the priming object was presented in its canonical view, and we emphasized conditions in which the reach-and-grasp response required a wrist rotation. We will, however, briefly describe the priming results for trajectories that did not involve a wrist rotation.

The method for identifying statistical outliers in the individual trial data delivered by the sensors is described in Data Filtering in Supplemental Method and Results. This procedure resulted in the loss of 16% of the trials. From the remaining data, mean trajectories and a three-dimensional representation of the standard error of the mean at each sampled point in the trajectory (which we call an error volume) were calculated by subject and condition for each sensor. Difference trajectories representing the change in the paths followed by the hand in the congruent versus incongruent priming conditions, with error volumes based on these changes, were then calculated for each subject. Finally, difference trajectories were averaged across subjects, and error volumes were computed. Further details are provided in Conformal Geometric Algebra in Supplemental Method and Results.

Results

The plots shown in Figure 2 illustrate the nature of the trajectories produced by our reach-and-grasp task and the computational challenges associated with evaluating differences between those trajectories. Figure 2a is a three-dimensional depiction of the position trajectory (i.e., position as a function of time) for each of five sensors, which we call a box plot. The plot shows all trials from one subject making a horizontal grasp from a vertical starting position when the action associated with the prime object matched the grasping action being produced (congruent condition). The three dimensions represent forward motion (y-axis), lateral motion (x-axis), and vertical motion (z-axis). The trajectories begin near the origin of the plot (i.e., x = y = z = 0) and move forward along the y-axis until the fingers curl around the response element (note the curvature in the sensors for the index and middle fingers). Trial-to-trial variability in movement is characterized by displacement among the individual trajectories. The plot also shows two-dimensional projections of the trajectory (in desaturated color) on each of the three standard planes (i.e., x-y, x-z, and y-z). These projections assist in visualizing the trajectory’s three-dimensional shape and allow one to clearly see how it changes along two dimensions at a time. For example, on the y-z plane (side view), one can easily see the curvature of the two fingers as they form themselves around the horizontal response element at the end of the movement.

Fig. 2.

Three-dimensional position and rotation trajectories (full color), with projections onto each two-dimensional coordinate plane (desaturated color). Position trajectories (a) and rotation trajectories (b) are shown for each of five sensors for 1 subject making a horizontal grasp from a vertical starting position for all trials in the congruent condition. The black vector in (b) represents the rotation of the index finger sensor at a specific instant in time (a point on one of the paths), as described in the text. The average position trajectory (c) and rotation trajectory (d) are shown for the congruent and incongruent conditions for each sensor for the same subject and action depicted in (a) and (b). The average position trajectory (e) and rotation trajectory (f) are shown for congruent and incongruent conditions for each sensor for a different subject making the same action as the first subject. The directional cones in (d) and (f) mark 10% increments of normalized time. The average trajectories shown in (c) through (f) include error volumes depicting 1 standard error of the mean (computed across trials for an individual subject), although these volumes are very small in (c) and (e). The scale in (a), (c), and (e) is in millimeters, and the radius of the spheres in (b), (d), and (f) is .05.

Figure 2b is a three-dimensional depiction of the rotation trajectory (i.e., rotation as a function of time) of each sensor over the course of the movement for the same trials shown in Figure 2a. We call this way of displaying rotation trajectories a ball plot. Each point on the rotation trajectory represents the momentary axis and degree of rotation experienced by a particular sensor. The trajectories in Figure 2b begin at the right side of the plot and progress leftward from there. These trajectories are interpreted as follows: Begin by drawing a vector from the origin to any point on a trajectory (e.g., the black vector shown in Fig. 2b). The vector lies on the axis of rotation, and the length of the vector represents the angular degree of rotation. The direction of rotation is given by the right-hand rule. To apply the rule, one extends the right-hand thumb along the vector, pointing in the same direction as the vector. Curling the fingers “around the vector” shows the direction of rotation.

In Figure 2b, the vector indicates that near the end of the movement, the index finger as it curled around the response element was still rotating into a horizontal position from the vertical starting point. This can be seen by consulting the x-y and y-z two-dimensional projections. In the x-y projection (overhead view, lying below the ball plot), the end point of the index finger sensor’s trajectory lies in the negative region of the x-axis and the positive region of the y-axis. This captures the fact that as this finger curled inward, it was still rotating into the horizontal position. In addition, the y-z projection, shown to the left of the ball plot (side view), indicates that this point also lies in the positive region of the z-axis, which implies that the curling action occurred while the finger retained some vertical aspect.

Because the notion of a ball plot is likely to be unfamiliar to many, we provide a second example of how to interpret rotation, applied again to the index finger. Consider the starting point of its trajectory shown in Figure 2b. As the ball plot and its two-dimensional projections imply, displacement of this starting point from the origin occurs almost exclusively along the x-axis. This means that if we construct a vector from the origin to the index finger’s starting position, the direction of rotation would primarily be around the x-axis pointing in the positive direction, which indicates that at the start of the movement, the index finger is moving upward and rotating toward the body.

Figure 2c shows the average position trajectory of each sensor for the congruent and incongruent conditions separately, and Figure 2d shows the corresponding average rotation trajectories. Data are for the same subject and action as in Figures 2a and 2b. Note that averaging across trajectories from different trials, which naturally vary in the time required to complete the action, necessitates defining trajectories in normalized time. That is, the full trajectory for a trial must be parceled into segments corresponding to particular proportions of the total time taken to complete the movement on that trial. Averaging across trials can then be done for each of these portions of the trajectories. Error volumes representing 1 standard error of the mean in three dimensions are plotted for these trajectories. These volumes were particularly small for the positional trajectory and therefore are occluded in the figure by the line drawn along the trajectory. The rotation trajectories also feature directional cones providing information on the temporal progression of the trajectory.

Figures 2e and 2f show the corresponding average trajectories and error volumes for a different subject. Notice that for both the position and rotation trajectories, the differences between congruent and incongruent conditions are much more subtle than the between-subjects differences. Our analytic approach calculates difference trajectories (i.e., the difference between congruent and incongruent conditions) within subjects before aggregating them across subjects, which thereby allows subtle differences between conditions to be detected even in the presence of large between-subjects variation in trajectories. Aspects of our mathematical representation that make this tractable are discussed in Conformal Geometric Algebra in Supplemental Method and Results.

We now discuss the prime object’s effect on a grasping action using average difference trajectories plotted in three-dimensional space. These trajectories show the position of a sensor in the incongruent condition at a given time point, relative to the position of that sensor in the congruent condition at that time. An effect of congruency will be revealed by a difference trajectory that progresses further away from the origin as the effect accrues over time. If there is no effect of the relationship between the object in working memory and the hand movement being executed, then the difference trajectory should remain near zero throughout the time course of the movement and should not extend beyond the origin by more than the plotted error volume. The result would be a path hovering near the origin, as shown in Figure 3.

Fig. 3.

Box plot showing an averaged difference trajectory and error ellipsoids in three dimensions, with projections onto each two-dimensional coordinate plane. (This plot is from a condition not reported in full detail in this article and shows no effect of congruency.) The origin is the ball in the middle of the box plot, indicated by a large black arrow, and represents a null difference. Note that the difference trajectory remains very close to the origin throughout its extent. The scale of the axes is in millimeters.

As can be seen from the difference trajectories displayed in Figures 4 and 5, the object in working memory had, indeed, a substantial impact on the reach-and-grasp response. These difference trajectories reveal positional and rotational differences (incongruent relative to congruent) and are shown as solid lines with directional cones placed at intervals of 10% of normalized time. Because we are plotting difference trajectories, the scale of the axes in Figure 4, though still in millimeters, is more fine-grained than in Figure 2. Error volumes corresponding to differences of 1 standard error of the mean are depicted as low-contrast ellipsoids around each trajectory and were computed independently at successive points along the trajectory. To assist with interpretation of these trajectories, we also provide two-dimensional projections as in Figure 2. These projections show the error volumes as overlapping ellipses.

Fig. 4.

Box plots showing averaged difference trajectories and error ellipsoids in three dimensions for a vertical grasp made from the horizontal starting position and a horizontal grasp made from the vertical starting position. Individual plots, as well as projections onto each two-dimensional coordinate plane, are shown for each sensor. Differences were computed by subtracting reach-and-grasp actions in the congruent condition from those in the incongruent condition. Negative differences imply a smaller value on a particular dimension for the incongruent than for the congruent condition. Error ellipsoids correspond to differences of 1 standard error of the mean. The ball at the origin represents a null difference (a large black arrow is shown to help locate this point). The directional cones mark 10% increments of normalized time. The scale of the axes is in millimeters.

Fig. 5.

Ball plots showing averaged difference trajectories and error volumes in three dimensions, with projections onto each two-dimensional coordinate plane, for the sensor on the back of the hand. The figure shows the results for (a) a vertical grasp made from the horizontal starting position and (b) a horizontal grasp made from the vertical starting position. An imaginary vector extending from the origin to a given point on the trajectory represents the axis (by its orientation) and degree (by its length) of rotation by which the sensor in the incongruent condition has been shifted away from its orientation in the congruent condition. The origin of the sphere represents no difference between congruency conditions. Error volumes correspond to differences of 1 standard error of the mean. The radius of the sphere is .05. (Videos S2 and S4 in the Supplemental Material show animations of the difference trajectories for rotation depicted in this figure.)

Consider first the action in which subjects moved from a horizontal starting position (fingers extended, palm facing down) to a vertical power grasp (Fig. 4, the five panels in the left column). Figure 4 shows that for all of the sensors except the index finger, the difference trajectory moved a substantial distance away from the origin (the centrally located ball indicated by the arrow) in the negative direction along the forward (y) axis, up to a maximum average exceeding 14 mm, before turning to the positive direction. As an example of this effect, consider the difference trajectory for the thumb. This trajectory begins near the origin and moves in the negative direction to a maximum of about 16 mm along the y-axis. The directional cones indicate that this relative lag continues to increase until about the midpoint of the action (the fourth of ten cones), then the difference between the congruent and incongruent condition is progressively reduced until the grasp is completed. This pattern implies that in roughly the first half of the movement in the incongruent condition, there was less forward progress relative to the congruent condition. Over the remainder of the movement, this difference necessarily diminished in normalized time because the hand landed in approximately the same position at the end of both congruent and incongruent trials. This effect is especially clear when looking at the two-dimensional projection on the x-y plane.

Notice that another effect is revealed both in the x-y projection and in the projection on the x-z plane. The trajectory is displaced to the right along the x-axis during the second half of the movement. The same rightward displacement occurs for all the other sensors as well, although to only a minor extent for the index finger. This excursion to the right in the incongruent condition appears to be due to an attraction to the alternate response element (horizontal handle), which was positioned to the right of midline and was compatible with the prime object presented in that condition.

A more interesting result concerns the effect of congruency on the thumb’s position over the course of the trajectory. As indicated by the projection on the x-z plane, the thumb was substantially lower in the incongruent than in the congruent condition (maximum average difference = 8 mm on the z-axis during the middle part of the movement), more so than is seen with the other sensors. This exaggerated downward position of the thumb provides evidence that when the priming object invites a horizontal grasp, the thumb momentarily remains in a pronated position as the hand moves into the vertical grasp. Positions of the middle finger along the x-axis provide intriguing evidence on how this interference was overcome. This sensor displays an exaggerated rightward excursion in the last quarter of the movement (maximum average difference = 6 mm on the x-axis), apparently counteracting the sustained pronation of the thumb. The segments of the mean difference trajectories described here are all more than 2 standard errors away from the origin.

The ball plot in Figure 5a represents differences in rotation between the congruent and incongruent conditions (i.e., rotation in the incongruent condition, relative to the congruent condition, as a function of normalized time) for the vertical grasp. Data are displayed for the sensor on the back of the hand because it provided the most stable estimate of rotational position. The form of the path, projected onto the horizontal (x-y) plane, provides the clearest interpretation of the rotational effects. In this plane, the y-axis projects from the origin toward the midline of the observer. To interpret the difference trajectory, one can apply the right-hand rule, just as was done when interpreting the rotation trajectories shown in Figure 2.

To apply the rule, place your right hand with the thumb extended along this axis pointing toward your midline, emulating a vector extending from the origin to a point on the trajectory. The curl of your fingers indicates the direction of rotation for the difference trajectory (i.e., from the viewer’s perspective, a counterclockwise rotation about the y-axis). This rotational difference indicates that in the incongruent condition, the sensor of interest lagged in its rotation from a horizontal starting position to the final vertical grasp. That is, relative to the position of the sensor in the congruent condition at a particular point in the trajectory, in the incongruent condition, the sensor was rotated counterclockwise away from that orientation (i.e., more toward a flat, horizontal position). In the latter part of the movement, this difference was resolved as the hand formed the vertical orientation required by the cued grasp posture. Difference trajectories in ball plots require some practice to read, so we have provided animations in the Supplemental Material (see Animation of Congruency Effect on Rotation in Supplemental Method and Results and Videos S1–S8) that depict the each difference trajectory as a rotating cube. Video S2 shows the difference trajectory corresponding to Figure 5a. In addition, Video S9 shows how the shape of the error volume for rotational difference trajectories varies depending on distance from the origin. To assist with interpretation of the three-dimensional difference trajectories for position and rotation, we have provided MATLAB files S1 to S14 in the Supplemental Material (see also Visualization Tools for Difference Trajectories in Supplemental Method and Results) so the trajectories can be viewed from different perspectives.

Of great importance, a pattern of effects complementary to those we have just discussed was obtained when the hand moved from a vertical start position to enact a horizontal grasp. For the position trajectories (Fig. 4, the five panels in the right column), there was again a lag in the forward motion of the hand in the incongruent condition, which can be seen most clearly in the x-y plane for the thumb sensor (maximum average = 12 mm on the y-axis). There was also a leftward deviation for all sensors toward the location of the vertical response element (located to the left of midline), which was compatible with the incongruent object prime. This deviation can be most clearly seen in either the x-y plane or the x-z plane for each of the sensors, but especially for the thumb (maximum average = 6 mm on the x-axis). In addition, the thumb and fingers remained relatively high on the z-axis in the incongruent condition through the first half of the movement (compatible with the vertical grasp afforded by the depicted object in that condition). To see this, note that in the y-z plane (corresponding to a side view of the box plot), the trajectories curve upward along the z-axis after about the first third of the movement. In the case of the index finger, for example, the maximum average upward deviation is nearly 6 mm. These positional displacements were more than 2 standard errors from the origin over much of the trajectory. Then the thumb descended (the downward deflection began between the fifth and sixth directional cones), followed by the fingers (between the sixth and seventh directional cones). The early descent of the thumb appears to have counteracted the tendency for the index and middle fingers to remain in a vertical position.

The ball plot confirms this description of events (Fig. 5b). Examining the y-z projection, one can see that the trajectory extends primarily away from the observer along the y-axis (with some elevation on the z-axis that remains within about 30°). A vector extending from the origin away from the observer to the most extreme point on the trajectory together with the right-hand rule indicates that the rotational difference consists of a clockwise rotation primarily around the y-axis. This direction of rotation for the difference trajectory is due to the fact that the position of the sensor in the incongruent condition, relative to the congruent condition, is closer to the upright starting position. In other words, the rotation of the back of the hand from upright to pronated is slower in the incongruent condition. We infer that the downward trajectory of the thumb, which begins prior to the downward trajectory of the other fingers, plays a role in counteracting the initial slowing of the hand’s rotation. The animation of Figure 5b in the Supplemental Material (Video S4) illustrates the direction and magnitude of the congruency effect on the rotation trajectory.

Depictions of the positional and rotational difference trajectories for actions that did not require a rotation of the hand are presented in Figures S3 and S4 in the Supplemental Material (see also Congruency Effects for Conditions With No Hand Rotation in Supplemental Method and Results). The only congruency effect on hand position that was apparent for those actions was in the incongruent condition, in which the hand drifted laterally toward the incorrect response element in the first half of the trajectory, consistent with what was seen with the actions requiring a rotation as described above. With respect to rotational differences, during a vertical grasp, the sensor on the back of the hand indicated that the hand rotated around the x-axis so that the fingers were elevated slightly in the incongruent relative to the congruent condition. When making a horizontal grasp, there was a tendency in the incongruent condition to rotate the hand slightly forward, away from the body (see Videos S7 and S8 in the Supplemental Material).

To address the question of whether the effects of the object context extended throughout the hand’s trajectory, rather than being confined to the early stages of movement, we plotted the combined translation and rotation effect size for each sensor as a function of normalized time in Figure 6. Figure 6a shows that for the vertical grasp made from a horizontal starting position, the congruency effect for all sensors but the index finger was greater than 2 standard errors of the mean even after the midpoint of the trajectory. Near the end of the trajectory, the effect for some sensors was still greater than 2 standard errors above zero. Note that the middle finger in the vertical-grasp condition shows some perturbations in the last part of the trajectory. These are likely caused by small movements of the sensor due to its imperfect attachment to the finger for some subjects tested in the horizontal starting condition. For the horizontal grasp made from the vertical starting position (Fig. 6b), the maximal effect size occurred in the last quarter of the trajectory for the sensor on the back of the hand, and the effect remained over 2 standard errors away from zero for that sensor even at the end of the movement.

Fig. 6.

Combined positional and rotational congruency effect over normalized movement time for all five sensors. Data are shown separately for the vertical grasp made from the horizontal starting position (a) and the horizontal grasp made from the vertical starting position (b). The effect-size metric can be interpreted as the standard error of the mean for the congruency effect combining both position and rotation components of the trajectory. An effect size of 2, for example, is different from 0 by 2 standard errors of the mean. Hotelling distance is a multivariate measure of effect size that is used when multiple dependent measures are combined.

Discussion

Previous reports have shown that the trajectory of the hand in a pointing task and of an eye movement to a target can deviate toward or away from the spatial location of a distractor (Song & Nakayama, 2008; Tipper, Howard, & Houghton, 1998; Van der Stigchel, 2010; Weaver, Lauwereyns, & Theeuwes, 2011). Computational models of trajectories that curve away from a distractor are based on the idea of localized areas of inhibition that operate on neuronal populations responsible for encoding the distractor location (Doyle & Walker, 2001; Tipper, Howard, & Paul, 2001). A further mechanism is required to counteract the influence of the distractor so that the trajectory curves back toward the target location (McSorley, Haggard, & Walker, 2004). Situations in which the hand or eye curves toward instead of away from a distractor occur when attention is forced to the distractor because of task demands (Song & Nakayama, 2008). For example, selecting a target using its unique color as the cue inevitably generates internal competition from among a set of homogeneously colored distractors. This competition is the result of lateral interactions between the neuronal populations that separately encode the locations of the target and distractors (McPeek & Keller, 2001).

In our experiment, attention was forced to the visual object that had to be held in memory while the reach-and-grasp action was programmed and executed. Remarkably, the competition we observed was generated at least in part by a mental representation of a potential action to an object held in working memory. Thus, analogously to visual search experiments that have examined hand and eye trajectories attracted toward a spatial distractor, the dynamics of the grasp posture of the hand in our task was affected by the “virtual grasp” associated with the object. This competition between the intended and the virtual grasps altered both the forward motion and rotation of the hand in principled ways. We observed three qualitatively distinct effects. When the grasping response required a rotation, an incongruent object in working memory appeared to generate a virtual counterforce that was compensated for during the movement. For example, pronation of the hand was altered when the distracting object invited supination, and vice versa. Regardless of whether rotation was required, during the first half of the reach action, an incongruent object in working memory slowed forward progress and attracted the hand toward the competing response element. The latter result converges nicely with previous demonstrations showing analogous effects for both hand and eye movements in two dimensions (McSorley et al., 2004; Song & Nakayama, 2008).

The evidence we obtained goes well beyond previous work that has shown effects of physically present distractor objects on the trajectory of a hand or eye movement toward a target location (McSorley et al., 2004; Song & Nakayama, 2008; Tipper et al., 1998; Weaver et al., 2011). The distracting action representation in our task was evoked by an object in memory that was no longer visible when the grasping action was planned and executed. Nevertheless, the entire trajectory of the movement, as well as the articulation of the hand, were altered under the competing influence of the distracting object (see Fig. 6).

Other research examining the effect of an irrelevant word or object on the trajectory of hand actions has relied on traditional kinematic measures, such as time to peak velocity and thumb-forefinger aperture (e.g., Gentilucci, Benuzzi, Bertolani, Daprati, & Gangitano, 2000; Gentilucci & Gangitano, 1998; Glover & Dixon, 2002; Glover et al., 2004). These results indicate that effects are confined to relatively early stages of the trajectory and dissipate during the course of the movement. Measures such as velocity and aperture, however, are inherently constrained by the fact that the hand must arrive at an end point with the thumb and forefinger spaced in accordance with the size of the grasped object. It is inevitable, then, that these kinematic measures eventually must be determined by the parameters of the target and can admit no influence of context on later stages of the movement. Our statistical methodology allowed us to track changes as small as a fraction of a degree in the rotation of the hand and a few millimeters in its position, and it clearly established that the on-line control of movement is continuously modulated by conceptually driven representations of action, in much the same way that movement is continuously informed by the presence of competing objects in space. Overcoming the competing affordances of an object in working memory when carrying out an intended action is much like resolving competition between action plans evoked by objects in space.

Footnotes

Acknowledgements

We thank Marnie Jedynak for assistance with data collection.

Declaration of Conflicting Interests

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

Funding

This work was supported by discovery grants from the Natural Sciences and Engineering Research Council of Canada to M. E. J. Masson, D. N. Bub, and P. F. Driessen and by National Science Foundation (NSF) Grant No. SBE-0542013 to the Temporal Dynamics of Learning Center, an NSF Science of Learning Center.

Supplemental Material

Additional supporting information may be found at

References

Bub

D. N.

Masson

M. E. J.

Cree

G. S.

(2008). Evocation of functional and volumetric gestural knowledge by objects and words. Cognition, 106, 27–58.

Chao

L. L.

Martin

(2000). Representation of manipulable man-made objects in the dorsal stream. NeuroImage, 12, 478–484.

Chapman

C. S.

Gallivan

J. P.

Wood

D. K.

Milne

J. L.

Culham

J. C.

Goodale

M. A.

(2010a). Reaching for the unknown: Multiple target encoding and real-time decision-making in a rapid reach task. Cognition, 116, 168–176.

Chapman

C. S.

Gallivan

J. P.

Wood

D. K.

Milne

J. L.

Culham

J. C.

Goodale

M. A.

(2010b). Short-term motor plasticity revealed in a visuomotor decision-making task. Behavioural Brain Research, 214, 130–134.

Cisek

(2007). Cortical mechanisms of action selection: The affordance competition hypothesis. Philosophical Transactions of the Royal Society B: Biological Sciences, 362, 1585–1599.

Cisek

Kalaska

J. F.

(2005). Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action. Neuron, 45, 801–814.

Doyle

Walker

(2001). Curved saccade trajectories: Voluntary and reflexive saccades curve away from irrelevant distractors. Experimental Brain Research, 139, 333–344.

Gentilucci

Benuzzi

Bertolani

Daprati

Gangitano

(2000). Language and motor control. Experimental Brain Research, 133, 468–490.

Gentilucci

Gangitano

(1998). Influence of automatic word reading on motor control. European Journal of Neuroscience, 10, 752–756.

10.

Glover

(2004). Separate visual representations in the planning and control of action. Behavioural and Brain Sciences, 27, 3–78.

11.

Glover

Dixon

(2002). Semantics affect the planning but not control of grasping. Experimental Brain Research, 146, 383–387.

12.

Glover

Rosenbaum

D. A.

Graham

Dixon

(2004). Grasping the meaning of words. Experimental Brain Research, 154, 103–108.

13.

Grèzes

Tucker

Armony

Ellis

Passingham

R. E.

(2003). Objects automatically potentiate action: An fMRI study of implicit processing. European Journal of Neuroscience, 17, 2735–2740.

14.

Handy

T. C.

Grafton

S. T.

Shroff

N. M.

Ketay

Gazzaniga

M. S.

(2003). Graspable objects grab attention when the potential for action is recognized. Nature Neuroscience, 6, 421–427.

15.

Hansen

Elliott

(2009). Three-dimensional manual responses to unexpected target perturbations during rapid aiming. Journal of Motor Behavior, 41, 16–29.

16.

Hansen

Elliott

Khan

M. A.

(2008). Quantifying the variability of three-dimensional aiming movements using ellipsoids. Motor Control, 12, 241–251.

17.

Jakobson

L. S.

Goodale

M. A.

(1991). Factors affecting higher-order movement planning: A kinematic analysis of human prehension. Experimental Brain Research, 86, 199–208.

18.

Liu

Chua

Enns

J. T.

(2008). Attention for perception and action: Task interference for action planning but not for online control. Experimental Brain Research, 185, 709–717.

19.

McPeek

R. M.

Keller

E. L.

(2001). Short-term priming, concurrent processing, and saccade curvature during a target selection task in the monkey. Vision Research, 41, 785–800.

20.

McSorley

Haggard

Walker

(2004). Distractor modulation of saccade trajectories: Spatial separation and symmetry effects. Experimental Brain Research, 155, 320–333.

21.

Milner

A. D.

Goodale

M. A.

(2008). Two visual systems re-viewed. Neuropsychologia, 46, 774–785.

22.

Santello

Flanders

Soechting

J. F.

(1998). Postural hand synergies for tool use. Journal of Neuroscience, 18, 10105–10115.

23.

Santello

Flanders

Soechting

J. F.

(2002). Patterns of hand motion during grasping and the influence of sensory guidance. Journal of Neuroscience, 22, 1426–1435.

24.

Santello

Soechting

J. F.

(1998). Gradual molding of the hand to object contours. Journal of Neurophysiology, 79, 1307–1320.

25.

Song

J.-H.

Nakayama

(2008). Target selection in visual search as revealed by movement trajectories. Vision Research, 48, 853–861.

26.

Sumner

Husain

(2008). At the edge of consciousness: Automatic motor activation and voluntary control. The Neuroscientist, 14, 474–486.

27.

Tipper

S. P.

Howard

L. A.

Houghton

(1998). Action-based mechanisms of attention. Philosophical Transactions of the Royal Society B: Biological Sciences, 353, 1385–1393.

28.

Tipper

S. P.

Howard

L. A.

Paul

M. A.

(2001). Reaching affects saccade trajectories. Experimental Brain Research, 136, 241–249.

29.

Van der Stigchel

(2010). Recent advances in the study of saccade trajectory deviations. Vision Research, 50, 1619–1627.

30.

Weaver

M. D.

Lauwereyns

Theeuwes

(2011). The effect of semantic information on saccade trajectory deviations. Vision Research, 51, 1124–1128.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

31.46 MB

3.57 MB

9.73 MB

0.00 MB

0.58 MB