Functional dissociation of multiple-object tracking mechanisms based on hemispheric asymmetries

1 Introduction

Healthy subjects show a surprisingly effortless ability to visually track a number of randomly moving objects over a time of several seconds (Pylyshyn, 1989). In the classical multiple object tracking (MOT) task, subjects are presented with several visually ind-istinguishable objects and are required to keep track of a subset of them while they move around the vi-sual field, before identifying all relevant targets (Pylyshyn, 2004; Scholl, Pylyshyn, & Feldman, 2001). The normal student population is reported to be able to track up to five objects simultaneously with an accuracy of around 80% (Alvarez & Franconeri, 2007; Franconeri, Alvarez, & Cavanagh, 2013; Pylyshyn, 1989). A number of theories have been put forward in order to explain this feature of the visual system including pre-attentive indexing (Pylyshyn, 1989), a rotating attentional spotlight (Hogendoorn, Carlson, & Verstraten, 2007) and mul-tifocal attention (Cavanagh & Alvarez, 2005). Common to all these theories is the reliance of the respective process on all individual spatial locations of the relevant targets as well as a monotonous increase of resource requirements with higher task demands.

However, a further feasible tracking mechanism has been proposed, operating solely on the configurational information of the convex hull, formed by the entire set of relevant items during motion (Yantis, 1992). Behavioral data supports the notion that such a high-order object representation would be maintained partly outside the usual retinal reference space, since local spatiotemporal transformations of individual items do not impair tracking performance in such cases (Liu et al., 2005). Further results show that serial tracking processes operating on individual spatial item information and mechanisms maintaining a morphing high-order object representation of the entire object set might exist simultaneously (Merkel, Hopf, Heinze, & Schoenfeld, 2015; Merkel et al., 2014). Electrophysiological data show an early visual component indicating an object-based, parallel process operating on the entire target set only, while a latter visual component seems to modulate with the congruity of the relevant target items and a set of probe items (Merkel et al., 2014), thus indicating an individual item-based tracking mechanism.

Interestingly, several studies suggest an application of the object-based tracking mechanism mostly for experienced subjects (Merkel, Hopf, & Schoenfeld, 2017; Oksama & Hyönä, 2008; Yantis, 1992). Nevertheless, the relative contributions of parallel and serial tracking processes determining tracking performances are still unknown.

Data of hemispheric asymmetry suggests a dissociation between object-based and space-based attentional mechanisms within the left and right parietal cortex respectively (Egly, Driver, & Rafal, 1994). Moreover, a general precedence of the right hemisphere over the left hemisphere in attentional function has been suggested (Mesulam, 1981). In this context, the application of serial and parallel tracking processes might translate into a difference in relative resource contribution of right and left attentional networks.

The current study aims to resolve the role of left and right hemispheres in different multiple object tracking mechanisms by having the MOT paradigm performed by left or right hemispheric patients. Electrophysiological markers for object-based tracking and single item tracking (Merkel et al., 2014) have been identified previously and are used in the current study to operationalize the contribution of object-based and space-based tracking mechanisms in left and right hemispheric patients.

2 Methods

2.2 Multiple object tracking task

For the MOT experiment, stimuli were presented against a black background subtending 21°×21° of visual angle. A central fixation cross of 0.5° was present throughout the experiment. Each trial started with the presentation of six visually identical items (black squares, 1°×1° with white outline) distributed randomly across the background (Fig. 1). Next, for 2 seconds three of the six items flashed three times, designating them relevant targets for the current trial to be tracked. Subsequently, all six items moved within the display on predefined semi-random, non-occluding, independent trajectories for 2800–3200 ms (180+-12 frames). Upon stopping, three of the six items color changed to white, designating them probes. The task for the subjects was to indicate by button press whether all three probes matched the relevant tracked targets or not. Participants had to press the right arrow button with their middle finger if all three probes matched all targets fully (Match3). On the other hand, if only two, one or none of the probes matched the tracked items, the left arrow button had to be pressed with the index finger (Match2, Match1, Match0).

OPEN IN VIEWER

Fig. 1

Experimental design. a) Three out of six visually identical objects were assigned targets at the beginning of each trial and had to be kept track of throughout a motion period of 2800 ms –3200 ms. Once movement ceased, three objects were probed and subjects had to decide whether b) none, one or two of those objects were targets (Match0, Match1, Match2). In this case a response using the index finger was required. If all probe items matched the target items (Match3), subjects had to respond with the middle finger.

Importantly, the items trajectories were calculated in a way that the last frame (during movement cas-eation) was visually identical across all trials. Thus, not only were the locations of all items during that last frame constant but the identity of the probes as well. Differently matching conditions between targets and probes were created by varying the assignment between target and probe items at the beginning of each trial. This design allows for a better comparison of electrophysiological responses towards differently matching probe trials, since the stimulus evoked potential, time-locked to that last frame, would be identical (Merkel et al., 2014). In short, semi-random non-occluding trajectories sharing an identical final position were calculated offline by creating a set of paths with identical starting positions, each ‘chasing’ a random location, that changes position every 2–60 frames, and reversing their final course. For each of the six final locations a set of several thousand trajectories were calculated offline and subsequently merged randomly into groups of six trajectories, each comprising one trial with the restriction that at each frame path locations can never fall below 2° in order to avoid occlusion (see Merkel, Hopf, & Schoenfeld, 2020).

Each patient performed 200 trials overall. Each of the 4 conditions (Match3, Match2, Match1, Match0) was presented 50 times, randomly distributed thro-ughout the session. Subjects were seated comfortably in a chair within a dimly lit room and stimuli were presented on a 17 inch LCD screen placed 70 cm in front of the patients. Subjects were asked to fixate the cross in the middle throughout the experiment. They were instructed to keep track of the three target items throughout each trial and respond to the probe as quickly and accurately as possible. Before the session, each subject was given a number of practice trials to get familiar with the task.

3 Results

The results seem to show a general slower re-sponse for right-hemispheric patients towards the probe stimulus than the left-hemispheric patients (Fig. 2a). However, a non-parametric Wilcoxon rank- sum test did not reveal a significant effect (U(11,10) = 42, p > 0.38). Furthermore, performing a rANOVA for each group separately confirmed no systematic variation of reaction times between the four match conditions neither for the left (F(3,30) = 2.617, p > 0.12, η²_p = 0.207) nor right (F(3,27) = 0.644, p > 0.53, η²_p = 0.067) hemispheric patients. The error rates did differ across the match conditions in both groups (F(3,57) = 10.5, p < 0.002, η²_p = 0.356) but not between groups (F(1,19) = 0.97, p > 0.75, η²_p = 0.005). Interestingly, both groups showed a different pattern of error rates towards the four match conditions (F(3,57) = 5.02, p < 0.029, η²_p = 0.209). The left hemispheric patients showed a linear increase in errors with increasing match between target and probe items, while the right lesion group showed a sole steep increase in errors only for the full match condition (Fig. 2b). Note that the task was a two alternative forced choice task, requiring the same response for all partly matching probes and a different response for the fully matching probe. Figure 2b displays the individual responses of all subjects towards all match conditions, as well. The response pattern of the right hemispheric group is of particular interest, since it suggests that the ‘partly matching’ response was applied mostly as a default response irrespective of general performance, being pressed throughout the experiment in 100% of the trials by three right hemispheric patients. In general no variation between the partly matching probes could be observed within this group (F(2,18) = 0.347, p > 0.6, η²_p = 0.037). However, within the right hemispheric group the full match condition did elicit correct responses in some subjects, thus having an error rate < 1 (t(9) = –3.83, p < 0.004, η²_p = 0.62) suggesting that some patients were able to clearly identify a full match trial.

OPEN IN VIEWER

Fig. 2

Behavioral results. a) Patients with right hemispheric lesions tended to be slower to respond to any of the match condition compared to the left hemispheric patients. Four of the ten patients with lesions on the right side showed substantially slower response times. b) Right and left hemispheric patients showed a different pattern of responses towards the match conditions. For the left hemispheric patients, between and within subject variability in the error rates for partly matching probes suggests a process operating on the individual tracked items. In contrast, the error rates of the right hemispheric patients suggest the adoption of using the ‘partly matching’ response as a default response throughout the experiment. Patients excluded from the VBLM are marked.

The pattern of the course of the mean ERP res-ponse amplitudes towards the different match probes for the two groups shares some of the properties of the error rate patterns (Fig. 3a). ERP amplitudes just failed to significantly vary across match conditions (F(3,57) = 3.003, p = 0.052, η²_p = 0.136). The two-factorial rANOVA did not reveal a significant interaction with the patient group (F(3,57) = 1.707, p > 0.17, η²_p = 0.08). No variation in the event related potential could be observed between match conditions in the right hemispheric group (F(3,27) = 0.279, p > 0.76, η²_p = 0.03). Mean amplitudes differed how-ever within the left hemispheric patients (F(3,30) =3.877, p < 0.034, η²_p = 0.279) showing more negative potentials during probes that are more congruent with the tracked targets (Fig. 3b).

Comparing the spatial distribution of the lesion maps between groups one can see a similar spatial extension of the lesions within the left and right STG respectively (Fig. 4). Although in two patients (pat1, pat2) with right hemispheric lesions the damage involves parts of the IPL, the spatial extent between the groups was not statistically significant (t(18) = 1.53, p > 0.14, η²_p = 0.115). After exclusion of the two right-hemispheric patients with parietal lesions, the lesion patterns between groups were virtually identical (Fig. 5a) also with respect to the size (t(16) = 0.88, p > 0.39, η²_p = 0.042). Figure 5b illustrates that the right-hemispheric parietal lesions in those two subjects did not drive the distinctive behavioral and electrophysiological patterns observed in that lesion group. The right-hemispheric pattern continued to show a different response towards the full match condition in an interaction effect with the left-hemispheric pattern that just felt short of exhibiting significance (F(3,51) = 3.39, p = 0.072, η²_p = 0.166). Likewise, no variation in match conditions between ERP-responses between 200 ms and 300 ms was observed for the right-hemispheric patients (excluding the two subjects exhibiting parietal lesions) (F(3,21) = 0.69, p > 0.51, η²_p = 0.089).

OPEN IN VIEWER

Fig. 3

Electrophysiologic responses. a) Occipital electrode sites showed a different response pattern towards the probe display within the time range between 200 ms and 300 ms for right and left hemispheric patients. b) While the probe display did not elicit a differential visual evoked response in patients with lesions in the right hemisphere, patients with left hemispheric lesions showed an increase in the negative potential with higher correspondence between probe and target set.

OPEN IN VIEWER

Fig. 4

Voxel-based lesion mapping. Lesions of all patients (save one) were semi-automatically delineated by manually thresholding and selecting automatically segmented hyper- and hypointensities within structural images of each patient. Lesion masked normalization was subsequently used to normalize those lesions onto a standardized brain template. The degree of anatomical overlap of the patient groups with lesions in the right- and left hemisphere are visualized separately in red and blue respectively. The main focus for both groups is centered within the insula and superior temporal gyrus. A second, more distributed focus for the right-hemispheric group can be seen within the inferior parietal lobe.

OPEN IN VIEWER

Fig. 5

A summary of the results excluding two right-hemispheric patients with severe damage to the parietal cortex. a) The spatial extend of the lesion without two right-hemispheric patients (pat1, pat2) is virtual identical for the two groups. b) The behavioral and electrophysiological patterns observed for the right-hemispheric patients do not seem to depend on an involvement of the parietal cortex.

Twentyone patients with middle cerebral artery stroke lesions within the left or right hemisphere per-formed in a multiple object tracking task. Event related potentials were recorded at the same time. Behavioral data suggest that patients with right-hemispheric lesions are mostly unable to track three out of six objects over a course of several seconds. Electrophysiological data of those subjects support this conclusion, since potentials evoked by different matching probes did not show any significant variation in amplitude.

In contrast, damage to the frontoparietal network within the left hemisphere seems to leave the general ability to track multiple independently moving objects in those effected patients mostly intact. The electrophysiological data for this group shows a parametrical amplitude increase towards conditions in which fewer targets are being probed. Such a pattern was previously shown to reflect a serial tracking process operating on the individual relevant objects of the target set (Merkel et al., 2014).

Voxel based lesion maps for each group show consistent damage centered within the white matter of the STG and Insula in left and right hemispheric patients. Apart from lesions within the right IPL, VBLM patterns for both groups are mostly symmetric and on average of similar extent. For the present patient groups the observed lesions most likely resulted in intrahemispheric disconnections of fronto-parietal attention subnetworks (Doricchi, Thiebaut de Schotten, Tomaiuolo, & Bartolomeo, 2008; Karnath, Fruhmann Berger, Küker, & Rorden, 2004). Such lesions do not require a direct involvement of the grey matter of IPL or SPL regions to cause severe attentional deficits (Doricchi & Tomaiuolo, 2003; Suchan et al., 2014). On the contrary, it has been shown that the integrity of white fiber long range connections like SLF II and AF, located within the white matter of the STG, are associated with the extent of the cognitive impairment during neglect syndrome (He et al., 2007; Karnath, Rorden, & Ticini, 2009; Thiebaut de Schotten et al., 2008) for both right and left-hemispheric patients (Beume et al., 2017; Suchan & Karnath, 2011). Such disconnection might even have a larger impact on attentional networks since small damage to such fiber bundles can affect large sections of cortical target regions (Bartolomeo, 2007) and would not be subject to compensation by perilesional cortical areas (Catani & Mesulam, 2008; Doricchi et al., 2008). The current results confirm this argument, since the behavioral and electrophysiological pattern associated with patients with right hemispheric lesions do not necessarily require damage to the right parietal cortex. Thus, the observed lesion patterns most likely account for most of the presently described attentional deficits, since those patterns are well in line with earlier reports on voxel-based lesion mapping and tractography studies in relation to neglect symptoms.

Especially, the detrimental effect of right-hem-ispheric lesions on the ability to maintain multiple items over time is well in line with a large body of literature observing a central role of the right hemisphere in visual spatial attention. The left parietal cortex is believed to only exert control over contralateral shifts of spatial attention, while the right parietal cortex is thought to entail a representation of both hemifields (Mesulam, 1981; Posner, Walker, Friedrich, & Rafal, 1984). Such an asymmetry of hemispheric engagement in which the right parietal cortex exerts control over the entire visual field has been shown with the use of electrophysiology (Heilman & Van Den Abell, 1980) and imaging techniques (Corbetta, Miezin, Shulman, & Petersen, 1993; Pollmann & Morrillo, 2003; Thiel, Zilles, & Fink, 2004; Vandenberghe et al., 1997). This crucial role of the right hemisphere in the control of attentional mechanisms is believed to translate at clinical level into much more severe cognitive symptoms in neglect patients with right-hemispheric lesions (Mesulam, Waxman, Geschwind, & Sabin, 1976).

Apart from a general larger right-hemispheric eng-agement in attentional functions, further asymmetries between left and right parietal cortex associated with more specific visual processes were also described. Reports of functional dissociations between the left and right hemispheres in visual perception are equally well in line with the current results. Consistently, a larger perceptional bias for abstract stimulus features within the left hemisphere versus more concrete stimulus properties processed by the right hemisphere was previously described (Burgund & Marsolek, 1997; Koutstaal et al., 2001; Marsolek, Nicholas, & Andresen, 2002; McMenamin, Deason, Steele, Koutstaal, & Marsolek, 2015). Similarly, perceptual discriminations within an object-centered reference frame are faster and more accurate for stimuli presented to the left hemisphere, while the opposite holds true for a viewer-centered presentation (Behrmann & Moscovitch, 1994). Another description frames the same dissociation as a difference in processing categorical and coordinate spatial visual information within the left and right hemisphere, respectively (Hellige, Taylor, & Eng, 1989; Kosslyn et al., 1989). The common idea behind these frameworks is a right hemispheric bias for a representation of spatial information within a metric coordinate system retaining quantitative spatial distances (Ivry & Robertson, 1998).

The majority of object tracking theories stipulate processes operating on the spatial information of the individual relevant items (Alvarez & Franconeri, 2007; Pylyshyn, 1989). Previous results indicate that tracking performance is mostly determined by speed and spatial separation between targets (Shim, Alvarez, & Jiang, 2008), thus absolute spatial distance measures of the individual items (Franconeri, Jonathan, & Scimeca, 2010). An omission of this metric spatial information by right-hemispheric lesions, especially by disconnecting the parietal cortex, could therefore disrupt general tracking abilities, as the current results indicate. Interestingly, some imaging studies investigating cortical engagement during multiple object tracking observed a slight right-hemispheric (although not significant) bias throughout the task (Culham, Cavanagh, & Kanwisher, 2001; Culham et al., 1998). A failure to use specific spatial visual information compared to more abstract representations in right-hemispheric patients has been reported previously during perceptual priming experiments (Schnaider Beeri, Vakil, Adonsky, & Levenkron, 2004). Moreover, right-sided patients performed worse than left-sided patients and controls in recognizing changes in spatial relations of multiple objects in a delayed match to sample task (Laeng, 1994).

Furthermore, the above mentioned hemispheric functional asymmetry is not limited to differences in perceptual biases but is believed to also provide a dissociation between space- and object-based attentional mechanisms. Shifting attention through the same distance of space entails a larger engagement of the left hemisphere if this region is bound by some feature (Egly et al., 1994; Fink, 1997). This left-hemispheric bias for object-based attentional shifts over space-based attentional shifts was confirmed in several imaging studies (Arrington, Carr, Mayer, & Rao, 2000; Wilson, Woldorff, & Mangun, 2005). Furthermore, enhanced BOLD responses can be observed only within the left parietal cortex during the discrimination of two features within one object compared to one feature of the same object (Vandenberghe et al., 1997). The functional dissociation of space- and object-based attention observed in these studies provides an additional background for the current results. Lesions within the right hemisphere result in severe impairments of maintaining multiple objects based on their spatial location. Thus, tracking for this group is almost impossible. Left-hemispheric patients on the other hand retain the ability to track individual item information. The electrophysiological data show a graded visual mismatch effect for probes with a decreasing congruity with the currently tracked targets.

Apart from tracking individual items it has been suggested, that multiple object tracking could be facilitated by maintaining an abstract representation of the convex hull of all relevant items through time (Yantis, 1992). Recently, electrophysiological (Merkel et al., 2017, Merkel et al., 2017, 2014) as well as neuroimaging data (Merkel et al., 2015) provided evidence for such a process. While recording electrophysiological responses during a tracking task similar to the current one, an object-based tracking mechanism has been indicated by an early negative amplitude variation only for probes that match the relevant targets fully (Merkel et al., 2014). The current results do not show such an electrophysiological component for the left-hemispheric patients, suggesting a mostly space-based attentional process responsible for performing in the tracking task. This is well in line with the notion of a left-hemispheric bias for object-based attentional mechanisms, which would be impaired in the patient group showing a disruption of the left fronto-parietal attention network.

The current study provides evidence for hemis-pheric-specific contributions of attentional processes enabling the maintenance of multiple objects through space and time. The right parietal cortex seems to be crucial in operating on individual spatial information of each item while the left hemisphere would provide resources, that could maintain abstract representations of the spatial configuration of all relevant items. Hereby, the former mechanism seems to be a prerequisite of the latter, as right hemispheric lesion impede object tracking in its entirety.