Counting Multidimensional Objects

Representing multifeatured objects is a fundamental task for the perceptual system. The initial representation of such an object seems to take the form of specialized feature “maps”—one for each feature (e.g., color, shape, and orientation). The information in each map is at least partly independent from the information in other maps (e.g., Treisman, 2006; Treisman & Gelade, 1980; Treisman & Schmidt, 1982). For example, when a scene contains a blue car next to a red bicycle, neurons in the color map would register the presence of the colors red and blue. At the same time, neurons in the shape map would register the shapes of a bicycle and a car. This kind of representation by itself does not contain information that allows the observer to know which color belongs to which object. The features must be bound into multifeatured object representations, in addition to activating the basic separate feature maps. If these integrated representations fail, perceptual errors in binding are experienced (e.g., Treisman & Schmidt, 1982).

Theoretically, the binding could be represented in two different ways. First, there could be a preexisting conjunction detector for each combination. Because the possible combinations of features are limitless, however, this solution is likely to work only for some familiar feature combinations, such as the color red and a strawberry shape, or the color yellow and a banana shape. For other, less frequent and less consistent feature combinations, a post hoc temporary representation of the conjunction must be formed (see Hommel & Colzato, 2009, for a discussion on this issue).

The object-file theory suggests a framework for this kind of post hoc binding. The theory (Kahneman, Treisman, & Gibbs, 1992) suggests that different features that characterize the same object are attached together temporarily via their location, which also acts as the retrieval cue for accessing the object. The theory distinguishes the long-term stored knowledge of object categories (or types), represented as nodes in a recognition network with just one representation for each category, and the temporary episodic representations (object files) that mediate “seeing” the specific objects currently present in their particular instantiation. The spatial location of each object file is its unique retrieval cue, or tag. Different locations that represent different objects are represented by distinctive tags. For example, for three separated objects, three distinctive location tags are created. ¹ A temporary object file that is created for each filled location specifies the features that are presently in that location. These files can be matched to the types in the recognition network in order to identify the objects that are present. This theory also allows the representation of multiple identical objects as separate object files with identical contents but different locations, even though there is only one type node for any given stored category.

Lately, we (Goldfarb & Treisman, 2012) have extended this theory and suggested that the representation of temporary object files acts as the platform from which the arithmetic system can count particular subsets of the items present. The perception of number is a basic ability that exists in human adults, human babies, and even some species of animals (e.g., Dehaene, Piazza, Pinel, & Cohen, 2003; McComb, Packer, & Pusey, 1994). People can generate a number label for the overall number of things in a scene (e.g., Piazza, Izard, Pinel, Le Bihan, & Dehaene, 2004; Piazza, Pinel, Le Bihan, & Dehaene, 2007), but can also count the numbers of objects in specific subsets among those things. For example, an observer can notice that there are four objects in a scene and that there are two cars and two bicycles in the same scene. The numbers in subsets are important not only in representing one’s surroundings, but also in allowing one to perform basic mathematical calculations (e.g., Halberda, Mazzocco, & Feigenson, 2008). The overall number of items present in a scene can be identified simply by counting the filled locations in a location map without binding the features they contain. In contrast, however, when an observer needs to count a specified subset of similar or identical objects, the defining feature (or features) of that subset must be specified. Thus, within our framework, the countable units need to be object files of bound features.

What might be the neural instantiation of these temporary multidimensional representations, or object files? One suggestion is that they might be encoded as synchrony of firing between the neurons representing the features that belong to the same object (e.g., Engel & Singer, 2001; Hommel & Colzato, 2009; Vogel, Woodman, & Luck, 2001; von der Malsburg, 1999). Support for this idea has been documented in several experiments with both human and animal participants. (For a review of the evidence, see Engel & Singer, 2001, and Raffone & Wolters, 2001; but see Lamme & Spekreijse, 1998, and Shadlen & Movshon, 1999, for discussion of some controversial assumptions of this theory). In this account, an object file is represented by the firing synchrony of the neurons that represent the features—such as color, size, and shape—that occupy the same spatial location. Each object file is individuated by its location tag, but its features are integrated by sharing the same neural synchrony within that particular location. It is important to note that binding occurs only within locations. Synchrony across different locations does not result in binding the features of different objects together.

We now provide an example of how the synchrony assumption might fit with the object-file theory and then derive a new prediction about counting items in subsets. We then report two experiments in which we tested this prediction.

In the example display shown in Figure 1a, the system needs to represent four letters that are in four different locations and printed in four different colors. The hypothesis is that neurons representing the color red and the X shape in Location 1 share a common synchrony, which we can label Synchrony Correlation a, or SCa. Similarly, the color green and the O shape in Location 2 share a common synchrony, SCb; the color blue and the T shape in Location 3 share a common synchrony, SCc; and the color brown and the S shape in Location 4 share a common synchrony, SCd. From this representation, the arithmetic system can abstract, for example, that there is one X or one green item. Now consider a second case, shown in Figure 1b. In this example, features are repeated. The identity of a feature can be represented only once, and each distinct object file that contains that feature is bound to this identity. This can be done by synchronizing the firing in each location with the identity of the feature the location contains. In the example in Figure 1b, the object files in Locations 1 and 4 can be synchronized with the X shape, and the object files in Locations 2 and 3 can be synchronized with the O shape. From this platform, the arithmetic system can conclude, for example, that there are two Xs or two Os.

OPEN IN VIEWER

Fig. 1.

Applying the neural-synchrony explanation to object files. In each example stimulus, four letters appear in four locations. If each location has a unique shape and color (a), a specific shape, a specific color, and a specific location can share a common synchrony. For example, neurons representing the X shape and the color red in Location 1 can share a common synchrony, which can be labeled Synchrony Correlation a (SCa). If only letter shapes have to be represented and the letters are repeated (b), the four letter shapes can be represented by synchrony in firing of neurons representing shape and location; locations that contain the same shape will have the same SC. However, if (c) the same letter shape appears in different colors in different locations and both colors and shapes need to be represented (a mixed shape-color association), the stimulus is logically impossible to represent. Neural synchrony can represent which shape is in each location, and it can also represent which color is in each location; however, it is impossible to simultaneously synchronize both the colors and the shapes in all their locations.

According to this theory, there are some situations that are logically impossible to represent. In the case shown in Figure 1c, there are repeated features, and in addition, both colors and shapes must be specified. As in the example in Figure 1a, shape and color must be synchronized in each location. However, in this case, features are repeated. As in Figure 1b, the firing rate associated with the X shape can be represented in Locations 1 and 4, and the firing rate associated with the O shape can be represented in Locations 2 and 3. However, another option is for the firing rate associated with the color red to be represented in Locations 1 and 3, and the firing rate associated with the color green to be represented in Locations 2 and 4. But, in this example, it is impossible to simultaneously synchronize both the colors and the shapes in all their locations.

Notice that in Figure 1c, if the value SCa is associated with the X shape and Locations 1 and 4, and the value SCb is associated with the O shape and Locations 2 and 3, then the observer can see the string “XOOX.” However, the correct colors of this string cannot be synchronized with their locations at the same time. The first X in the string is red; hence, it is clear that the X shape needs to be synchronized with the color red. However, if the system tries to synchronize the red with the X by giving the color red the value SCa, then the color red will also be “seen” in Location 4, even though the object in this location is green. Hence, in this example, the specific color, shape, and location of each object cannot be simultaneously synchronized. It follows that the system can either know how many Xs there are or how many reds there are, but it cannot know both at the same time. Does this theoretically assumed difficulty create a real behavioral problem? In the following experiments, we examined this question by comparing the perception of mixed color-shape associations with that of unique color-shape associations.

Experiment 1

In Experiment 1, we asked participants to compare the numbers of instances in two subsets, each defined by a value on one of two different dimensions: a color or a shape. The participants’ task was to decide if there were more Xs than red objects, if there were more red objects than Xs, or if the number of Xs was equal to the number of red objects. The experiment had two conditions: In one, it was possible to synchronize the relevant features on the two dimensions simultaneously, according to the assumptions we have just described (i.e., there was a unique color-shape association for the relevant features, X and red; see Fig. 2a). Thus, both the Xs and the reds could be synchronized with their locations. In the other condition, such synchronization was impossible (i.e., there were mixed color-shape associations for X and red; see Fig. 2b). We compared both response times (RTs) and error rates in the two conditions.

OPEN IN VIEWER

Fig. 2.

Illustration of the two conditions in Experiment 1. In one condition (a), the target feature X was uniquely associated with the color green, and the target feature red was uniquely associated with the O shape. Therefore, when both X and red needed to be represented, two different synchrony correlations (SCa and SCb) could simultaneously indicate the locations in which each target feature appeared. In the other condition (b), the target feature X was associated with both red and green, and the target feature red was associated with both X and O shapes. Therefore, it was not possible to simultaneously synchronize both the color red and the shape X in all their locations.

Method

Participants

Eleven undergraduate students from Princeton University participated in the experiment in partial fulfillment of course requirements. All had normal or corrected-to-normal vision.

Stimuli

The stimulus displays consisted of letter strings composed of the letters X and O, colored in red and green. The size of each letter was approximately 1.2°. Each string of five letters appeared in the center of a white screen. Each feature (X, O, red, or green) could appear either two or three times. On some trials, there were more Xs than reds; on others, there were more reds than Xs; and on others, there were equal numbers of Xs and reds. In the unique-association condition, the Xs and the reds never shared a common location (i.e., all Xs were green). In the mixed-associations condition, X and red co-occurred in at least one location: When there were more Xs than reds, either one or two of the three Xs were green; when there were more reds than Xs, either one or two of the three Os were red; and when there were equal numbers of Xs and reds, one X was green. Figure 3 shows the complete set of stimuli. The order of the letters was randomized in each display.

OPEN IN VIEWER

Fig. 3.

The nine sets of stimuli used in Experiment 1. In each of the two conditions (unique association, mixed associations), there were more Xs than reds on some trials, more reds than Xs on other trials, and equal numbers of Xs and reds on still other trials. The order of the objects in the strings (each shape and color combination) was randomized in the actual displays.

Procedure

Stimulus presentation and data collection were controlled by a Dell computer with an Intel Xeon central processor. Stimuli were presented on a Dell 19-in. monitor. A keyboard was placed on a table between the participants and the monitor, which was approximately 65 cm from where they were seated. Stickers with the labels “same,” “more Xs,” and “more reds” were pasted on the keys “g,” “h,” and “j,” respectively. Participants completed the experiment individually. They were instructed to compare the number of Xs with the number of red letters, and they were told that the possible responses were “more Xs,” “more reds,” or “the same.” Participants were asked to respond as fast as possible but to avoid mistakes. Each trial started with a 1,000-ms white display that was followed by a letter string. The letter strings were chosen randomly for each participant from a list in which 50% of the trials had equal numbers of Xs and reds, 25% had more reds than Xs, and 25% had more Xs than reds; half of the trials chosen belonged to the unique-association condition, and half belonged to the mixed-associations condition. The string on each trial disappeared when the participant responded, and then the next trial began. The computer registered the participant’s responses, as well as the RT (in milliseconds) from the onset of each string to the participant’s response.

Before the beginning of the experimental block, participants were given 4 practice trials, regardless of how many errors they made. They then performed a block of 80 experimental trials.

Results and discussion

The overall mean error rate was 0.5% for the unique-association condition and 6.7% for the mixed-associations condition. The difference between conditions was significant, t(10) = 3.34, p < .01. For the trials responded to correctly, the mean RT was calculated for each participant in each condition. A paired-samples t test was applied to these data. Responses to letter strings with unique color-shape associations were significantly faster (2,464 ms) than responses to letter strings with mixed color-shape associations (3,232 ms). This 768-ms effect was significant, t(10) = 4.77, p < .001.

To sum up, a large effect was observed for both the RT and the error data. Strings that had unique color-shape associations and therefore—according to our assumptions—could be synchronized neurally were counted faster and with fewer errors in comparison with strings that had mixed color-shape associations and therefore—according to our assumptions—could not be simultaneously synchronized.