Putting Effort Into Infant Cognition

Abstract

Working memory allows people to manipulate information in support of ongoing tasks and provides a work space for cognitive processes such as learning, reasoning, and decision making. How well working memory works depends, in part, on effort. Someone who pays attention at the right time and place will have better memory and improved performance on memory tasks. In adult cognitive research, participants’ devotion of maximal task-focused effort is often taken for granted, but in infant studies, researchers cannot make that assumption. In this article, we showcase how pupillometry can provide an easy-to-obtain physiological measure of cognitive effort that allows us to better understand infants’ emerging abilities. In our work, we use pupillometry to measure trial-by-trial fluctuations of effort, establishing that, just as in adults, such fluctuations influence how well infants can encode information in visual working memory. We hope that by using physiological measures such as pupil dilation, there will be a renewed effort to investigate the interaction between infants’ attentive states and cognition.

Keywords

infant visual working memory cognitive effort task-evoked pupil response

Tasks have goals. Meeting those goals depends on ability but also effort. When we talk about cognitive tasks, then, we can talk about cognitive effort. This notion was well captured by Kahneman (1973):

[The] intensive aspects of attention . . . must be distinguished from arousal. Thus, the schoolboy who pays attention . . . is performing work, expending his limited resources, and the more attention he pays, the harder he works. . . . The intensive aspect of attention corresponds to effort rather than to mere wakefulness. (p. 4)

In studies with adults and older children, researchers can control effort through instructions and incentives, but infant researchers do not have this luxury. Instead, we typically use generalized assessments (fussiness, sleepiness, inability to habituate, or valid data proportions) to exclude infants who do not seem to be on task (Slaughter & Suddendorf, 2007). Even then, most of what we have learned about infants’ cognitive abilities comes from measures of their looking patterns, but looking does not imply seeing; it may just be a blank stare (Aslin, 2012). And the stakes are high. Averaging inattentive moments with on-task ones can systematically underestimate infants’ abilities and unnecessarily limit theories of cognitive development. Luckily, we have a window onto cognitive effort: the pupil.

Pupillometry as a Measure of Cognitive Effort

The pupillary light reflex was first described by the 10th-century Persian physician Al-Razi. The diameter of the pupil is also modulated by stress and arousal via the autonomic nervous system. Importantly, there is a third, higher-level cortical network that contributes to pupil regulation, and this network underlies the effort exerted from moment to moment on an ongoing task. More than 50 years ago, Hess and Polt (1964) showed that the pupil was a marker of task difficulty and found greater dilation in more difficult mental multiplication tasks. Since then, cognitive psychologists have used the pupil as a sensitive, involuntary measure of effort and attention in a variety of domains; more than 400 research articles have been published in the past 10 years—reviews highlight this trend in both adults (van der Wel & van Steenbergen, 2018) and infants (Laeng, Sirois, & Gredebäck, 2012). In all of this work, controlling for other effects on pupil diameter (e.g., luminance or emotional triggers) is always the first design step. After that is done, the pupil provides a marker of task-relevant effort, for instance, in maintaining information during visual search (Porter, Troscianko, & Gilchrist, 2007) and multiple object tracking (Alnæs et al., 2014). The pupil provides a valid measure of effort because it reflects the activity of the locus coeruleus–noradrenergic (LC-NE) neuromodulatory system. (The LC is a small nucleus in the brain stem; it is the sole source of cortical noradrenaline and projects to widely distributed areas of the brain, especially the frontal cortex.) This system plays a causal role in regulating task engagement and optimizing performance (Aston-Jones & Cohen, 2005; Sara, 2009). Two components of LC activity have been isolated—tonic and phasic—that are positively reflected in obligatory, time-linked tonic and phasic pupil responses. Tonic activity marks a relatively slow-changing modulation of general arousal (“wakefulness,” in Kahneman’s, 1973, formulation); low levels are associated with drowsiness and high levels with distractibility. In contrast, phasic activity marks a rapid, task-evoked modulation of a focused attentional state that optimizes performance.

Pupillometry has been used successfully in a few different domains of cognitive development. For instance, pupillometry has corroborated that infants have specific expectations about their physical and social world and that when those expectations are violated, effort is needed to process the unexpected: For example, infants show greater pupil dilation when viewing physically impossible events (Jackson & Sirois, 2009). However, there has been little developmental research using pupillometry to assess effort in working memory. This is surprising given that working memory and the control of attention are intimately tied (Kane & Engle, 2003), and the use of the pupillometry in the study of memory has a rich history (Kahneman & Beatty, 1966). In adults, the pupil has been used to measure the influence of effort on trial-by-trial working memory performance (Unsworth & Robison, 2017) and exploited to determine the relative effort of older and younger adults in working memory tasks (e.g., older adults need to try harder to achieve the same performance as younger ones; Piquado, Isaacowitz, & Wingfield, 2010). Importantly, Chiew and Braver’s (2014) work demonstrated that it is effort per se (which they manipulated through rewards), not memory load directly or positive affect (looking forward to receiving a reward), that drives the pupil.

Effort Modulates Memory

Accounting for effort is especially important in the study of working memory because this is the system responsible for maintaining information in an active state in service to an ongoing task (Baddeley, 1992). Early developmental work used heart rate to identify periods of sustained attention when the encoding of visual information into memory was especially efficient (when 3- to 6-month-olds were shown a stimulus for 5 s during periods of sustained attention, this was sufficient to elicit a novelty preference equivalent to a baseline exposure time of 20 s; Richards, 1997). The link between infants’ sustained attention and subsequent memory encoding was also demonstrated using event-related potentials (ERPs; Richards, 2003) and electroencephalograms (Begus, Southgate, & Gliga, 2015). These methods are useful but lack temporal sensitivity (heart rate) or are relatively cumbersome to use with infants (electroencephalogram). Pupillometry offers a practical, validated measure in which responses can be collected at a subsecond time scale (the lower bound for task-evoked change is likely 220 ms; see Mathôt, Fabius, Van Heusden, & Van der Stigchel, 2018) during unrestrained viewing of stimuli on the screen of an eye tracker, which is particularly important in infant studies.

That said, there had not been any use of pupillometry to assess infants’ effort in working memory tasks until very recently. There had been some work on long-term memory and effort; a recent study found that 7-month-olds but not 4-month-olds showed larger pupil dilation to previously seen items compared with novel ones (Hellmer, Söderlund, & Gredebäck, 2018), replicating the effect found in adults. In addition, Sonne, Kingo, and Krøjgaard (2017) found a positive correlation between pupil dilation and memory retrieval in an imitation task in 20-month-old toddlers over a 2-week period. Until the work of our group (Cheng, Kaldy, & Blaser, 2019a), the relationship between effort (as measured by the pupil response) and working memory had been studied only in school-age children (7 years and up) who can reliably follow verbal instructions (E. L. Johnson, Miller Singley, Peckham, Johnson, & Bunge, 2014; Karatekin, Couperus, & Marcus, 2004). Our work established that just as in adults (Unsworth & Robison, 2017), moment-to-moment fluctuations in cognitive effort modulate infants’ working memory performance. For instance, in a group of 13-month-old infants from a recent study of ours (Cheng et al., 2019a), the highest quartile of participants—based on their average pupil dilation during the presentation of to-be-remembered stimuli—achieved 66% correct, and the lowest quartile performed at chance.

Effort Affects Visual Working Memory (VWM) Performance in Infants

To study infants’ working memory, we designed a novel delayed-match-retrieval (DMR) paradigm (Kaldy, Guillory, & Blaser, 2016), a memory game based loosely on the card game Concentration, adapted for infants and based on anticipatory eye movements (see Fig. 1a). This paradigm inverts the classic delayed-match-to-sample task that has been used by memory researchers for close to 80 years. In the classic task, a sample object is presented (e.g., a star) and then removed. After a delay period, two objects are presented: a match (e.g., another star) and a nonmatch (e.g., a heart). An adult would be asked, or an animal would be trained with rewards, to pick out the match. Importantly, DMR inverts this procedure: Potential matches or nonmatches are presented first and then covered. After a delay, the sample is revealed; the participant then needs to find the (now hidden) match. This inversion means that high performance is achieved only when all object-location bindings are successfully maintained in memory during the delay. Infants are encouraged to do this through visual rewards that are presented at the location of the match (after the end of the response period). Each time, to-be-remembered objects are selected from a small set with replacement, ensuring that items in working memory have to be updated in each trial. We present the events on the screen of an eye tracker and measure which of the face-down cards (match or nonmatch) infants look at first after the sample was exposed, in anticipation of the reward animation. Compared with traditional change-detection tasks used with infants (Ross-Sheehy, Oakes, & Luck, 2003), this task has some more naturalistic elements; for example, the cards flip instead of having a sudden onset and offset, and the retention period (2–5 s) is in the time frame of everyday tasks that infants face. But note that infants’ performance is limited by how well they can learn the simple rule of the game (“Find the match!”). We have shown that 10-month-old infants (but not 8-month-olds) can perform above chance with two to-be-remembered objects in this task (Kaldy et al., 2016) and that 25-month-olds can track the identities of objects even if they moved during occlusion (Cheng, Kaldy, & Blaser, 2019b).

Fig. 1.

Illustration of and results from the delayed-match-retrieval paradigm. On each trial (a), 13-month-old infants (N = 22) watched an animation in which three face-down cards entered the screen. Two of the cards then flipped face up sequentially to show different faces (e.g., a ball and a dog) and then flipped back face down. The third card, which matched one of the two (now face down) cards, then flipped face up. A delay of 3 s then ensued, during which eye movements and pupil diameter were monitored. After this 3-s response phase, a brief reward animation occurred at the location of the match card (simultaneously, the card was flipped face up). This was designed to encourage infants to fixate on the location of the (face down) match in anticipation of the reward. During the response phase, before the onset of the reward, if infants fixated the match before the nonmatch, this was coded as a correct response. Group-based pupil dilation across time (b) is shown separately for infants whose performance was above and below the median. Infants were median split on the basis of their visual working memory (VWM) performance. We found that infants who performed better overall in the memory task had significantly larger pupil dilation during encoding. The scatterplot (c) shows individual infants’ VWM performance as a function of their average pupil diameter during encoding and maintenance (the second half of the period shown in b). The best-fitting regression line is also shown. Figure adapted from Cheng, Kaldy, and Blaser (2019a).

To assess the role of cognitive effort in infants’ memory performance, we tested 13-month-olds in the DMR paradigm and analyzed both their behavioral and pupil responses (Cheng et al., 2019a). We found that infants who had larger pupils during the memory-encoding phase of trials had better subsequent memory performance (Figs. 1b and 1c). (The relationship between pupil size and performance was present on a trial-by-trial basis as well; trials in which pupil dilation was larger during encoding were more likely to end in success.) Looking at this relationship across all participants allowed us to determine the function relating pupil dilation to performance: Every 0.1 mm of pupil dilation driven by task-relevant effort resulted in an approximately 4% increase in performance. Recently, Ross-Sheehy and Eschman (2019) analyzed pupil responses in infants (and adults) during a change-detection task with a 500-ms delay and found a differential response to the trial outcome (i.e., the pupil differentiated a change outcome from a no-change outcome). However, they did not use the pupil to map the relationship between infants’ effort (i.e., as measured by the pupil during memory encoding) and memory performance.

What causes these moment-to-moment fluctuations in effort and what determines individual differences in the frequency of high-effort moments are especially important questions to address in future studies. Several studies have looked at individual differences in different aspects of infants’ attentional control and their developmental trajectory. For example, infants’ focused attention in play-based tasks at 9 to 12 months can predict their effortful control scores on parent-report surveys 1 to 2 years later (Johansson, Marciszko, Gredebäck, Nyström, & Bohlin, 2015; Kochanska, Murray, & Harlan, 2000). These findings suggest that the early ability to control and exert task-relevant effort has a certain amount of individual stability and longitudinal predictive value. Pupillometry could be a fruitful method to build a mechanistic model of effortful control development from infancy to childhood in future longitudinal studies.

Accounting for the Role of Effort

There are two ways in which accounting for effort and differences in effort between contexts, individuals, and groups can affect theories of cognitive development. First, as discussed above, ignoring variability in effort may lead to a systematic underestimation of infants’ abilities. In the domain of working memory, this means underestimating capacity. By isolating high-effort moments, we can determine best-case capacity, and this best-case memory may support cognitive abilities that the average, underestimated value does not. For instance, current estimates of VWM capacity seem to show that infants can reliably maintain only one object in VWM until about 8 months of age (Kaldy & Leslie, 2005; Ross-Sheehy et al., 2003). These limits would radically constrain higher cognitive processing in infants. It would mean, for example, that a young infant could not evaluate binary relationships—same/different, bigger/smaller, old/new—on the basis of representations that are held in memory. This is in conflict with some highly influential findings in the infant-cognition literature. For example, Johnson and colleagues (2009) reported that 5-month-old infants could learn a rule that was implemented on three serially presented multimodal objects. In addition, Wynn’s (1992) seminal two-object tracking task could not be solved by 5-month-olds if their VWM could hold only a single object. We anticipate that once the modulatory effect of effort is accounted for in future studies of VWM capacity development, these quantitative estimates will be revised upward, helping to resolve these discrepancies.

Second, because poor performance in a task may reflect lower effort rather than weaker ability, one should resist attributing differences in performance to differences in domain-specific cognitive processes before accounting for the effect of differences in effort. For instance, in a visual search study, we found that 2-year-olds diagnosed with autism spectrum disorder (ASD) could reliably outperform typically developing (age-matched) toddlers. We tentatively attributed this finding to visual attentional differences (perceptual enhancement), but subsequent pupillometric analyses revealed that pupil diameter during search predicted success at finding the target in both groups. We found that toddlers diagnosed with ASD were more frequently in a high-effort mode during search than the typically developing control subjects (Blaser, Eglington, Carter, & Kaldy, 2014). That is, toddlers with ASD outperformed control subjects not because they had better visual skills or visual search strategies but because they tried harder more often. Even in cases in which it does not explain observed differences, accounting for effort helps to refine theoretical accounts. For instance, in adults, whereas greater effort (manipulated by incentives and as measured by the pupil) is associated with better performance in a complex span working memory task, it cannot fully explain the difference in performance between high-span and low-span individuals (Heitz, Schrock, Payne, & Engle, 2008).

Future Directions

Cognitive effort is a ubiquitous modulator of task performance capable of determining success or failure. Recently, the construct of cognitive effort has become central in computational studies of decision making and other higher-level cognitive processes (e.g., Shenhav et al., 2017). In developmental work, we see two areas of inquiry in which leveraging pupillometry as a measure of effort will be of critical importance: (a) determining the contexts (tasks, stimuli, and incentives), individual differences, and group traits that determine patterns of effort and (b) using epochs of maximal effort to improve estimates of underlying competence and reassess developmental trajectories of cognitive processes. We expect these inquiries to be especially fruitful in the domain of working memory and decision making, processes that require effortful cognitive control.

Footnotes

Transparency

Action Editor: Randall W. Engle

Editor: Randall W. Engle

ORCID iD

Zsuzsa Kaldy

References

Alnæs

Sneve

M. H.

Espeseth

Endestad

van de Pavert

S. H. P.

Laeng

(2014). Pupil size signals mental effort deployed during multiple object tracking and predicts brain activity in the dorsal attention network and the locus coeruleus. Journal of Vision, 14(4), Article 1. doi:10.1167/14.4.1

Aslin

R. N.

(2012). Infant eyes: A window on cognitive development. Infancy, 17, 126–140.

Aston-Jones

Cohen

J. D.

(2005). An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annual Review of Neuroscience, 28, 403–450.

Baddeley

(1992). Working memory. Science, 255, 556–559.

Begus

Southgate

Gliga

(2015). Neural mechanisms of infant learning: Differences in frontal theta activity during object exploration modulate subsequent object recognition. Biology Letters, 11(5), Article 20150041. doi:10.1098/rsbl.2015.0041

Blaser

Eglington

Carter

A. S.

Kaldy

(2014). Pupillometry reveals a mechanism for the autism spectrum disorder (ASD) advantage in visual tasks. Scientific Reports, 4, Article 4301. doi:10.1038/srep04301

Cheng

Kaldy

Blaser

(2019a). Focused attention predicts visual working memory performance in 13-month-old infants: A pupillometric study. Developmental Cognitive Neuroscience, 36, Article 100616. doi:10.1016/j.dcn.2019.100616

Cheng

Kaldy

Blaser

(2019b). Two-year-olds succeed at MIT: Multiple identity tracking in 20- and 25-month-old infants. Journal of Experimental Child Psychology, 187, Article 104649. doi:10.1016/j.jecp.2019.06.002

Chiew

K. S.

Braver

T. S.

(2014). Dissociable influences of reward motivation and positive emotion on cognitive control. Cognitive, Affective, & Behavioral Neuroscience, 14, 509–529.

10.

Heitz

R. P.

Schrock

J. C.

Payne

T. W.

Engle

R. W.

(2008). Effects of incentive on working memory capacity: Behavioral and pupillometric data. Psychophysiology, 45, 119–129.

11.

Hellmer

Söderlund

Gredebäck

(2018). The eye of the retriever: Developing episodic memory mechanisms in preverbal infants assessed through pupil dilation. Developmental Science, 21(2), Article e12520. doi:10.1111/desc.12520

12.

Hess

E. H.

Polt

J. M.

(1964). Pupil size in relation to mental activity during simple problem-solving. Science, 143, 1190–1192.

13.

Jackson

Sirois

(2009). Infant cognition: Going full factorial with pupil dilation. Developmental Science, 12, 670–679.

14.

Johansson

Marciszko

Gredebäck

Nyström

Bohlin

(2015). Sustained attention in infancy as a longitudinal predictor of self-regulatory functions. Infant Behavior & Development, 41, 1–11.

15.

Johnson

E. L.

Miller Singley

A. T.

Peckham

A. D.

Johnson

S. L.

Bunge

S. A.

(2014). Task-evoked pupillometry provides a window into the development of short-term memory capacity. Frontiers in Psychology, 5, Article 218. doi:10.3389/fpsyg.2014.00218

16.

Johnson

Fernandes

Frank

Kirkham

Marcus

Rabagliati

Slemmer

(2009). Abstract rule learning for visual sequences in 8- and 11-month-olds. Infancy, 14, 2–18.

17.

Kahneman

(1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.

18.

Kahneman

Beatty

(1966). Pupil diameter and load on memory. Science, 154, 1583–1585.

19.

Kaldy

Guillory

S. B.

Blaser

(2016). Delayed match retrieval: A novel anticipation-based visual working memory paradigm. Developmental Science, 19, 892–900.

20.

Kaldy

Leslie

A. M.

(2005). A memory span of one? Object identification in 6.5-month-old infants. Cognition, 97, 153–177.

21.

Kane

M. J.

Engle

R. W.

(2003). Working-memory capacity and the control of attention: The contributions of goal neglect, response competition, and task set to Stroop interference. Journal of Experimental Psychology. General, 132(1), 47–70.

22.

Karatekin

Couperus

J. W.

Marcus

D. J.

(2004). Attention allocation in the dual-task paradigm as measured through behavioral and psychophysiological responses. Psychophysiology, 41, 175–185.

23.

Kochanska

Murray

K. T.

Harlan

E. T.

(2000). Effortful control in early childhood: Continuity and change, antecedents, and implications for social development. Developmental Psychology, 36, 220–232.

24.

Laeng

Sirois

Gredebäck

(2012). Pupillometry: A window to the preconscious? Perspectives on Psychological Science, 7, 18–27.

25.

Mathôt

Fabius

Van Heusden

Van der Stigchel

(2018). Safe and sensible preprocessing and baseline correction of pupil-size data. Behavior Research Methods, 50, 94–106.

26.

Piquado

Isaacowitz

Wingfield

(2010). Pupillometry as a measure of cognitive effort in younger and older adults. Psychophysiology, 47, 560–569.

27.

Porter

Troscianko

Gilchrist

I. D.

(2007). Effort during visual search and counting: Insights from pupillometry. Quarterly Journal of Experimental Psychology, 60, 211–229.

28.

Richards

J. E.

(1997). Effects of attention on infants’ preference for briefly exposed visual stimuli in the paired-comparison recognition-memory paradigm. Developmental Psychology, 33, 22–31.

29.

Richards

J. E.

(2003). Attention affects the recognition of briefly presented visual stimuli in infants: An ERP study. Developmental Science, 6, 312–328.

30.

Ross-Sheehy

Eschman

(2019). Assessing visual STM in infants and adults: Eye movements and pupil dynamics reflect memory maintenance. Visual Cognition, 27, 78–92.

31.

Ross-Sheehy

Oakes

L. M.

Luck

S. J.

(2003). The development of visual short-term memory capacity in infants. Child Development, 74, 1807–1822.

32.

Sara

S. J.

(2009). The locus coeruleus and noradrenergic modulation of cognition. Nature Reviews Neuroscience, 10, 211–223.

33.

Shenhav

Musslick

Lieder

Kool

Griffiths

T. L.

Cohen

J. D.

Botvinick

M. M.

(2017). Toward a rational and mechanistic account of mental effort. Annual Review of Neuroscience, 40, 99–124.

34.

Slaughter

Suddendorf

(2007). Participant loss due to “fussiness” in infant visual paradigms: A review of the last 20 years. Infant Behavior & Development, 30, 505–514.

35.

Sonne

Kingo

O. S.

Krøjgaard

(2017). Bound to remember: Infants show superior memory for objects presented at event boundaries. Scandinavian Journal of Psychology, 58, 107–113.

36.

Unsworth

Robison

M. K.

(2017). The importance of arousal for variation in working memory capacity and attention control: A latent variable pupillometry study. Journal of Experimental Psychology. Learning, Memory, and Cognition, 43, 1962–1987.

37.

van der Wel

van Steenbergen

. (2018). Pupil dilation as an index of effort in cognitive control tasks: A review. Psychonomic Bulletin & Review, 25, 2005–2015.

38.

Wynn

(1992). Addition and subtraction by human infants. Nature, 358, 749–750.

Putting Effort Into Infant Cognition

Abstract

Keywords

Pupillometry as a Measure of Cognitive Effort

Effort Modulates Memory

Effort Affects Visual Working Memory (VWM) Performance in Infants

Accounting for the Role of Effort

Future Directions

Recommended Reading

Footnotes

Transparency

ORCID iD

References