Abstract
Meaning-making and temporal memory are closely intertwined, yet we still do not know how the overall understanding of complex events affects retrospective temporal judgments. The present study investigated the effect of a manipulation of the temporal linearity of a narrative on the subsequent memory-for-time performance. Participants indicated the time of occurrence of short video clips extracted from a previously encoded movie on a horizontal timeline representing the movie duration. Importantly, a group of participants (N = 20) watched the original movie, which depicts events occurring in chronological order, whereas another group (N = 30) watched a scrambled version of the same movie in which the temporal linearity was lost. This procedure allowed us to measure the quantity and direction of the temporal memory bias. The scrambled presentation produced a mild and general impairment of recognition memory compared to the linear presentation. More importantly, it biased temporal judgments as a function of the direction and amount of discrepancy between the story and the viewing time, in accordance with an automatic reshaping of temporal memory caused by a chronological representation of the storyline. This effect could be distinguished from a tendency to move judgments toward the center of the timeline, independently from the specific scrambling arrangement, consistent with the idea that the non-linearity of the story also generally increased the degree of temporal uncertainty. Taken together, our results provide further evidence that temporal memories are automatically reconstructed according to the general meaning of the events.
Introduction
The ability to place past events in time is essential for making sense of the world and modelling experience. Indeed, we have an inherent chronological sense of the past, according to which events are preceded and followed by other events, and this feature contributes strongly to the sense that our lives unfold in time (Friedman, 1993, 2004). However, it is not clear whether the global meaning of the event itself, once established, can influence our retrospective judgments of temporal memory. In particular, it may be possible that the temporal placement of specific events in memory is altered by the overall interpretation of what happened and the attribution of semantic/contextual bindings between different events. For instance, when your friends throw you a surprise party, you suddenly remember all the clues that should have led you to guess it earlier. How does this change the temporal memory of what happened? Reconstructive theories of memory (e.g., Bartlett, 1932; Conway & Pleydell-Pearce, 2000; Friedman, 1993; Loftus, 1993; Ross, 1989; Schacter, 2012; Spiro, 1980) suggest that temporal information is not directly associated with the event, but is inferred by combining aspects of the memory with meaningful temporal patterns (Friedman, 1993, 2004, 2005), which can be both personal/autobiographical (e.g., places where one has lived) and non-personal (e.g., conventional/social) in nature. For example, when participants are asked to recall the time of an event that occurred a few months ago, they reconstruct its location in time by relying on its contiguity with a routine event (Friedman, 1987) or on specific aspects of the remembered event that serve as a temporal clues for inferential processes (e.g., the weather to locate the event within a summer vacation; Friedman & Wilkins, 1985).
Recently, research on how we encode and retrieve complex event representations, including their temporal organization, has been aided by the use of popular movies and other types of narratives (Chang et al., 2022; Lee, Chen & Bellana, 2020; Nastase et al., 2019), as the use of visually rich and dynamic lifelike stimuli allows for both experimental control and ecological validity (Furman et al., 2007). This approach is also based on the idea that “fundamental aspects of event cognition are shared for real and fictional events” (Radvansky & Zacks, 2014), and that it would therefore be possible to use narrative material to investigate how people understand and remember everyday events. Using such narrative material, several studies have suggested that the meaning-making process may have some influence on our temporal memory judgments. For example, narrative comprehension is thought to rely on the accumulation of ongoing information, the segmentation into sub-events (“situation models” or “event models”; Zacks & Radvansky, 1998), and the construction of a coherent representation (Song et al., 2021). It has been shown that participants are more precise at judging the temporal occurrence of information presented near event boundaries (Frisoni et al., 2021), and the age-related decline observed for order memory is associated with an impoverished ability to segment events (Zacks et al., 2006). However, while these studies have focused on the ‘local’ effect of comprehension (i.e., segmentation into subevents), it is still unclear how the ‘global’ event representation (i.e., the overall meaning of the episode) affects temporal memory.
Schematic or script knowledge (Alba & Hasher, 1983; Bartlett, 1932; Ghosh & Gilboa, 2014; Rumelhart, 1980; Schank & Abelson, 1977) is also thought to influence both the encoding and the retrieval of complex events by conveying the temporal structure of stereotypical situations. For example, people prefer to recall script actions (e.g., eating in a restaurant) in their familiar order (Bower et al., 1979). Furthermore, research on “story grammar” (Kintsch & van Dijk, 1978; Mandler, 1978; Rumelhart, 1975; Stein & Glenn, 1979; Thorndyke, 1977) has shown that we tend to recall narratives according to a conventional/coherent storyline by making rational distortions (Bartlett, 1932) or adding parts to their story recall (Bower et al., 1979). Looking for evidence for the effect of story schema on temporal memory for complex events, in previous work, we compared the ability to judge the time-of-occurrence of video clips extracted from a previously encoded movie under different conditions that varied in the amount of cutting at encoding (complete vs. incomplete movie; Frisoni et al., 2021) but also in the position of the cut (final vs. middle cut; Frisoni et al., 2022). We found that removing the final or the middle part of the movie from the encoding session resulted in a systematic bias in memory for time. The results of these studies indicate that temporal memory judgments for movie scenes can be biased to make room for missing information and thus fit a standard template, or story schema, regardless of what was actually encoded. Importantly, while these results suggest the involvement of reconstructive memory processes when participants fill in unseen, but highly likely, story parts, the evidence for an actual event reconstruction mechanism is only indirect. A more direct way of identifying reconstructive processes in temporal memory would be to explicitly manipulate the temporal order of the narrative, for example by presenting a movie in which scenes are presented out of order, and testing the extent to which memory for time performance reflects the chronological reconstruction of the unfolding events as inferred from story comprehension. Looking at previous investigations on the matter, there is some qualitative evidence that people tend to recall stories in their canonical form (Mandler, 1978) but, to our knowledge, this process has not been investigated quantitatively. In particular, the quantitative approach would make it possible to test directly whether and to what extent temporal judgments change as a function of the mismatch between the time of occurrence of an event and its position in an overall meaningful representation. Most importantly, these results have been found using material such as short written stories, but to our knowledge there is limited evidence that they can be extended to more ecological material.
For example, according to the so-called “iconicity assumption” (Zwaan, 1996), readers assume that the order in which situations are narrated corresponds to the chronological order of the events. In other words, the assumption reflects an expectation of temporal linearity, i.e., the fact that events are narrated in the (chronological) order in which they occurred. We believe that the coherence and logic of the story’s meaning is closely related to the degree of temporal linearity, which facilitates the understanding of cause-effect relationships within the story. However, the two temporal orders are sometimes incongruent in everyday communication (e.g., conversations), and the manipulation of linearity is increasingly being used by screenwriters and directors (e.g., consider non-linear movies such as Pulp Fiction (Tarantino, 1994) or Tenet (Nolan, 2020) or Tenet, in which events are presented out of sequence, with massive use of flashbacks, flash-forwards and parallel narratives) because it contributes to the aesthetic experience of the viewer (Cutting, 2021).
Importantly, movies provide a relatively straightforward way of investigating the relationship between the global meaning of events (hereafter defined as “story time”) and the events’ actual time of occurrence (“viewing time”). A recent study using audiovisual material has provided evidence that people can rely on story time when the two temporal flows are dissociated (Xu et al., 2019). Specifically, participants were asked to watch scrambled video narratives and then judge which of two images was presented earlier in the video (i.e., viewing time) or occurred earlier in the story (i.e., story time). Half of the trials were presented in a congruent condition (matching viewing time and story time), while the other half were presented in an incongruent condition. The authors found that viewing time judgments were biased by the story time when participants were provided with temporal landmarks taken from the story world. This might be taken to suggest that making sense of the experience and remembering when an event occurred are inextricably intertwined. However, this effect does not provide information about the location of events in time, but it is limited to the temporal expectation that successive movie scenes describe successive events. Instead, our main hypothesis is that temporal memory judgments are based on a meaningful temporal pattern (Friedman, 1993, 2004, 2005), which, in this case (i.e., the case of ecological encoding material about movies and video narratives), reflects a meaningful and coherent representation of the story (i.e., storyline).
The aim of the present study is to quantify the potential reconstruction effect on temporal memory for complex events produced by the mismatch between the time of occurrence of an event and its position in an overall meaningful representation. Specifically, we investigated the effect of encoding a “scrambled” narrative, i.e., a movie divided into 6 randomly re-arranged parts, on temporal judgments about the encoded material to examine the relationship between the understanding of a meaningful event and memory for time. Participants viewed either a linear or a scrambled movie and were subsequently asked to indicate the time of occurrence (“viewing time” judgment) of short video clips extracted from the movie using a visual analog scale representing the timeline of stimulus presentation. According to a non-reconstructive/fidelity account of memory for time, a linear relationship would be expected between the time of occurrence judgments and viewing time (viewing time-based retrieval; Figure 1(A)). In other words, if observers simply rely on what they have been shown, and thus on the mere time of occurrence, their judgments should be based on viewing time only. In this case, memory-for-time performance in the scrambled condition should exactly parallel performance in the original linear version of the task (Frisoni et al., 2021). If, instead, temporal judgments are based on a story representation that includes both temporal and non-temporal knowledge about the encoded material, the mismatch between viewing and story time should generate a systematic shift of temporal judgments towards the storyline. In other words, the amount and direction of positioning errors should be directly related to the amount and direction of the shift between the viewing and the story time (reconstructive effect; Figure 1(B)). Lastly, an additional distinction must be made between the fidelity/reconstructive accounts and another potential effect of the scrambling procedure on temporal memory judgments (central attraction effect; Figure 1(C)). This alternative account predicts a non-specific decrease in performance due to the incongruence between viewing time and story time (Xu & Kwok, 2019), leading to temporal confusion. In particular, unlike the first two accounts, which are associated with a specific temporal pattern of temporal memory judgments, the temporal confusion created by the temporal scrambling may lead to a general bias towards placing clips towards the center of the timeline. One problem in disentangling the reconstructive from the central attraction effect is that the former account also predicts a relative midline attraction effect, since the scrambling procedure inevitably causes initial parts to be shifted forward in time and vice versa. Therefore, to disentangle the two accounts, different subgroups of participants were exposed to different versions of the scrambled movie. Specifically, we counterbalanced the order of presentation of the different story parts so that each of the six parts could be presented in each of the six possible positions in different subgroups of subjects, while keeping the average amount of story part displacement constant across subgroups. The rationale for this manipulation was that the precise order of presentation of the parts should only matter for the reconstructive account, whereas the confusion account always makes the same prediction, regardless of the particular scrambling pattern. The figure illustrates the three predictions for the “scrambled version c” (see Figure 2). On the left column, the sequence of shots from left to right indicates the viewing time, whereas red numbers indicate the “story time”. The arrows indicate the expected systematic errors corresponding to the predictions. Rightward arrows represent a forward shift (overestimation), whereas leftward arrows represent a backward shift (underestimation) in temporal memory judgments. The length of the arrow indicates the expected amount of shift. On the right column, each prediction is translated in terms of relative error. Black numbers represent the viewing parts. (A). Viewing time-based retrieval. Each story part is remembered according to the viewing time, that is, the time it was viewed by the subject. (B). Reconstructive effect. Each story part is over- or underestimated depending on the difference between the linear and the scrambled presentation. (C). Central attraction effect. Each story part is over- or underestimated depending on the distance from the center of the analog scale, independent of the story time.
Method
Participants
Thirty right-handed volunteers (26 females; aged 20–30; mean age: 21.5 years), with no familiarity with the movie show, participated in the study. Participants were naive as to the purpose of the experiment, reported normal or corrected-to-normal vision, and gave informed consent prior to the experiment according to the guidelines of the Human Studies Committee of G. D’Annunzio Chieti University. Participants were randomly assigned to one of six conditions, resulting in 5 participants per condition. The study consisted of an encoding session (duration: approx. 90 min) followed by a retrieval session (duration: approx. 30 min). The two sessions were separated by a ∼24 hrs interval (Figure 2(A)-(B)). Experimental paradigm & scrambling procedure. The figure illustrates the structure of the experimental paradigm and the scrambling procedure. (A). In the encoding phase, participants were presented with an episode from the TV series “Sherlock” (B). In the retrieval phase, participants were presented with a series of 2 secs video clips, extracted either from the previously encoded movie or from an unseen, related, movie, and were asked to perform an item recognition judgment (old/new) followed by a temporal memory judgment using the touchpad of their personal laptop. For clips judged as old, participants were also asked to indicate the time of occurrence of the video clip using a visual analog scale representing the entire duration of the video. An ITI of 2 secs with a central fixation cross preceded the next trial. (C). To create scrambled movies, the linear episode (viewing time = story time) was divided into six parts. The parts were recombined and presented in scrambled order (viewing time ≠ story time). In the figure, each letter (a–f) on the left corresponds to a “scrambled” version of the movie whereas each shot represents a story part. The red numbers (1–6) indicate the “story parts”, whereas the black horizontal arrow indicates the viewing time (thus the order of the cells from left to right indicates the order of the “viewing parts”). As a result of the scrambling procedure, a given story part could be seen in all the possible temporal positions across different groups of subjects.
Materials and Procedure
We created six scrambled versions of the same TV episode (BBC Sherlock, Season 1, Episode 1, “A Study in Pink”; duration: 87:30 min) by splitting the original movie (which can be defined as the “linear” version of the story) into six parts, which were then edited together in a random order (see Figure 2(C)). In other words, each version contained the entire episode, with identical parts presented out of order. This procedure allowed us to assign each “story part” to each possible “viewing” position (see Figure 2(C)) and, thus, to systematically assess the effect of temporal scrambling on memory for time performance. The experimental stimuli for the retrieval session consisted of a series of 2 sec video clips extracted from the encoded Sherlock episode (old stimuli) or another, unseen, episode of the same show (Episode 2, “The Blind Banker”) (new stimuli). Both old [N = 108 + 6 for practice] and new [N = 108 + 6] stimuli were sampled uniformly every 43.75 s (this value refers to the center of the clip) from the beginning of the movie.
The encoding and retrieval phases of the “scrambling movie” condition were performed remotely due to COVID-19 restrictions. Participants were instructed to complete both experimental sessions on their laptop in a quiet room and to maintain a distance of approximately 60 cm from the screen. In the encoding session (Figure 2(A)), participants were presented with a scrambled version of the Sherlock episode. None of the participants watched the original/linear movie. Participants were asked to watch the video only once, without pausing, and were not informed of the nature of the following task. In the retrieval session (Figure 2(B)), participants were asked to provide judgments about the time of occurrence of video-clips extracted from the seen (scrambled) episode using a visual analog scale. The stimuli were presented using the online survey platform Qualtrics XM. Each trial started with the presentation of the clip at the center of the screen. Participants made an item recognition judgement (old/new) using the touchpad. Following old responses, participants were asked to provide a memory for time judgment on a central horizontal grey line (size: 0.9° × 19.8° visual angle) consisting of 720 consecutive segments, each corresponding to 7.29 s of the movie. Participants used the touchpad to select a point on the line that indicated the time at which the clip occurred in the encoded video (i.e., time of occurrence/viewing time). The timeline remained on the screen until a judgment was given, followed by an intertrial interval (ITI) of 2 s. No feedback was provided.
Importantly, the instructions made it clear that the line represented the duration of the video that was shown in the encoding session (with the extreme left corresponding to the beginning of the video, and the extreme right corresponding to the end of the video), and that the task was to indicate the position of each clip according to viewing time. At the end of the experiment, participants were interviewed to test whether any potential bias could be accounted for a conscious/intentional distortion (e.g., any potential distortion could have reflected a misinterpretation of the task), and to ensure that they had not voluntarily placed the clips according to a linear narrative (timeline of the story/story time), but simply according to the time of occurrence in relation to the encoded video (timeline of the video/viewing time). The procedure for the remote experiment was as follows: each participant received the specific temporally scrambled version of the movie via an online platform and had to download it. The experimenter received a notification when the download was complete. The following day, the experimenter sent the link to the memory task. Clicking on the link started the experiment on the participant’s laptop. At the end, the dataset and the running times were stored on the platform.
For the “linear movie” condition, we used the dataset from the study by Frisoni and colleagues (2021; Exp. 1, N = 20). In this study, the movie was presented in the original format. Crucially, the original movie presents the sequence of events in chronological order and the plot presents a conventional and coherent trajectory, with a strong causal structure (introduction, complication, development and climax; Cutting, 2021; Frisoni, 2021). The procedure of the linear presentation was like the one used in the present study, except for the order of the story parts at encoding (the movie was presented in its original format) and the experimental setting (i.e., the experiment was conducted in the laboratory using E-Prime 2.0 software, Psychology Software Tools).
Data Analysis
To quantify variations of performance as a function of the clip position, we divided the movie into six equal parts of ∼15 min each (hereafter referred to “Story Parts”) and used the part as an independent variable in a 1-way repeated-measures analysis of variance (ANOVAs). This arbitrary division was chosen as a compromise between temporal sampling and the presence of a sufficient number of clips in each part (N = 18). Analyses of item recognition accuracy were conducted selectively on old clips (hit ratio) while analyses of memory for time judgments were restricted to correctly recognized old clips (hits). The precision of memory for time was calculated as the temporal distance, quantified in seconds, between the line segment selected by the subject and the correct segment for each trial. The placement error was calculated with respect to the center of each video clip. Two parameters were considered: absolute and relative error. Compared to absolute error, relative error further considers error direction (forward or backward) and allows an estimate of directional bias.
First, to test for the presence of a general effect of the scrambling procedure on episodic memory, we compared the task performance, in terms of item recognition and temporal precision, in the scrambled versus the linear condition (data from our previous study, Frisoni et al., 2021, Experiment 1). Item recognition memory in the scrambled condition was calculated by averaging across all the scrambled versions. We conducted a Mixed ANOVA with Encoding Type (Linear vs. Scrambled) as the between-subject factor and Viewing Part (Part 1–6, independent of “Story Part”) as the within-subject factor on the hit rate, where a hit corresponds to a correctly identified old item. Recognition memory for each Viewing Part for the Scrambled condition was averaged across subgroups (e.g., Viewing Part 1 across the a-f versions, see Figure 2(C)). To test more directly whether recognition performance depended on the Story Part, regardless of the specific scrambled arrangement at encoding (Ferguson et al., 2017), we created a “Linearized Scrambled” condition from the Scrambled condition by averaging recognition and temporal memory scores for each Story part, irrespective of Viewing Part (e.g., Story Part 1 across the a-f versions, see Figure 2(C)). Next, we tested for the presence of a general effect of the scrambling procedure on temporal memory, independent of the scrambling type, by comparing the amount of absolute and relative error between the linear and the scrambled conditions. A Mixed ANOVA with Encoding Type (Linear vs. Scrambled) as the between-subject factor and Viewing Part (Part 1–6) as the within-subject factor was conducted for each error index. A series of one-sample t-tests (two-tailed) with Bonferroni correction (N = 6) were also conducted for each Viewing Part in each encoding condition to determine the presence of a significant under- or overestimation bias of the temporal judgments.
As for the item recognition, the same analyses were performed with the Viewing Part factor replaced by the Story Part factor (i.e., Linearized scrambled), to test whether the precision of temporal memory depended on the Story Part, independent of viewing time.
Second, we focused exclusively on the Scrambled condition to test whether the precision of temporal memory varied as a function of either the relative distance between the viewing and the story time, consistent with a “reconstructive hypothesis” or the relative distance from the middle of the visual analog scale, consistent with a “central attraction hypothesis”. For each scrambled version (a-f versions), we took the mean relative error (calculated across 5 subjects) for each of the six viewing parts (resulting in a total of 36 cells). To test the “reconstructive hypothesis”, we calculated the distance between each “Viewing Part” and its position in the linear story (“viewing/story time distance”). For example, considering the scrambled version “a” (see Figure 2(C)), the distance for Part 1 is equal to zero (because Viewing Part 1 corresponds to Story Part 1) whereas the distance for Part 2 is equal to +2 (because Viewing Part 2 should be shifted towards Story Part 4). This way, for each of the 36 cells, we obtained a “viewing/story time distance” value ranging from −5 to +5. Importantly, these values reflected not only the amount (small vs. large), but also the direction (negative vs. positive) of the hypothesized temporal reconstruction. We then performed a Spearman Rank Order Correlation Test between the relative error and the temporal distance between viewing and story time (viewing/story time distance). A similar procedure was used to test the “central attraction hypothesis”. This time, we calculated the distance between each “Viewing Part” and the center of the timeline (“center distance”). More specifically, Viewing Parts 3 and 4 were considered to be in the center of the timeline, and thus had a distance value of 0, Viewing Parts 2 and 5 had a distance value of +1 and −1 respectively, and viewing Parts 1 and 6 had a distance value of +2 and −2 respectively. Thus, for each of the 36 cells, we obtained a “center distance” value ranging from −2 to +2. For example, considering the scrambled version “a” (see Figure 2(C)), the distance for Part 1 is +2 (because Viewing Part 1 should be shifted towards Viewing Part 3 to be placed in the center), whereas the distance for Part 3 is equal to zero (because Viewing Part 3 is considered to be in the center). Again, we performed a Spearman Rank Order Correlation Test between the relative error and the “center distance”.
Finally, to isolate the role of reconstructive processes from the central attraction bias, we estimated the presence of a reconstructive effect holding viewing time constant. To this end, we performed a trial-by-trial Spearman Rank Order Correlation between the “viewing/story time distance” described above, and the relative error for each clip of each Story Part (calculated across 5 subjects), separately for each of the six possible Viewing Parts. The use of this procedure ensures that the eventual presence of a significant correlation, supporting the “reconstructive hypothesis”, cannot be explained by the effect of the “center distance”, which is held constant in each of the six analyses (one for each Viewing Part). Note that for each Viewing Part, the analysis included data obtained from different groups of subjects, and thus different Story Parts (1–6). As a result, each correlation test was performed over 108 points, 18 trials by 6 Story Parts.
Results
Two mixed model ANOVAs tested the effect of the scrambling procedure on item recognition memory, independent of scrambling type. The first analysis (Figure 3) revealed a main effect of Encoding Type [F (1,48) = 4.89, p < .05] and Viewing Part [F (5,240) = 3.91, p < .005] and a significant interaction effect [F (5,240) = 4.44, p < .001]. The effect of Encoding Type was explained by greater recognition memory for the Linear (mean ± SD: 0.91 ± 0.04, black line) compared to the Scrambled condition (0.84 ± 0.12, grey line), suggesting that the scrambling procedure had an overall negative impact on recognition memory. The main effect of the Viewing Part was explained by greater accuracy for Part 3 (0.80 ± 0.17) compared to Part 1 (0.77 ± 0.22) and 4 (0.79 ± 0.19; p < .05). However, when the analysis was conducted by averaging recognition scores for each Story part, irrespective of Viewing Part (Linearized scrambled, red line in Figure 3), no significant interaction effect between Encoding Type and Story Part [F (5, 240) = 0.46, p = .80] was observed, suggesting that recognition performance crucially depends on the specific story part (or stimulus material), irrespective of its temporal position at encoding. Taken together, these results indicate that recognition performance was worse in the scrambled compared to the linear condition, but this general effect did not alter the relative pattern of recognition performance across story parts. The figure illustrates the group averaged recognition memory as a function of viewing/story part. The “Linear” (black) condition refers to the original version of the movie (Frisoni et al., 2021), and thus to the correspondence between the sequence of story (red axis) and viewing (x-axis) parts. The “Scrambled” (grey) condition refers to the “Viewing Parts” across groups, regardless of the “Story Parts” order. The “Linearized Scrambled” (red) condition refers to the “Story Parts” across groups, regardless of the order of the “Viewing Parts”. Error bars indicate SEM.
Two mixed model ANOVAs with Encoding Type as the between-subject factor tested the general effect of the scrambling procedure on the absolute error, one with Viewing Part and the other with Story Part as the within-subject factor (Figure 4). Comparing the linear (black line) with the scrambled (grey line) presentation in the first analysis, we found a main effect of Encoding Type [F (1, 48) = 44.70, p < .0001] and Viewing Part [F (5,240) = 2.66, p < .05] and a significant interaction effect [F (5,240) = 5.05, p < .001]. The interaction effect was explained by greater accuracy for Part 1 (mean ± SD: 371 ± 184 s) and Part 6 (328 ± 171 s) in the Linear condition compared to Part 1 (932 ± 680) and 6 (1391 ± 811) in the Scrambled one (all p < .005). Thus, the scrambling procedure reduced temporal memory precision and further abolished the advantage for extreme movie parts (first, last), observed with the linear presentation (Frisoni et al., 2021). A trend towards a significant interaction effect [F (5,240) = 2.24, p = .05] was also observed when Story Part was used as a within-subject factor (Linearized Scrambled, red line), further suggesting that, unlike for item recognition performance, the scrambling procedure affected the relative pattern of temporal precision across story parts. Furthermore, it is noteworthy that the greater temporal precision associated with the extreme (vs. middle) parts of the movie found in the Linear condition (see Frisoni et al., 2021; Exp. 1) is absent both when Viewing Time is isolated from Story Time (i.e., Scrambled condition, Figure 4, grey) and when Story Time is isolated from Viewing Time (i.e. Linearized condition, Figure 4, red). Thus, the congruence between Viewing and Story Time seems to be a prerequisite for the observation of this effect. The figure illustrates the group averaged absolute (on the left) and relative (on the right) errors as a function of viewing/story part. The “Linear” (black) condition refers to the original version of the movie (Frisoni et al., 2021), and thus to the correspondence between the sequence of story (red axis) and viewing (x-axis) parts. The “Scrambled” (grey) condition refers to the “Viewing Parts” across groups, regardless of the “Story Parts” order. The “Linearized Scrambled” (red) condition refers to the “Story Parts” across groups, regardless of the order of the “Viewing Parts”. Error bars indicate SEM.
The same analyses were conducted for the relative error, which provides information about systematic biases towards an over- or an underestimation (Figure 4). Using Viewing Part as a within-subject factor, we found a significant main effect of Viewing Part [F (5, 240) = 32.37, p < .0001] and an interaction between Encoding Type and Viewing Part [F (5, 240) = 23.38, p < .0001]. In particular, while no Viewing Part showed a consistent bias towards an over- or an underestimation in the Linear presentation (black line, all p > .05, Bonferroni corrected), clips extracted from the first half of the scrambled movie (Viewing Parts 1–3) were overestimated (grey line, all p < .005), whereas those extracted from the second half (Viewing Parts 4–6) were underestimated (all p < .0005; Figure 4). A significant interaction effect [F (5,240) = 2.61; p < .05] was also present when Story Part was used as the within-subject factor (Linearized Scrambled, red line), further indicating that the scrambling procedure affected the relative pattern of temporal precision across story parts. In particular, the effect of scrambled presentation, observed across scrambling versions, may reflect a general bias towards the center of the timeline, but also a specific reshaping that depends on the order of presentation of the story parts.
We tested the two hypotheses taking into account the mean relative error for each Viewing Part and the corresponding distance from its position in the linear story (viewing/story time distance) and from the center of the timeline (center distance). A significant correlation was found between mean relative error and both the viewing/story time distance (rs = 0.83, p < .0001, Figure 5, on the left) and the center distance (rs = 0.82, p < .0001, Figure 5, on the right), a result that likely reflects the collinearity between the two distance measures, since the scrambling procedure inevitably shifts the initial parts forward in time and vice versa. Comparison of reconstructive and central effects. The scatter plots show the relationship between the relative positioning error (y-axis) for each Viewing Part (grey dots; N = 36) as a function of the distance between the Viewing and the Story Parts (viewing/story time distance, on the left) and the distance from the center of the timeline (center distance, on the right).
To disentangle the role of the reconstructive process from a general central attraction bias, we estimated the potential contribution of the viewing/story time distance while controlling for the center distance. For each scrambled version of the movie, a trial-by-trial Spearman Rank Order Correlation was conducted between the group-level relative error for each HIT clips and the direction/amount of reconstructive shift (viewing/story time distance). A significant correlation was observed for almost all the Viewing Parts [Part 1 (rs = 0.28, p < .005), Part 3 (rs = 0.49, p < .0001), Part 4 (rs = 0.24, p < .05), Part 5 (rs = 0.35, p < .0005), Part 6 (rs = 0.49, p < .0001)], with the exception of Part 2 (rs = 0.12, p = .22) (Figure 6). This result indicates that both the amount and direction of the relative error were directly related to the shift from the presented order to the narrative order (or to a more linear story time). In other words, what we have termed “error” is instead an index of spontaneous reorganization of the encoded material. Indeed, while we have no evidence as to how conscious the central tendency bias was (as we did not interview participants about this issue), none of the subjects reported being aware of the reconstructive bias or treating the line as the timeline of the story. Each figure shows the relative error for a Viewing Part (scrambled) as a function of the direction/amount of reconstructive shift. Positive and negative values indicate over- and underestimation of the clip position respectively. If a Story Part is presented before its actual story position, it must be moved forward (overestimation; Viewing Parts 1-2-3 in the top row). Conversely, if it is presented after its actual story position, it must be moved backwards (underestimation; Viewing Parts 4-5-6 on the bottom row). Circles represent individual video clips (trials). Red numbers represent Story Parts. For example, in the top row, the third figure on the right (titled “Viewing Part 3”) shows what happens at retrieval to each Story Part (red numbers) that is presented at encoding as Viewing Part 3: Story Part 1 (red) is shifted backward from Part 3 to Part 1, so it is – 2 (black numbers); Story Part 3 (red) is already in its place, thus the reconstructive shift is equal to zero (black); Story Part 6 (red) is shifted forward from Part 3 to Part 6, so it is + 3 (black).
Discussion
People have a poor memory for temporal information per se, and previous research has suggested that they use what they have to improve their judgment. For example, they use knowledge of the calendar or the seasons of the year to reconstruct when something happened, or they try to make sense of things in order to remember them more truthfully. But if this is the case, how powerful are these processes? Are they conscious? Is it possible to quantify their impact on temporal memory? To answer these questions, we tested temporal memory for complex events using ecological material and manipulating the temporal order in which the narrative events are presented at encoding, i.e. artificially creating a temporally non-linear movie. Indeed, the use of non-linear narratives in the movie industry has been around since the beginning (e.g., “Destiny” by Lang, 1921) and has become increasingly popular in modern cinema, probably reflecting the changing times and/or the general increase of “narrational complexity” in popular movies (Cutting, 2021). We might speculate that this cinematic device is so appealing because it relies on the viewer’s meaning-making process, which is automatically at work when watching and remembering events, and the elegant interplay of meaning (story time) and narrated events (viewing time) probably explains the fascination of viewers and directors with non-linear narratives.
After the presentation of a “scrambled” movie, i.e., a movie divided into 6 randomly re-arranged parts, we asked participants to make temporal judgments about the encoded material in order to examine the relationship between the understanding of a meaningful event and memory for time. Specifically, we compared the precision of judgments about the time of occurrence of video clips (i.e., viewing time) extracted from a previously encoded movie across two encoding conditions: linear presentation (story time = viewing time) versus temporally scrambled presentation (story time ≠ viewing time). Different scrambling versions were administered to different groups of subjects to further distinguish between a central tendency effect (possibly reflecting temporal uncertainty), simply caused by the scrambling procedure, and a more specific reconstruction effect that should scale as a function of the particular arrangement of the story parts at encoding. The pattern of results is consistent with a combined manifestation of the two effects. Temporal judgments in the scrambled presentation condition were systematically biased as a function of the direction and the amount of discrepancy between the story and the viewing time, consistent with an automatic reshaping of temporal memory caused by the chronological representation of the storyline. However, the scrambling manipulation also resulted in a general tendency to shift judgments towards the center of the timeline, regardless of the specific scrambling arrangement. This suggests that the non-linearity of the story increased the degree of uncertainty when making temporal judgments (temporal uncertainty), which in turn led participants to adjust their estimates towards the center of the timeline.
The finding that participants’ judgments were automatically reshaped by story time is consistent with reconstructive theories of temporal memory (Friedman, 1993, 2004, 2005). According to these accounts, the time of occurrence of events is inferred at retrieval by combining contextual aspects of the memory (e.g., the environment and one’s internal state associated with the event during its encoding) with the general knowledge of natural, social/conventional, or personal time patterns (Brown et al., 1985; Brown et al., 1986; Friedman & Wilkins, 1985; Lieury et al., 1980; Linton, 1975; Ribot, 1901; Underwood, 1977). Thus, contextual information provides temporal clues for inferential processes. Crucially, however, personal experience/prior knowledge of time patterns is essential for treating these contextual associations as temporal information. For example, in a study by Friedman and Wilkins (1985; 1987), participants were asked to recall the occurrence of news events on different time scales. They found that months and hours were identified with both higher accuracy and confidence than years or weeks (the so-called “scale effect”). In other words, forgetting the year of occurrence did not imply forgetting the within-year location (i.e., month) of the same event. This indicates that memory judgments are not made by relying on information that is uniformly lost over time (i.e., distance-based processes of memory for time; see Friedman, 1993), but rather by reconstructing the position within a reference template. Our findings suggest that people spontaneously adopt and rely on the meaningful structure of the episode to organize temporal memory. Using a scrambling procedure, we showed that temporal memory judgments do not simply undergo a coarse reorganization, but rather are strictly calibrated as a function of the mismatch between the time of occurrence of the event and its time within the story template. Put differently, temporal memory seems to be something derived from the meaning attributed to an episode. It is not simply that people use rudimentary temporal cues to infer the time of an event (Friedman, 1993), but that the temporal structure seems to be a trace of the structure attributed to the episode. And people do not seem to be aware of the bias in general, nor of the fact that each of their responses seems to directly refer to a coherent template.
However, the present results do not allow us to further specify when the mechanism underlying the template creation takes place. A reconstructive process might already be at work during encoding, as soon as participants recognize that the story time has been temporally rearranged. In this case, the gradual building of a linear representation of the story would serve to facilitate event comprehension and the integration of new material into the working event model as the participant proceeds through the narrative. This linear representation of story time, already formed at encoding, would then be automatically reactivated at retrieval, even when the task is to judge the viewing time rather than the story time of the clip. However, a temporal reorganization could also occur at retrieval, reflecting the tendency to reorganize events in a more coherent fashion, a process that normally aids memory performance. For example, such a chronological reconstruction of story time would aid memory judgments about the meaning, rather than the viewing time, of the narrative.
Narrative Comprehension and Construction of the Story Representation
In the present study, we used narrative comprehension as a proxy for comprehension of real-life events. The construction of narratives seems to be an intensely human business. We strive to make sense of the world, and we need to remember our past as a set of coherent/meaningful representations (also referred to as “lifetime periods”; Conway, 1996) rather than as a jigsaw puzzle of fragments. There is evidence that temporally distant events are better remembered when they form a coherent narrative (Cohn-Sheehy et al., 2022), suggesting that people can form coherent links between non-adjacent events during memory encoding (Event Horizon Model; Radvansky & Zacks, 2014, 2017). Theories of text comprehension converge on the idea that readers attempt to integrate a causal chain of events into a cohesive representation, by creating and updating abstract models of what is happening in the text (Gernsbacher, 1990; Kintsch, 1998; Trabasso & Sperry, 1985; Zacks & Radvansky, 1998; Zwaan et al., 1995). Indeed, there is evidence that information about events depicted by picture stories (i.e., the gist; Bransford & Franks, 1971; 1972) is easier to remember than surface information (Gernsbacher, 1985) and that comprehenders spend more time looking at the first picture compared to the others (Gernsbacher, 1983), suggesting that they are setting the stage for the creation of an abstract model that is later stored in memory. Furthermore, irrelevant information (i.e., information that is not coherent with the larger representation) tends to be suppressed (Gernsbacher, 1995), while salient incoming information is embedded into the developing mental structure. Finally, it has been shown that reading times increase when people have to update the model, suggesting that the representation of a sequential succession of events is facilitated when the iconicity assumption is met (Zwaan, 1996).
A growing body of research in cognitive neuroscience has recently investigated the neural bases of narrative comprehension during movie watching (Hasson, Furman, Clark, Dudai, Davachi, 2008; Lee & Chen, 2022). In an fMRI study by Song and colleagues (2021), participants were presented with scrambled movies and were asked to press an “Aha” button whenever they experienced a moment of comprehension about the overall meaning of the story (e.g., when a sudden insight into previously presented events occurred). They found that the default mode network (Hassabis & Maguire, 2009; Yeshurun et al., 2021) played a key role during these moments of narrative integration, which were reliable across observers. In addition, large-scale brain states and interactions between brain regions were dynamically reconfigured as comprehension progressed. In another fMRI study by Kauttonen and colleagues (2018), participants watched the movie “Memento” (Nolan, 2000), which can be considered a prototype of a non-linear narrative, in which short segments of the story are also presented twice (i.e., narrative cues) to enable an online reconstruction of the story. They found that a distributed pattern of neural activity in various frontal and parietal regions emerged during the presentation of these narrative cues, reflecting the integration of previous scenes with ongoing events, and thus the construction of a linear/coherent representation of the story. Taken together, these studies suggest that people make sense of what they read or see by integrating temporally distant events piece-by-piece and thus building a coherent representation. To infer the location of events in time, temporal memory would hinge on a time pattern (Friedman, 1993) that is linearly assembled during comprehension and later retrieved in this form. Although we acknowledge that we did not directly test the extent to which participants understood the story, the presence of a reconstructive effect is clearly indicated by the fact that different parts of the film are automatically shifted proportionally, both in terms of quantity and direction, to a particular position in a time pattern. Future experiments could investigate how individual differences in story comprehension affect temporal memory, or what happens when the construction of a larger representation is severely hampered, for example by using denser scrambling or Surrealist movies.
Schematic Knowledge and Memory for Time
Another way of making sense of experience is through the use of schematic prior knowledge (Alba & Hasher, 1983; Bartlett, 1932; Ghosh & Gilboa, 2014; Rumelhart, 1980; Schank & Abelson, 1977). Indeed, schemas represent common elements of repeated situations and their typical temporal sequence and are thought to influence both event understanding and remembering. Moreover, they allow us to automatically generate realistic expectations about the world and to remember events in a more coherent way (e.g., the sequence of events is reconstructed according to the likely temporal unfolding of events based on previous similar experiences). In particular, memory for time may operate through schema-based reconstruction, which is considered the central retrieval process in schema theory (Abbott et al., 1985; Bower et al., 1979; Bower & Clark-Meyers, 1980; Migueles & García-Bajos, 2012). Indeed, because of the encoding processes assumed by schema theory (e.g., the selection of relevant information), the retrieved information cannot be a replica of the original event. Therefore, people would attempt to reconstruct the event by generating schema-congruent information (Bower et al., 1979; Brockway et al., 1974; Cofer et al., 1976; Spiro, 1980). For instance, the reconstruction process could explain the “typicality effect” (Graesser et al., 1979; Graesser et al., 1980), i.e. the failure to discriminate between old and new events that are typical of a given script (i.e. schematic knowledge about the common temporal sequence of events in routine activities, such as going to the restaurant or visiting the doctor; Schank & Abelson, 1977; Baldassano et al., 2018). For example, in a study by Bower and colleagues (1979), participants who read several similar stories from the same script falsely recalled portions of the story that weren’t present in the original story, and also recombined elements of the different stories with a common event representation during the recall process.
There is evidence that schematic knowledge can also affect memory for narratives. In his pioneering work, Bartlett (1932) showed that schemas led to numerous distortions in the recall of stories and that participants were unaware of these errors (Zangwill, 1972). In particular, the most important distortion was the ‘rationalization’, the tendency to make the story more coherent and acceptable/conventional from the reader’s point of view (for example, by omitting highly ambiguous or supernatural elements of the story). Despite the limitations of this pioneering work (e.g., the lack of quantitative statistical tests and the use of vague instructions; see Gauld & Stephenson, 1967), subsequent well-controlled studies have provided support for Bartlett’s main findings (see Bergman & Roediger, 1999). In addition, there is some evidence that participants tend to remember scrambled and interleaved stories (i.e. narratives in which the events of two or more episodes are broken up in parts and then intermixed) in a canonical/conventional form (Mandler & Johnson, 1977; Rumelhart, 1975; Thorndyke, 1977), with a narrative trajectory from the beginning (the setup) through the middle (plot complication and development) to the end (resolution, climax, and epilogue; see Cutting, 2021). In our previous studies, we have shown the effect of schematic reconstruction on memory retrieval (Frisoni et al., 2021, 2022). In particular, when presented with an incomplete movie, participants tend to add the missing parts of the story to their temporal representation of the event. Similarly, the temporal reshaping observed in the present study might be based on the activation of a story schema during retrieval. Specifically, to reconstruct the temporal locations of events, participants would use a pre-existing story schema, i.e., a general kind of knowledge about typical story components and how narrative events are likely to unfold, acquired through multiple presentations of similar material. However, whereas the effect of schemas is traditionally associated with a scaffolding role of memory, or at least with coarse distortions, what we have shown in the present study is a highly precise mechanism of temporal memory calibration. In summary, in both cases (the specific storyline constructed during encoding, or the general story schema activated during retrieval) participants used a time pattern to reconstruct the temporal location of movie scenes. Moreover, the two possibilities are not mutually exclusive. In other words, once we have constructed a storyline representation during encoding, the story schema could serve to strengthen temporal memory judgments during memory retrieval.
Finally, it should be noted that schematic knowledge might also modulate memory traces during the retention period, as the representation of the event undergoes memory consolidation (Gilboa & Marlatte, 2017; Tompary & Davachi, 2017). Indeed, it has been shown that new memories tend to be incorporated into prior relevant schemas (McClelland et al., 1995; van Kesteren et al., 2010; Winocur et al., 2010), and schematization is thought to increase progressively with consolidation (Richter et al., 2019; Spens & Burgess, 2024). This phenomenon contributes to learning speed and improves subsequent memory performance for schema-related information (Bransford & Johnson, 1972; Zwaan & Radvansky, 1998). In other words, the linear representation of the story may reflect the increased influence of a linear story schema during consolidation. Our previous findings (Frisoni et al., 2022) argue against this possibility, or, at least, suggest that consolidation processes are not necessary for the reshaping to occur. Specifically, we provided evidence that the schematic effect (i.e., the tendency to increasingly underestimate the time of occurrence of the video clips as a function of their proximity to the missing part of the movie) was also present when the task was performed immediately after the encoding session and did not increase as a function of the retention interval. Since the present study also tested a 24-h delay (a period compatible with rapid, but not slow, consolidation effects; Bayley et al., 2005; Tse et al., 2007; Runyan et al., 2019), the influence of schema-based consolidation over a longer period cannot be ruled out (human system consolidation; Squire, 1992).
Central Tendency Effect and General Decrease in Recognition Memory
The present findings also revealed that the scrambling procedure induced a general bias towards the center of the timeline, regardless of the order in which the story parts were presented. Specifically, the non-linearity of the story might have increased the degree of uncertainty in making judgments, and thus the tendency to calibrate responses around the midpoint. The increased difficulty/uncertainty in the scrambled task (as indicated by the significant difference in the absolute error between the linear and scrambled tasks) is likely due to the interference/incongruence between viewing time and story time (Xu & Kwok, 2019), which would lead to temporal confusion. Although we did not directly measure memory confidence, participants might have been less confident in making their temporal judgments due to a general conflict between story and viewing time. Alternatively, temporal confusion might also be related to poorer comprehension of non-linear stories. However, there is evidence that participants make the same kind of summaries whether they are presented with linear or scrambled stories (Kintsch et al., 1977).
Regarding the central tendency effect in response to uncertainty, this effect is reminiscent of the bias observed in perceptual tasks, such as auditory intensity (Berliner et al., 1977) or visual length discrimination (Ashourian & Loewenstein, 2011). In such cases, it has been shown that participants tend to choose a response that is shifted towards a central value of the presented stimulus set, overestimating small stimuli and underestimating large stimuli to reduce the overall error (“regression effect”, Stevens & Greenbaum, 1966; “contraction bias”, Poulton, 1979; “central tendency bias”, Huttenlocher et al., 2000), when they know the range of responses. There is some evidence that the effect is greater on low confidence trials, and thus serves to “regularize” responses and to reduce the influence of sensory noise (Xiang et al., 2021). Moreover, recent research suggests that this bias extends into the time domain (Tal-Perry & Yuval-Greenberg, 2022). Taken together, our findings suggest that when making temporal memory judgments under uncertainty, participants skew their reconstruction of the time of events towards the center of the distribution.
Another finding of our study is that, although recognition memory was slightly worse in the scrambled (vs. linear) condition, the effect did not vary as a function of the story part. In other words, the scrambling procedure had only a general effect on the overall level of recognition memory, but the specific pattern of recognition performance for each story part appeared to be consistent across conditions (see Figure 3). Again, this general decrease can be accounted for by the interference between temporal levels (i.e., viewing time and story time; Xu & Kwok, 2019). Importantly, however, our findings demonstrate the independence between time-of-occurrence and recognition memory judgments. This is consistent with previous studies (Ferguson et al., 2017) showing that recognition memory for dynamic temporal events is not affected by the disrupting the global structure of the story. This effect may be related to the different levels of representation of narrative material (Zacks & Radvansky, 1998), meaning that surface information and the high-level representation of the story are dissociable (Kintsch, 1998; Schmalhofer & Glavanov, 1986). It is also possible that recognition judgments for sensory information are preserved for scrambled events. In other words, recognition judgments might be predominantly based on the recognition of sensory information for which no discrepancy is found between the linear and scrambled conditions. This would be consistent with the idea that events unfolding at different timescales are nested in a cortical hierarchy (i.e., “temporal receptive window”; Hasson, Yang, Vallines, Heeger, & Rubin, 2008), with short events in early sensory regions and longer events in higher-order brain areas. Overall, these results suggest that the scrambling procedure had an overall negative impact on both temporal and recognition memory, although recombination and recognition of events from each story part remained stable.
Study Limitations
One limitation of the present study is the sample size. As a matter of fact, while the numerosity of the scrambled paradigm (N = 30) is an issue for those analyses that were performed on the whole sample, only a limited number of subjects (N = 5) were exposed to each of the six possible scrambling conditions. This limitation applies to the analysis performed to distinguish between the reconstructive and the central tendency bias. To deal with this issue, we took advantage of the availability of single trial estimates of performance provided by the current paradigm and we found highly significant effects (i.e. high correlation values between the group-level relative error for each HIT clips and the direction/amount of reconstructive shift) except for the second viewing part. Thus, while we acknowledge that the inclusion of more participants per scrambling condition would increase the generalizability of our findings, we believe that the pattern of results presented in this manuscript is robust. However, future studies with larger sample sizes may help to replicate the present findings. Another limitation to the generalizability of the findings is that our group consisted mainly of female participants. Indeed, male participants might show a greater or lesser tendency towards a reconstructive bias, and this should be investigated in future studies using the same paradigm.
A further limitation concerns the experimental control of remotely conducted studies. Even though we gave precise instructions to watch the movie only once, with no interruption, and informal interviews with the subjects reassured us about their compliance with the instructions, we cannot exclude the possibility that the movie was paused or rewatched for specific parts. On the one hand, the possibility that subjects watch the material during the retention interval applies also to other laboratory studies using a 24 hrs between encoding and retrieval. On the other end, the choice of a long retention interval reduces potential primacy or recency effects in memory performance.
Conclusion
The present study suggests that temporal memories of what has been seen are reconstructed on the basis of the overall meaning of the events. In other words, the idea that memory does not occur “as a video replay” should be taken very literally.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a grant (n. 384/20) from BIAL Foundation to M.F. and was conducted under the framework of the Departments of Excellence 2018–2022 initiative of the Italian Ministry of Education, University and Research for the Department of Neuroscience, Imaging and Clinical Sciences (DNISC) of the University of Chieti-Pescara.
Data Availability Statement
Data will be made available on request.