Face Recognition Under Adverse Viewing Conditions: Implications for Eyewitness Testimony

A Preliminary Note on the SDT

The SDT (Green & Swets, 1966; Macmillan & Creelman, 2005) describes the psychological processes underlying a wide range of phenomena in perception and cognition, especially in memory. For example, when remembering faces, memory tests often require an observer to distinguish seen from unseen faces. Every response in such memory tests is, in fact, influenced by both the observer's memory ability (known as sensitivity) and their general tendency to give one type of response, be it “seen” or “unseen” (known as bias).

The key advantage of using SDT is that it allows researchers to estimate sensitivity and bias separately based on multiple memory responses, because SDT considers the two as independent psychological processes. Under the SDT framework, responses to a seen-versus-unseen face memory task can be categorized into four possible outcomes:

Hit: a “seen” response to a seen face

Characterizing Performance by Sensitivity and Bias

Face memory performance is commonly measured using accuracy, which combines hit and correct rejection rates into one measure. This method is oversimplified and can be problematic, because a specific level of accuracy (e.g., 75% accurate) can result from many possible combinations of sensitivity and bias levels. For instance, lower accuracy in identifying masked than unmasked faces (e.g., Freud et al., 2020) can be a result of bias rather than lower sensitivity (Or et al., 2023). Meanwhile, face memory accuracy “improvements” may result from higher hit rates alone, without reducing false alarms (Hockley, Hemsworth & Consoli, 1999).

Thus, reporting accuracy alone may not provide sufficient information in understanding the observer's perceptual and decision-making processes. This is crucial to the reliability of eyewitness testimony. For instance, a biased memory response could lead to misidentification of an innocent person as the perpetrator (or the perpetrator being set free in the opposite direction of bias). Unfortunately, many studies focused solely on accuracy. While some reported sensitivity values, few reported response biases.

Confidence, Accuracy, and Eyewitness Testimony

Beyond sensitivity, bias, and accuracy, another important measure is confidence. Confidence refers to the subjective judgment about one's own accuracy when performing a task. It is measured as a rating tied to the memory decision, with choices among two (e.g., low or high confidence), several (e.g., on a scale of 1–7), or pseudocontinuous (e.g., on a scale of 0–100) ratings. Ideally, the confidence rating given to a memory decision represents the decision's reliability. However, human memory, decisions, and confidence judgments are often imperfect due to cognitive limitations. Although confidence and accuracy are generally linked (i.e., high-confidence memory responses are, generally speaking, accurate), many factors could lessen or possibly reverse this confidence–accuracy relationship.

In the context of eyewitness testimony, such a confidence–accuracy relationship depends on multiple conditions. An extensive review found a strong relationship (positive correlation) between eyewitness confidence and identification accuracy when police lineup conditions are appropriate or “pristine,” for example, by including only one suspect per lineup, preventing the suspect from standing out in the lineup, using double-blind testing, etc. (Wixted & Wells, 2017). However, the review highlights that a confidence–accuracy relationship depends on the lineup condition, as nonpristine conditions can compromise this relationship. Thus, the review recommends adopting these pristine lineup conditions and avoiding nonpristine ones.

Individual differences can also affect the confidence–accuracy relationship. An online, mock-crime study shows that the probability of making a high-confidence error depends on a collection of factors. This includes decision time, eyewitness's general ability to remember faces, and the type of justification given after choosing the suspect in the face-identification task (Grabman et al., 2019). As these factors are largely related to how well faces are represented in memory, the study's findings, in general, support the importance of face-representation quality: The better a face image is represented in the brain, the more likely the observer is going to be accurate when giving a high-confidence response.

Accuracy, confidence, and, critically, the relationship between the two depend on many other factors. Lighting and disguises are two important factors that have rarely been examined in previous research. Nevertheless, this is an important future direction for both experimental and applied research, as it bridges the theories (e.g., face perception and memory) to real-world applications (e.g., eyewitness testimony and lineups).

Uniform Lighting: Dim versus Bright

Face recognition and memory are, in general, impaired under uniformly dim lighting. Early face memory research reported that dimmer lighting led to lower accuracy and confidence in identifying faces than did brighter lighting (DiNardo & Rainey, 1989; Yarmey, 1986). More recently (Nyman et al., 2019), a simulation of eyewitness lineups across three broad-range lighting levels (0.7–300 lx; simulating starlight conditions to a brightly lit office space) found face recognition in lower lighting conditions to be significantly less accurate, alongside a slower response and lower confidence. Across all lighting levels, higher confidence was associated with higher accuracy, demonstrating a positive confidence–accuracy relationship. Interestingly, accurate responses were faster only under higher lighting levels. The poorer face recognition performance under dimmer lighting can be attributed to a reduction in the amount of available information being extracted from dimmer faces, following observations from the impact of dim lighting on nonface picture recognition (e.g., Loftus, 1985).

Nonuniform Lighting: Direction of Lighting and Cast Shadow

Although research on nonuniform lighting has been limited, some evidence indicates that changes in lighting direction impair face recognition. For instance, face matching performance was found to be poorer when matching a top-illuminated face with a bottom-illuminated face, than when matching two faces illuminated from the same direction (Hill & Bruce, 1996). In a series of face memory and face matching tasks (Braje, 2003; Braje et al., 1998), changes in lighting direction also led to lower sensitivity and slower responses. Cast shadows in these studies did not seem to affect sensitivity, though sometimes led to slower responses in face recognition. However, cast shadows have been found to improve recognition performance for nonface objects (Castiello, 2001; Tarr et al., 1998). More research is needed to understand the effect of nonuniform lighting on face recognition.

Congruence in Lighting Between Face Memory Encoding and Retrieval

Face memory performance is generally better when uniform lighting levels match between study and test, but performance is impaired when they do not match. For instance, when bright lighting was used during the face memory test, face recognition performance was poorer when faces were initially studied under dim lighting than under bright lighting (De Jong et al., 2005; Wagenaar & Van der Schrier, 1996). However, these studies did not present faces under dim lighting during the face memory test. Thus, it was unclear whether police lineups should be set in dim lighting in order to match the lighting of crimes happening in the dark.

More recently, another study systematically examined how changes in lighting between study and test affected face memory (Lim et al., 2022). Lighting was incongruent when faces were studied under bright lighting and then tested under dim lighting (Bright/Dim) and vice versa (Dim/Bright). Otherwise, lighting was congruent (i.e., Bright/Bright or Dim/Dim). In general, sensitivity was lower under incongruent than congruent lighting. Observers also showed conservative biases (i.e., more likely to indicate faces as “unseen”) under incongruent lighting but remained unbiased under congruent lighting. Nevertheless, performances were similar between the Bright/Bright and Dim/Dim conditions.

These findings are consistent with the encoding specificity principle found in memory studies, where incongruent encoding and retrieval impairs memory performance (Shapiro & Penrod, 1986; Tulving & Thomson, 1973), also during eyewitness identification (Carlson et al., 2021).

In contrast, an earlier study on the effect of uniform lighting changes suggested that brighter lighting at either encoding or retrieval stages resulted in higher sensitivity, with similar bias levels across conditions (DiNardo & Rainey, 1991). These findings run contrary to the encoding specificity principle, though it can be explained by the narrow range of lighting levels, both of which were too dim. For instance, their “bright” conditions still fell within the range of lighting on a full moon night.

Research on the effect of nonuniform lighting on face recognition also supports the encoding specificity principle, where congruent lighting directions between study and test resulted in better face recognition performance than incongruent lighting directions (Braje 2003; Braje et al., 1998; Hill & Bruce, 1996).

Nevertheless, research on the effects of lighting on face recognition remains scarce. It is especially unclear how the brain processes facial information under extremely low lighting levels, as in some possible crime scenes. More research in this area is needed.

The Upper Facial Region: Eyes

Research points to the essential role of the eye region in providing major diagnostic information for face recognition. For example, face-selective neurons were found to be especially sensitive to the eyes (Freiwald, Tsao & Livingstone, 2009; Issa & DiCarlo, 2012; Nemrodov et al., 2014; Nemrodov & Itier, 2011; Schyns, Bonnar & Gosselin, 2002; Vinette, Gosselin & Schyns, 2004). Humans often direct their gaze firstly to the eyes, which have been shown to maximize information retrieval from faces (Barton et al., 2006; Henderson, Williams & Falk, 2005; Or, Peterson & Eckstein, 2015; Peterson & Eckstein, 2012), allowing faces to be recognized within the first couple of eye movements (Hsiao & Cottrell, 2008). Face recognition also improves when the eye region information is effectively used (Hills, Ross & Lewis, 2011; Royer et al., 2018; Sekiguchi, 2011).

As a result, occluding the upper facial region generally impairs face recognition. Earlier studies, which digitally removed or occluded the eyes, found various kinds of impairment on face memory, including increased recognition error and looking time (McKelvie, 1976), slowing of familiarity judgment on celebrity faces (Roberts & Bruce, 1988), or a decrease in its accuracy (Sadr, Jarudi & Sinha, 2003). The eyebrows (in addition to the eyes) have been found to be crucial in familiarity judgment (Sadr et al., 2003).

Later studies used real-life images or more advanced photo-editing techniques to introduce more realistic disguises, though results generally remained the same in that face recognition is impaired with upper region disguises. For instance, sunglasses impair face memory (Hockley et al., 1999; Nguyen & Pezdek, 2017, Or et al., 2023), regardless of the race of the face image (Nguyen & Pezdek, 2017). Sunglasses also impair face matching performance, even for “super-recognizers” (e.g., Ramon, 2021) with above-average accuracy in face recognition (Bennetts et al., 2022; Graham & Ritchie, 2019; Noyes et al., 2021). Regular glasses, which do not occlude but modify the appearance of the upper face region, can also lead to poorer face memory (Righi, Peissig & Tarr, 2012; Terry, 1993; 1994) and face matching performance, especially when a face with glasses is matched with one without (Graham & Ritchie, 2019; Kramer & Ritchie, 2016). Face matching performance also declined when matching a face wearing sunglasses with one wearing regular glasses (Graham & Ritchie, 2019).

Some studies show that more disguises on the upper facial region lead to greater impairment on face recognition. For instance, faces with more upper area covered (e.g., with sunglasses and toques, or plain stockings down to the nose) led to poorer performance and lower confidence in face recognition than did faces with a single type of disguise in simulated police lineups (Mansour et al., 2020). Changes in hairstyles (e.g., with wigs), in addition to having glasses, also impair face memory (Righi et al., 2012).

The Lower Facial Region: Nose and Mouth

Research following the COVID-pandemic has agreed that a sanitary face mask, which covers the mouth and part of the nose, impairs face recognition in face memory tests (Freud et al., 2020; 2022; Garcia-Marques, Oliveira & Nunes, 2022; Guerra et al., 2022; Hsiao, Liao & Tso, 2022; Marini et al., 2021; Or et al., 2023; Stajduhar et al., 2022) and face matching tasks (Bennetts et al., 2022; Carragher & Hancock, 2020; Estudillo, Hills & Wong, 2021; Noyes et al., 2021). The impairment is reflected by lower accuracy, lower sensitivity, and/or a larger bias. Despite extensive exposure due to pandemic requirements, accuracy in recognizing masked faces in face memory tests has not improved over the years (Freud et al., 2022). In contrast, performance for matching masked faces may be improved by diagnostic feature training (Carragher et al., 2022).

Other Forms of Disguises

A limited number of studies also suggest that face recognition may be impaired by occluding the head outline (e.g., headscarf: Megreya, Memon & Havard, 2012; Toseeb, Keeble & Bryant, 2012; Wang et al., 2015), disguising the entire face by a patterned/plain stocking (Davies & Flin, 1984; Mansour et al., 2020), or adding facial hair (beard/mustache: Dhamecha et al., 2014; Foley & Foley, 1998; Terry, 1994). Their impacts on eyewitness identification remain to be understood.

Comparing Disguises on the Upper and Lower Facial Regions

Although research suggests that both upper and lower region disguises impair face recognition, it remains unclear which area of disguise has a larger impact. Early studies seem to suggest that occluding the lower facial region does not impair face recognition as much as occluding the upper region does. For instance, blacking out the mouth resulted in fewer recognition errors than blacking out the eyes did in face memory tests, though both led to similar confidence ratings (McKelvie, 1976). These results suggest that accuracy and confidence may not always correlate. Blacking out the lower region also does not slow down familiarity judgment (Roberts & Bruce, 1988), but blocking the lower head outline and features down from nostrils (e.g., using bandanas) seems to impact face memory more for African Americans than Caucasian Americans (Nguyen & Pezdek, 2017).

However, more recent studies comparing the effects of sunglasses and sanitary masks showed inconsistent results. For example, one study on average observers and super-recognizers found poorer face matching performance for masked faces than faces with sunglasses (Noyes et al., 2021), whereas another study found comparable performances (Bennetts et al., 2022). Face memory has worsened less with masks than sunglasses, to the extent that sensitivity only declined under incongruent masking (Or et al., 2023). This is consistent with early studies suggesting a smaller impact from occluding the lower facial region than the upper region. Nevertheless, future studies should investigate further the relative impacts of upper and lower region disguises and their implications on eyewitness identification.

Congruence in Disguise and Encoding Specificity

Many face recognition studies have investigated the effects of congruence in disguise across memory stages. In general, when disguises are incongruent between study and test (i.e., studying disguised faces, testing with undisguised faces, or studying undisguised faces, testing with disguised faces), performance declines more than when disguises are congruent. The next section reviews such effects on accuracy, confidence, and bias.

Incongruent Disguises Lower Accuracy?

Face memory accuracy, in general, declines more under incongruent than congruent disguises, regardless of disguise in the upper or lower facial region (masking eyes or mouth: McKelvie, 1976; ski masks: Manley, Chan & Wells, 2019; sanitary masks: Garcia-Marques et al., 2022; Hsiao et al., 2022; Marini et al., 2021; sunglasses or sanitary masks: Or et al., 2023; and regular glasses: Righi et al., 2012; Terry, 1993; 1994). These findings suggest that the effects of disguises align with the encoding specificity principle (but see, e.g., Freud et al., 2020 and Guerra et al., 2022, for the lack of encoding specificity), which is further supported by studies using face matching tasks (glasses/sunglasses: Bennetts et al., 2022; Graham & Ritchie, 2019; Kramer & Ritchie, 2016; Noyes et al., 2021).

Incongruent Disguises Lower Confidence?

Observers tend to be less confident when disguises are incongruent between study and test. When disguise is present in only one of the memory stages, confidence is lower than when it is present in both study and test, as shown over time: in early lab studies that blacked out facial features (e.g., McKelvie, 1976), in later studies using realistic disguises such as sunglasses (e.g., Hockley et al., 1999), and in recent studies using mock-lineup configurations (Manley et al., 2019). In a more recent study (Manley, Chan, & Wells, 2022), beyond a general, positive correlation between confidence and accuracy regardless of masking, higher confidence ratings have been found for an all-masked lineup than a no-mask lineup. This finding supports the encoding specificity principle as the perpetrator always wore masks during study (encoding phase). Thus, the encoding specificity principle may also apply to memory confidence.

Incongruent Disguises Affect Response Bias?

In contrast, response bias may be more affected by the presence of disguise itself than congruence. Generally, decisions in face memory tasks become more liberal (i.e., more likely to indicate faces as “seen”) when the face is disguised (Garcia-Marques et al., 2022; Guerra et al., 2022; Or et al., 2023), though the direction of bias (whether conservative or liberal) due to disguise is inconsistent across face matching studies (Bennetts et al., 2022; Graham & Ritchie, 2019; Kramer & Ritchie, 2016). In addition, when a mixed list of masked and unmasked faces were studied, decisions became more liberal (with a lower sensitivity) toward masked faces than unmasked faces in the test (Garcia-Marques et al., 2022). This suggests that biases in face memory tasks can be systematically manipulated based on encoding-retrieval congruence.