Face to face: Comparing ChatGPT with human performance on face matching

In November 2022, OpenAI's ChatGPT was released to the public and we were in awe of its ability to tell jokes and do our assignments for us. However, with its fabricated citations (Walters & Wilder, 2023) and overuse of words like ‘delves’ and ‘significant’ (Kobak et al., 2024), we soon realised that our fear of artificial intelligence and the impending ‘rise of the machines’ may have been a tad premature. Here, with the latest advances as of September 2023, I argue that perhaps a little fear may still be warranted.

ChatGPT's newest large language model (LLM) – GPT-4V(ision) – can process visual input. Preceding models relied on extensive training with text data sourced from the internet, allowing them to learn the structure within these data. Now, through exposure to vast numbers of image-text pairs, the latest model can ‘see’, recognise and interpret uploaded images. Crucially, if any future Terminator is going to be successful in its mission, it will need to discriminate between people to target key members of the resistance. Therefore, I wanted to determine how well ChatGPT performed with face images in particular, building on previous research showing human levels of accuracy when identifying mental states from only the eye region of faces (Elyoseph et al., 2024).

Face matching involves deciding whether two photos depict the same person or two different people. This can be challenging when the faces are unfamiliar since the photos themselves lack information regarding how appearance can vary as a function of lighting, expression, hairstyle and so on (Hancock et al., 2000). Researchers have created several such tasks, varying in both trial difficulty and how unconstrained or controlled the images are, and so I tested ChatGPT on a selection of these (see Figure 1), with the results presented in Table 1. ¹

OPEN IN VIEWER

Figure 1.

An example from the Glasgow Face Matching Test, showing ChatGPT's correct response.

OPEN IN VIEWER

Table 1.

Human and ChatGPT performance on several tests of face matching.

Test	Version	Trials	Human accuracy (%)	ChatGPT accuracy (%)
Glasgow Face Matching Test (Burton et al., 2010)	Short	40	81.3 (9.7)	92.5
Glasgow Face Matching Test 2 (White et al., 2022)	Short	80	75.0 (10.0)	76.3
Kent Face Matching Test (Fysh & Bindemann, 2018)	Short	40	65.9 (9.6)	62.5
Models Face Matching Test (Dowsett & Burton, 2015)	Long	120	64.5 (7.5) a	88.3
Oxford Face Matching Test (Stantic et al., 2022)	Long	200	74.1 (5.4)	77.0
Expertise in Facial Comparison Test (White et al., 2015)	Short	84	77.9 (8.2) b	81.0

Note. Human accuracies reported as M (SD).

^aValues from typical perceivers featured in Bobak et al. (2016).

^bValues from Stacchi et al. (2020).

I found that ChatGPT's performance was comparable with human abilities, making it less accurate than state-of-the-art deep convolutional neural networks (DCNNs; National Institute of Standards and Technology, 2024; Phillips et al., 2018). However, the difference is that ChatGPT is based on a transformer neural network, first converting images into text-like representations. Crucially, ChatGPT can interpret any image rather than being limited to a specific domain (unlike DCNNs) and is freely available for use by Terminators (and the general public).

Having shown that ChatGPT demonstrates competent face matching abilities, further study might investigate the underlying process. For instance, are some facial features (e.g., the eyes) more heavily weighted during comparisons? Are some transformations (e.g., head rotation, expression change) more difficult to handle? How do ChatGPT's errors relate to human decisions? The possibilities for combining LLMs and face perception research are numerous and represent an exciting new avenue for exploration. Indeed, by probing the nature of its internal representations, we may uncover both similarities and differences with human face processing, resulting in new insights much like those emerging from research with DCNNs (e.g., Parde et al., 2019). In the meantime, while August 29, 1997 (aka ‘Judgement Day’) passed without incident, I recommend caution as we continue to witness the evolution of ChatGPT in the near future.