Face presentation attack detection: Research opportunities and perspectives

2.1 Taxonomy of presentation attacks

Any of the presentation attacks is aimed at creating a facial artefact. According to ISO/IEC 30107-1, the biometric characteristic or object is termed the Presentation Attack Instrument (PAI) [1]. The PAI is broadly classified into two types: artificial and human. Artificial PAI refer to an artificial means of generating the PAI, and it is classified as complete artificial PAI (print photograph, video, 3D mask, etc.) and partial artificial PAI (for example, partially visible face or sunglasses). Human PAI involve the human characteristics, such as lifeless, altered (by cosmetic surgery), non-conformant (facial expression), coerced (the use of the face of an unconscious human), and conformant (zero-effort impostor attempts).

Jain et al. [59] classified artificial attacks against biometric systems into two large groups:

1.

Zero-effort (indirect or accidental) attacks, when the biometric traits of an intruder may be sufficiently similar to a legitimately enrolled individual. Zero-effort attacks try to take advantage of the False Acceptance Rate (FAR). This situation is largely a problem of discriminative methods.

2.

Adversarial (direct or intentional) attacks, when an intruder is able to masquerade as a registered individual, using physical or digital artefacts of a legitimately registered user. Another scenario is when an individual deliberately manipulates his/her biometric trait in order to avoid detection by an automated biometric system.

Also, when supporting a biometric system, it is necessary to take into account some objective attacks: circumvention of system, repudiation in access, collusion between individuals, coercion, Denial of Service (DoS), etc.

The ISO/IEC 30107 standard introduced the term “presentation attacks” instead of direct attacks as: “Presentation of an artefact or human characteristic to the biometric capture subsystem in a fashion that could interfere with the intended policy of the biometric system” [60].

The most ancient practice to avoid being recognized is the use of masks or facial disguises, which can be associated with 3D attacks. However, artificial facial recognition systems often capture image or video that is interpreted as a 2D space rather than a 3D representation of a person. That is why relatively cheap 2D spoofing attacks are more popular compared to 3D spoofing attacks, causing a lot of attention to development of 2D PAD methods.

If an intruder tries to trick face recognition system into impersonation, such attacks are called impersonation attacks. If a person uses tricks to avoid being recognized by the system, but not necessarily by impersonating a legitimate user, then this type of presentation attacks is called an obfuscation attack. Whether the attack is impersonation or obfuscation depends on the intruder’s goal: an intruder can use any type of presentation attack.

Presentation attacks may be classified into two groups: 2D spoofs and 3D spoofs, as shown in Fig. 1. 2D spoofs include print (image) attacks, replay (video) attacks and adversarial attacks, while 3D spoofs involve mask attacks and disguises.

OPEN IN VIEWER

Figure 1.

Taxonomy of face spoofing attacks.

Print attack means representing a photograph of the genuine user to a face recognition system. Photograph may be taken from social networks, the Internet or captured by a digital camera without the consent of the genuine user. Photograph can be printed on a paper (printed copy) or is displayed on the screen of a mobile phone or a tablet (digital-photo attacks). Advanced type of image attack is the use of high resolution printed photographs with holes for the eyes and mouth (photographic masks), when the intruder can blink and move his/her lips.

Replay attacks represent a video clip of the genuine user captured by any digital device (mobile phone, tablet or laptop). This type of attacks is a more sophisticated version of the print attacks and involves richer spatio-temporal information instead of the spatial information of a still image. Objectively, replay attacks are harder to detect because of the genuine temporal information. Adversarial attacks are of a more recent type, when some wearable things become the source of the spoof. They aim to deceive deep neural networks.

Mask attacks are among to the most complex, as they provide the complete 3D structure of the face. Thus, the use of depth cues cannot be a solution to prevent print and replay attacks presented on flat surfaces. In the previous decade, this type of attacks was much less common that 2D spoofs due to expensive silicon masks. Nowadays, the situation has changed: some companies provide 3D face models at a reasonable price, and self-manufacturing a face mask is also becoming more feasible and easier thanks to relatively cheap 3D scanners, scanning software and 3D printers.

The PAD methods can be implemented at the level of sensors (hardware-based) and at the level of features (software-based). Currently, they are shifted to software-based methods. The PAD methods are often addressed by well-known facial cues [61], such as motion (blinking of eyes), involuntary movements of parts of face and head, surface texture of the skin, and the depth information of the head. Ming et al. [40] proposed a topology of facial PAD methods based on cues, including liveness cue-based methods (motion cue-based and rPPG-based), texture cue-based methods (static and dynamic), 3D geometric cue-based methods (3D shape and pseudo-depth map), multiple cues-based methods (different combinations of liveness, texture and 3D geometry), and methods using new trends (Neural Architecture Search (NAS), zero-shot learning, domain adaption, etc.). The latter methods are driven by the development of deep learning models. All groups of methods mentioned above are effective against print and replay attacks, while rPPG-based, dynamic texture-based and liveness + 3D geometry (rPPG + pseudo-depth map) based methods are recommended against 3D mask attacks.

In literature, 2D PAD methods are more highlighted and developed due to the large number of publicly available databases. Mask attacks began to be systematically studied only after the appearance of the first mask-specific database [62]. The COVID-19 pandemic has made an additional contribution – facial recognition using a medical mask, as well as mask attacks. It is also worth noting that a special case is the recognition of twins, which may refer to a zero-effort attack.

2.2 Lambertian modeling

Many PAD methods are based on the intuitive idea that the fake image passes twice through the camera system and once through the print system. This implies more serious distortion of the fake image relative to the real image, or in other words, a lower quality of the fake image. It should be noted that the first databases included obviously distorted fake images, and relatively simple algorithms based on hand-crafted features could distinguish them. Nowadays, the situation has changed fundamentally, and databases collect only high-quality fake images and video clips. Even printed images have good resolution thanks to modern printers.

Tan et al. [7] were the first who identified the importance of Lambertian modeling for face anti-spoofing. They obtained rough approximations of illuminance and reflectance parts. Further, in some publications, attention was paid to the mathematical formulation of the PAD task [63, 64, 65]. However, most authors prefer to deal with the consequences of the physical representation and do not take into account the Lambertian modeling. This can be explained by the fact that the difference in reflectance between real and fake images or frames can be captured by special sensors, but not by the most common RGB-cameras. Moreover, modern databases provide high-quality fake images and video clips. This has led to the failure of many algorithms with hand-crafted features and provokes the search for multiple cues in various modalities.

According to the Lambertian reflectance assumption, the face surface is modeled as the ideal diffuse reflectors [18]. In other words, Lambert’s cosine law describes the intensity of a face image I ⁢ ( x , y ) as follows:

I ⁢ ( x , y ) = f c ⁢ ( x , y ) ⁢ ρ ⁢ ( x , y ) ⁢ I 𝑙𝑖𝑔ℎ𝑡 ⁢ cos ⁡ θ ,(1)

where f c ⁢ ( x , y ) is the coefficient depending on the camera used at the point with coordinates ( x , y ), ρ ⁢ ( x , y ) is the reflectance coefficient, which represents the diffuse reflectivity of the surface at that wavelength, I 𝑙𝑖𝑔ℎ𝑡 is the intensity of the incoming light at a particular wavelength, θ is the angle between the surface normal n and the incoming light ray s, cos ⁡ θ = 𝐧 ⋅ 𝐬 .

Equation (1) can be rewritten for real image I r ⁢ ( x , y ) and fake image I f ⁢ ( x , y ) under assumption that f c ⁢ ( x , y ) is a constant:

I r ⁢ ( x , y ) = ρ r ⁢ ( x , y ) ⁢ I 𝑙𝑖𝑔ℎ𝑡 ⁢ ( 𝐧 r ⋅ 𝐬 ) ,(2) I f ⁢ ( x , y ) = ρ f ⁢ ( x , y ) ⁢ I 𝑙𝑖𝑔ℎ𝑡 ⁢ ( 𝐧 f ⋅ 𝐬 ) .(3)

Due to the fact that the lighting conditions during the capture of real and fake images or video sequences are unknown, various authors make additional assumptions. Thus, Tan et al. [7] supposed that the differences between the human skin and a photograph (or a video clip) can be made evident by comparing their surface properties under the same lighting conditions (s = const). They estimated the surface reflectance property ρ r / ρ f and the surface normal n r /n f for a real face and a fake image using variational Retinex-based method and Difference of Gaussian (DoG)-based method. Chan et al. [63] adopted Lambertian reflection model to extract standard deviation and mean as features. Liu et al. [66] extracted the normal cues from facial reflection frames based on the Lambertian model using the random light CAPTCHA and then trained an end-to-end multi-task CNN to perform liveness classification and light CAPTCHA regression simultaneously. Ebihara et al. [64] proposed the SpecDiff descriptor constructed by leveraging two types of reflection: specular reflections from the iris region and diffuse reflections from the entire face region. Specular reflections had a specific intensity distribution depending on liveness (with or without a flash) due to the curved human iris, which has a structure resembling glass beads. Diffuse reflections represented the 3D structure of a subject’s face against 3D masking attacks. The SpecDiff descriptor was used as a classifier in the ResNet4 deep network. Face PAD models for print and replay attacks were designed in [67]. In this work, the intensity at a given pixel ( x , y ) of a given bona fide face is considered as the linear combination of luminance I r L ⁢ ( x , y ) and chrominance I r C ⁢ ( x , y ) components:

I r ⁢ ( x , y ) = I r L ⁢ ( x , y ) + I r C ⁢ ( x , y ) = ρ r ⁢ ( x , y ) ⁢ I 𝑙𝑖𝑔ℎ𝑡 + C r ⁢ ( x , y ) ⁢ D C ,(4)

where C r ⁢ ( x , y ) presents color gamut of human skin, D c is the color distortion parameter of the capture device.

The color distortion analysis of the print and display (or replay) artefacts were simulated in the same manner as Eq. (4), but with the additional color distortion factors of printing and digital display conversion. Then chromatic co-occurrences of Local Binary Patterns (LBPs) were calculated based on the chrominance and luminance components of bona fide faces and facial artefacts under various luminance conditions and cameras. Such estimates were used as sub-classifiers for the ensemble learning method.

It should be noted that adversarial attacks are modeled differently, using an adversarial example perturbation by adding random noise to image (digital adversarial attack) [68] or showing special glasses, stickers, hats, etc., on the photo (objective adversarial attack) [69].

3.1.1 Print attack detection

Methods for print attack detection belong to dynamic and static approaches. The dynamic approach assumes that a still photograph is recorded as a short video, while static approach analyzes a still photograph as a single snapshot.

Printed photos were the first instrument to fool the face recognition system. The first algorithms analyzed a still photograph in temporal domain in order to confirm or reject a person’s identification. Some of these methods were based on motion detection in specific face regions, such as eye blinking, changes of mimic wrinkles near the mouth or involuntary lip movement [23]. Other methods detected face and head gestures (for example, nodding, smiling, changing gaze directions) [27]. The third group of methods analyzed foreground/background motion correlation using optical flow between the consequence still images [70, 71]. These methods are highly effective for detecting print attacks, but useful against replay attacks.

To overcome this shortcoming, some dynamic liveness detection techniques have been proposed to detect replay attacks, such as using 3D facial structure by analyzing multiple 2D images [72], context-based analysis (background and foreground motion) [73], evaluating noise produced during the recapture [74], or employing face texture dynamic analysis [75].

Dynamic-based PAD methods can be applied in systems where only a single face image of the user is available (passport related applications). Also not all frames are suitable for analysis, and real-time application is questionable. Such shortcomings limit the application of this approach in practice, but sometimes it provides better accuracy than static-based PAD methods.

Analysis of a still photograph is more attractive because it is faster overall. Most of the static methods analyze the texture of the face using various image processing algorithms, traditional and deep learning. Traditional methods have been based on the Fourier spectrum, multiple difference of Gaussian filters, LBPs, Gabor wavelets, histogram oriented gradients, and so on.

Nguyen et al. [76] proposed to use the hybrid features as a combination of handcrafted and deep features against print attacks including cut photos. Multi-level local binary patterns were used to measure the skin details on face region, such as edges, corners, and blobs by extracting a 3732-component feature vector. The VGG-19 model provided a binary classification: “real” and “presentation attack” images, forming a 4096-component feature vector. The high dimensionality of the common feature vector necessitated the use of the principal component analysis followed by the Support Vector Machine (SVM) classifier. For samples from the NUAA database and the CASIA database, the ACER values reached 11.247% and 2.174%, respectively.

However, the accuracy of these methods can suffer due to different illumination conditions, and other cues are usually used in conjunction with them.

3.1.2 Replay attack detection

Replay attack means playing a short video of an authorized user on a digital device such as mobile phone, laptop or tablet. The intruder is standing in front of the camera of face biometric system. Thus, the replay attack provides dynamic biometric features, in other words, the motion of the user. This type of attack is also possible due to the easy access to a person’s video in social networks.

Patel et al. [77] integrated deep texture features and eye-blink cue as countermeasures against replay and print presentation attacks. The CaffeNet model (with 5 convolution layers and 3 fully connected layers) was used as a backbone in the architecture with two branches, one for face texture analysis and the other for whole frame texture analysis. The generic-to-specific transfer learning scheme was designed to train the network due to the small sizes of existing face spoof databases. Such approach resulted in 13.8% of the Half Total Error Rate (HTER). Eye difference images between successive frames were calculated by cropping the regions of the left and right eyes separately to 40–50 pixels. The voting scheme was then used to generate a decision per video clip. After the eye-blink cue was fused with the deep texture features, HTER on replay attacks achieved 12.4%.

Feng et al. [78] estimated image quality cues provided by shearlet decomposition and averaged motion cues as the dense optical flows of face video and scene video to detect replay and 3D mask attacks. Then these three types of cues were fused using a hierarchical neural network to improve the generalization ability for both 2D and 3D spoofing detection. The REPLAY-ATTACK database and the 3D-MAD database were used in the experiments. With a bottleneck feature fusion, this method achieved a HTER of 0% on the 3D-MAD database.

The FASNet, which followed the VGG-16 architecture, was pre-trained with transfer learning to detect replay and mask attacks [79]. The top layers of the VGG-16 model were changed to perform the binary classification for the anti-spoofing task. The FASNet reached an HTER of 1.2% and 0.00% on the REPLAY-ATTACK database and the 3D-MAD database, respectively.

Dynamic liveness cues are often used in practice. Eye-blinking, chin and lip movement as a multiple-motion-cue-based method was developed in [80]. Bekhouche et al. [81] proposed spatiotemporal CNN with pyramid bottleneck blocks for eye blinking detection in a wild scenario. Killioğlu et al. [82] presented liveness detection method based on pupil tracking. Application of rPPG signals helps to estimate pulse or heart rate [3, 83, 84], or skin blood flow [85].

Chen et al. [86] suggested a three-way classification to distinguish real face, fake face and background for detecting the print and relay attacks. They designed face anti-spoofing region-based convolutional neural network based on Faster region-based CNN with addition of a Crystal Loss function to the original multi-task loss function. Also, the Retinex-based LBPs were presented to handle the different illumination conditions in face spoofing detection. These two detectors were further cascaded and achieved promising performances on the CASIA-FASD, REPLAY-ATTACK and OULU-NPU databases. In continuation of this direction, Peng et al. [87] proposed a two-stream vision transformers framework based on transfer learning, where the outputs of the RGB stream and the multi-scale Retinex with color restoration stream were fused by self-attention fusion module. The output of the fusion features was then classified by the softmax function as a live or spoof face in the video. Experiments on OULU-NPU, CASIA-MFSD and REPLAY-ATTACK databases have shown that this approach outperforms many methods in intra-database testing and provides good generalization performance in cross-database testing. Yu et al. [20] extended the central difference CNN to a multimodal version, exploring three modalities (RGB, depth and infrared). The CASIA-SURF CeFA, a large-scale cross-ethnicity face anti-spoofing database, was used in experiments.

Rehman et al. [88] enhanced the discriminative ability of deep features in face PAD, adding a perturbation layer (a learnable preprocessing layer for low-level deep features). In other words, the authors induced the information of the hand-crafted features into the deep features of a selected layer in CNN, called perturbation layer. Their experiments showed that the face PAD performance was improved in both intra-database and cross-database scenarios using CASIA, REPLAY-ATTACK and OULU-NPU databases.

Cai et al. [89] proposed two-branch framework based on CNN and Recurrent Neural network (RNN) to extract both global and local information, respectively, based on a single frame. For patch selection for RNN processing, they adopted deep reinforcement learning, which overperformed the max SoftMax scores (denoted as the MAX-SCORES method) and the method of selecting patches randomly (denoted as the RANDOM method). Six publicly available face presentation attack databases including CASIA, IDIAP REPLAY-ATTACK, MSU Mobile, OULU-NPU, SiW, and Rose-Youtu databases, were utilized in the experiments.

Roy et al. [90] employed a bi-directional feature pyramid network based on the EfficientDet detection architecture. In addition, they leveraged the properties of 2D Fourier spectra by adding an auxiliary branch in their baseline network. The results were tested on the OULU-NPU and REPLAY-Mobile databases. Karmakar et al. [91] proposed to detect the spontaneous facial micro-expressions by analyzing the higher frequency in multidimensional Fourier transform. This method has been applied for detecting replay attacks to distinguish between live and fake facial video streams.

Li et al. [92] detected replay and 3D mask attacks based on differences of motion direction and intensity, as well as differences of texture information between live and fake faces. The proposed architecture included optical flow and texture module, two attention modules (channel attention and region attention), and CNN network. Attention modules generated the corresponding maps, which were fed into the CNN network for feature extraction and image sequence classification. Yu et al. [93] developed a novel deep forgery detector called Patch-DFD based on facial patch mapping and bilinear interpolation with max-pooling operation. Then 5 fixed size feature maps were fused by a local voting strategy. Method has been tested against replay attacks on DeepfakeTIMIT, Celeb-DF (v2), and FaceForensics + ⁣ + databases.

Muhammad et al. [94] attempted to generalize the detection of print attacks and replay attack by proposing a temporal sequence sampling method to accumulate appearance and dynamic information of video sequences into single RGB images. First, the input video was divided into N equal video clips, and the inter-frame 2D affine motion was estimated based on the trajectories of SURF (Speeded Up Robust Features) keypoint matches. Second, the estimated inter-frame 2D affine motion was removed from the video frames, converting the resulting clips into single RGB images. The proposed self-supervised learning scheme helped in fine-tuning a 2D CNN to learn more meaningful representations from the video clips. The method has been tested on REPLAY-ATTACK, CASIA-FAS, MSU Mobile, and OULU-NPU databases. The main drawback was the unreal-time response caused by processing a two seconds video.

3.1.3 Adversarial attack detection

Szegedy et al. [95] were the first who demonstrated that deep learning models are vulnerable to adversarial (small, often quasi-imperceptible perturbations) examples, when the original samples are taken from the data distribution. Deep learning-based face recognition systems are no exception: they are vulnerable to adversarial attacks from both the digital and physical domains. Adversarial attacks have recently emerged that aim to mislead a classifier in a deep learning-based face recognition system [96]. These perturbations are physically imperceptible to human vision, generating synthetic images by adversarial methods [97]. Perturbations are utilized in two modes “no target” (hiding the identity of the user) and “dodging” (accessing the identity of the target user).

The method of 3D printing glasses to create physical attacks is a widespread adversarial attack [69, 98]. Zhou et al. [99] proposed an infrared-based stealthy facial morphing of the subject using “infrared lighting cap”. Nguyen et al. [100] developed a more convenient method to create adversarial attacks using light projections, using a commercially available webcam and a projector. The proposed method generated a digital adversary template using one or more target images available to the intruder, and this digital sample was then projected onto the opponent’s face in the physical area to either impersonate the target (impersonation) or avoid recognition (obfuscation).

Countermeasures against adversarial attacks can be roughly divided into two groups: rectification methods and methods for adversarial detection. Rectification methods increase the robustness of the system [101], while methods for adversarial detection analyze the behavior of the model, detecting anomalous events [102]. The robustness of a model can be increased via adversarial training or regularization methods that train deep learning models. Detection subsystems are often implemented as binary detectors that distinguish genuine and adversarial inputs. A recent detection method based on predictions has shown a good trade-off between detection efficiency and attack resistance [103]. It is based on a k-NN scheme or several RBF-SVMs applied to specific layers of a deep neural network, in other words, to intermediate representations. Massoli et al. [104] captured the evolution of the features extracted from the input at intermediate layers, created a trajectory in feature space, and analyzed the behavior of the original samples and the manipulated samples. A series of experiments, including a realistic scenario with a face recognition system, showed a high degree of resilience against adversarial attacks.

Deb et al. [105] proposed a new self-supervised adversarial defense framework called FaceGuard that can automatically detect, localize, and purify a wide variety of adversarial faces without using pre-computed adversarial training samples.

3.1.4 3D mask PAD methods

Most existing PAD methods proposed for 2D fake surfaces are useless for detecting 3D masks because the 3D mask usually looks more like human skin [61]. Even photographic masks with cut out areas of the eyes and mouth can deceive the recognition system.

Face mask materials can be divided into rigid/non-rigid or soft/flexible [106]. Relatively inexpensive rigid masks can be made from paper, resin or plastic. Soft masks often use latex and silicone materials. They are close to reproducing the color and texture of human skin and can adapt to different sizes, shapes and movements of the face. Currently, some mask materials, which are used in the manufacture of 3D masks, can accurately match the topology of the face, making 3D mask attack detection increasingly difficult and sometimes impossible [107]. Thus, the production of ThatsMyFace mask requires 3D reconstruction and 3D printing techniques to create a custom face mask. At that time, the RealF mask based on the three-Dimension Photo Form (3DPF) technique looks very similar as the genuine face. Advances in manufacturing 3D masks make the 3D mask attack very popular in practice.

Existing 3D mask PAD methods find the difference between real face skin and mask materials. They work at the digital level (texture based, shape based, and deep features based methods) and sensor level (the reflectance/multispectral properties based and other cues/liveness based methods, see in Section 3.2).

Texture based methods utilize various types of LBPs in conjunction with Linear Discriminant Analysis (LDA) [61], linear SVM [108], Euclidean distance classifier [109] and other similar classifiers. Agarwal et al. [110] suggested to use Haralick texture features and got good results on 2D/3D mask spoofing databases including 3DMAD database.

Shape-based methods extract discriminative features using image transformation, shape descriptors or reconstructing geometry cues. Kose et al. [111] described 3D face shape with warping parameters and extracted LBP features on 2D images and 2.5D depth maps for mask attack detection. Tang and Chen [112] used principal curvature measures and meshedSIFT-based features [113] to describe the 3D triangular mesh surface of the face. Hamdan and Mokhtar [114] applied the angular radial transformation to extract shape features from the RGB images, which were then passed to a Maximum Likelihood (ML) classifier. The same authors combined the Legendre moment invariant decomposition of the RGB image with the LDA projection, also using the ML classifier [115]. Wang et al. [116] reconstructed depth cues from RGB images using a 3D morphable model and then looked for the geometry differences between real faces and masks based on the extracted normal features.

One of the first works in deep feature based methods investigated iris, face and fingerprint modalities, trying to optimize a three-layered CNN architecture and filter optimization [117]. The 3DMAD database was one of nine databases (3 for iris, 2 for face and 4 for fingerprints) used in the experiments. This approach was too broad and inefficient, despite the high reported results. Lecena et al. [79] presented a pre-trained modified VGG-16 model (called as FAS-Net) for recognizing photo, video or mask attacks using only static features. The approach was tested on the REPLAY-ATTACK and 3DMAD public databases. On the REPLAY-ATTACK database the accuracy achieved 99.04% and HTER 1.20%. For the 3DMAD, the accuracy was of 100.00% and HTER 0.00%. Manjani et al. [118] collected a Silicone Mask Attack Database (SMAD) (see Section 3.3) and proposed a multilevel deep dictionary for the face PAD. The feature learning was independent of knowledge of attack types and did not exploit particular distinguishing attributes. The algorithm encoded the features of the presented samples from low to high level using SVM as a classifier. Experiments were performed on the SMAD, 3DMAD, CASIA-FASD, UVAD, and REPLAY-ATTACK databases, showing promising results in both intra-database and cross-database protocols.

Liu and Kumar [119] tested 5 CNN architectures and Siamese CNN on their own dataset including a total 13 different 3D face masked subjects and 9 different real subjects. Visible and Near-InfraRed (NIR) images were collected. Experimental results have shown that the NIR image can provide better performance than visible image based images. At the same time, the multispectral imaging provided better performance than using visible and NIR images separately. Shao et al. [120] developed a novel feature learning model to learn discriminative deep dynamic textures for 3D mask face anti-spoofing. The main assumption is that a real face displays different facial motion patterns compared to the 3D mask. These subtle facial motions can be captured by convolutional layers, forming multiple deep dynamic textures. The proposed joint discriminative learning strategy was further incorporated in the learning model to jointly learn the spatial- and channel-discriminability of the deep dynamic textures. Intra-database and cross-database evaluation on the 3DMAD database and their own supplementary dataset indicated the efficiency and robustness of the proposed method.

Chen et al. [121] developed a multi-modal dynamics fusion network for 3D face mask anti-spoofing. Dynamic texture and shape clues were encoded with a two-branch deep CNN model at different rates and intervals for a more comprehensive description. Various poses were also considered. This approach has been evaluated on the 3DMAD, HKBU-MARs V1 and SMAD public benchmarks. Birla and Gupta [122] developed the PATRON system, which utilized the respiration rate extracted from video and face alignment to detect 3D mask attack. The authors applied multi-kurtosis optimization to extract the respiratory signal from the resultant temporal signals. The experiments with samples from the public database HKBU-MARsV1 + gave an error of 14.7%.

Furthermore, some methods tried to combine deep learning based features with hand-craft features and achieved outstanding results in mask spoofing detection [120]. This strategy can adaptively weight the learned features to make better discriminative deep dynamic textures more meaningful.

3.1.5 Detection of disguised, makeup, plastic surgery attacks, and morphing attacks

Disguised, makeup and plastic surgery are direct attacks, intentionally or unintentionally impersonate or obfuscate. Unintentional disguises include sunglasses, hats, or scarves. Disguised parts of the face can significantly interfere with recognition [123]. Disguise attacks are used in border crossing and airport security applications [124]. Easily available and cheaper makeup is similar to disguised attacks, but is harder to identify as it closely resembles the real face [125].

Facial plastic surgery is divided into two categories – reconstructive and cosmetic. Reconstructive facial plastic surgery corrects anomalies in facial features, while cosmetic facial plastic surgery improves the visual aspect of facial structures and characteristics. After undergoing cosmetic surgery, a person may attempt to fake a facial biometric system, which would be a plastic surgery attack. After plastic surgery, regions of the face, including the nose, eyes, lips, ears, or bone structure, are reshaped to obtain the desired appearance or as a result of some disease treatment surgeries [31]. In this case, biometric data must be updated in the database of the recognition system.

Facial automated border control systems at airports and seaports, which are widely deployed in many countries, compare the gate image and the biometric reference with the passenger’s electronic machine readable travel document chip, computing a similarity score [126]. The fact that the original biometric reference is not stored in a special database provokes a new type of presentation attacks called morphing attacks, which involves two or more persons. First, an intruder (e.g., a wanted fugitive) and an accomplice (e.g., a citizen) take their own face photos and create a morphed face image with two captured photos. Second, they improve the visual quality of the morphed face image by optimizing the morphing parameters and using some post-processing operations to reduce the blending artefacts. Third, they evaluate the morphed face image using publicly available face verification engines, and the accomplice applies for a passport by using the morphed image and the accomplice’s own personal information. As a result, the intruder may cross an automated border control system using an accomplice’s electronic machine readable travel document. Peng et al. [127] proposed to detect a face morphing based on the watchlist as reference. This recognition scheme is traditional and is aimed at comparing facial feature vectors from a suspect image and the biometric reference contained in the watchlist. The authors wrote that their approach has better generalization ability to unseen types of face morphing attacks and morphing parameters than existing methods. They investigated the variations in facial expressions, face angles, and image qualities using their own collected database for face morphing attacks.

4.1 Transfer learning with domain generalization and domain adaptation

Domain generalization methods for PAD extract common differentiation features to improve generalization. However, it is difficult to find a compact and generalized feature space for fake faces due to the wide variety of fake face distributions from different domains. Jia et al. [51] proposed an end-to-end single-side domain generalization framework to improve the generalization ability of PAD by making the feature distribution of the real faces compact, while the feature distribution of the fake faces of different domains became more separated. Spatial and temporal auxiliary supervision was based on face depth as spatial cues and rPPG signals (pulse) as temporal cues. Autoencoders can be used to align the distributions of source domains for generalized features [138]. Shao et al. [48] exploited the shared and discriminative data across multiple PAD domains to automatically search and learn a generalized feature space. Their multi-adversarial deep domain generalization was performed under a dual-force triplet-mining constraint, but not as end-to-end solution.

Chen et al. [139] introduced a novel camera invariant face anti-spoofing model, which consists of two branches: the feature invariant branch and the feature discrimination augmentation branch. This model was evaluated on four face anti-spoofing databases: CASIA-FASD, REPLAY-ATTACK, OULU-NPU and MSU-MFSD with good intra-database and cross-database results.

Liu et al. [140] proposed a framework for extracting domain-invariant features via adversarial learning and combining low-rank decomposition methods to detect replay attacks. This approach improved the model’s generalization ability to unseen scenarios. The extensive experiments demonstrated that the proposed method achieved SOTA results on four public databases, including CASIA-MFSD, MSU-MFSD, REPLAY-ATTACK, and OULU-NPU.

Domain adaptation methods minimize the distribution discrepancy between source and target domain by using unlabeled target data [141, 142, 143]. These methods explicitly explore the relationship among multiple source domains without gaining access to any target data through unseen attacks. Most of the traditional domain adaptation methods focus on minimizing the distribution discrepancies across multiple source domains to extract domain-invariant features. Thus, Wang et al. [144] proposed an unsupervised domain adaptation with disentangled representation approach to improve the generalization capability of PAD into new scenarios. El-Din et al. [145] proposed a novel end-to-end domain adaptation-based architecture utilizing deep embedding clustering of target domain to generalize face PAD. Jia et al. [146] proposed a unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing. Two modules were developed: one module to minimize the marginal distribution between the source and the target domains to extract domain-invariant features, and another module to make the features of the same class compact.

4.2 Anomaly detection

The PAD statement can be formulated in such a way that genuine face images are considered normal samples, while all possible attacks form the anomalous sample space. The main hypothesis is that the genuine face class has lower variance in the feature distribution and forms a close cluster. At the same time, attacks have a higher variance and can be considered as anomalies in the feature space. Any samples outside the margin of the genuine cluster are considered anomalies, e.g. the attacks. Thus, unseen attacks can be detected more accurate.

Even through many PAD methods are based on the binary classifiers, they have some disadvantages, mentioned below, with respect to the one-class classifiers approach:

•

Building a decision boundary between classes is a non-trivial task.

•

The generalization of two-class classifiers in the presence of novel or unseen attacks is low.

•

Difficulties in collecting training databases for binary classifiers, especially 3D mask databases [54].

•

Spoofing attacks are not predictable due to their nature or capture devices.

•

Client-specific solutions cannot be applied to methods with binary classifiers [147].

Nikisins et al. [148] proposed a face PAD system interpreting print and replay attacks as anomaly detection. A One-Class Classification (OCC) problem was solved using image quality measures and a Gaussian Mixture Model trained to represent the probability distribution of bona-fide samples. A challenging aggregated database composed of three publicly available databases (REPLAY-ATTACK, REPLAY-Mobile and MSU-MFSD) was introduced.

Fatemifar et al. [149] investigated the client specific models in the context of anomaly detection, as opposite to the first study of using client-specific information for spoofing detection proposed in [150]. They used four one-class anomaly detectors based on support vector data description, sparse representation based classifier, the Mahalanobis distance, and a Gaussian mixture model. Experiments involving three spoofing databases (REPLAY-ATTACK, REPLAY-Mobile, and Rose-Youtu) confirmed the merits of the client-specific anomaly detection approach. In the further study [151], new score normalisation method was proposed to normalise the output of individual outlier detectors before fusion. Li et al. [152] proposed a CNN-based method for face anti-spoofing against unseen type of attacks as a one-class classification. They developed a distance-based loss function (hypersphere loss function) and organized the training process as an end-to-end supervision. Hypersphere loss function identified the attacks directly without using an additional classifier. Feng et al. [153] proposed a framework consisting of a spoof cue generator and an auxiliary classifier. The generator minimized the spoof cues of live samples, while an auxiliary classifier served as a spoof cue amplifier to make the spoof cues more discriminative.

Anomaly-based detectors based on genuine-access data only have generalization performance advantages over binary spoofing attack detection methods. Fatemifar et al. [154, 155] leveraged the fusion of multiple anomaly classifiers using weighted averaging. Their novel three-stage optimisation method combined a hybrid optimisation method using genetic algorithm and pattern search to explore the weight space, a two-sided score normalisation method to improve the anomaly detection performance, and an ensemble pruning method to improve the generalisation performance. In addition, client-specific information was incorporated to train the proposed model. Experiments with REPLAY-ATTACK, REPLAY-Mobile and Rose-Youtu databases have demonstrated the effectiveness of SOTA anomaly-based and multiclass approaches.

Pérez-Cabo et al. [156] introduced anomaly detection strategy based on deep metric learning using just still images. For regularization, they used a triplet focal loss in the form of a “metric softmax” loss and obtained promising experimental results. Baweja et al. [157] presented another deep-learning solution for anomaly detection, in which both classifier and feature representations are learned together end-to-end. For this purpose, they introduced a pseudo-negative class, which was modeled using a Gaussian distribution followed by the application of pair-wise confusion loss to further regularize the training process. This approach was tested on four publicly available databases: REPLAY-ATTACK, Rose-Youtu, OULU-NPU, and SiW databases. Favorskaya and Pakhirka [158] have built visual maps aimed at finding specific visual anomalies caused by print and replay attacks, using a hybrid approach with creating the visual maps by traditional methods and deep learning for better generalization of the final result.

4.3 Few-shot and zero-shot learning

Few-shot learning is the process of learning from few samples with the supervised data (labeled target samples). Few-shot learning is especially useful when training requires large scale databases. When the number of labeled samples for target class is zero, few-shot learning is called zero-shot learning. Therefore, zero-shot learning is very suitable for detecting unseen or novel attacks. Few-shot learning aims at learning from very few instances from unseen categories, while zero-shot learning requires recognition unseen categories by learning only from the description or semantic information of these unseen categories.

However, currently few-shot and zero-shot learning for the PAD methods is underdeveloped. It should be noted that existing few-shot learning methods often use prior knowledge from one single modality, while zero-shot learning methods frequently explore multi-modality data [159].

Liu et al. [55] investigated the problem of zero-shot face anti-spoofing for 13 types of spoof attacks, including print, replay, 3D mask, makeup, and so on. The proposed deep tree network divided the spoof samples into the most similar spoof clusters (semantic sub-groups) without supervision and also studied the features in a hierarchical way. Each tree node consists of a convolutional residual unit, which is a block with convolutional layers and the short-cut connection. Tree routing unit defines a node routing function to route a data sample to one of the child nodes according to the largest data variation. The leaf node consists of a supervised feature learning module, which concatenates classification supervision and pixel-wise supervision to learn spoofing features.

Qin et al. [160] proposed a novel adaptive inner-update meta face anti-spoofing method to tackle a zero- and few-shot learning problem through meta-learning inspired by the model-agnostic meta-learning [161]. Meta-learner is focused on unseen attack detection by training on predefined genuine and fake facial images and providing better discrimination. First, the proposed method was trained by a meta-learner on zero- and few-shot PAD training tasks simultaneously. Second, an adaptive inner-update strategy improved the performance. The authors also proposed three zero- and few-shot PAD benchmarks and conducted experiments on both the proposed benchmarks and existing zero-shot protocols.

4.4 Auxiliary methods

The auxiliary estimation models are trained with some reliable auxiliary information, which is no longer a simple binary label. Auxiliary information has the advantage that it can be better generalized to the PAD task, and it does help to distinguish fake images from bona fide images. Depth, rPPG, reflection, noise, and disparity are the main auxiliary features.

Atoum et al. [13] were the first to consider the estimated face depth as a training signal. They suggested a two-stream CNN-based face anti-spoofing method for detecting print and replay attacks. A deep end-to-end network was used as the patch-based CNN stream, extracting scores of randomly extracted facial patches. A fully convolutional network estimated the depth of a face image, assuming that a print or replay presentation attack has a flat depth map, while live faces contain a normal face depth. The outputs of the both streams were fused for the final solution. The proposed method was evaluated using three benchmark databases: CASIA-MFSD, MSU-USSA, and REPLAY-ATTACK, achieving HTER values 2.27%, 0.21%, and 0.72%, respectively. Later these authors proposed a method for estimating the dense depth map for a live or spoof face image for mobile PAD scenarios [162]. Rehman et al. [163] studied the dynamic disparity features for depth estimation by introducing a disparity layer within CNN. This approach was tested on a collected stereo camera-based face anti-spoofing database.

Liu et al. [14] focused on the known spoof patterns across spatial and temporal domains rather than cues extraction to detect print and replay attacks. Their network combined the CNN and RNN architectures in a coherent way. The CNN part supervised for differences in depth maps and extracted feature maps in parallel. Then the depth map and the feature map were aligned with the proposed non-rigid registration layer. The RNN part was trained with the aligned maps and the rPPG supervision. The Average Classification Error Rate (ACER) in cross testing reached 10–14%.

Niu et al. [164] introduced the real-time rPPG method for continuous heart rate measurement from facial videos. The proposed multi-patch ROI strategy removed outlier signals followed by spatio-temporal analysis, replacing binary labels to supervise the CNN and RNN, respectively. Such auxiliary information is often used for reliable classification.

Lin et al. [84] proposed a generalized method exploiting both rPPG and texture features in terms of 2D and 3D mask attacks. The rPPG information in the form of multi-scale long-term statistical spectral features with variant granularities was incorporated with contextual patch-based CNN. This method has been tested to detect all three attack types using 3DMAD, HKBU-MARs V1, MSU-MFSD, CASIA-FASD, and OULU-NPU databases.

Jourabloo et al. [165] simulated noise patterns as auxiliary information by decomposing a spoof face into a spoof noise and a live face, and then using the spoof noise for classification. Such decomposition is based on the hypothesis that a spoof image can be generated by a bona fide image with multiplicative noise and additive noise. However, noise modeling is inefficient due to the lack of ground truth information. Xu et al. [166] proposed using the bona fide image of the corresponding subject as a type of ground truth in the training set. For this purpose, a metric-learned end-to-end network was developed to simulate face spoofing noise. Yang et al. [16] developed a spatio-temporal anti-spoofing network that extracted both global temporal and local spatial cues to distinguish live faces versus spoof faces.

Wang et al. [167] developed a new method to estimate depth information from multiple RGB frames, efficiently encoding spatiotemporal information for replay attack detection. Optical flow guided feature blocks take single-frame features from two consecutive frames to estimate short-term motion. The extracted features are then fed into the convolution gated recurrent units to obtain long-term motion information and output the residual of single-frame facial depth. The combined estimated multi-frame depth maps were supervised by the depth loss and binary loss. Four databases OULU-NPU, SiW, CASIA-MFSD, and REPLAY-ATTACK were used in the experiments. Intra-testing of the OULU-NPU database and the SiW database reached 9.2% and 3.1% ACER, respectively, while cross-testing between the CASIA-MFSD database and the REPLAY-ATTACK database yielded 17.5%, and 24.0% HTER, respectively.

George and Marcel [168] used pixel-wise binary supervision instead of using synthesized depth values as an auxiliary features. Furthermore, both binary and pixel-wise binary supervision was used by adding a fully connected layer on top of the pixel-wise map to protect against replay attacks. Kim et al. [169] proposed to apply a depth map and a reflection map as bipartite auxiliary supervision for PAD task. The developed bipartite auxiliary supervision network extracted and fused these auxiliary features to detect presentation attacks, in particular print and replay attacks.

Yu et al. [17] introduced a Central Difference Convolutional Network (CDCN), in which central difference convolution was able to extract the invariant detailed spoofing features, such as lattice artefacts. They proposed an extended version of CDCN consisting of the searched backbone network and multi-scale attention fusion module for aggregating the multilevel central difference convolutional features. These authors also presented a multi-modal variant using two fusion strategies for three modalities (RGB, depth and infrared) [20]. Yu et al. [19] proposed a bilateral convolutional network capturing intrinsic material-based patterns via aggregating multi-level bilateral macro- and micro-information. Thus, traditional bilateral filtering was integrated with deep network. Multilevel feature refinement module in the form of a three-head supervision combined depth (shape), reflection and patch (texture) information.

4.5 Multi-spectral methods

Most direct attacks are carried out using visible range sensors as the easiest way to fool any facial recognition system. This means that a sensor operating in different range can provide more cues for the PAD methods. Thus, the SWIR camera can be successfully used to detect 3D mask attacks [134]. As mentioned in [37], four different modalities are often used for this purpose, i.e. the visible range, SWIR, Mid-Wave InfraRed (MWIR), and Long-Wave InfraRed (LWIR). As is well known, the infrared spectrum consists of active (NIR and SWIR) and passive (also called thermal) (MWIR and LWIR) IR ranges. The NIR face images detect illumination direction changes and help in low-light conditions, while the SWIR range is suited for surveillance. Passive or thermal IR range is emission dominant, and thermal images capture thermal signatures of the skin tissue.

Nikisins et al. [170] proposed a multi-channel face PAD approach based on three types of devices. They created a stack of gray-scale, NIR, and depth facial images or a stack of images of facial features and then applied a set of pre-trained multi-channel encoders and a multi-layer perceptron for classification. For training multi-channel CNN with three modalities, they used the domain adaptation technique, transferring the knowledge of facial appearance from RGB to multi-channel domain.

Jiang et al. [171] presented a dataset of paired visible and NIR images with distance, pose, expression and session variations for print and 3D mask attacks. They also developed a multilevel CNN processing paired visible and NIR images. Multispectral data (color imagery, NIR imagery and thermal imagery) against the custom silicone mask attacks was used in [172]. Wang et al. [173] demonstrated that their end-to-end CNN with four branches with RGB, depth, IR, and fusion modal inputs, exploiting multi-modal fusion approach via spatial and channel attention overperformed other CNNs for detecting print attack. Jiang et al. [174] proposed a novel multiple categories image translation generative adversarial network that generated corresponding NIR images for visible live and spoof face images instead of using near-infrared equipment. Liveness detection using NIR and RGB images is also a popular approach to detect replay attacks [175].

George et al. [176] developed a multi-channel CNN (color, infrared, depth, and thermal channels) using transfer learning from a pre-trained face recognition network LightCNN. They also created a Wide Multi-Channel presentation Attack (WMCA) database that captures by multiple devices and channels and contains a wide variety of 2D and 3D presentation attacks. A two stream CNN that worked in RGB space and multi-scale Retinex as an illumination-invariant space has been proposed to detect print and replay attacks over both intra-database and inter-database protocols in [177].

Castelblanco et al. [178] investigated two dynamic face-authentication challenges (the camera close-up and head-rotation) to solve the task of liveness detection and face verification. A set of neural-network models based on the CNN and Siamese neural network architectures has been proposed. Their own dataset of 177 live videos recorded by 41 different subjects included multiple types of media-based attacks (printed- attacks, screen-attacks, 3D masks, videos acquired from public social media, deep fakes), for a total of 243 attacks in uncontrolled scenarios.

6.1 Evaluation metrics

Anjos et al. [207] proposed to use Half Total Error Rate (HTER) as an evaluation metric for various PAD methods in 2011. Like any recognition system, the PAD system has two types of errors: false rejection, when the real accesses are rejected, and false acceptance, when the attacks are accepted. The HTER metric combines the False Rejection Rate (FRR) and the False Acceptance Rate (FAR) as follows:

𝐻𝑇𝐸𝑅 = 𝐹𝑅𝑅 + 𝐹𝐴𝑅 2 ,(5)

where FAR and FRR are defined as

𝐹𝐴𝑅 = 𝐹𝑃 𝐹𝑃 + 𝑇𝑁 𝐹𝑅𝑅 = 𝐹𝑁 𝐹𝑁 + 𝑇𝑃 .(6)