Robust And Discriminant Local Color Pattern (RADLCP): A novel color descriptor for face recognition

1. Introduction

The prominence of local descriptors is justified by results achieved by them in different applications. The accuracy and computational complexity of these local descriptors are quite remarkable. Some of these applications in which these descriptors are tested and evaluated are Face Recognition (FR), Object Recognition (OR), Palmprint Recognition (PR), Texture Analysis (TA), Scene Matching (SM) and Ear Recognition (ER). Apart from these the local descriptors has also proven its ability in the multimodal biometric systems. In multimodal biometric systems two or more biometric traits are used for building the feature size. The local descriptors forms their feature size by extracting and joining the features from different image portions. These image portions are ear, eyes, nose, mouth and forehead. Most of the local approaches builds their methodology by deploying the small patches on the pixels regions. This makes local descriptors much effective and efficient than the global approaches, where they operates on the entire image for the feature extraction. Although the task of feature reduction is well performed from the global descriptors. In unconstrained conditions the local descriptors performs better than the global descriptors. The hybrid one is better than either of two. The unconstrained conditions are caused by the various image transformations and these are light, emotion, pose, blur, noise, corruption and occlusion. One of the successful local descriptor exists in literature is Local Binary Pattern (LBP) [2]. LBP was originally invented for TA and then it is used successfully in other applications also. LBP is the first descriptor which works on the gray pixels by using the small 3 × 3 patch. This methodology proves very effective and achieves impressive results. In LBP, the neighbors are transformed to 1 by comparing the neighbors with center pixel else 0 is given as the label. By putting weights and adding values builds the LBP image from which the histogram size is derived. LBP possesses various advantages and these are invariance monotonic property and less complex algorithm. After LBP emergence, various disadvantages are also observed by the researchers and these are limited spatial patch, large size generation after the feature extraction, noisy function (thresholding) and ineffective in worse lightning changes.

This results in various LBP variants and these LBP variants beats the performance of LBP and many others. Description of some of the LBP variants are defined as: Shakoor et al. developed the feature selection and mapping of LBP to resolve feature size problem. Precisely by proposing mapping methods the feature reduction is done and then these features are mapped into the histogram. All developed methods are light & rotation invariant. Furthermore the discriminant features are selected by using the method called as constrained method. On various datasets the proposed methods proves its potency [3]. Karanwal et al. proposed the Triangle LBP (TLBP) and Orthogonal LBP (OLBP) for two different challenges i.e. pose and expression variations. TLBP feature extraction is done in the vertical and horizontal directions by utilizing 5 × 3 and 3 × 5 image patches by triangle rotation in 180^∘ and 0^∘ degree directions. For extracting OLBP features the orthogonal locations are used. To build a discriminant feature size both features are merged and called as TAO-LBP. On five datasets TAO-LBP justify it’s potent [4].

In passing years, the deep learning methods has attracted the lot of attention in FR application. The robustness and discriminativity achieved by them (in unconstrained conditions) is the prime reason behind this. On most cases, the deep learning methods proves better than the local and global methods. The influential structure and concept makes this possible. Some of the popular deep learning methods used successfully in literature are AlexNet, CNN, VGG and LetNet. CNN build its size by using the final layers (completely connected) of pre-trained CNN models. The other layers activation function can also be used for extracting the features. These are softmax function and classification layers. There are several demerits observed in the deep learning methods and these are: their computational cost is on the larger side, training data requirement in huge amount and difficulty in adapting the parameter settings. These factors cannot allow to impose the deep learning framework in the proposed work. Besides these there are some local descriptors which proven out better than these deep learning methods in the unconstrained conditions, in terms of accuracy and computational cost. This allows to develop the novel local descriptor in different unconstrained conditions. Literature also suggests the incorporation of local descriptors with deep learning methods. The results of such methods are quite astonishing.

In recent times, the color based local descriptors has gathered significant attention in FR. The color based descriptors outperforms comprehensively the gray scale based descriptors and also the other ones. There are different ways in which the color based feature descriptors are created. Some uses the traditional color formats for feature extraction like RGB, YCbCr, YIQ, Lab etc. Others make the hybrid combination from the different color formats for feature extraction. All these work attain good results with respect to the challenge they were implemented. The main objective is to find the discriminant and the robust hybrid format for the feature extraction. In proposed work the robust and discriminant hybrid format is created and then used for feature extraction. The extraction of features are done by three well known descriptors and then steps of compression and matching are done. The major contributions of the invented work are given in the next paragraph.

In [1] Karanwal et al. introduced the novel color descriptor in Face Recognition (FR) called as Fused Local Color Pattern (FLCP). In FLCP, the RGB color format is utilized for extracting features. From R, G and B channels, the MRELBP-NI [5], 6 × 6 MB-LBP [6] and RD-LBP [7] are imposed for feature extraction and then all are integrated to form the FLCP size. FLCP beats the accuracy of various methods. The one major shortcoming observed in [1] is that the basic format RGB is used for extracting features. Literature suggests that other hybrid format achieves better recognition rates than RGB. Motivated from literature, the proposed work uses the hybrid color space format RCrQ for feature extraction. In this format R channel is taken from RGB, Cr channel is taken from YCbCr and Q channel is taken from YIQ. On R channel, MRELBP-NI is imposed for extracting features, On Cr channel 6 × 6 MB-LBP is imposed and on Q channel RD-LBP is imposed for extracting features. Then all channel features are joined to build the robust and discriminant feature called as Robust And Discriminant Local Color Pattern (RADLCP). Compression and matching is assisted from PCA [8] and SVMs [9]. For evaluating results GT [10] face dataset is used. Results proves the potency of the RADLCP in contrast to the gray scale based implemented descriptors. RADLCP also beats the results of FLCP. Several literature techniques are also outclassed by RADLCP. For evaluating all the results MATLAB R2021a is used.

Road map: Some of the related works are discussed in Section 2, description of all descriptors are defined in Section 3, results are evaluated and posted in Section 4, discussions are given in Section 5 with conclusions and future directions in Section 6.

2. Related works

Khanna et al. proposed novel method for Expression Recognition (ER), by using LBP and STFT techniques. For acquiring local set of features LBP is used and for acquiring frequency set of features the STFT is used. The amalgamated features of LBP and STFT is made compacted by FDR, chi square test and variance threshold. For matching those features the SVMs is used. Results conducted on various datasets proves the capability of merged feature [11]. Latha et al. imposed the new color descriptor MCLBP for Image Retrieval (IR). For strengthing the CBIR system, the HSV color format statistical features are generated by using standard deviation and mean for developing the feature size. By using MLCP and HSV color format there is immense improvement in the recognition accuracy. On various benchmark datasets the invented method proves its potent by defeating the accuracy of various others [12]. Shakoor et al. developed the feature selection and mapping of LBP to resolve feature size problem. Precisely by proposing mapping methods the feature reduction is done and then these features are mapped into the histogram. All developed methods are light & rotation invariant. Furthermore the discriminant features are selected by using the method called as constrained method. On various datasets the proposed methods proves its potency [3]. Karanwal et al. [4] proposed the Triangle LBP (TLBP) and Orthogonal LBP (OLBP) for two different challenges i.e. pose and expression variations. TLBP feature extraction is done in the vertical and horizontal directions by utilizing 5 × 3 and 3 × 5 image patches by triangle rotation in 180^∘ and 0^∘ degree directions. For extracting OLBP features the orthogonal locations are used. To build a discriminant feature size both features are merged and called as TAO-LBP. On five datasets TAO-LBP justify it’s potent. Luo et al. developed the novel method Improved LBP (ILBP) to overcome the shortcomings of LBP. In ILBP, the descriptor is developed with 2 operators and these are LBP based on ranking magnitude and segmentation operator of the global threshold. In contrast to other methods the ILBP much finer than others [13]. Karanwal et al. presented the ILBP for FR. ILBP is proposed to complement the LBP, as LBP doesn’t build the balanced transformed image for feature extraction. ILBP considered two major steps and these are (i) mean statistic is utilized for the neighbor’s comparison and (ii) negative threshold is used for comparison. As a results the transformed image generates much finer histogram feature than LBP and various others. ORL and GT are the two datasets utilized for the evaluation purpose [14]. Wajih et al. introduced novel method Center Symmetric LBCNN (CS-LBCNN) for recognition of handwritten bilingual digit. CS-LBCNN addresses the issue of LBCNN. Additionally the improved version of CS-LBPCNN is also produced to resolves the issue of 0 thresholding function. This improved version is called as TCS-LBPCNN. The developed methods are compared to the various other methods to check its efficacy. The developed methods proves better than the various existing LBCNN models [15]. Karanwal et al. introduced two variants of LBP in FR called as ND-LBP and NM-LBP. In former descriptor neighbor pixels are compared lined up in the direction clockwise, for building its feature size. In latter one, the comparison is done among neighbors and mean to build its feature size. Further the features of ND-LBP and NM-LBP are merged to build the discriminant face descriptor called as ND-LBP+NM-LBP. For compaction and matching PCA and SVMs are used. Results done on ORL and GT datasets shows that the fused descriptor conquer the accuracy of individual descriptors and various methods from literature [16].

Wei et al. presented the novel IR method by integrating the deep and texture features. Initially HOG features are generated based on LBP by utilizing full directional derivative (IInd order), in which more gradient details are extracted through emotion simplification method of derivative (full directional). Then quaternion emotion technique is proposed for color image and Hu moments are generated, which is the color image representation by merging the color and texture details. Ultimately the deep features are derived from the enhanced VGG. The all above defined features are joined for retrieval operation. On there datasets, the invented method justifies its ability [17]. Vu et al. proposed the FR which is mask based, by merging the CNN and LBP. First, the RetinaFace (CNN technique) is used as the efficient and fast encoder, which learns jointly the extra-supervised and self-supervised features from multiple scales. Furthermore, LBP features derived from nose, eyebrows, forehead and eyes are joined with the earlier extracted features from RetinaFace. Results conducted on various datasets confirms the capability of the developed method, which beats the accuracy of the various techniques [18]. Sajwan et al. discovered the novel color IR method by using Diagonal LBP (DLBP) and the new distance metric. In DLBP, the diagonal LBP features are extracted and the color format used is L*a*b*. On Wang-1k dataset, the DLBP justifies its capability be defeating various other methods [19]. Ganguly et al. [20] developed the Copy Move Forgery Detection (CMFD) method by using LTrP. First the image is decomposed in different blocks and then LTrP features are extracted from every block. All are integrated to form single vector. Then lexicographic sorting is performed for the features generated from the blocks and same blocks are recognized by feature matching from neighbor blocks. The falsely matched blocks are eliminated by utilizing the outlier removal concept called as Shift Vector Aided (SVA). Results on two datasets shows that the invented methods procures better accuracy than the compared ones. Lu et al. presented the color based LBP feature for FR called as TCLBP. In persisting techniques, the inter-channel details are encoded on color channel pairs by deploying the similar spatial structure as used in the encoding of intra channel. As a result the high dimensional size is created and it is not effective in encoding the details of inter channel. Furthermore the pixel difference values along color components are not proper measure if they are numerically not comparable. In TCLBP, the inter-channel details are encoded more efficiently and effectively. On four datasets the TCLBP justify its ability by defeating various others [21].

Karanwal et al. imposed the three LBP variants in FR called as MLBP, MnLBP and CLP. MLBP form its code by comparing the neighbors with the whole patch mean. MnLBP form its code by comparing the neighbors with whole patch median. Both mean and median achieves better results than the LBP, in which center pixel is used for comparison. Further, to make the more informative and effective face descriptor, the features of LBP, MLBP and MnLBP are merged in one framework called as CLP. CLP beats the results of alone descriptors and it also conquers several literature methods. PCA and SVMs are utilized for the compaction and matching. The datasets used for evaluation are ORL and GT [22]. Karanwal et al. presented the MB-ZZLBP in FR. In first step, the mean computation is done from different regions (2 × 2) of 6 × 6 patch. Then zigzag oriented pixels are collated with each other. Specifically, the differentiation is conducted among the higher & lower order pixels (higher-lower). For difference value higher or similar to 0, the 1 is given as the label else 0 is given. Then by putting weights, the MB-ZZLBP code is formed. Results on various datasets proves the capability of the MB-ZZLBP [23]. Chaudhari et al. proposed the method for classification of banana disease by utilizing LBP and GLCM. LBP acquire local features therefore it generates good performance. GLCM is utilized for making the size of feature. Results proves the invented method is better than others [24]. Karanwal et al. introduced the BILBD descriptor for FR, by utilizing the services from three descriptors. These three are MRELBP-NI, RD-LBP and LPQ. In MRELBP-NI, the micro and macro essential are captured effectively therefore it is discriminant than the other descriptors. RD-LBP acquire the radial details, through the pixel difference between higher and lower scales. LPQ acquire the frequency domain details by using STFT in each neighborhood patch. All these three are effective methods therefore there features are merged into one framework. The merged one is called as BILBD. Results on 3 datasets justify its potent by defeating the various methods [25]. Arican et al. presented RGB-D descriptor for OR. In RGB-D, the extraction of features is accomplished by using Bag of Words (BoW), which is novel and efficient technique persisting in literature. This technique develops far better accuracy than the original. The RGB-D results are very encouraging [26]. Shu et al. developed the novel color descriptor MCLBP for TA. To reflect the dependency and correlation between different channels, the MCLBP joins the texture properties of single channel with multi-channel color details. Further, the color differences are decomposed into signs and magnitude of color difference which extends MCLBP to MCLBP+M. Finally the ultimate feature size is produced by joining the sign and magnitude features of color difference. On five datasets the descriptors proves its capability by defeating the various others [27].

3. Discussion of the supplemental descriptors

This work extends the work invented in [1]. In [1], a novel descriptor FLCP is introduced for FR in different unconstrained conditions. FLCP is the color based descriptor and its features are extracted from RGB color format. The R component is preserved for MRELBP-NI feature extraction, G component is preserved from 6 × 6 MB-LBP feature extraction and B component is preserved from RD-LBP feature extraction. Further all component features are merged to form the FLCP size. FLCP beats the results of various descriptors. After evaluating carefully the descriptor invented in [1], the one major shortcoming observed is that color format RGB is used for extracting features. If more robust and discriminant hybrid color format is used then accuracy improvement is assured.

In proposed work a novel descriptor RADLCP is introduced for the FR by using the robust color format. Precisely, more discriminant hybrid color format RCrQ is created by taking R, Cr and Q components from RGB, YCbCr and YIQ color formats. Then R component is preserved for MRELBP-NI feature extraction, Cr component is preserved from 6 × 6 MB-LBP feature extraction and Q component is preserved from RD-LBP feature extraction. Further all component features are merged to form RADLCP size. The compaction and classification steps are attained from PCA and SVMs. SVMs is very effective technique and used in many applications to evaluate the results therefore SVMs is considered for the evaluation purpose. Results are conducted on GT face dataset.

The results analysis is performed in two phases. In first phase, the four descriptors are implemented and used for comparison. These four are MRELBP-NI, 6 × 6 MB-LBP, RD-LBP and FLCP. First three are gray scale descriptors and last one is the color descriptor. RADLCP outclassed the ACC of all compared descriptors. The results achieved by RADLCP is [89.50% 91.71% 93.33%] on TRG = 7:9. The other descriptors procures the ACC rate of [77.25% 78.28% 81.66%], [76.50% 78.28% 81.00%], [74.00% 76.57% 80.33%] and [87.50% 90.28% 91.66%]. All these descriptors are tested on same parameter settings in which the RADLCP is evaluated. In second phase, 21 techniques are picked from literature and used for comparison. These 21 techniques are tested on the same TRG value in which the RADLCP is tested. Although their implementation protocols may be different. RADLCP conquer the ACC of all 21 techniques on the respective training size. This proves the potent of RADLCP in both phases. RADLCP proven as the robust and discriminant face descriptor.

6. Conclusions and future directions

This work introduced the novel local color descriptor for FR so-called RADLCP. RADLCP is the novel color local descriptor which has been inspired from the fusion of multiple descriptors in color space. The proposed work used the robust and discriminant hybrid color format for making the RADLCP size. In [1] Karanwal et al. introduced the novel color descriptor in Face Recognition (FR) called as Fused Local Color Pattern (FLCP). In FLCP, the RGB color format is utilized for extracting features. From R, G and B channels, the MRELBP-NI, 6 × 6 MB-LBP and RD-LBP are imposed for feature extraction and then all are integrated to form the FLCP size. FLCP beats the ACC of various methods. The individual gray scale descriptors are outclassed and various literature methods are also outclassed. The one major shortcoming observed in [1] is that the basic format RGB is used for extracting features. Literature suggests that other hybrid format achieves better recognition rates than RGB.

Motivated from literature, the proposed work uses the hybrid color space format RCrQ for feature extraction. In this format R channel is taken from RGB, Cr channel is taken from YCbCr and Q channel is taken from YIQ. On R channel, MRELBP-NI is imposed for extracting features, On Cr channel 6 × 6 MB-LBP is imposed and on Q channel RD-LBP is imposed for extracting features. Then all channel features are joined to build the robust and discriminant feature called as Robust And Discriminant Local Color Pattern (RADLCP). Compression and matching is assisted from PCA and SVMs. For evaluating results GT face dataset is used. Results proves the potency of the RADLCP in contrast to the gray scale based implemented descriptors. RADLCP also beats the results of FLCP. Several literature techniques are also outclassed by RADLCP. RADLCP achieves the ACC of [89.50% 91.71% 93.33%] on TRG=7:9. The other descriptors procures the ACC rate of [77.25% 78.28% 81.66%], [76.50% 78.28% 81.00%], [74.00% 76.57% 80.33%] and [87.50% 90.28% 91.66%]. All these descriptors are tested on same parameter settings in which the RADLCP is evaluated. The 21 techniques are picked from literature and used for comparison. These 21 techniques are tested on the same TRG value on which the RADLCP is tested. Although their implementation protocols may be different. RADLCP conquer the ACC of all 21 techniques on the respective training size. This proves the potent of RADLCP in both phases. RADLCP proven as the robust and discriminant face descriptor. For evaluating all the results MATLAB R2021a is used.

There are some places where some important essentials are missed out in proposed work. First, extraction of features regionally are not performed in the proposed work. Second, some large scale datasets can be used in the proposed work third, testing on other applications. Fourth, some better global methods could be used for feature compression. Fifth, testing on the other challenges such as noise, blur and age. Sixth, the usage of some other classifiers such as polynomial (SVMs), Nearest Neighbor (NN), Naïve Bayes (NB) and Neural Networks (NNs). All these points develops the scope of the future research and which will be completed in the forthcoming article. Additionally, the other novel local descriptor will be assured in the upcoming article. This novel descriptor will be robust and discriminant in various unconstrained conditions.