Deep learning and robotics enabled approach for audio based emotional pragmatics deficits identification in social communication disorders

OPEN IN VIEWER Graphical abstract

The aim of this study is to develop Deep Learning (DL) enabled robotic systems to identify audio-based emotional pragmatics deficits in individuals with social pragmatic communication deficits. The novelty of the work stems from its integration of deep learning with a robotics platform for identifying emotional pragmatics deficits. In this study, the proposed methodology utilizes the implementation of machine and DL-based classification techniques, which have been applied to a collection of open-source datasets to identify audio emotions. The application of pre-processing and converting audio signals of different emotions utilizing Mel-Frequency Cepstral Coefficients (MFCC) resulted in improved emotion classification. The data generated using MFCC were used for the training of machine or DL models. The trained models were then tested on a randomly selected dataset. DL has been proven to be more effective in the identification of emotions using robotic structure. As the data generated by MFCC is of a single dimension, therefore, one-dimensional DL algorithms, such as 1D-Convolution Neural Network, Long Short-Term Memory, and Bidirectional-Long Short-Term Memory, were utilized. In comparison to other algorithms, bidirectional Long Short-Term Memory model has resulted in higher accuracy (96.24%), loss (0.2524 in value), precision (92.87%), and recall (92.87%) in comparison to other machine and DL algorithms. Further, the proposed model was deployed on the robotic structure for real-time detection for improvement of social-emotional pragmatic responses in individuals with deficits. The approach can serve as a potential tool for the individuals with pragmatic communication deficits.