Deep acoustic learning on unmanned aerial vehicles for real-time human and drone detection

1. Introduction

Sensing technologies are central to many civil applications, including event monitoring, object classification, search and rescue, and surveillance of both urban environments and disaster zones. The choice of sensors varies based on application needs and environmental constraints. In hostile environments, acoustic sensing and processing prove suitable where other sensor data are inadequate, inapplicable, or unreliable. Examples include applications such as reconnaissance and surveillance with unmanned aerial vehicles (UAVs) for contrasting intrusions,^1–3 and search and rescue with adverse weather or even in hostile environments. ⁴ In such conditions, speech and audio data fusion with video, as described in Cecchinato et al., ⁵ Toma et al.,^6,7 may be infeasible, even if multiple cameras are used as in Liu et al., ⁸ particularly in environments with occlusions, in disaster areas^9,10 or in cases where the target is outside the field of view (FoV). Indeed, in scenarios with low visibility, camera related sensors cannot enable data processing algorithms to achieve significant performance. For this reason, applications based solely on visual data cannot guarantee the desired performance. Similarly, integrating visual and acoustic data may not be appropriate, as it increases algorithmic complexity without benefiting from camera capabilities. In these conditions, the use of only acoustic data overcomes these limitations.

Recently, within the fields of deep learning (DL) and deep neural networks (DNNs), as demonstrated in studies,^11–15 acoustic source localization and recognition (LR) algorithms have received increasing attention in the literature, as highlighted in Kolamunna et al., ¹⁶ Al-Emadi et al., ¹⁷ Kolamunna et al. ¹⁸ The usage of DL and DNNs in acoustic data processing could support acoustic scene awareness. In particular, regarding the application of UAVs, recent work analyzed the combined sound signals emitted by both UAV components, like propellers and electrical engines, and mechanical vibrations. ¹⁸ That study demonstrated that the combined noise features an acoustic signature sufficiently unique to identify specific UAVs among different UAV categories. DL and DNNs are also being investigated in various multi-channel acoustic processing applications, such as those in Choi and Chang, ¹⁹ He et al., ²⁰ Salvati et al., ²¹ where multichannel spectral phase information serves as input to a convolutional neural network (CNN) for the direction of arrival (DOA) estimation. The use of DL in audio processing applications for improving or designing new multi-channel LR algorithms has been investigated only recently,^19,20,22 and its application is being explored in many data-based contexts, as shown in Salvati et al., ²¹ Farhadi et al. ²³ Further investigations about localization of speakers based on DL networks can be found in Varanasi et al., ²⁴ Chakrabarty and Habets, ²⁵ Wang et al. ²⁶

In this study, we introduce and evaluate CNN-based models for feature-based LR tasks involving acoustic signals. The input stages process the phase values computed from the covariance matrix and the module of the frequency spectrum of the multichannel acoustic signals. The output stages perform regression where sound source localization (SSL), in terms of DOA estimation, is required and classification where acoustic source recognition, in terms of class estimation, is required. This is investigated in two real-world scenarios addressing UAV-based sensing and processing of: (1) voice emitted by humans, and (2) propeller noise emitted by UAVs. Specifically, in the experimental framework, UAVs gather information from the environmental acoustic scene using their onboard acoustic sensors, which consist of microphone arrays with different configurations. In both Scenario 1 and Scenario 2, the task of the acoustic method is the LR, namely localization and recognition, of an acoustic source. Localization provides DOA estimation of the human voice in Scenario 1 and of the propeller noise in Scenario 2. Recognition differentiates voice types in Scenario 1 (normal or loud) and distinguishes sounds from different UAVs in Scenario 2. This research is conducted within the context of autonomous systems where sensory data can form the basis for acoustic scene awareness and decision-making. The sensory data are represented by acoustic signals. Two different real datasets with acoustic signals are collected for the two specific scenarios where different acoustic sources are considered as targets. The first dataset consists of voice signals recorded using a small 4-microphone linear array mounted on a UAV (Parrot Bebop quadcopter). The human target is positioned in front of the UAV at various distances and angles and emits vocal signals in both normal and loud voice modes. To generate the second dataset, propeller noise recordings were produced using two different UAVs (a DJI Matrice 200 and an F450 custom-built quadcopter) as targets. The data is collected by a medium-sized 19-microphone spherical array and includes two sound categories according to the target UAVs. In both cases, the DOA and category of the acoustic sources are unknown parameters to be estimated. To benchmark the performance of the LR method, DOA estimation errors, sound recognition accuracy, and computing time are measured under different parameter settings.

In these scenarios, when acoustic data are collected by microphone arrays with small inter-microphone distances installed on multirotor UAVs as described in Cobos et al., ²⁷ Oh et al., ²⁸ Salvati et al., ²⁹ processing becomes challenging due to inadequate spatial resolution, limited signal-to-noise enhancement, and insufficient spatial information. The ego-noise generated by the propellers, electrical engines, and mechanical vibrations of the UAV carrying the microphone array further challenges LR algorithm performance. Considering these limitations, we implemented an automatic acoustic source LR algorithm using a data-driven approach. For this purpose, two alternative DL schemes based on CNNs are exploited for estimation of both DOA and category of the acoustic source. A feature-based approach for learning the models is considered.

The primary motivation for this research is the analysis of the acoustic scene exclusively through audio processing in scenarios that still require adequate investigation in the literature, particularly within the DL framework. Additionally, in scenarios where video information is unreliable or unavailable, integrating audio processing with video analysis becomes infeasible, and algorithms based solely on audio processing must guarantee fully accurate results. In summary, this study introduces two real-world application scenarios based solely on acoustic data processing: (1) speaking subject LR from UAVs, and (2) UAV LR from UAVs, both reproduced entirely at our university. Real acoustic sensors, UAVs, and acoustic sources are deployed for generating two real datasets representing both the acoustic scenes. Deep models using a data-driven approach are then investigated for acoustic source LR. These models are capable of analyzing the multi-channel acoustic signals even in the presence of ego-noise. The evaluation of the LR models is conducted on datasets from both Scenario 1 and Scenario 2, collected in real environments. LR estimation accuracy and computational complexity have been investigated and compared with both the conventional steered response power with the phase transform (SRP-PHAT) method and a state-of-the-art (SOTA) CNN model^30,31 to evaluate the performance and the real-time capability of our approach. Increased dataset diversity is also investigated for Scenario 1 with 2 to 6 speakers, and for Scenario 2 with 6 different UAVs.

The rest of the paper is organized as follows: discussion on related work from the literature is provided in Sect. 2; two real-world scenarios based on LR from UAVs are introduced in Sect. 3; the general architecture of the proposed system with the logical structure of the algorithm is described in Sect. 4; acoustic sensors and experimental datasets based on acoustic data related to the two scenarios are presented in Sect. 5; experiments and results of the LR algorithm based on the two DNN models for a data-driven approach are shown in Sect. 6; a detailed analysis of the outcomes is then reported in Sect. 7; conclusion of this study and future work are the final parts of the manuscript in Sect. 8.

2. Related work

From the analysis of the state-of-the-art, four primary categories, among others, have been identified based on the acoustic applications addressed: 1) human voice localization, 2) human voice recognition, 3) UAV localization, and 4) UAV recognition.

- The first category concerns voice recognition from UAVs. A voice recognition-based detection system for search and rescue operations in large-scale disasters, such as major earthquakes, is presented in Yamazaki et al. ³² A speaker installed on the UAV produces sounds to elicit responses from victims, who are then detected by a recognizer that captures their vocal reactions. The captured acoustic signal consisting of the human voice also contains ego-noise and other outdoor environmental sounds.

A speaker identity verification through voice recognition is presented in Abdulghani et al. ³³ for security and privacy of voice-controlled UAVs. Features are first computed from the audio signals as mel-frequency cepstral coefficients (MFCC), and then a feature matching is applied. The authors employed DL as a soft computing tool capable of enabling intelligent systems that emulate human behavior.

A sound source separation and identification algorithm for processing noise-contaminated acoustic signals is presented in Morito et al. ³⁴ Audio data are gathered using a microphone array embedded in a UAV to detect human voices quickly and over a wide area during disaster situations. The authors propose a partially-shared DNN (PS-DNN) that can learn from a limited amount of annotated data.

- The second category is voice localization from UAVs. An algorithm proposed for drone-based search and rescue operations during disaster management is investigated in Banerjee et al. ³⁵ to estimate the coordinates of the speech source. Specifically, a person requiring assistance (acoustic source) screams, and the drone with an onboard microphone array captures the vocal signal emitted by the human. An approach is proposed to analyze the captured audio signals for acoustic source localization. The ego-noise (comprising noise related to UAV motion, propeller noise, electrical motor noise, and other stationary structural noise) challenges the performance of the SSL algorithm based on time difference of arrival.

The publicly-available DREGON dataset introduced in Strauss et al. ³⁶ was developed for SSL research purposes. It was generated using a microphone array embedded in a UAV. The dataset contains both clean and noisy in-flight audio recordings annotated with the 3D position of the target sound source. It can be used for emerging tasks of UAV-embedded SSL. The study conducted on the dataset showed promising localization performance for broadband acoustic sources in the presence of high noise levels, while speech localization remained challenging under extreme noise conditions.

The authors of the work related to the IEEE Signal Processing Cup 2019 Student Competition, ³⁷ highlight that, although UAVs equipped with acoustic sensors such as microphone arrays could significantly aid in localizing people during emergencies –where video acquisition is severely limited due to reduced or absent visual information caused by poor lighting conditions (e.g. nocturnal or foggy conditions) or by obstacles limiting the FoV– UAV-based acoustic localization has not yet been sufficiently investigated.

- The third category relates to UAV recognition. The drone recognition system studied in Solis et al., ³⁸ based on audio signals generated by UAVs, employs MFCCs as audio features and uses both a support vector machine (SVM) and a CNN for recognizing UAV-generated audio. A small UAV audio dataset was created.

UAV identification under extreme environmental conditions with large dataset requirements is addressed in McCoy et al., ³⁹ where an ensemble DL framework is proposed to counter unauthorized or malicious UAVs. The UAV classification is based on hybrid synthetic and deep features computed from acoustic signals fused with data from other sensors.

UAV recognition in Lee et al. ⁴⁰ is performed using CNN models trained on audio spectrograms along with other data, with the CNN output probabilities subsequently processed by multinomial logistic regression. For the experiments, datasets were both collected through field measurements of real UAVs using audio microphones for acoustic processing and obtained from open online repositories.

- The fourth category corresponds to UAV localization. The drone acoustic detection system (DADS) from Stevens Institute of Technology uses DOA estimation and localization to track UAVs based on their propeller noise. ⁴¹ The system is based on microphones arranged in a tetrahedron. A 16-channel two-tier cross array, the OptiNav 40-microphone phased array, and parabolic and shot gun microphones were also considered. The multirotor UAVs employed in testing were DJI models Phantom 4, M600, and S1000.

A UAV acoustic source localization algorithm based on ESPRIT (estimation of signal parameters via rotational invariance techniques) with Toeplitz matrix reconstruction is proposed in Hu et al. ⁴² Indoor and outdoor tests were conducted with a 12-channel spherical microphone array and a circular micro-electro-mechanical systems (MEMS) microphone array designed by the authors.

The objective in Chang et al. ⁴³ is DOA estimation of an intruding UAV through its acoustic signature, using harmonics extracted from received sound signals. The proposed method first estimates the harmonic frequencies corresponding to the frequency-domain acoustic signal of the UAV. A classifier for multiple signals is then applied to compute DOA estimates for the set of harmonics. A weighted sum of the estimated DOAs is then computed as the drone’s DOA estimate, where the weights are proportional to the harmonic energies.

Recent and detailed overviews on SSL in drone audition are also provided in Chevtchenko et al., ⁴⁴ Martinez-Carranza and Rascon. ⁴⁵

As mentioned above, this study introduces two real-world application scenarios based solely on acoustic data processing: (1) speaking subject LR from UAVs, and (2) UAV LR from UAVs, which are described later in this manuscript. These scenarios were reproduced at our university to generate two real datasets, according to them. Real acoustic sensors, UAVs, and acoustic sources are deployed. This allowed the proposed DL models designed for acoustic source LR to be experimentally validated. In fact, two deep models capable of extracting information from acoustic scenes using a data-driven approach are proposed in this research.

The motivation for considering the analysis of the acoustic scene alone is that the application of multi-channel audio processing for LR tasks requires further investigation in the literature, especially within the rapidly developing field of DL. Moreover, this research addresses the LR problem in scenarios where video information is unreliable or unavailable, such as during nocturnal or foggy conditions. In these cases, audio-based algorithms alone should provide fully accurate analysis of the surrounding scene. An iterative diagonal unloading (IDU) beamforming based on the identification of the dominant signal for DOA estimation in acoustic multi-channel signal processing is studied in Salvati et al. ⁴⁶

Each cited work addresses specific scenarios, models, and issues. We conclude this section with a brief account of recent works most similar to our approach. Hence, the comparison is restricted to the recently mentioned CNN-based solutions. Among those contributions, several studies propose systems for predicting the DOA of sound sources. For instance, the approach in Choi and Chang ¹⁹ focuses on a search and rescue application that employs a simple stereo microphone onboard a UAV and proposes a parametric multi-channel Wiener filter to address UAV ego-noise. Then, power level-based features are extracted (e.g., power level ratio (PLR), power level difference (PLD), and power level summation (PLS)) and fed into a CNN to predict the DOA of the sound source. The CNN architecture comprises a chain of six convolution blocks with two-dimensional convolutions, batch normalization, rectified linear unit (ReLU) activation, and max-pooling layer and two fully connected blocks. In He et al., ²⁰ within the context of multi-speaker DOA estimation, the issue of insufficient labeled training data is addressed by leveraging data augmentation and weakly-supervised domain adaptation. Simulation is used to generate source domain data, while collected real data are annotated with the number of sound sources as weak labels. The real data are then augmented by mixing single-source segments. Finally, weakly-supervised domain adaptation is applied to models pre-trained on simulated data. This approach, according to experiments with real robot audio data conducted by the authors, achieves performance similar to that of fully-labeled real data scenarios. The problem of determining DOA of sound sources as a classification task is addressed also in Varanasi et al., ²⁴ where two CNNs are used to infer elevation and azimuth of sound sources by leveraging on Spherical Harmonic Decomposition. This decomposition enables the extraction of two feature sets containing information about the elevation and azimuth of the sound source. Finally, the authors of Wang et al. ²⁶ focus on DOA estimation in noisy and reverberant environments by exploiting DNNs to identify speech-dominant time-frequency units with relatively clean phase. This is particularly useful for DOA estimation with a DNN trained using only monaural spectral information. This yields a model directly applicable to microphone arrays with diverse geometries.

From the above-mentioned studies, it emerges that one the novel contributions of this work is an approach that combines sound classification and DOA estimation in a complete LR system using two types of non-trivial microphone arrays. Moreover, the associated CNNs are relatively lightweight, using 3-4 convolutions for Scenario 1 (two separate CNNs for sound classification and DOA estimation) and 7 convolutions for Scenario 2 (employing a data fusion strategy to combine both types of predictions). This is a consequence of the effectiveness of the chosen features.

In conclusion, our study primarily focuses on applying the LR algorithm to real scenarios featuring multi-channel acoustic data for two different targets: human voice and UAV sound. We present two datasets collected according to these scenarios, where two microphone array configurations have been employed: a 4-microphone linear array with 16kHz sample rate and a 19-microphone spherical array with 48kHz sample rate). In the revised version of our study, multiple UAVs and voices have been employed to increase data diversity. The algorithm is designed to be suitable for embedded applications. Complexity analysis and comparison with the conventional SRP-PHAT and a SOTA CNN in^30,31 are conducted to verify the performance of the DNN models.

3. Two real-world scenarios with UAVs based on acoustic signal processing

In this section, the two real-world scenarios for analyzing the acoustic scene –(1) a speaking subject LR from UAVs and (2) a UAV LR from UAVs– are described in detail. The term “real-world” refers to non-simulated scenarios representing applications that exist in reality (as opposed to hypothetical scenarios). The two datasets and the extended versions are collected in real environments.

3.2. Scenario 2: acoustic data-based UAV LR from UAVs

An acquisition UAV gathers information from the surrounding acoustic scene using its onboard microphone array. The acoustic sensor is mounted onboard the UAV. It can capture acoustic signals from sources in the surrounding area. In this scenario, the acoustic source is a UAV that performs flight maneuvers in the observed area. It generates propeller noise that propagates throughout the surrounding area.

The acoustic signal received at the microphone array consists of two main components: the propeller noise emitted by the target source (UAV) and the propeller noise emitted by the acquisition UAV where the microphone array is mounted (ego-noise). The collected signals can then be processed through signal processing methods for different tasks, such as detection, recognition, and localization of the target UAV (sound source) by means of the intrinsic characteristics of its propeller noise. Possible application areas include UAV-UAV interaction and counter-UAV operations, Figure 1, where acquisition and processing of acoustic sensory information for UAV LR may enable situation awareness.

OPEN IN VIEWER

Figure 1.

An example of Scenario 2 designed for counter-UAV applications, where a sentinel UAV and interceptor UAVs detect an intruder UAV by acquiring information from the surrounding acoustic scene through microphone arrays.

Acoustic data-based speaking subject LR from UAVs (scenario 1) and acoustic data-based UAV LR from UAVs (scenario 2) are the two real-world scenarios investigated in this work. In both scenarios, the primary cause of performance degradation is the ego-noise generated by the propellers. Several studies on UAV propeller noise characterization can be found in the literature, such as in Insausti et al., ⁴⁷ Moshkov, ⁴⁸ Podsédkowski et al., ⁴⁹ Kingan et al. ⁵⁰

The general system architecture is represented in Figure 2, showing the physical representation of the system –consisting of an acoustic source (either a UAV or a human) and a UAV equipped with a multi-microphone array (upper part of Figure 2)– along with the logical structure of the algorithm for DOA estimation Θ ^ (localization) and class prediction C ^ (recognition) computed through a CNN-based deep network (bottom part of Figure 2). An alternative scheme could employ two different branches without the fusion layer to keep the two paths separate: DOA estimation based on the feature computed as the phase of each element of the covariance matrix ∡ Φ ^ ( k , f ) , where k denotes the time index and f denotes the frequency bin, and the class prediction based on the feature computed as the module of the signal spectrum | X ( f ) | , where f denotes the frequency bin. Both variants are investigated in this research. A formal and detailed description of the different components of the algorithm’s logical structure is provided in the following sections, from acoustic array formulation to CNN-based DNN models.

OPEN IN VIEWER

Figure 2.

General architecture of the system. Physical representation of the system (top) and logical structure of the algorithm (bottom).

4.1. Acoustic array formulation

The acoustic array formulation considers a microphone array with M omnidirectional sensors and an acoustic source in the far-field. Ω = [ Θ ] is the DOA of the acoustic wave in the array, where Θ denotes the azimuth angle. In a single-source scenario, the data model in the short-time Fourier transform (STFT) domain, relative to the multi-channel signal, can be expressed as:

x ( k , f ) = a ( f , Ω ) S ( k , f ) + v e ( k , f ) + v ( k , f )

(1)

where k is the frame time index, f denotes the frequency bin, a ( f , Ω ) is the array steering vector for the source direction Ω , S ( k , f ) represents the source signal relative to the reference sensor, v e ( k , f ) is the non-stationary UAV ego-noise consisting of multiple narrowband harmonic noise generated by both motors and broadband aerodynamic noise of the UAV propellers, and the term v ( k , f ) represents the additive noise as spatially white Gaussian noise characterized by zero mean and variance σ 2 for all microphone sensors.

The covariance matrix of the array signal can be computed as Φ ( k , f ) = E { x ( k , f ) x H ( k , f ) } , which is symmetric and positive definite, where E { ⋅ } denotes the mathematical expectation. The covariance matrix is computed in the frequency range ( f m i n , f m a x ) . The elements in the matrix Φ ( k , f ) denote the correlation between pairs of microphones. Specifically, in array processing, a covariance matrix represents the correlations between signals received by different elements of an array, such as microphones, and is essential for characterizing desired signals and suppressing undesired interference and noise. The covariance matrix is derived from the output vector of the array, which is often complex-valued and represents the signals received by the array. It is organized as a square matrix that containing the variances of each sensor output along its main diagonal and the covariances between pairs of sensors outputs off the diagonal. The covariance matrix can be used in algorithms for interference and noise characterization, adaptive filtering, target localization, and data-driven estimation. Since the covariance matrix is often unknown in practical applications, it has to be estimated. Its estimated model can be computed by averaging over blocks of the multi-channel signal:

Φ ^ ( k , f ) = 1 B ∑ k b = 0 B − 1 x ( k − k b , f ) x H ( k − k b , f )

(2)

where the time offset k b ∈ { 0 , … , B − 1 } ( B denotes the mean number of snapshots) and the frequency f is in the range ( f m i n , f m a x ) . However, due to finite sample size (number of snapshots), signal model mismatches, and the non-stationary nature of the source, there is always some mismatch between the estimated and true covariance matrix. Further details can be found in Toma et al.^31,51,52

On the other hand, the spectrum X ( f ) of the multi-channel signal can be computed for each snapshot.

4.4. CNN-based DNN models

In this study, two deep network architectures are investigated and proposed. The first structure in Figure 3 is employed in Scenario 1. The regression network is separated from the classification network. Sound DOA estimation is based on covariance matrix features. Sound classification uses the single-channel spectrum module features. Both branches pass through a convolution layer before the corresponding head networks for regression and classification, respectively. The layer composition and corresponding parameters are also listed in the scheme.

OPEN IN VIEWER

Figure 3.

Features-based DNN scheme for data-driven acoustic source LR designed for the Scenario 1. The layer composition and parameters are visualized in the scheme.

The second structure in Figure 4 is employed in Scenario 2. Two parallel branches process the input features –specifically the phase values computed from the estimated covariance matrix, ∡ Φ ^ ( k , f ) , and the module of the single-channel acoustic spectrum, | X ( f ) | – through convolution, flatten and linear layers. A convolution fusion layer joins the two branches immediately before the two head networks for sound DOA estimation and sound classification, respectively. Sound DOA estimation is obtained through regression, and sound classification is implemented using a classifier head. The layer composition and corresponding parameters are also listed in the scheme.

OPEN IN VIEWER

Figure 4.

Features-based DNN scheme for data-driven acoustic source LR designed for the Scenario 2. The layer composition and parameters are visualized in the scheme.

Furthermore, network and learning parameter values are also detailed in Section 6.3 to facilitate understanding of the experiments.

The motivation for introducing these two deep networks is to substantially reduce the computational complexity of the algorithm by replacing portions of the computationally intensive multi-channel array processing with data-driven deep learning models without compromising LR algorithm performance.

5. Acoustic sensors and experimental datasets

Experiments with acoustic signals have been conducted with to validate and demonstrate the proposed method. In this section, we introduce two microphone arrays (a 4-microphone linear array and a 19-microphone spherical array) and two acoustic signal datasets (a human voice generated by speech dataset and a UAV noise generated by propellers dataset) used in the experiments. The three UAVs used in the experiments and mentioned in this section are: (1) a Parrot Bebop, ⁵³ (2) a DJI Matrice 200, ⁵⁴ and (3) a F450 custom build quadcopter. ⁵⁵

5.1. Acoustic sensors

1) 4-microphone linear array. The Sony PlayStation Eye (PS3 Eye) ⁵⁶ is a small and lightweight device, as shown in Figure 5(a). In addition to a camera, the PS3 Eye features a built-in four-capsule linear microphone array, enabling technologies for multidirectional voice location tracking, echo cancellation, and background noise suppression. The acoustic array is designed to have the microphones equidistant with inter-microphone distance and total length equal to 2cm and 6cm, respectively. The peripheral can be used in applications such as speech recognition (SR) and audio chat in noisy environments without requiring a headset. The PS3 Eye microphone array operates with each channel processing 16-bit samples at a sampling rate of 16KHz, and a signal-to-noise ratio of 90dB. Several technologies are available for the PS3 Eye. Among these is the PSVR (PlayStation voice recognition), a SR library that is intended to support about 20 different languages.

OPEN IN VIEWER

Figure 5.

a) Sony PlayStation Eye (PS3 Eye) where the 4-microphone linear array is visible on top of the device and b) Zylia ZM-1 consisting of 19 microphones organized in a spherical array.

2) 19-microphone spherical array. The Zylia ZM-1 ⁵⁷ is a multichannel microphonic device used for acquisition. As a compact spherical array, it consists of 19 digital omnidirectional microphonic capsules (XENSIV) based on MEMS technology from German-based Infineon Technologies forming a sphere with a diameter of 9 cm, Figure 5(b). The nominal signal-to-noise ratio is 69dB, the dynamic range is 105dB, and the output linearity is guaranteed up to 130dB. The Zylia ZM-1 is capable of capturing the entire surrounding acoustic scene in 3D. The sample rate and the resolution are 48kHz and 24 bit, respectively. The acoustic gain is adjustable in the range from 0 to 70dB. The microphones do not require re-calibration due to their matched and constant over-time parameters. The front of the sphere is marked with a painted dark-red dot.

5.2. Acoustic datasets

The two experimental datasets described in this section and the related annotation files are freely available at. ⁵⁸

1) Data consisting of human voice generated by speech. The dataset was collected according to the Scenario 1 introduced in Sect. 3.1 and corresponds to the experimental setup represented in Figure 6. It features a human subject whose pre-recorded vocal signals were alternatively emitted by two acoustic loudspeakers placed in front of the drone. The two loudspeakers were positioned symmetrically with respect to the drone’s frontal direction and changed positions in terms of angle and distance, uttering the same sentence one at time in two different voice modes: normal and loud. A set of 30 positions was used for each of the two categories (normal voice and loud voice) according to the following angles ( Θ [degrees] ) and distances ( d [meters] ) in a 2D space: [ − 5 , 5 ] , [ − 10 , 10 ] , [ − 15 , 15 ] , [ − 20 , 20 ] , and [ − 25 , 25 ] degrees at 2 , 3 , and 4 meters. The voice sequence was repeated at each of these positions.

OPEN IN VIEWER

Figure 6.

Setup for Scenario 1, where the acoustic signal generated by human voice is captured by the 4-microphone PS3 Eye array. The main scheme of the experiment is shown in the lower-left part of the image, indicating the positions of the target human (speaker). The horizontal distance values ( d [meters] ) are 2 , 3 , and 4 meters. The azimuth angles values ( Θ [degrees] ) are [ − 5 , 5 ] , [ − 10 , 10 ] , [ − 15 , 15 ] , [ − 20 , 20 ] , and [ − 25 , 25 ] degrees.

An example of an acoustic signal generated by propeller noise of a target UAV (Parrot Bebop) and collected by the PS3 Eye is shown in Figure 7 where all the four channels are visualized. The time duration of the visualized recording is 410 seconds and the sampling rate is 16 kHz.

OPEN IN VIEWER

Figure 7.

Acoustic signal generated by human voice of a target speaker and collected by the PS3 Eye sensor all the 4 channels are visualized in the image). The time duration of the visualized recording is 410 seconds and the sampling rate is 16 kHz.

The ego-noise is a pre-recorded signal previously obtained by recording the propeller noise of a Parrot Bebop UAV using the 4-microphone PS3 Eye array, as shown in Figure 8. In the synthesis method, the ego-noise is summed up by time domain superposition to the signals relative to the targets previously described. The signal-to-ego-noise ratio (SNR) can be set according to the specifications of the experiment. To increase the diversity of the dataset, a software tool for producing different voices from acoustic recordings in the Scenario 1 previously described was utilized. It applies acoustic transformations to the voice signal to change its characteristics, simulating different voices and yielding a number of speakers (more than 10 speakers can be obtained). The drawbacks of this method are the loss of spatial information, normal/loud characteristics, and channel separation. The extended diversity dataset is employed for testing the multi-voice classification performance of the DNN-based recognition algorithm.

OPEN IN VIEWER

Figure 8.

Ego-noise signal generated by propeller noise of a hovering Parrot Bebop UAV and collected by the PS3 Eye array mounted on the UAV (all the 4 channels are visualized in the image).

2) Data consisting of UAV noise generated by propellers. The dataset was collected according to Scenario 2, introduced in Sect. 3.2, and corresponds to the experimental setup shown in Figure 9. The acoustic signals, sequentially emitted by UAVs generating propeller noise while hovering at six specific positions defined by their coordinates, were recorded using the Zylia. Due to its weight, it remains unsafe for a lightweight UAV to carry the Zylia. The altitude of the spherical acoustic array, measured relative to the ground, was 1.60m. Two UAVs with different characteristics, serving as unidentified acoustic sources, flew in the area near the acoustic sensor (first, the DJI Matrice 200, and then the F450, in two separate acquisition sessions). Both UAVs were in hovering mode during acquisition at six coordinate points, according to the following azimuth angles ( Θ [degrees] ) and distances ( d [meters] ) in a 3D space: − 45 , 0 , and + 45 degrees at 5.016 and 10.179 meters. The elevation angles ( ϕ [degrees] ) were: 4.574 and 10.758 degrees.

OPEN IN VIEWER

Figure 9.

Setup for Scenario 2, where the acoustic signal generated by hovering UAV propellers is captured by the 19-microphone Zylia ZM-1 array. The main scheme of the experiment is shown in the lower-right part of the image, indicating the positions of the target UAVs. The 3D distance values ( d [meters] ) are 5.016 and 10.179 meters. The azimuth angles values ( Θ [degrees] ) are − 45 , 0 , and + 45 degrees. The elevation angles values ( ϕ [degrees] ) are 4.574 and 10.758 degrees.

The front direction relative to the Zylia is the reference direction. The experimental setup that reproduces the real acoustic scene is shown in Figure 9. The Zylia is placed in the middle of a meadow at the university campus near the gym building. To increase dataset diversity, four additional UAV targets were employed following the same procedure according to Scenario 2 specifications. The four additional UAVs were: (1) an Aurelia X6 Hexacopter, (2) a DJI Phantom 4 Quadcopter, (3) a DJI Mini 4K Quadcopter, and (4) a Yuneec H520E Hexacopter resulting in a total of six UAV targets (including the two UAVs described previously). This results in a multi-class problem, and the extended diversity dataset is used to test the multi-UAV classification performance of the DNN-based recognition algorithm. Regarding environmental noise, reverberation from obstacles, and other challenging conditions, dataset acquisitions were conducted in open spaces (near roads, industrial buildings, trees, etc.) under outdoor conditions where reverberation and various environmental sounds from the surrounding area were also captured, undoubtedly affecting LR algorithm performance. To create the dataset with acoustic signals consisting of propeller noise recordings according to the six configurations for the hovering UAVs, audio signals collected using the spherical acoustic array were recorded as mp4 files. The recording time for each configuration was 19-30 seconds, the Zylia audio gain was set to 0dB, the data format was 32-bit float, the FFT length was 2048 samples, and the sample-rate was 48kHz. The sounddevice Python module is used in the registration software.

An example of an acoustic signal generated by propeller noise from a target UAV (Matrice 200) and collected by the Zylia ZM-1 is shown in Figure 10 where 8 out of 19 channels are visualized. The time duration of the visualized recording is 33 seconds and the sampling rate is 48 kHz. Tests on the Scenario 2 setup were conducted with the Zylia ZM-1 mounted below an F450 UAV, as shown in Figure 11(a), for ego-noise measurements. Additionally, real ego-noise was collected by flying a UAV (a Holybro X500 V2 quadcopter) as an alternative ego-noise source over the Zylia at a safe distance from the spherical array as shown in Figure 11(b). Since it is unsafe for the F450 to carry the Zylia, and lighter spherical arrays and planar arrays do not offer comparable quality and performance, the hovering state of the acquisition UAV was simulated both by fixing the UAV to a stable support and by flying the UAV over the Zylia. In this manner, the ego-noise is a pre-recorded signal previously obtained by recording the propeller noise of the F450 UAV using the 19 microphone Zylia array. An example is shown in Figure 12. After SNR calibration, in the synthesis method the ego-noise is summed up by time domain superposition to the signals relative to the targets previously described. This simulates the hovering state of the F450. The SNR can be set according to the specifications of the experiment.

OPEN IN VIEWER

Figure 10.

Acoustic signal generated by propeller noise of a target UAV (Matrice 200) and collected by the Zylia ZM-1 sensor (only 8 out of 19 channels are visualized in the image). The time duration of the visualized recording is 33 seconds and the sampling rate is 48 kHz.

OPEN IN VIEWER

Figure 11.

Zylia ZM-1 a) mounted below a fixed F450 quadcopter b) below a flying Holybro X500 V2 quadcopter for testing the Scenario 2 setup and ego-noise measurements.

OPEN IN VIEWER

Figure 12.

Ego-noise signal generated by propeller noise of a hovering F450 UAV and collected by the Zylia ZM-1 array mounted on the UAV (only 8 out of 19 channels are visualized in the image).

This work comprises three core components: system setup, dataset acquisition and processing, and deep learning-based acoustic scene interpretation for localization and recognition tasks. The proposed models are designed for real-world deployment in scenarios where traditional sensing may be limited, such as search-and-rescue or UAV monitoring. By analyzing multichannel audio data, the models extract spatial and semantic information from acoustic scenes, demonstrating robustness even under high ego-noise conditions. The network’s ability to detect and discriminate acoustic signatures of the signals using the CNN and other model layers enables the DNN to mitigate the impact of noise.

Results were obtained under different signal-to-noise conditions (SNR1 and SNR2) and batch sizes (BS1 and BS2) for both scenarios. Validation and testing produced similar values for each metric, SNR, and batch size configuration, for both azimuth angle estimation and class prediction across both scenarios. Overall, a higher ego-noise level (SNR2) slightly increases azimuth prediction errors, although this effect is more pronounced in Scenario 1 than in Scenario 2. A similar pattern is observed for classification accuracy. In Scenario 1, the batch size parameter has essentially no influence. However, when considering testing ( t ) of the deep models in this scenario, a smaller batch size (BS1) results in slightly better performance for both localization and recognition. A larger batch size (BS2) substantially worsens the localization performance in Scenario 2. However, it has no significant influence on recognition accuracy in this scenario. Therefore, using a smaller batch size (BS1) results in better performance, particularly for localization. The conventional SRP-PHAT method and a state-of-the-art CNN model were used in the comparative analysis for DOA estimation performance, and computational complexity was evaluated by measuring single-frame processing time for our method, the conventional, and the SOTA-based approaches to demonstrate the effective real-time capability of the DNN-based algorithm compared to the other two methods. A study of dataset diversity was also conducted to demonstrate classification performance with 2 to 6 speakers in Scenario 1 and 6 different UAVs in Scenario 2.

Representations of acoustic data as they propagate through the DNN are visualized for Scenario 2 to investigate the DNN and how it produces DOA and sound class predictions. Understanding these representations offers two advantages: highlighting the internal processes at the layer level within the DNN and enabling DNN optimization based on these representations. Once fully investigated within the DL field, this study may yield important findings and applications for contexts such as acoustic scene awareness from UAVs for real-time search-and-rescue operations and counter-UAV intrusion detection.

8. Conclusion and future work

This research presented experimental UAV-based acoustic scene awareness through LR in counter-UAV and search-and-rescue applications. Using acoustic sensors –a 19-microphone spherical array and a 4-microphone linear array– information from the acoustic scene can be extracted and analyzed. The motivation lies in the fact that analysis of the acoustic scene exclusively through multi-channel audio processing for LR tasks requires further investigation, especially within the context of rapid developing DL. Specifically, this study addresses the LR problem in scenarios where video information is unreliable or unavailable. In such cases, audio-based algorithms must guarantee accurate results. A real experimental framework based on acoustic data acquisition by UAVs was utilized for LR of acoustic sources in two scenarios: (1) LR of target human voice from UAVs for search-and-rescue applications (the human voice is the source) and (2) LR of target UAVs from UAVs for counter-UAV applications (the propeller noise is the source). Within a data-driven framework, two feature-based DNN models based on CNNs were investigated and utilized to analyze the data gathered from the corresponding acoustic scenes and perform LR tasks involving DOA estimation (localization) and class prediction (recognition). These models are capable of processing the multi-channel acoustic signals even in the presence of UAV ego-noise.

The results demonstrated the validity of the algorithm for both scenarios at different signal-to-noise levels. Moreover, (1) comparison with both the conventional SRP-PHAT method and a selected SOTA CNN model for evaluation of DOA estimation accuracy, (2) computational time analysis demonstrating the real-time capability of the DNN-based approach, and (3) multi-class classification using an increased diversity dataset with 2 to 6 speakers in Scenario 1 and 6 UAVs in Scenario 2 were also conducted in the LR study. Future work will expand the range of acoustic sources and sensing devices to improve model generalizability. Additionally, ongoing research will explore both the optimization of deep neural network architectures, hyperparameters, and interpretability through internal feature visualization and the potential of combining multi-channel audio processing with deep learning. Environmental influences caused by reverberation from walls, buildings, trees, and other obstacles will also be investigated in future work to fully characterize their impact on LR tasks. To improve the suitability of the LR method for UAV-embedded platforms, strategies for reducing the computational complexity –such as model pruning, quantization, and signal downsampling– should be considered. These efforts aim to enhance the practical deployment of audio-based scene awareness in critical applications such as search-and-rescue and UAV intrusion detection.