Precise acoustic drone localization and tracking via drone noise: Steered response power

Abstract

This study introduces a new method for precise acoustic drone localization and tracking via the noise generated by the drone. Drone noises include harmonic frequency components which are related to the rotational speed of the propeller and the number of blades. This study integrates utilization of the frequency components around harmonics into Steered Response Power - Phase Transform (SRP-PHAT). First, a custom discrete Fourier transform (namely DFT-Harmonics) is defined which concentrates only on the vicinities of harmonics to capture the frequency components possibly related to the drone sound. Then, DFT-Harmonics is integrated into SRP-PHAT, which is named SRP-Harmonics. The benefits of SRP-Harmonics are explained and illustrated via SRP maps and videos. Experiments with real microphone array data show that SRP-Harmonics is precise in localizing and tracking a drone, while the ordinary SRP-PHAT can not be reasonably successful. Moreover, SRP-Harmonics after Kaiser window can maintain its performance even when significant level of artificial white noise or natural wind noise exists in the data.

Keywords

Acoustic drone localization drone noise and sound drone tracking microphone array wind and gust noise unmanned aerial vehicle positioning trajectory estimation

Introduction

Unmanned aerial vehicles (UAV) or namely drones are today utilized worldwide in commercial, defense or civil applications. The potential missions and the variety of possible applications of small drones are inevitably increasing.^1,2 The current and prospective opportunities provided by the drones within social, environmental or commercial areas increase the popularity of them.³ Of course, this popularity inevitably brings many problems such as the growing controversy about drone trespass.⁴ Because of the need for avoidance of drone trespass in civilian environments or in defense industry, drone detection, localization and tracking are listed among popular research questions.⁵ This research area is so popular that several authors attempt to collect all the related studies to construct literature reviews on this subject.^5–9 Four major categories namely radar, visual, acoustic, and radio-frequency sensing systems are cited to classify these studies.⁵ Use of drone sounds for detection or tracking can be attractive because of not requiring line-of-sight, independency from visual or light conditions and the low costs to build the system.⁷ Acoustic detection or localization of the drones can be critical also for the drones which may not emit any radio wave.⁶

The majority of the acoustic studies regarding drones are about detection or classification of drones. In the literature, the features utilized in detection of drone sound can be listed as follows: short time energy, temporal centroid, zero crossing rate, spectral centroid, spectral roll-off, Linear Predictive Cepstral Coefficients and finally Mel-Frequency Cepstral Coefficients. And the classifiers employed during detection of drone sound can be listed as follows: Gaussian Mixture Model, Hidden Markov Model (HMM), K-Nearest Neighbors, Support Vector Machines, Q-factor wavelet transform, Skinny Pattern, Convolutional Neural Network, Recurrent Neural Network and finally Light-Weight Convolutional Neural Network.^10–19 All these studies depend on machine learning (mainly deep learning), and differ in the features that are used in the systems.

Beyond detection or classification, drone localization and tracking were also studied in the literature. Generalized Cross Correlation - Phase Transform (GCC-PHAT) based time difference of arrival (TDOA) estimations or direction of arrival (DOA) estimations for localization, and Kalman Filter or HMM systems for tracking have been previously utilized.^20,21 Indoor acoustic localization systems were constructed to allow localization of drones in GPS-denied environments.²² Fang et al.²³ used a different technological setup (fiber-optic acoustic sensors) rather than ordinary microphone array to localize the drones. Furthermore, there exist studies about acoustic source and noise localization around aerial vehicle components and engines to be able better design these components.^24–27

Periodic or harmonic components within the sounds of aerial vehicles were previously exploited in detection, direction finding and localization of these vehicles. There are a couple of sources of UAV noise that can be harmonic,²⁸ but the noise of propeller and rotor is often the loudest harmonic noise source. Magliozzi et al.²⁸ stated that the propeller noise can be considered as the combination of harmonic components and broadband noise. The harmonic components are directly related to the rotational speed of the propeller and the number of blades. The harmonics were utilized to reduce the computational complexity of the direction finding algorithms firstly for ground vehicles²⁹ and then for drones.^30,31 Moreover, Harmonic Line Association (HLA) was used to classify drone sounds.^32,33

Combined schemes for drone detection and localization were also developed where HLA was used for only drone detection not for localization.^34,35 Itare et al.³⁶ employed time-domain delay-and-sum beamforming for DOA estimation of a drone via frequency components in accordance with harmonics. Finally, Djurek et al.³⁷ stated that attempting to build acoustic signatures for drones based on only the harmonics and their ratios is not adequate. They said that the frequency components close to harmonics should be also taken into account, because there always exist frequency components around the harmonics that are related to the rotation of electric motors and quadcopter blades (i.e. the narrow band random components as defined by Magliozzi et al.²⁸).

Unlike all these studies, this study integrates utilization of frequency components around harmonics into Steered Response Power - Phase Transform (SRP-PHAT), and the new proposed method is named as Steered Response Power around Harmonics or simply SRP-Harmonics. First, a custom discrete Fourier transform (DFT) is defined and named as DFT-Harmonics. DFT-Harmonics only and densely concentrates on the vicinities of harmonics. Next, integration of DFT-Harmonics and SRP-PHAT is proposed. The new method (i.e. SRP-Harmonics) and its benefits are explained and illustrated via SRP maps. Finally, by means of experiments with real microphone array data, it is shown that SRP-Harmonics is quite effective in localizing and tracking a drone. In the conclusion, the effectiveness and practicability of the proposed method and the future research directions are discussed.

Steered response power around harmonics (SRP-Harmonics)

As mentioned previously, this study integrates exploitation of the frequency components around harmonics into SRP-PHAT method. In order to make this integration, first a custom DFT is defined, and it is named as DFT-Harmonics. Next, estimating TDOA via GCC-PHAT based on DFT-Harmonics is illustrated to conceptually introduce the new localization framework. Next, SRP-Harmonics (i.e. integration of DFT-Harmonics and SRP-PHAT) is described. Finally, before moving on to localization experiments, the harmonic components of the noise of DJI Phantom 4 and the issue of wind noise are briefly discussed.

A custom DFT to concentrate only on vicinities of harmonics: DFT-Harmonics

In this section, the idea is to develop a custom DFT which concentrates only on the vicinities of the harmonics of a periodic signal (e.g. drone sound). The ordinary DFT and the inverse transform have been previously defined as follows:³⁸

X (k) = \sum_{n = 0}^{N - 1} x (n) e^{- j \frac{2 π k n}{N}}

(1)

x (n) = \frac{1}{N} \sum_{k = 0}^{N - 1} X (k) e^{j \frac{2 π k n}{N}}

(2)

where

N

is the length of the input signal and

k

is the index of DFT. The continuous time frequency

f

is related to

k

as follows:

f = k \frac{f_{s}}{N}

(3)

where

f_{s}

is the sampling frequency of the digital audio or signal.

In this study, a custom DFT which concentrates only on the vicinities of harmonic frequencies of the periodic signals is proposed, so it is simply referred to as DFT-Harmonics. DFT-Harmonics can be defined as follows:

X (i) = \sum_{n = 0}^{N - 1} x (n) e^{- j \frac{2 π h_{i} n}{N}}

(4)

where

h_{i}

is the sequence of frequency components around harmonics. To define the sequence

h_{i}

, let us first define

{\tilde{h}}_{i}

which is the positive half of it:

{\tilde{h}}_{i} = H_{⌈ \frac{i}{(2 β / Δ) + 1} ⌉} + \mod (i - 1, 2 β / Δ + 1) \cdot Δ - β, i = 1, 2, \dots, L

(5)

where

Δ

is the resolution around harmonics,

β

is the absolute maximum deviation from the center of the harmonics and

⌈ ⌉

is the ceiling operator. Therefore, the number of the frequency components around a single harmonic is simply

(2 β / Δ) + 1

H_{n}

is the sequence of the positive harmonics where

H_{1}

corresponds to 1st harmonic that is the fundamental frequency. If

M

is the total number of positive harmonics that are included in calculations, then the total number of the frequency components in this half sequence can be defined as follows:

L = (\frac{2 β}{Δ} + 1) \cdot M

(6)

Finally, the complete sequence $h_{i}$ which also includes the negative frequency components can be defined as,

h_{i} = {\begin{cases} \begin{array}{l} - {\tilde{h}}_{L - i + 1} & i \leq L \end{array} \\ \begin{array}{l} {\tilde{h}}_{i - L} & i > L \end{array} \end{cases}, i = 1, 2, \dots, 2 L

(7)

In the rest of this study, the parameters are designated as $Δ = 0.1$ and $β = 3$ . These numbers are selected to comprehensively and densely include all the frequency components around harmonics by taking into account especially the bandwidths of harmonics of drones. They can be set to any other numbers depending on the trade-off between speed of algorithm and accuracy of localization. The sequence $h_{i}$ becomes as follows:

({- H}_{M} - 3, {- H}_{M} - 2.9, \dots, {- H}_{M} + 3, \dots, H_{M} + 3)

(8)

Finally, the inverse transform for DFT-Harmonics can be defined as follows,

x (n) = \frac{Δ}{N} \sum_{i = 1}^{2 L} X (i) e^{j \frac{2 π h_{i} n}{N}}

(9)

The continuous time frequency $f$ can be written as the function of $i$ that is the index of DFT-Harmonics as follows:

f (i) = h_{i} \frac{f_{s}}{N}

(10)

In order to more clearly define the new method and to more accurately depict its difference compared to the conventional method, before analyzing the drone sound, the new concepts will be first explained via a nearly-perfect periodic signal that is a steady /AA/ vowel sound from a male voice for 0.36 s. The reason of this attempt is that the harmonics of this almost periodic signal are more significant, so this will result in more clear SRP maps. The harmonics of the drone sound will be discussed later.

Figure 1(a) illustrates the ordinary DFT of this signal against $f$ . The ordinary DFT obtains the values of frequency components for all the frequency range. However, calculating the values of frequency components which are not close to harmonics appears to be a waste because the signal has no significant spectral power in these regions. Moreover, in case of external or environmental noise, these regions will considerably be dominated by these external noise components, so will have a negative effect on localization. Therefore, DFT-Harmonics takes into account the frequency components which are only located around the harmonics of the signal.

Figure 1.

(a) The ordinary DFT of a nearly-perfect periodic sound, (b) the components of DFT-harmonics around a single harmonic.

Figure 1(b) illustrates the difference between the ordinary DFT and DFT-Harmonics. It shows the vicinity of the second harmonic which is labeled by the dashed red rectangle shown in Figure 1(a). Only a few frequency components of the ordinary DFT hit the second harmonic. On the other hand, in DFT-Harmonics, the frequency components are densely located around it as shown in Figure 1(b). Consequently, DFT-Harmonics will be proficient in comprising all the important frequency components related to the rotation of electric motors and quadcopter blades by taking into account the harmonic components as well as the narrow band random components.

GCC-PHAT and localization via DFT-Harmonics

To obtain the time delay or namely TDOA (in samples) between two signals $x_{1} (n)$ and $x_{2} (n)$ which are obtained by two different microphones located at spatially different positions, GCC-PHAT suggests us to calculate the Fourier transform of the cross-correlation between two signals as follows:³⁹

R_{x_{1} x_{2}} (k) = \frac{X_{1} (k) X_{2}^{*} (k)}{| X_{1} (k) X_{2}^{*} (k) |}

(11)

where

X_{2}^{*} (k)

is the complex conjugate of

X_{2} (k)

. Then, the cross-correlation function in time-domain that is

r_{x_{1} x_{2}} (n)

can be obtained via inverse Fourier transform,

r_{x_{1} x_{2}} (n) = F^{- 1} {R_{x_{1} x_{2}} (k)} \approx δ (n - n_{d})

(12)

As can be seen in Figure 2(a), the cross-correlation function is supposed to be a delayed impulse function where $n_{d}$ is the sample delay between $x_{1} (n)$ and $x_{2} (n)$ . Consequently, the time delay (in seconds) between two signals is equal to $n_{d} / f_{s}$ . Each time-delay corresponds to a hyperbola on two-dimensional space.⁴⁰ Therefore after obtaining multiple time-delays via multiple microphones, the location of target can be determined via intersection of the corresponding hyperbolas in noise-free environments as shown in Figure 2(b).

Figure 2.

(a) The cross-correlation function obtained by using ordinary DFT corresponds to a delayed impulse function, (b) the location of the source can be determined by intersection of hyperbolas, (c) the cross-correlation function obtained by using DFT-Harmonics corresponds to a delayed impulse train multiplied by a $s i n c$ function, (d) the grid of hyperbolas corresponding to the local maxima in the cross-correlation function obtained by DFT-harmonics.

This study utilizes DFT-Harmonics instead of the ordinary DFT, therefore the new function ${R^{'}}_{x_{1} x_{2}} (k)$ can be viewed as a filtered version of $R_{x_{1} x_{2}} (k)$ as follows:

{R^{'}}_{x_{1} x_{2}} (k) = R_{x_{1} x_{2}} (k) \cdot [W_{β} (k) * S_{H_{1}} (k)]

(13)

where

W_{β} (k)

is the rectangular function with the cutoff value β and

S_{H_{1}} (k)

is the impulse train in frequency domain with the period

H_{1}

that is the 1st harmonic of the input signal. Finally,

*

is the convolution operator. This filtering effect occurs because of concentrating only on vicinities of harmonics of the input signal.

From this perspective, the new time-domain cross-correlation function is supposed to be as follows:

{r^{'}}_{x_{1} x_{2}} (n) \approx δ (n - n_{d}) * [\frac{2 β}{H_{1}} \cdot s i n c (\frac{2 β}{N} n) \cdot S_{(N / H_{1})} (n)]

(14)

where

S_{(N / H_{1})} (n)

is the impulse train in time domain with the period

N / H_{1}

. As can be seen in Figure 2(c), the new cross-correlation function is a delayed impulse train multiplied by a

s i n c

function.

The side effect of utilization of DFT-Harmonics is the formations of local maxima in the time-domain cross correlation function. If it is considered that all these local maxima correspond to also other hyperbolas in two dimensional space, the scheme of intersections of hyperbolas becomes a grid as shown Figure 2(d). This grid will explain the overall view of the SRP map obtained via DFT-Harmonics shown in the next section. At the first glance, it may be considered that formation of this grid would confuse the localization process, however it will be shown that the local maxima will not create any significant issue and SRP-Harmonics can effectively localize the source.

Ordinary SRP-PHAT versus SRP-Harmonics

SRP-PHAT is a powerful localization method which is based on the proposition that the location of the source emits more energy than all other locations in the search space. Before providing formula of SRP-PHAT, the function which maps the point $\vec{q}$ in the search space to the TDOA (in samples) between two distinct microphone positions ${\vec{b}}_{i}$ and ${\vec{b}}_{j}$ should be defined as follows:⁴¹

n_{i, j} (\vec{q}) = ⌈ f_{s} \frac{‖ \vec{q} - {\vec{b}}_{i} ‖ - ‖ \vec{q} - {\vec{b}}_{j} ‖}{c} ⌉

(15)

where

c

is the propagation speed of the signal in the space. Therefore, SRP-PHAT was previously defined as follows:⁴²

P (\vec{q}) = \sum_{i = 1}^{B - 1} \sum_{j = i + 1}^{B} r_{x_{i} x_{j}} (n_{i, j} (\vec{q}))

(16)

where

B

is the total number of microphones in the localization system. Finally, the location estimation of the target can be found as follows:

\hat{q} = \underset{\vec{q}}{argmax} (P (\vec{q}))

(17)

In this study, equation (16) is reorganized by inserting the correlation function ${r^{'}}_{x_{i} x_{j}}$ obtained via DFT-Harmonics to obtain SRP-Harmonics. Therefore, SRP-Harmonics can be calculated as follows:

P_{H a r m o n i c s} (\vec{q}) = \sum_{i = 1}^{B - 1} \sum_{j = i + 1}^{B} {r^{'}}_{x_{i} x_{j}} (n_{i, j} (\vec{q}))

(18)

Figure 3 demonstrates the SRP maps for different signal-to-noise ratios (SNR) of a sample artificial localization scheme based on again the same signal mentioned above. In this localization scheme, four different microphones are located at (−10, −10), (−10, 10), (10, −10) and (10, 10) m. In the simulation, white Gaussian noises with different SNR values are added to the data. Finally, the source is attempted to be localized via SRP-PHAT and SRP-Harmonics. In the SRP maps shown in Figure 3, the resolution is 0.1 m and the darker color represents the higher power value.

Figure 3.

The SRP maps of SRP-PHAT and SRP-Harmonics for different SNR values in an artificial localization scheme with nearly-perfect periodic sound.

Figure 3(a) shows the map of the ordinary SRP-PHAT for noise-free condition. As can be seen, the hyperbolas can be clearly observed in this map, and these hyperbolas intersect each other at the position of the target (i.e. the source). As a result, the maximum point of the SRP map occurs exactly at the target position. Figure 3(d) shows the map of the SRP-Harmonics for noise-free condition. As can be seen, the grid of hyperbolas mentioned in the previous section is visible in this map. The dominant hyperbolas intersect each other at the position of the target, so the maximum point of the SRP-Harmonics occurs also exactly at the target position. From this perspective, both methods appear to be equal, however the advantage of the SRP-Harmonics will be apparent when the noise level increases.

Figure 3(b) shows the map of the ordinary SRP-PHAT for SNR value of −10 dB. As can be seen, the SRP map is now imprecise, yet it is still possible to observe the intersection of hyperbolas at the target position. Consequently, the maximum point still occurs at the target position. On the other hand, Figure 3(e) shows the map of the SRP-Harmonics for SNR value of −10 dB. Unlike the map of ordinary SRP-PHAT, this map is quite clear and it appears that it has not been affected by this high level of noise. Again, the maximum point of the SRP-Harmonics occurs at the target position.

Figure 3(c) shows the map of the ordinary SRP-PHAT for SNR value of −15 dB. Now, the SRP map is significantly imprecise, so the maximum point of the SRP map occurs at a false position. On the other hand, the map of the SRP-Harmonics shown in Figure 3(f) is relatively clean compared to Figure 3(c). And, the maximum point of the SRP-Harmonics still occurs at the target position. Therefore, SRP-Harmonics is an effective noise-resilient method for the signals having significant harmonics. In the next sections, it will be shown that SRP-Harmonics is a proper and effective method for drone localization and tracking, because the drone noise also possesses apparent harmonics.

Harmonics of drone sound, wind noise and effect of Kaiser windows

As mentioned previously, the overall aim of this study is to build a method to effectively localize the drones through their noises. In the previous section, it is mentioned that SRP-Harmonics can effectively localize the signals having significant harmonics. In the literature, it was mentioned that the sounds of aerial vehicles have significant harmonic components. When the sound of the DJI Phantom 4 is recorded outdoor with the sampling rate 48 kHz, the spectrogram of this humming sound is obtained as shown in Figure 4(a). As can be seen, the existence of horizontal parallel lines that are visible especially under 1 kHz indicates the presence of harmonics.

Figure 4.

(a) Spectrogram of the drone sound, (b) the auto-correlation function, (c) frequency spectrum of the drone sound in which harmonics are apparent.

In the autocorrelation function of the drone sound shown in Figure 4(b), there exists a significant peak which occurs at 5.1 ms, therefore the fundamental frequency of the drone sound is 196 Hz (i.e. 5880 rpm of motor by keeping in mind that the propeller has two blades²⁸). Figure 4(c) shows the spectrum of the drone sound where the harmonics are quite visible.

Of course, the signal should be windowed during its analysis to be able deal with short time intervals of the signal each time. The side lobes of the frequency response of rectangular windows are relatively high, which creates the problem of spectral leakage.³⁸ Therefore, the spectral view of the harmonics of the drone can be improved by utilizing window functions such as Hamming, Hanning and Kaiser windows so on. In this study, Kaiser windows are preferred because their frequency response can be adjusted by setting the β_Kaiser to a desired value.

At this point, it will be discussed that Kaiser windows not only enhance the harmonics of drone sound, but also, as a side effect, significantly attenuate the negative effect of the wind or gust. As shown in Figure 5(a), the effect of wind or gust can be defined as a low frequency noise with a very large amplitude. The large amplitude of the wind noise results in very significant low-frequency components in the spectrum. If rectangular windows are utilized in case of wind, the frequency components of wind noise have very high side lobes which suppress the harmonics of the drone sound as shown in the red plot of Figure 5(b). On the other hand, by applying other windows which have lower side lobes, this negative effect of wind or gust can be attenuated. If Kaiser window with β_Kaiser = 10 is applied, then the side lobes of wind are quite attenuated, and the harmonics of drone sound become apparent as shown in the blue plot of Figure 5(b).

Figure 5.

(a) Drone sound together with wind (or gust) noise in time domain, (b) drone sound together with wind (or gust) noise in frequency domain and the effect of utilizing of Kaiser windows.

Simulations and experiments

In this chapter, by using the real microphone array data, it will be demonstrated that SRP-Harmonics is quite effective in localizing and tracking a drone by means of the noise that it generates. First, localization of a stable drone is discussed. Next, the resilience of SRP-Harmonics is discussed via artificial white Gaussian noise, artificial wind noise and natural wind noise experiments. Finally, tracking a moving drone in two and three dimensional spaces is discussed.

Experiments of localization of a stable drone via microphone array

In this section, the experiments of localizing a stable drone via real microphone array data are presented. The microphone array used to record the data is the TASCAM® US1800, the microphones are Audio-Technica® AT2031, and finally the drone is DJI Phantom 4 as shown in Figure 6(a). The microphones are located outdoor linearly with 3.30 m distance. The heights of the microphones are adjusted to 1.2 m via adjustable microphone stands as depicted in Figure 6(b). When auto take off is applied, DJI Phantom 4 takes off and hovers at an altitude of 1.2 m that is the same height with the microphones. Consequently, accepting the center of the microphones as the origin and the alignment of the microphones as the x-axis, an outdoor two dimensional localization setup is established. For three dimensional localization, the drone is positioned to a random altitude.

Figure 6.

The equipment used in the experiments and the experimental setup.

In a sample experiment, the drone is located at the point (9, 14) m via 50 m measuring tape, and its sound has been recorded by the microphone array for 11 s after take-off. The number of harmonics included in calculations for SRP-Harmonics is 7. The fundamental frequency of the drone sound is calculated and updated for each time window to be able to more effectively apply SRP-Harmonics. When the window length is set to be 0.5 s, SRP maps (with 0.05 m resolution) shown in Figure 7 are obtained. As can be seen, while SRP-Harmonics (Figure 7(b)) is successfully localizing the drone, the ordinary SRP-PHAT (Figure 7(a)) can not be quite precise about its location. Moreover, if the window shift length is set to 0.1 s, the video at the link https://youtu.be/I0q1NTxCJig is obtained which contains all the maps of SRP-Harmonics through 11 s. As can be seen throughout video, SRP-Harmonics can accurately localize the drone.

Figure 7.

(a) SRP map of SRP-PHAT for localization of a stable drone, (b) SRP map of SRP-Harmonics for localization of a stable drone (https://youtu.be/I0q1NTxCJig).

Experiments with artificial white Gaussian noise, artificial and natural wind noise

To statistically inspect the noise resilience of the SRP-Harmonics, the drone is localized at 55 different locations as a grid from −12.5 to 12.5 m with 2.5 m increment along x-axis and from 10 to 18 m with 2 m increment along y-axis. For each location, the sound of the drone hovering at the height of 1.2 m is recorded for around 10 s. The window length is set to 0.5 s and the shift length is set to 0.1 s. With these parameters, the drone is localized via SRP-PHAT and SRP-Harmonics, and root mean squared (RMS) localization errors of all these windows are calculated.

Firstly, artificial white Gaussian noise is added to the microphone array data. As can be seen in Figure 8(a), SRP-Harmonics can successfully localize the drone and maintain its success up to 40 dB SNR. On the other hand, the ordinary SRP-PHAT can not effectively localize the drone in any noise level. As mentioned previously, Kaiser windows enhance the harmonics of the drone sound. When Kaiser Windows with β_Kaiser = 10 are applied before SRP-PHAT, it is observed that it significantly improves the performance of SRP-PHAT yet it is still not quite effective. Finally, utilization of Kaiser Windows also improves the performance of SRP-Harmonics. As seen, SRP-Harmonics after Kaiser Window maintains its initial success up to 30 dB SNR.

Figure 8.

(a) RMS localization error in case of artificial white Gaussian noise, (b) RMS localization error in case of artificial wind noise, (c) spectrogram and localization error of SRP-Harmonics in case of natural wind noise, (d) spectrogram and localization error of SRP-Harmonics after Kaiser window in case of natural wind noise (https://youtu.be/ON8T0jqWGdY).

Secondly, artificial wind noise is added to the microphone array data. Random noise signals are created which include frequency components only below 100 Hz to simulate the wind noise. As can be seen in Figure 8(b), SRP-Harmonics can successfully localize and maintain its success up to 10 dB SNR. SRP-Harmonics after Kaiser Window maintains its initial success up to 0 dB SNR. Utilization of Kaiser windows significantly improves the performance of SRP-Harmonics, because it attenuates the negative effect of wind noise as mentioned previously. Finally, both the SRP-PHAT and the SRP-PHAT after Kaiser window can not precisely localize the drone.

Lastly, the wind screens of the microphones are deliberately removed to record the sound of drone together with natural wind noise. Figure 8(c) shows a sample spectrogram where wind (or gust) blows after 7.5 s and it creates around 0 dB wind noise, which is estimated by comparing the power spectral densities before and during the wind. As seen in Figure 8(c), the localization performance of SRP-Harmonics decreases in case of this level of wind. However, as seen in Figure 8(d), utilization of Kaiser windows makes the drone harmonics quite apparent. Therefore, the localization performance of SRP-Harmonics after Kaiser window can maintain its success in spite of this natural wind (or gust) noise. The video at the link https://youtu.be/ON8T0jqWGdY demonstrates these spectrograms together with the recorded audio.

Experiments of tracking a moving drone via microphone array

In this section, the experiments about tracking a moving drone are presented. First the drone takes off, then a random direction is arranged via the orientation stick of the remote controller. Finally, by pushing the movement stick exactly forward and keeping its position stable, the drone is allowed to make a linear movement with a random constant speed. During this movement, its sound is recorded via the microphone array. The trajectory that the drone followed and its speed are later obtained via DJI GO app.

A sample movement of the drone is illustrated via red dots in Figure 9(c). The speed of the drone is 2.85 m/s. In this sample experiment, both the window and shift length are 0.2 s. Figure 9(a) and (b) show the SRP maps of SRP-Harmonics for two different time instants. As can be seen in these SRP maps, the global maximum points coincide with the current drone positions. In Figure 9(c), it can be seen that throughout the movement of the drone, SRP-Harmonics produced outlying estimations for only two frames. These outlying points occur because the movement of the drone causes instability in TDOA values. The relative distances of the moving drone to each microphone keep changing within a time window, therefore this issue can occasionally confuse the localization algorithm and produce outliers. In Figure 9(c), all the other localization points coincide with the trajectory of the drone. The video at the link https://youtu.be/WAaoHe-U3Bg contains all the maps of SRP-Harmonics as a movie throughout the motion of the drone.

Figure 9.

(a) and (b) are the SRP maps of SRP-Harmonics for two different time instants while the drone is moving (https://youtu.be/WAaoHe-U3Bg), (c) the estimated locations versus the trajectory of the drone.

To statistically inspect the success of the SRP-Harmonics in drone tracking, the sound of the drone is recorded for 96 different linear motions with random initial point and speed. By each time applying different levels of force to the movement stick in the remote controller, it is aimed to obtain a wide range of drone speed that is between 1.5 m/s and 10 m/s. In these experiments, the window length is reduced to 0.1 s and the shift length is set to 0.05 s. Using Intel® Core™ i7 processor at turbo boost (4.6 GHz), the average execution time of localization with these specifications is found to be 0.034 s when the matrices representing TDOA in samples (equation (15)) throughout the search space are previously recorded on the computer for the current measurement geometry. Therefore, these specifications are ideal for real-time localization for this processor capacity.

In Figure 10(a), the box plots of the percent of outlying estimations for different levels of drone speeds can be seen. Even though the rate of outlying estimations can reach important values, it is proposed that the remaining points which coincide with the trajectory of the drone are quite enough to adequately estimate the speed and the trajectory of the drone via Linear Least Squares (LLS). The outlying estimations can be eliminated progressively throughout the motion of the drone. The process initiates with agglomerative hierarchical clustering after receiving a couple of location estimations (i.e. 5 estimations or more). Because the outlying estimations have a tendency to randomly scatter throughout the search region, the estimations associated with the drone trajectory stay close to each other compared to outlying points. Consequently, the initial location estimations which are not outlying are used in calculating the first estimations about the parameters of the drone trajectory. After this point, by detecting and eliminating outlying estimations every time, the LLS based estimations of the speed and the trajectory of the drone are sequentially updated.

Figure 10.

(a) The box plots regarding drone tracking when SRP-Harmonics is calculated for the whole range, (b) the box plots regarding drone tracking when SRP-Harmonics is dynamically confined into a 4 × 4 m area around the point estimated via trajectory parameters.

The statistical results of these experiments are summarized in Figure 10(a). Different linear motions of the drone are divided into three groups based on the speed of the drone. Speed Group #1, #2 and #3 include the motions within the speed range 1.5 – 4 m/s, 4 – 7 m/s and 7 – 10 m/s respectively. As can be seen in Figure 10(a), increasing the drone speed results in an increasing number of outlying estimations. However, as mentioned before, remaining estimations can be sufficient to satisfactorily estimate the velocity and the trajectory of the drone. Similarly, as can be seen in Figure 10(a), increasing the drone speed also raises the error in estimations of the velocity and the entry (or the exit) point of the drone to the region of interest.

Finally, these estimations can be enhanced by dynamically confining the SRP-Harmonics area instead of calculating the SRP-Harmonics every time throughout the whole range. After obtaining the first guess about the trajectory parameters, the SRP-Harmonics calculations are now confined into a 4 × 4 m area around the points estimated via trajectory parameters. As seen in Figure 10(b), this method significantly reduces the rate of outlying estimations. When the box plots in Figure 10(a) and (b) are compared to each other, it can be observed that confining the area of SRP-Harmonics also considerably reduces the errors in velocity estimations and somewhat reduces the errors in entry (or exit) point.

Drone localization and tracking in three dimensional space

In this part, drone localization and tracking in three dimensional space via microphone array are presented. For three dimensional localization, the drone is positioned to a random altitude as explained in Figure 6(b) and a triangular measurement geometry is constructed where the microphones stay at (−4.8, 0, 1.2), (4.8, 0, 1.2), (0, 0, 1.2), and (0, 3.2, 1.2) m. In the previous sections which assume two dimensional space, localization via SRP-Harmonics can be easily visualized via SRP-Harmonics surfaces or namely the maps. However, in this section which deals with three dimensional space, localization via SRP-Harmonics can only be visualized via SRP-Harmonics volumes instead of surfaces. Figure 11(a) depicts the SRP-Harmonics volume via the isosurfaces corresponding to 50% of the global maximum value. As seen, there exists an isosurface enclosing the location of the drone. The video at the link https://youtu.be/R4E7kIYuU8Q shows that this isosurface is also enclosing the global maximum, so finally, global maximum coincides with the drone location in three dimensional space. As seen in Figure 11(b), the SRP-Harmonics can successfully localize the moving drone (with a speed of 5.9 m/s) in three-dimensional space with a few outlying estimations.

Figure 11.

Three-dimensional drone localization and tracking (https://youtu.be/R4E7kIYuU8Q).

To statistically inspect the success of the SRP-Harmonics in drone tracking within a three-dimensional space, similar to the previous section, the sound of the drone is recorded for 65 different linear motions with random initial point, speed and random altitude (i.e. 1.2 to 6 m). In these experiments, the drone speeds are between 4 and 10 m/s, so they are corresponding to Speed Group #2 and #3 defined in the previous section. It can be said that SRP-Harmonics can adequately estimate the parameters of the trajectory of the drone when the box plots in Figure 11(c) is observed. Figure 11(c) also compares the estimation errors of two-dimensional versus three-dimensional localization. Finally, it can be observed that estimation errors of three-dimensional tracking are slightly higher than those of two-dimensional tracking.

Conclusion

In this study, an effective method for acoustic drone localization and tracking is introduced by integrating utilization of the harmonics into SRP-PHAT. While making this integration, this study also takes into account the frequency components close to the harmonics (i.e. the narrow band random components). Consequently, a successful localization scheme is obtained to acoustically localize the drone. The validity and reliability of the proposed method in acoustic drone localization and tracking are demonstrated by the real microphone array data. The overall characteristics and behavior of SRP-Harmonics have been depicted by visual materials namely SRP maps and videos of SRP maps to justify its success. For a stable measurement geometry, when the matrices representing TDOA in samples throughout the search space are previously recorded on the computer, real-time localization and tracking by SRP-Harmonics are feasible in two or three-dimensional space.

Due to the limitations on the number of available microphones and the lengths of available XLR cables, there were constraints on the measurement geometry in this study. A lot of microphones can be scattered throughout a region and possibly with three-dimensional configurations, so the localization distance can be significantly expanded within three-dimensional space. Moreover, utilization of more sophisticated microphones which are suitable especially for outdoor recording can also increase the range of localization. Nevertheless, the range of localization in the experiments of this study is also critically important to localize and track the drones which are trespassing or coming closer toward an area or a building for any reason.

SRP-Harmonics is also suitable to be adapted to many other applications. Localization of ground vehicles is also possible as long as they generate sounds including apparent harmonic components. Localization of human voice is also possible via SRP-Harmonics by employing the voiced parts of the speech. SRP-Harmonics can also facilitate detection and recognition of the drones when a library including the fundamental frequencies of different types of drones is available. Finally, localization of multiple drones is also possible especially when the drones have different fundamental frequencies. To sum up, considering again the reasons of employing acoustic localization for drones (i.e. not requiring line-of-sight, independency from light conditions, low costs, ability of localization of the drones not emitting any radio wave), this study contributes to this area by introducing an effective and viable method for drone localization. Until the researchers achieve to develop entirely silent aerial vehicles¹ through advanced drone noise analyses,⁴³ future studies will continue to study the acoustic localization of the drones.

Supplemental Material

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Eskişehir Technical University Scientific Research Projects Commission under project number 24ADP051.

ORCID iD

Seçkin Uluskan

Supplemental Material

Supplemental material for this article is available online.

References

Serré

Gourdain

Jardin

, et al. Towards silent micro-air vehicles: optimization of a low Reynolds number rotor in hover. Int J Aeroacoustics 2019; 18(8): 690–710.

Malgoezar

Vieira

Snellen

, et al. Experimental characterization of noise radiation from a ducted propeller of an unmanned aerial vehicle. Int J Aeroacoustics 2019; 18(4–5): 372–391.

Merkert

Bushell

. Managing the drone revolution: a systematic literature review into the current use of airborne drones and future strategic directions for their effective control. J Air Transport Manag 2020; 89: 101929.

Skorup

. The growing legal controversy about drone trespass, https://rtp.fedsoc.org/blog/the-growing-legal-controversy-about-drone-trespass/ (2021).

Taha

Shoufan

. Machine learning-based drone detection and classification: state-of-the-art in research. IEEE Access 2019; 7: 138669–138682.

Kaleem

Rehmani

. Amateur drone monitoring: state-of-the-art architectures, key enabling technologies, and future research directions. IEEE Wireless Commun 2018; 25(2): 150–159.

Khan

Menouar

Eldeeb

, et al. On the detection of unauthorized drones—techniques and future perspectives: a review. IEEE Sensor J 2022; 22(12): 11439–11455.

Matić

Kosjer

Lebl

, et al. Methods for drone detection and jamming. In: Proceedings of the 10th international conference on information society and technology (ICIST), Lecce, Italy, 4–5 June 2020, pp. 16–21.

Guvenc

Koohifar

Singh

, et al. Detection, tracking, and interdiction for amateur drones. IEEE Commun Mag 2018; 56(4): 75–81.

10.

Mezei

Molnár

. Drone sound detection by correlation. In: IEEE 11th international symposium on applied computational intelligence and informatics (SACI), Timisoara, Romania, 12–14 May 2016, pp. 509–518.

11.

Jeon

Shin

Lee

, et al. Empirical study of drone sound detection in real-life environment with deep neural networks. In 25th European signal processing conference (EUSIPCO), Kos, Greece, 28 August 2017–2 September 2017, pp. 1858–1862.

12.

Bernardini

Mangiatordi

Pallotti

, et al. Drone detection by acoustic signature identification. Electr Imaging 2017; 29: 60–64.

13.

Kim

Park

Ahn

, et al. Real-time UAV sound detection and analysis system. In: IEEE sensors applications symposium (SAS), Glassboro, NJ, 13–15 March 2017, pp. 1–5.

14.

Shi

Ahmad

, et al. Hidden Markov model based drone sound recognition using MFCC technique in practical noisy environments. J Commun Netw 2018; 20(5): 509–518.

15.

Anwar

Kaleem

Jamalipour

. Machine learning inspired sound-based amateur drone detection for public safety applications. IEEE Trans Veh Technol 2019; 68(3): 2526–2534.

16.

Aydın

Kızılay

. Development of a new light-weight convolutional neural network for acoustic-based amateur drone detection. Appl Acoust 2022; 193: 108773.

17.

Akbal

Dogan

, et al. An automated accurate sound-based amateur drone detection method based on skinny pattern. Digit Signal Process 2023; 136: 104012.

18.

Utebayeva

Ilipbayeva

Matson

. Practical study of recurrent neural networks for efficient real-time drone sound detection: a review. Drones 2022; 7(1): 26.

19.

Dong

Liu

. Drone sound detection system based on feature result-level fusion using deep learning. Multimed Tool Appl 2023; 82(1): 149–171.

20.

Chang

Yang

, et al. A surveillance system for drone localization and tracking using acoustic arrays. In 10th sensor array and multichannel signal processing workshop (SAM), Sheffield, UK, 8–11 July 2018, pp. 573–577.

21.

Guo

Ahmad

Chang

. Classification, positioning, and tracking of drones by HMM using acoustic circular microphone array beamforming. EURASIP J Wirel Commun Netw 2020; 2020: 9–19.

22.

Famili

Stavrou

Wang

, et al. Rail: robust acoustic indoor localization for drones. In: IEEE 95th vehicular technology conference, Helsinki, Finland, 19–22 June 2022, pp. 1–6.

23.

Fang

, et al. Drone detection and localization using enhanced fiber-optic acoustic sensor and distributed acoustic sensing technology. J Lightwave Technol 2023; 41(3): 822–831.

24.

Kaja

Srinivasan

Chaitanya

, et al. Data-driven neural networks for source localization and reconstruction using a planar array. Int J Aeroacoustics 2022; 21(8): 684–707.

25.

Siller

König

Funke

, et al. Acoustic source localization on a model engine jet with different nozzle configurations and wing installation. Int J Aeroacoustics 2017; 16(4–5): 403–417.

26.

Bao

Chen

Zhao

, et al. Numerical study of acoustic source localization of rotor using a novel discrete noise analysis strategy. Int J Aeroacoustics 2023; 22(3–4): 371–394.

27.

Yang

Sun

Guo

, et al. A fast 3D-MUSIC method for near-field sound source localization based on the bat algorithm. Int J Aeroacoustics 2022; 21(3–4): 98–114.

28.

Magliozzi

Hanson

Amiet

. Propeller and propfan noise. In: Hubbard

(ed) Aeroacoustics of flight vehicles: theory and practice. Volume 1: noise sources. Washington, DC: NASA, 1991, pp. 1–61.

29.

Pham

Sadler

Fong

, et al. High-resolution acoustic direction-finding algorithm to detect and track ground vehicles. In 20th army science conference. Singapore: World Scientific, 1997, pp. 16–20.

30.

Yang

Chang

, et al. DOA estimation using amateur drones harmonic acoustic signals. In: IEEE 10th sensor array and multichannel signal processing workshop (SAM), Sheffield, UK, 8–11 July 2018, pp. 587–591.

31.

Chang

Yang

Shi

, et al. Feature extracted DOA estimation algorithm using acoustic array for drone surveillance. In: 87th vehicular technology conference, Porto, Portugal, 3–6 June 2018, pp. 1–5.

32.

Jeong

Lim

, et al. Drone sound identification and classification by harmonic line association based feature vector extraction. J of Adv Navig Techn 2016; 20(6): 604–611.

33.

Ahmad

Shi

, et al. SVM-based drone sound recognition using the combination of HLA and WPT techniques in practical noisy environment. KSII Trans on Internet and Infor Sys 2019; 13(10): 5078–5094.

34.

Sedunov

Haddad

Salloum

, et al. Stevens drone detection acoustic system and experiments in acoustics UAV tracking. In IEEE international symposium on technologies for homeland security, Woburn, MA, 5–6 November 2019, pp. 1–7.

35.

Shi

Chang

Yang

, et al. An acoustic-based surveillance system for amateur drones detection and localization. IEEE Trans Veh Technol 2020; 69(3): 2731–2739.

36.

Itare

Thomas

Raoof

, et al. Acoustic estimation of the direction of arrival of an unmanned aerial vehicle based on frequency tracking in the time-frequency plane. Sensors 2022; 22(11): 4021.

37.

Djurek

Petosic

Grubesa

, et al. Analysis of a quadcopter’s acoustic signature in different flight regimes. IEEE Access 2020; 8: 10662–10670.

38.

Proakis

Manolakis

. Digital signal processing: principles, algorithms, and applications. Upper Saddle River, NJ: Prentice-Hall, Inc, 1996.

39.

Knapp

Carter

. The generalized correlation method for estimation of time delay. IEEE Trans Acoust 1976; 24(4): 320–327.

40.

Pine

Cheney

. The geometry of far-field passive source localization with TDOA and FDOA. IEEE Trans Aero Electron Syst 2021; 57(6): 3782–3790.

41.

Vera-Diaz

Pizarro

Macias-Guarasa

. Acoustic source localization with deep generalized cross correlations. Signal Process 2021; 187: 108169.

42.

Cobos

Marti

Lopez

. A modified SRP-PHAT functional for robust real-time sound source localization with scalable spatial sampling. IEEE Signal Process Lett 2011; 18(1): 71–74.

43.

Jiang

Zhong

Zhang

, et al. Boundary element analysis on the fuselage scattering of drone noise. Int J Aeroacoustics 2022; 21(1–2): 43–56.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

Precise acoustic drone localization and tracking via drone noise: Steered response power - phase transform around harmonics

Abstract

Keywords

Introduction

Steered response power around harmonics (SRP-Harmonics)

A custom DFT to concentrate only on vicinities of harmonics: DFT-Harmonics

GCC-PHAT and localization via DFT-Harmonics

Ordinary SRP-PHAT versus SRP-Harmonics

Harmonics of drone sound, wind noise and effect of Kaiser windows

Simulations and experiments

Experiments of localization of a stable drone via microphone array

Experiments with artificial white Gaussian noise, artificial and natural wind noise

Experiments of tracking a moving drone via microphone array

Drone localization and tracking in three dimensional space

Conclusion

Supplemental Material

Supplemental Material

Supplemental Material

Supplemental Material

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

Supplemental Material

References

Supplementary Material