A layered approach to detect elephants in live surveillance video streams using convolution neural networks

Abstract

Human-Elephant Conflict (HEC) and its mitigation have always been a serious conservation issue in India. It occurs mainly due to the encroachment of forests by humans as part of societal development. Consequently, these human settlements are highly affected by the intrusion of wild elephants as they cause extensive crop-raiding, injuries and even death in many cases. HEC is a growing problem in rural areas of India which shares a border with forests and other elephant habitats. Based on the studies, it is very explicit that HEC is an important conservation issue which affects the peaceful co-existence of both humans and elephants near the forest areas. The desirable solution for this problem would be to facilitate co-existence among humans and elephants, but this often fails because of technical difficulties. Hence, this paper presents an end-to-end technological solution to facilitate smoother coexistence of humans and elephants. The proposed work deploys a live video surveillance system along with deep learning strategies to effectively detect the presence of elephants. From the numerical analysis, it is revealed that the post-training accuracy of the deep learning model used in the proposed approach is evaluated at 98.7% and outperforms an an out-of-the-box image detector. The layered approach used in the proposed work improves resource management which is a major bottleneck in real-time deployment scenarios.

Keywords

Human elephant conflict machine learning convolutional neural network support vector machine

1 Introduction

The loss and fragmentation of elephants habitat due to expanding human activity leads to serious conservation problems in India. This continuously escalated the Human-Elephant conflicts (HEC) which range from crop-raiding, injuries, and deaths to humans caused by elephants, and elephants being killed by humans to prevent crop-loss, and land encroachment. In addition to this, established elephant trails are now impacted by roads and railways. This leads elephants to be often killed by trains as they attempt to cross the railway lines, and elephants charging on humans and vehicles when they attempt to cross trails.

Various conventional deterrent systems such as chili and beehive fencing have been developed in the past to prevent HEC. On the other hand, technology-based methods have been developed to track as well as to caution the presence of the elephant in specific regions such as satellite tracking of elephants using global positioning systems (GPS). These methods were primarily developed to study the behaviour of elephants or to develop early warning systems. But, most of these techniques were invasive, not cost-effective and become complex when scaled up and applied for a larger population and a wider geographical area. This revealed the path for the rise of non-invasive techniques like acoustic, seismic and vision-based detection methods.

Studies have shown that elephants may remain silent or can learn to be silent and still for long periods of time [1]. Seismic vibrations produced by the elephants suffers when the ground has moisture. Also, seismic noises analogous to that made by elephants can be produced by external sources. These reasons clearly show the need for an effective early warning system which contributes to the efficient detection of elephants utilizing the positive aspects of visual detection techniques [2]. This paper proposes a robust vision-based method that uses deep learning techniques to accurately detect the presence of elephants.

2 Related works

India’s forests hold 60% of the Asian elephant population. The habitat ranges for elephants in the country are more or less fragmented and impacted due to increased human activity [3]. HEC is an important conservation issue in India especially in the forest fringes where protected areas are covered by highly fertile and cultivable lands [4]. The HEC that ranges from crop raiding, injuries and death to humans caused by elephants, and sometimes killing the animals to prevent crop-loss, and land encroachment create severe localized economic and socio-political issues [5]. In addition, untoward developmental pressures including the construction of roads, railroads, dams, and canals in the forested regions in the country have increased the magnitude of the problem. The present management plans for HEC in India fundamentally include forceful rehabilitation of people especially the indigenous communities inhabiting forested regions; conventional methods such as electric fencing, digging a trench and using biological repellent such as chili or other plant parts to drive away from the elephants [6]. None of these are found to be sustainable and economically viable solutions since they do not address the problem holistically. Development of systems that provide detection, tracking of the animal and communication with the stakeholders are considered to be a path-breaking invention to solve the HEC [7]. These technologies were exposed to fundamental improvisation and transformed into invasive initially than to noninvasive elephant detecting systems [8]. The invasive elephant detecting systems that are cost-intensive and non-scalable techniques, such as satellite tracking, GPS, light sensors with laser beams and vibration-based sensors, were replaced by reliable, low cost and scalable methods such as acoustic and visual monitoring system in many parts of the world [9]. However, in India, the application of non-invasive elephant detecting system to mitigate HEC is still in its infancy. The traditional methods are still in use and the issues related to HEC are severe in most part of peninsular India [10].

Most of the conventional detecting systems are either involving a manual effort by invasively tracking the elephants by using radio collar, GPS or by using infrared sensors [11]. These are not cost-effective and become complex when scaled up and applied for a larger population and a wider geographical area. Another major concern with respect to invasive techniques is the maintenance and monitoring for its proper functioning. Also, in case of failure, how much easier it is to replace the one which has trouble or malfunctioning is to be considered. Hence, the techniques which are non-invasive are appreciable and preferred over invasive techniques.

The elephant detecting systems today utilize any one of the non-invasive methods with stimuli like visuals, vocal, or seismic signals in detecting the presence of the elephants [12]. Elephant calls are rare events and they may remain silent or can learn to be silent for a long time. Because of this, using vocalization based method may be difficult in identifying its presence [13]. The seismic vibration-based detection methods suffer when the ground has moisture. Also, seismic noises analogous to that made by elephants can be produced by external sources. The elephants produce seismic signals through rumbles and foot stomps; the amplitude of these signals is considerably above the background noise [15]. This makes elephants potentially detectable by the system. These signals along with acoustic cues are used to communicate to the herds. With the available technology, it is difficult to read these signals beyond a certain range. Studies carried out in this sector confirm that the seismic signals can be accurately detected at a range of about 40 meters [16]. The signal may get attenuated when the distance is increased thereby affecting the signal strength. Hence, it might not be effective at broader surveillance. Hence, with this stimulus, it would be difficult to zero in the presence of elephants.

The acoustic signals produced by the elephants over other stimuli as they have the tendency to travel for a longer distance in the medium [17]. The studies reveal that these signals may even travel over 2 km in the air. These signals enable the elephant detection possible even without its physical presence in the close proximity to the detection system. This is a strong fact supporting us to actively consider using acoustic signals for the elephant presence detection [18]. Though the acoustic signals appear to be the better choice, the challenge with respect to noise filtration has to be handled [19]. Also, the suppression of other acoustic signals raised by other animals and birds in the deployed region should be addressed. These properties of the elephant make using only vocalization based method difficult in identifying its presence [20]. These reasons clearly show the need for efficient detection of elephants utilizing the positive aspects of visual detection techniques [14].

Some image-based detection techniques use range and segmentation of elephant image from the landscape are to detect presence of elephants. Zeppelzauer et al. [5] have developed a multi-modal early warning system which primarily uses a automatic vision based module for detection of elephants and to track them in the wild video recordings. The authors have developed a method to identify elephants based on the color model of their body. The background color model is built from the remaining color of the elephant’s surroundings. These models were used to train an SVM classifier to predict the presence of elephant in the wild video recordings. Such approaches face issues when deployed in environment with limited computational resources. Hence, it is noted that there is a need for an effective real-time approach for detecting elephants utilizing the positive aspects of visual detection techniques.

3 Proposed system

The system proposed in this paper aims at detecting the presence of elephants using non-invasive video surveillance method as shown in Figure 1. This method employs a layered approach to process the live video streams obtained from different surveillance cameras placed near timberline off the forest. It uses a composition of classic machine learning and deep learning algorithms to effectively identify the presence of the elephants by minimizing the false positive and true negative predictions. The different computational layers stacked in this system primarily focus on offloading the intensive computation happens at the edge of the deployed system which in turn result in improved response time.

Fig. 1

System Design.

The first layer preprocesses the captured image frame for the subsequent layers. The second layer uses machine learning algorithms to identify the presence of anomalies in the image if any. The third layer is a computationally intensive layer and it will be activated only if there are anomalies found in the image by the second layer. This layer confirms the presence of an elephant by using a pre-trained convolutional neural network (CNN) model, which takes the preprocessed image as an input and runs a single shot detection on it [21]. The final layer is activated if the neural network detects elephants in the image; this layer either sends an alert message to the concerned authorities or logs the visual information.

The areas where such systems are deployed demand for smaller and lighter computers with a minimal technological footprint which can blend with the environment. This naturally favors an ultra-small computer like an Arduino board or a Raspberry Pi. The compromise that should be made here is the amount of computational power such devices would provide. The layering approach used in this system provides conditionally activated layers which in turn reduces the average computational load on the microprocessor thereby making for efficient resource and power usage.

The detection accuracy often undermines the success of a system which is deployed in similar scenarios. Since the system will be deployed in the timberline areas which demand the system to make the decisions automatically with minimal human intervention. These preconditions demand the features taken into consideration are to be minimal and important as the real response times are critical and computing power is limited. The architecture of the proposed system is shown in Figure 2.

Fig. 2

Proposed System.

4 Layered approach of the proposed system

The layers which have been implemented to build the system are described in detail in the subsequent subsections.

4.1 The first layer: preprocessing

This layer of the system preprocesses the input data for the subsequent layers. Individual frames are broken down from the continuous live video feed and are converted into two-dimensional arrays containing the pixel values [22]. The image is then resized to a smaller dimension, to increase the processing time in the next layers. Since the support vector machine (SVM) in the next layer was trained on 64X64 grayscale images, every frame is converted to grayscale and is resized to match, and then flattened. Flattening is a process where a 2D matrix is converted to a 1XN matrix, by appending each row of the 2D matrix one after another.

4.2 The second layer: checking for anomalies

This layer uses the output from the first layer and passes it through a pre-trained SVM. It is highly selective and sensitive to the area where the system is being deployed. The layer is trained on thousands of images (with slight variation) of the said area. The parameters of the SVM used for training are listed in Table 1. The SVM model tries to predict if there are any anomalies in the current frame [23]. Anomalies can include anything which is out of the ordinary for the given location where the system is deployed; it could be wild animals, humans, and other out-of-place objects. This layer is very light computationally and does not involve a lot of processing power either - therefore all incoming frames are checked for anomalies against this layer. If an anomaly is detected, the control gets passed on to the next layer, which confirms if the anomaly was indeed an elephant or not. This layer is the key layer to the entire system as it acts as a filter for all incoming frames, restricting the frames which make it to the next layer. The post-training accuracy of this model after evaluated is 98.7%.

Table 1
Parameters of the SVM used for training

Parameter Value

C 10

Kernel RBF

Degree 3

Parameter	Value
C	10
Kernel	RBF
Degree	3

4.3 The third layer: confirmation using a neural network

A pre-trained CNN (trained on thousands of images of elephants) is now employed to detect the presence of elephants. This is a computationally intensive task on a small computer like the Raspberry Pi. This CNN uses the MobileNet architecture along with the single-shot detection algorithm (well known for image related detections) and predicts a probability or the confidence of an elephant being present [24]. The layer is capable of detecting multiple elephants in a single frame, even if the entire elephant isn’t in the frame. As this layer is only activated when the second layer detects an anomaly, a lot of energy and time is saved on average. After training, CNN was tested against a variety of images. For clear and obvious images, the detection confidence was more than 95% in most cases. For occluded and obstructed images, the neural network was able to detect the elephants albeit the confidence scores were lowered as shown in Figure 3.

Fig. 3

Confidence scores in challenging scenarios: a) 83% b) 15% c) 27% d) 52%.

4.4 The fourth layer: logging

If there was an elephant detected in the previous layer, this layer draws bounding boxes wherever an elephant was detected in the current frame. Other information like time, location and date is saved along with the image for retrieval and analysis later. Alerting systems (SMS alert etc.) can be attached to this layer, thus adding the potential for end-to-end deployment. The comparative analysis of an out-of-the-box image detector and proposed CNN detector is shown in Table 2. It reveals that the proposed approach is 1.5 times faster than the out-of-the-box image detector.

Table 2
Comparison between proposed model and out-of-the-box model

Type of neural network True positives False positives Average processing time (per image)

Out-of-the-box image detector 46 1 0.36s

Proposed CNN detector 44 4 0.24s

Type of neural network	True positives	False positives	Average processing time (per image)
Out-of-the-box image detector	46	1	0.36s
Proposed CNN detector	44	4	0.24s

5 Evaluation of system

5.1 Description of dataset

The data set consisting of images used to train and test the model was collected from real-time video recordings and various Internet sources. The recordings were performed in various regions of which includes Nagerhole and Bandipur national parks on different days under different lighting and climatic conditions. The entire data set consists of 1792 curated images of which 67.96% images have elephants in arbitrary positions and angles including images with multiple elephants in the frame. The collected images were further compiled, standardized, preprocessed and annotated with necessary metadata to form the data set.

5.2 Performance of the proposed system

The detection time is of the essence in such systems, and therefore it is critical that the system performs as quickly as it can. The trade-off between detection time and computing capacity has to be made. The following results were obtained on running this system on a Raspberry Pi 3 with the following specifications.

CPU: 1.2 GHz quad-core ARM Cortex A53 (ARMv8 Instruction Set)

GPU: Broadcom VideoCore IV @ 400 MHz

Memory: 1 GB LPDDR2-900 SDRAM

The average time taken for the SVM to process one frame from the camera (t_svm) is 0.12 s. The average time taken for CNN to run detection on one frame (t_cnn) is 2.1 s. Miscellaneous tasks and alerting on one frame (t_m) takes 0.2 s. Therefore, the total time (T) taken to process one frame which contains some anomaly will be: $T = t_{svm} + t_{cnn} + t_{m} = 2.42 s$ (1) Let an example scenario where the total number of frames captured (n) is 100, contain elephants in (n_e =) 10 frames and some other anomaly in (n_a =)10 frames. The probability of spotting an elephant in this scenario is 10% which is relatively high.

If the anomaly detector (SVM) was not present, the total time to process all frames (T₀) in seconds can be calculated by: $T_{0} = (t_{cnn} + t_{m} * (n_{e} + n_{a}) / n)) * n$ (2) Substituting values in this equation, the total time in seconds will be: $T_{0} = (2.1 + 0.2 * 0.2) * 100 = 214$ (3)

With the anomaly detector (SVM), total time to process all frames (T₁) in seconds can be calculated by:

$\begin{matrix} T_{1} = (t_{cnn} + t_{m} + t_{svm}) * (n_{e} + n_{a}) \\ + t_{svm} * (n - n_{e} - n_{a}) \end{matrix}$ (4) Substituting values in this equation, the total time in seconds will be: $T_{1} = (2.1 + 0.2 + 0.12) * 20 + 0.12 * 80 = 58$ (5)

Comparing the time taken in each case: $\frac{T_{1}}{T_{0}} = 3.69$ (6)

It can be seen that the second layer boosts the speed of the entire system by almost 4 times while keeping the power and processor consumption low. It filters out the frames which do not have any chance of containing an elephant. Since the camera’s effective frame rate depends on the average time of processing one frame, $\frac{n}{T_{1}} = \frac{100}{58} = 1.72 seconds per frame$ (7) i.e 0.58 frames per second on average.

The performance of the machine learning model in detecting the presence of elephants in the dataset is described in the confusion matrix shown in Table 3.

Table 3

Confusion Matrix

5.3 Detection accuracy

The test results of the proposed method is shown in Table 4 in comparison with the state-of-the-art method [5] and it is observed that the proposed method outperforms. The proposed method has fairly high true positive rate, low false positive rate with a better accuracy. The test dataset for the methods are different and the result of the other method is collected from the respective paper.

Table 4
Comparison of test results

Method Dataset size FPR Accuracy

Zeppelzauer et al.[5] 715 2.5% 91.7%

Proposed Method 1792 3.1% 92.7%

Method	Dataset size	FPR	Accuracy
Zeppelzauer et al.[5]	715	2.5%	91.7%
Proposed Method	1792	3.1%	92.7%

5.4 Discussion

When the image contains elephant(s) which are hidden behind vegetation, the model tends to produce true negatives. Similarly, when a herd of elephants are in the frame, the model is not able to correctly predict the number of elephants. These are popular problems in the domain of computational image processing viz occlusion and cluttering. Our future work primarily focus on providing solutions for the aforesaid issues.

6 Conclusion

The paper proposes an approach for detecting elephants in live video surveillance using a CNN. The proposed approach reduces the average computational load on the hardware thereby making way for effective resource and power usage while not compromising on the detection accuracy. Once the detection is confirmed, a trigger can be passed to an alert module to intimate the locals about the presence of elephants. Our proposed work mainly focuses and emphasizes the normal wellbeing of both elephants and locals in the area where it is getting deployed. Deploying the system in the identified region would help reduce the number of HEC incidents. This can contribute to an improvement in conservation, management, and research on elephants. Also, this system can also be added to automate any of the existing camera-based surveillance systems.

References

Noga

S.R.

, Kolawole

O.D.

, Thakadu

O.T.

and Masunga

G.S.

, Wildlife officials only care about animals’: Farmers’ perceptions of a ministry-based extension delivery system in mitigating human-wildlife conflicts in the okavango delta, Botswana, Journal of rural studies61 (2018), 216–226.

Marais

J.C.

, Automated elephant detection and classification from aerial infrared and colour images using deep learning. PhD thesis, Stellenbosch: Stellenbosch University, 2018.

Ogra

M.V.

, Human–wildlife conflict and gender in protected area borderlands: a case study of costs, perceptions, and vulnerabilities from uttarakhand (uttaranchal), India, Geoforum39(3) (2008), 1408–1422.

Rangarajan

, Desai

, Sukumar

, Easa

P.S.

, Menon

, Vincent

, Ganguly

, Talukdar

B.K.

, Singh

, Mudappa

, et al., Gajah. securing the future for elephants in India, 2010.

Zeppelzauer

and Stoeger

A.S.

, Establishing the fundamentals for an elephant early warning and monitoring system, BMC research notes8(1) (2015), 409.

Koirala

R.K.

, Ji

, Aryal

, Rothman

and Raubenheimer

, Dispersal and ranging patterns of the asian elephant (elephas maximus) in relation to their interactions with humans in Nepal, Ethology Ecology & Evolution28(2) (2016), 221–231.

Jukan

, Masip-Bruin

and Amla

, Smart computing and sensing technologies for animal welfare: a systematic review, ACM Computing Surveys (CSUR)50(1) (2017), 10.

Goswami

V.R.

and Vasudev

, Triage of conservation needs: the juxtaposition of conflict mitigation and connectivity considerations in heterogeneous, human-dominated landscapes, Frontiers in Ecology and Evolution4 (2017), 144.

Kamminga

, Ayele

, Meratnia

and Havinga

, Poaching detection technologies: a survey, Sensors18(5) (2018), 1474.

10.

Ramakrishnan

, Ilakkia

, Karthick

and Veeramani

, The role of elephants in the forest ecosystem and its conservation problems in southern India. In Indian Hotspots, pages 317–343. Springer, 2018.

11.

Marvin

D.C.

, Koh

L.P.

, Lynam

A.J.

, Wich

, Davies

A.B.

, Krishnamurthy

, Stokes

, Starkey

and Asner

G.P.

, Integrating technologies for scalable ecology and conservation, Global Ecology and Conservation7 (2016), 262–275.

12.

Ramesh

, Mathi

, Pulari

S.R.

and Krishnamoorthy

, An automated vision-based method to detect elephants for mitigation of human-elephant conflicts. In 2017 International conference on advances in computing, communications and informatics (ICACCI), pages 2284–2288. IEEE, 2017.

13.

Clemins

P.J.

, Johnson

M.T.

, Leong

K.M.

and Savage

, Automatic classification and speaker identification of african elephant (loxodonta africana) vocalizations, The Journal of the Acoustical Society of America117(2) (2005), 956–963.

14.

Mangai

N.M.S.

, Karthigaikumar

, Vinod

S.T.

and Chandy

D.A.

, Fpga implementation of elephant recognition in infrared images to reduce the computational time, Journal of Ambient Intelligence and Humanized Computing, pages 1–16, 2018.

15.

Gunther

R.H.

, O’Connell-Rodwell

C.E.

and Klemperer

S.L.

, Seismic waves from elephant vocalizations: A possible communication mode, Geophysical Research Letters31(11) (2004).

16.

Wood

J.D.

, O’Connell-Rodwell

C.E.

and Klemperer

S.L.

, Methodological insights: Using seismic sensors to detect elephants and other large mammals: A potential census technique, Journal of Applied Ecology42(3) (2005), 587–594.

17.

Poole

J.H.

, Behavioral contexts of elephant acoustic communication. The Amboseli elephants: a long-term perspective on a long-lived mammal. Chicago: The University of Chicago, pages 125–161, 2011.

18.

Roche

J.M.

, Peters

and Dale

, “Your tone says it all”: The processing and interpretation of affective language, Speech Communication66 (2015), 47–64.

19.

Elvitigala

, Tennakoon

, Hamza

, Lokuge

, Zoysa

K.D.

, Keppitiyagama

, Iyer

, Hewage

and Voigt

, Towards a sensor system to tame the human elephant conflict. In 2015 IEEE Sensors Applications Symposium (SAS), pages 1–4. IEEE, 2015.

20.

Anni

J.S.

and Sangaiah

A.K.

, Elephant tracking with seismic sensors: A technical perceptive review, Jurnal Teknologi74(1) (2015).

21.

Chen

H.G.

, Jayasuriya

, Yang

, Stephen

, Sivaramakrishnan

, Veeraraghavan

and Molnar

, Asp vision: Optically computing the first layer of convolutional neural networks using angle sensitive pixels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 903–912, 2016.

22.

Krishnamoorthy

and Mathi

, An enhanced method for object removal using exemplar-based image inpainting. In 2017 International Conference on Computer Communication and Informatics (ICCCI), pages 1–5. IEEE, 2017.

23.

Zhang

, Yu

, Sui

, Zhao

and Zhang

, Object tracking with multi-view support vector machines, IEEE Transactions on Multimedia17(3) (2015), 265–278.

24.

Howard

A.G.

, Zhu

, Chen

, Kalenichenko

, Wang

, Weyand

, Andreetto

and Adam

, Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.