Neural-logic multi-agent system for flood event detection

Abstract

This paper shows the capabilities offered by an integrated neural-logic multi-agent system (MAS). Our case study encompasses logical agents and a deep learning (DL) component, to devise a system specialised in monitoring flood events for civil protection purposes. More precisely, we describe a prototypical framework consisting of a set of intelligent agents, which perform various tasks and communicate with each other to efficiently generate alerts during flood crisis events. Alerts are only delivered when at least two separates sources agree on an event on the same zone, i.e. aerial images and severe weather reports. Images are segmented by a neural network trained over eight classes of topographical entities. The resulting mask is analysed by a Logic Image Descriptor (LID) which then submit the perception to a logical agent.

Keywords

Neural-logic multi-agent systems image segmentation computational geometry perception fusion

1 Introduction

The increasing severity of weather events, particularly storms and heavy rainfall, has caused significant damage worldwide and has contributed to the growing problem of global flooding [11, 23]. One of the major challenges in addressing these disasters is the ability to quickly and accurately identify and respond to them. Traditional methods of monitoring and analysis, such as manual observation and evaluation of meteorological data, are often not sufficient for this task [25]. To address this challenge, many organisations and governments are turning to advanced technology, such as deep learning (DL), to assist with the identification and analysis of flood events [13].

DL approaches use complex algorithms and large datasets to enable the precise extraction and classification of visual characteristics in images and other data. This can be useful for identifying patterns and trends in weather data, as well as for monitoring changes in real-time (see for example the UN Global Pulse project 1 , or the Extreme Earth project 2 ). However, the increasing severity of weather events, particularly those related to flooding, has highlighted the need for advanced, specialised systems for the identification, the monitoring, and analysis of these disasters. One reason is that the performance of DL systems can be affected by overfitting, limited training data, and the presence of similar but distinct classes in the training data. In the context of weather disasters, the images used for training and classification can be limited, and the circumstances present in the images can be highly variable, making it challenging for a DL system to achieve high levels of accuracy. In addition, natural disasters frequently necessitate more in-depth analyses, such as determining the extent of flooding or the potential impact on infrastructure and communities. Moreover, human operators alone are not able to cope in real- time with the very many alerts that arrive daily if not hourly. Hybrid systems, which combine the capabilities of multiple technologies, i.e. those of DL algorithms with those of intelligent logical agents, may offer a solution to this challenge, providing more accurate and comprehensive models for the detection and reporting of flood events.

Accordingly, in this paper we propose extensions to our previous work illustrated in [18]. In particular, we have developed a preliminar prototype (reference in [19]) in which a multi-agent system (MAS) is able to autonomously collect weather alerts, activate a DL module to categorise the related images, filter the results, and alert human operators only if there is a reasonable confidence that a risky situation is in effect arising. To do this, we implemented a system using a combination of logical agents and a DL component. The system uses aerial images and weather reports as information sources. The neural network (NN) is trained on eight classes of topographic entities to segment the images. Once the images have been segmented, a logical image descriptor (LID) is used to generate a logical description of the predicates for the segmented mask. This description is then submitted to a logical agent, which performs the reasoning. We adopted, for our system, computational-logic-based agents endowed with a logical semantics. This because a logic basis provides (at least potentially) verifiability and explainability, and thus trustworthiness [7] 3 . In our system, reasoning is performed using a perception-fusion approach instead of a data-fusion one. In a perception-fusion approach, the agents in the MAS use their own perceptions to reason about the environment and make decisions collectively. This means that each agent independently processes the raw sensor data it receives from its environment and shares its conclusions with the other agents, which are then combined to form a collective understanding of the environment. In contrast, in a data-fusion approach, the agents rely on a centralized data processing unit to collect and process data from all the agents before making decisions.

In summary, our system is designed with the primary goal of providing accurate and reliable alerts or pre-alerts for severe weather situations. To achieve this, we have employed a perception-fusion approach that integrates logical agents and a deep learning component to enable effective reasoning with various forms of information. our system represents a significant step towards our long-term objective of developing an advanced version capable of rapidly identifying flood events and alerting relevant authorities. This would facilitate the swift mobilization of rescue efforts and other essential resources. In conclusion, this system is a first example of a class of real-time applications where a stream of real-time data must be analyzed, filtered, evaluated, and provided to the user together with explanations, to enable prompt countermeasures.

Accordingly, the paper is structured as follows. Section 2 gives an outline of some systems for monitoring or detecting catastrophic events, both in the context of deep learning and MAS. Section 3 provides an overview of the system architecture, while detailed explanations of each MAS agent are provided in Sections 4, 5, 6, and 7. Section 4 describes the Neural agent and its components in details; this section illustrates the experiments performed for the analysis of aerial images using a NN and the implementation of a specific module called LID for generating logical descriptions of segmented images. Section 5 details the implementation of the Perceptor agent and provides some measures of the agent’s correctness in determining whether or not a region is flooded. Section 6 describes the Weather agent, which, as we will see, at the present stage is simulated (as at present, due to related costs, we do not have actual access to weather information). Section 7 provides an overview of the Alert agent, the agent in charge of notifying the proper authorities of any potential flood warnings. In Section 8 we report the discussion of our system and the possible future works. Finally, Section 9 summarises the final remarks and the contribution of our work.

2 Related works

The use of technology to mitigate the effects of natural disasters has been the focus of considerable research in recent years. In particular, the detection and monitoring of flood events has been a key area of interest, given the increasing frequency and severity of such disasters. In this section, we provide an overview of related work in this field, focussing on deep learning approaches, multi-agent systems and their potential combination.

Neural Networks. Several studies have explored the use of DL algorithms for the detection and monitoring of flood events. For example, in [27], several CNN-based models with different encoder backbones are proposed for the segmentation of water bodies using a variety of input data sources, including RGB images, infrared images, etc. Similarly, in [21] a semantic segmentation technique for autonomously detecting floods is employed. In [28], the authors propose a methodology that integrates different machine learning models to automatically identify damaged areas in aerial images. In this work, a DL model is used for automatic feature extraction from remote sensing images and multiple ML classifiers are used to automatically predict flood areas from unseen image data. Additionally, in [16], different DL models were trained for flood image classification to develop more accurate and efficient disaster response and recovery systems.

Although DL-based approaches have shown promise in flood detection, these can be limited by the availability of training data and the highly variable features of the images, which make it difficult for a deep learning system to achieve high levels of accuracy. In addition, more in-depth analyses are often required in these situations, such as determining the extent of the flood and its potential impact on infrastructure and communities.

MAS These challenges have prompted several studies to propose the use of MAS to improve flood detection and monitoring.

One such study, conducted in [1] proposed a MAS to simulate and estimate water levels in an urban catchment area. The system was designed to simulate water level saturation using the MAS approach instead of traditional hydrological modelling. This was achieved by a method that can simulate the uncertainties of the flow regime and calculate the depth of flow at any time, enabling the optimisation of reservoir storage capacity to allocate excess water. In [10] a MAS approach is developed to improve flood forecasting and warnings for a decision support system. This study seeks to provide technical support to the flood monitoring and warning authorities.

Among the work in this field, the work proposed in [17] is of relevance. In this work a flood disaster response simulator is introduced to be used as a benchmark for the coordination of autonomous agents in different MAS approaches.

Hybridization Although MAS-based approaches have proven promising for flood detection and monitoring, we believe they can be improved by considering a module within them in which a DL-based model is employed. Indeed, the integration of NN systems within multi-agent systems could ensure the specialisation of one or more agents in recognising particular patterns and intrinsic features in the data that might be difficult for purely symbolic systems to detect. In the context of monitoring catastrophic events, this specialisation could involve the analysis of images, audio, and video. However, while hybrid systems are promising for the detection and monitoring of such events, there is still a need for advancement in research to try to create systems capable of accurately and efficiently analysing the elements involved in catastrophic events in real-time.

Following this, in this paper we propose a specialised system that combines the capabilities of DL algorithms with those of intelligent logical agents to provide a comprehensive model for the detection and reporting of flood events.

3 Multi-Agent-System

3.1 DALI

The MAS employed in the present work has been implemented using DALI [4–6, 8], which is a programming language developed for constructing logical agents that can behave and make decisions similarly to humans. DALI is a superset of Prolog 4 , and thus any Prolog (logic) program can be seen as a DALI agent, devoid however of agent-oriented proper features. Agent-orientation in DALI agents comes from having events as first-class objects in the language, and from including two new kinds of rules: reactive rules—which are designed specifically to enable the agents to interact with and respond to their surroundings—, and proactive rules—by means of which an agent can take initiatives and initiate activities when deemed suitable by the agent itself.

Specifically, there are four types of events that are explicitly represented in the DALI syntax, and may occur in an agent’s code to trigger agent’s behavior: external events, internal events, present events, and past events. In this particular context, we only consider external and internal events. External events are designated by the postfix E and are caused by the agent’s environment (they correspond to exogenous stimula). When an external event occurs, the DALI agent can choose to react to it by utilising a reactive rule with the external event in its head. Internal events constitute a peculiar feature of a DALI agent, and allow it to act proactively independently of the environment and other agents. Internal events are designated by the postfix I and are described using two rules: the first contains the conditions that must be true for the reaction in the second rule to occur.

Actions are the means by which DALI agents can affect their environment, potentially in response to an external or internal event. Actions can optionally have preconditions. If an action has preconditions, it is defined using an action rule, otherwise, it is simply an action atom denoted by the postfix A.

DALI also includes a communication architecture with three levels. The first level implements a FIPA-compliant communication protocol 5 . The second level allows agents to communicate with each other using messages, and the third level enables agents to communicate with other agents and external systems via the support of an external ontology.

3.2 FloodMAS architecture

The architecture of the system that we implemented in this proposal is shown in Fig. 1. Peculiarly, FloodMAS is a framework that integrates a convolutional neural network (CNN) component into a MAS.

Fig. 1

FloodMAS architecture.

For the implementation of our architecture, we implemented the following agents:

[Neural agent] — is a Python agent designed to analyse aerial images and supply a logic description of them to the Perceptor agent. The agent is made up of two main components: i) a CNN for analysing the elements within aerial images, and ii) the LID that examines the output of the CNN and provides a description of the different elements within the output image. The CNN is trained to identify various elements, such as houses, vehicles, trees, and so on. This is accomplished by generating a mask of the original image that highlights the presence of these elements. The LID then uses morphological operations and calculations to analyse the mask and provide a description of the adjacent regions (see Section 4 for a more detailed description).

[Perceptor agent] — is an agent written in the DALI programming language. This agent receives a set of logical facts from the Neural agent and, using a set of rules, determines whether or not segmented regions included within the image are flooded. In practice, these rules are intended to enable the automated analysis of aerial images to the purpose of identifying and classifying regions based on their water and flood status.

[Weather agent] — is a Python agent designed to provide weather information to the Alert agent making use of an API.

The weather information provided by this agent is processed and then translated by the Alert agent—which is responsible for disseminating the alerts to relevant parties. This system allows for the efficient and effective communication of weather-related information and enables appropriate actions to be taken in response to potentially hazardous weather events.

[Alert agent] — is a DALI agent that receives information from both the Perceptor agent and the Weather agent. The primary responsibility of this agent is to receive information, examine it, and provide alert levels to the appropriate authorities. As stated in [18], this agent performs two action: (i) alert, and (ii) pre-alert. Following is a description of these two actions:

Alert: if Perceptor agent and Weather agent report flooding issues in the area of interest, then Alert agent notifies authorities in real-time via the send_message action.

Pre-alert: if either agent provides notification of a potential flooding issue; in this case, Alert agent notifies the authorities via the inform action.

The distinction between these two actions stems from the fact that if both agents – the Perceptor agent and the Alert agent – agree on the presence of flood events, then the situation is more likely to really occur, and the Alert agent must then urgently notify the appropriate authorities; if, on the other hand, only one of the two agents reports the presence of a disaster, the probability of flooding will be significantly lower, and the agent notifies the authorities, who will ultimately decide whether or not the phenomenon has occurred.

4 Neural agent

This section provides a detailed description of the two components that are part of the Neural agent (see Fig. 2). The first part of this section will describe the NN model used, as well as the training dataset and experiment results. In the second part, the morphological operations and calculations that are performed on the NN’s output will be described. These operations and calculations are used to obtain a description of the output in the form of DALI (i.e., Prolog) facts, which can then be provided to the MAS. As shown in Fig. 2, this module is identified as the Logic Image Description (LID) generator.

Fig. 2

Neural agent.

4.1 PSP-Net

In our work, as well as in [18], we utilised the PSP-Net model to conduct image segmentation on aerial images.

The PSP-Net model [29] is a powerful tool for image semantic segmentation, a process that allows a NN to identify and classify different objects in an image. This is achieved by organising the pixels in the image in a semantically meaningful way, grouping pixels that belong to the same object class together and assigning them a common color or gray level. This allows the NN to understand the content of the image and make predictions about the objects it contains.

The PSP-Net model is composed of two main components: an encoder and a decoder. The encoder is responsible for extracting features from the input image, using a CNN and a pyramide pooling module. The pyramide pooling module is a key component of the PSP-Net model, as it helps to capture the global context of the image by combining features at multiple scales. This is accomplished by pooling the feature map generated by the CNN into different dimensions and then up-sampling the pooled features to match the size of the original feature map. The resulting maps are then concatenated with the original feature map and passed to the decoder.

The decoder is an additional module that processes the concatenated feature maps to make predictions about the objects within the image. It uses the extracted features to generate the final segmented image, where each object is assigned a unique color or gray level. The decoder is therefore essential for turning the extracted features into meaningful predictions about the objects present in the image.

The use of this model, allow us to take advantage of the model’s ability to capture global context and make accurate predictions about the objects in an image.

4.1.1 PSP-Net experiment

As also reported in [27], the available datasets for semantic segmentation of very high resolution satellite and aerial images are limited. The main reason stems from licences that limit the redistribution of purchased imagery. Publicly available datasets include ISPRS [22], Dstl [15], and Deepglobe [9]. Nevertheless, only Dstl and Deepglobe datasets contain water classes and only the latter covers samples outside urban areas. Frequently, the size and shape of the water bodies is limited and none of them considers a floodwater class. Recently released datasets for natural disaster applications include the SpaceNet 8 dataset [14], which focuses on the detection of flooded roads and buildings, and the FloodNet dataset [20].

For our experiment, we employed the FloodNet dataset, which contains images captured during flights in Texas and Louisiana from August 30 to September 3, 2017. These flights were conducted at 61 meters in altitude, yielding images with exceptional spatial resolution. The dataset includes images of roads, buildings, vehicles, as well as attribute data indicating whether an object was flooded or not. This dataset includes a total of 2343 images, which are distributed as shown in Table 1 (cf. 1 column). The dataset is divided into the following ten classes: background, building-flooded, building-non-flooded, road-flooded, road-non-flooded, water, tree, vehicle, pool, and grass. Initially, all images containing only a single class are removed, such as a field image containing only grass or an image containing only water. This produces a change in the dataset’s distribution, as shown in Table 1 (cf. column 2). Following that, the class distribution is modified, reducing the number of classes to eight: background, building, road, water, tree, vehicle, pool, and grass. As in [18], we decided to change the class distribution by removing the distinction between flood and non-flood, as we only want the NN to be able to recognise objects within the image and not determine whether a flooding event has occurred or not. In contrast to the previous experiment proposal, the dataset is not arbitrarily divided into train and test data in a 70-30 ratio, but rather the partitioning presented at the FloodNet-Challenge 6 is utilised.

Table 1
Floodnet data distribution

Dataset Before After

Train 1445 1361

Test 448 426

Validation 450 434

Tot. 2343 2221

Dataset	Before	After
Train	1445	1361
Test	448	426
Validation	450	434
Tot.	2343	2221

For our NN experiment, we decided to use the albumentations library 7 to perform transformations on our images. The image augmentation step is composed of several individual transformations that are applied to the images in our training set. First, we use a transformation to resize all of the images to have a height and width of 384 x 384 pixels. This ensures that all of the images have the same dimensions, which is a necessary condition for training a NN. Next, we use a random transformation selector, which randomly applies one of the following transformations: brightness/contrast adjustment, color adjustment, and noise injection. The intent of altering brightness and colour is to support the NN in learning more robust characteristics. The noise injection transformation, on the other hand, adds random noise to the images. This can help the NN generalise more effectively and avoid data overfitting. Lastly, we normalise our images by referencing ImageNet 8 weights (widely-used benchmark in computer vision). Normalisation ensures that all of the pixel values in the images are within a similar range, which can help the NN to converge faster and perform better. For the test set, we only resize and normalise the images, as we do not want to modify the images in any way. This allows us to evaluate the performance of the NN on unmodified data.

In our experiment, the CNN was trained for 200 iterations with a batch size of 8. The loss function employed during training is a weighted sum of the Binary Cross Entropy (BCE) loss and the Dice loss [24]. In mathematical terms, the Dice loss is defined as:

$DiceLoss = 1 - \frac{2 * \sum_{i = 1}^{n} (y_{i} * \hat{y_{i}}) + δ_{i}}{\sum_{i = 1}^{n} y_{i} + \sum_{i = 1}^{n} \hat{y_{i}} + δ_{i}}$ (1)

where y_i and $\hat{y_{i}}$ are the ground truth and predicted pixel values, respectively, and δ is a small constant value added to the numerator and denominator to prevent division by zero.

The Dice loss quantifies the overlap between the actual and expected segmentations; it is often used as a loss function in training DL models for semantic segmentation tasks, where the objective is to predict a pixel-by-pixel class label for each image in the dataset. This loss is computed by averaging the loss values for each batch. Given that it is differentiable, the Dice loss function is a valuable tool for training DL models via gradient descent. In addition, it is sensitive to the spatial overlap between predicted and ground-truth segmentations, a crucial factor for many semantic segmentation tasks. However, it is susceptible to class imbalance and does not account for false negatives and positives. Consequently, it is frequently combined with other loss functions to enhance model performance. In our case, we used the BCE loss, as it can be used to train a model to predict the class probabilities for each pixel in an image.

The BCE loss is calculated as the negative log likelihood of the predicted class probabilities, given the ground truth labels. It is defined as: $BCE = - \frac{1}{N} \sum_{i = 1}^{N} y_{i} log (\hat{y_{i}}) + (1 - y_{i}) log (1 - \hat{y_{i}})$ (2)

where y_i and $\hat{y_{i}}$ are the ground truth and predicted class probabilities, respectively, and N is the number of classes.

As a result, the overall loss function is defined as:

$Loss = BCE * α + DiceLoss * (1 - α)$ (3)

where BCE is the binary cross-entropy loss, DiceLoss is the Dice loss, and α is a hyperparameter that controls the relative contribution of the BCE loss to the overall loss.

Fig. 3 depicts the loss function trends during NN training. In both the train and test sets, the loss decreases as the number of epochs increases. In the case of the test set, learning slows significantly after the 100th epoch compared to the train set, with slight overfitting observed between the 120th and 250th epochs.

Fig. 3

Loss trend during experiment.

We utilised two metrics to evaluate the performance of our model: accuracy and mean Intersection-over-Union (mIoU). Observe the pattern of the metrics by consulting Fig. 4 and Fig. 5.

Fig. 4

Accuracy trend during experiment.

Fig. 5

mIoU trend during experiment.

The mIoU metric was chosen in addition to the accuracy metric since it is a commonly employed metric for evaluating the performance of semantic segmentation models. This metric measures the average overlap between predicted and ground-truth segmentations for each class in the dataset, as follows:

$mIoU = \frac{1}{n} \sum_{i = 1}^{n} \frac{{TP}_{i}}{{TP}_{i} + {FP}_{i} + {FN}_{i}}$ (4)

where n is the number of classes, TP_i is the number of true positive pixels for class i, FP_i is the number of false positive pixels for class i, and FN_i is the number of false negative pixels for class i.

This formula calculates the IoU for each class and then takes the mean over all classes. The IoU for each class is calculated as the number of true positive pixels (correctly classified as belonging to the given class) divided by the sum of the true positive pixels, the false positive pixels (incorrectly classified as belonging to the given class), and the false negative pixels (incorrectly classified as not belonging to the given class). The mean IoU is then calculated as the mean of the IoUs for all classes. To calculate the IoU index, by similarity, we used the Jaccard index.

The Jaccard index is defined as the size of the intersection of two sets divided by the size of the union of the two sets. It can be used to calculate the IoU by treating the predicted segmentation and ground truth segmentation as sets of pixels, with each pixel represented as a binary value (i.e., a value of 0 or 1 indicating whether the pixel belongs to the set). Mathematically:

$IoU = \frac{\sum_{i = 1}^{n} (y_{i} * \hat{y_{i}})}{\sum_{i = 1}^{n} y_{i} + \sum_{i = 1}^{n} \hat{y_{i}} - \sum_{i = 1}^{n} (y_{i} * \hat{y_{i}})}$ (5)

where y_i and $\hat{y_{i}}$ are the ground truth and predicted pixel values, respectively, and n is the number of pixels in the image.

If we examine Table 2, we can observe that the accuracy values are greater than mIoU values in all the train, the test, and validation scenarios. This could be due to the fact that the NN makes relatively few errors, but these errors tend to occur for larger regions. For instance, if the NN accurately labels the majority of pixels in an image but assigns a large region of pixels incorrectly, it could result in high accuracy but low mIoU. Another possible explanation is that the NN has difficulty distinguishing between certain classes. For instance, if there are two visually similar classes, the NN may have issues properly labelling pixels belonging to one of these classes, despite correctly labelling the majority of other pixels. This could result in a relatively low mIoU even with a high overall accuracy.

Table 2

Experiment results

Dataset	Loss	Accuracy	mIoU
Train	0.10	92%	73%
Test	0.14	90%	68%
Validation	0.15	89%	67%

In the future, to enhance the mIoU, it may be necessary to experiment various techniques, such as adjusting the NN architecture, employing different loss functions, incorporating more training data, etc.

Figure 6 illustrates an example of the model’s output. It is noticeable that the model successfully classifies and segments images (see comparison of ground truth mask and predicted mask). However, the segmentation contours of the predicted mask appear smoother than those of the image’s ground truth mask, an aspect that can be enhanced in the future.

Fig. 6

PSP-Net output.

4.2 LID generator

As described at the beginning of this section, in this part we describe the various calculations and operations performed to transform the output of our NN into a set of descriptive facts. Notice that, simple as it is, our method represents a novelty in the literature, where at present no such method can be found.

First, the mask produced by the model is analysed using the skimage package [26] in Python, which calculates different attributes for each region within the images, such as bounding boxes, pixel coordinates, area, centroids, and so on.

We use centroids and coordinates to determine the circumradius of each region. The circumradius can be identified by determining the greatest distance between the region’s centroid and any of its coordinates. This is done by subtracting the centroid coordinates from the coordinates of each point in the region, squaring the result, and summing it across all coordinates in the region. The maximum of these distances is then taken and the square root is calculated to obtain the circumradius of the region, as expressed in the following formula:

$radius = \sqrt{max (\sum_{i = 1}^{n} {(r_{{co}_{i}} - r_{ce})}^{2})}$ (6)

where r_{co
_i} is the i-th coordinate of region r and r_ce is the centroid of region r.

Once the circumradius of each image region has been calculated, we attempt to select adjacent regions based on their spatial relationships. However, before to determining adjacencies, we decide to eliminate the very small water regions that were sometimes observed in the images. The objective of this procedure is to remove water regions that are too small from the mask. To do so, we sum the areas of all water regions in the mask. If the total area is less than a threshold value, which is set to 1% of the total number of pixels in the mask, we remove these regions from the mask.

The area of a region is defined as the number of pixels in the region. To calculate the area of a region, skimage creates a binary image in which all pixels belonging to the region are set to 1 and all other pixels are set to 0. The area of the region is then calculated as the sum of all non-zero pixels in this binary image. To remove small water regions from the mask, we perform the steps reported in Algorithm 1.

Algorithm 1. Remove Small Water Regions
1: procedure SMALLWATERREGIONS
2: sum ← 0
3: water ← 3
4: threshold ← total _ pixels × 10^-2
5: forregion in maskdo
6: iflabel = = waterthen
7: sum ← sum + area (region)
8: end if
9: end for
10: ifsum < thresholdthen
11: Drop
12: end if
13: end procedure

Once the excessively small water regions have been omitted, the adjacent regions are calculated. In this work, we use a method for finding adjacent regions using the centroids and circumradii of the regions (see Fig. 7). Our approach is also based on the observation that adjacent regions are typically spatially overlapping and proximate to each other. To capture these characteristics, we use two measures: the overlap of the bounding boxes of the regions and the proximity of their centroids. To determine the adjacency of regions, we first check if the bounding boxes of the regions overlap. This is done by comparing the minimum and maximum x and y coordinates of the bounding boxes. Specifically, let R be a region with bounding box defined by the coordinates (x₁, y₁, x₂, y₂), where (x₁, y₁) and (x₂, y₂) are the coordinates of the top-left and bottom-right corners of the bounding box, respectively. Let O be another region with bounding box defined by the coordinates $(x_{1}^{'}, y_{1}^{'}, x_{2}^{'}, y_{2}^{'})$ . The overlap between the two bounding boxes can be determined by the following conditions:

Fig. 7

Adjacency representation. Circles 1,2, and 3 circumvent the polygons defined by the irregular shapes generated in the mask by the NN. r₁, r₂, r₃ are their respective radii, centered in their respective centroids of the polygons. The adjacency condition between two polygons X and Y holds when the sum of the r_x + r_y is less than the distance between their centroids.

${\begin{matrix} x_{1} \leq x_{2}^{'} \\ x_{2} \geq x_{1}^{'} \\ y_{1} \leq y_{2}^{'} \\ y_{2} \geq y_{1}^{'} ∥ \end{matrix}$ (7)

If all of the above conditions are satisfied, then the bounding boxes of R and O overlap. If any of the conditions is not satisfied, then the bounding boxes do not overlap.

If the bounding boxes of two regions overlap, we calculate the distance between their centroids using the Euclidean distance 9 formula. This distance, d, is calculated by finding the difference in the x-coordinates and y-coordinates of the two regions. Specifically, we use the following formula:

$d = \sqrt{{(r_{x_{1}} - r_{x_{2}})}^{2} + {(r_{y_{1}} - r_{y_{2}})}^{2}}$ (8)

where r_{x
₁} and r_{y
₁} are the x-coordinate and y-coordinate of the centroid of the first region, and r_{x
₂} and r_{y
₂} are the x-coordinate and y-coordinate of the centroid of the second region.

Once we have calculated the distance d, we can determine whether the two regions are adjacent by comparing it to the sum of their circumradii (see Fig. 7). If the distance is less than this sum, then the regions are touching and are considered adjacent:

$d \leq r_{r_{1}} + r_{r_{2}}$ (9)

where r_{r
₁} and r_{r
₂} are the circumradii of the first and second region, respectively (see Eq. 6). If the condition is true, then the regions are adjacent.

At the end of this procedure, we obtain the LID output, which consists of two.PL files containing logical facts (see Fig. 8) and is sent to the Perceptor agent.

Fig. 8

LID output: Prolog facts describing the geometrical relations of the mask regions. Each region reports the region ID, class ID, and pixel coordinates (x, y) of its centroid calcutated by the skimage Python library. Here is shown just an extraction from a typical LID file, only the first 10 regions, which may contain hundreds of regions, depending on the complexity of the original aerial image. At the end of the LID file the adjacency facts are listed, as result of the computational geometry algorithm.

Observing Fig. 8, it is evident that each region has distinct arguments. The first argument is a region identifier, the second is the region label, and the third and fourth are the centroids’ x and y coordinates of the image. The file containing the facts of adjacent regions, on the other hand, includes two arguments that are most likely the two identifiers of the regions found in the file containing the information of the regions. Thus, the Perceptor agent learns, for instance, that region 0 is adjacent to regions 5, and region 1 to 5 and 6. And so forth.

5 Perceptor Agent

As described previously, the Perceptor agent receives information from the Neural agent and processes it to determine whether flooding is occurring in a given area (see Section 4).

To assess the presence of flooded or non-flooded areas, the following set of rules are implemented:

water(Region):-

region(Region, 3,_,_).

flooded(Region, Label):-

region(Region, Label,_,_),

(Label \= 7),

findall(Adjacent,

(adjacent(Region, Adjacent),

water(Adjacent)),

AdjWaterRegions),

length(AdjWaterRegions, N),

N > = 2.

has_flooded_regions:-

flooded(_,_).

In practice, two predicates are employed: water and flooded. The water predicate is defined such that a region is classified as water if the region predicate is true for the region and the region’s label equals 3 (i.e. water). This allows for the identification of regions within the aerial image that are bodies of water.

The flooded predicate is more complex, requiring multiple conditions to be met for its definition. A region is classified as flooded if the region predicate is true, the label is not 7 (i.e. grass), and at least two adjacent regions are classified as water. This definition enables the identification of regions experiencing flooding when water has spread beyond the boundaries of a defined body of water into adjacent areas.

Finally, the has_flooded_regions predicate is defined to be true if there is at least one flooded region. This enables the determination of whether a particular aerial image contains any flooded regions.

Overall, these predicates provide a useful set of tools for the automated analysis of aerial images and the classification of water and flood events. However, this simple kind of inference is to be intended as a proof of concept of the fact that results returned by NN can be processed, filtered and reasoned about so as to elicit the desired information. Such information might sometimes be obtained via a larger NN, that would however be more difficult to train, assuming that a suitable dataset can be found.

During the implementation of this agent, several goodness-of-fit metrics were evaluated to determine the agent’s ability to distinguish between flooding and non-flooding situations. This was made possible by the availability of ground truth values for the label flooded or non-flooded. Specifically, True Positive (TP), False Positive (FP), and True Negative (TN), False Negative (FN) values were evaluated (see Table 3). Thanks to these values, it was possible to determine the following precision and recall values for our agent:

Train set:

Precision: 57%

Recall: 83%

Test set:

Precision: 58%

Recall: 73%

Validation set:

Precision: 57%

Recall: 77%

Table 3
Confusion matrices for the Perceptor Agent

Train Ground Truth Total

Negative Positive

Predicted Negative 1095 27 1122

Positive 101 138 239

Total 1196 165 1361

Train Ground Truth Total

Negative Positive

Predicted Negative 347 14 361

Positive 27 38 65

Total 374 52 426

Validation Ground Truth Total

Negative Positive

Predicted Negative 348 12 360

Positive 32 42 74

Total 380 54 434

Train	Ground Truth	Total
Predicted	Negative	1095	27	1122
	Positive	101	138	239
Total	1196	165	1361

Train	Ground Truth	Total
Predicted	Negative	347	14	361
	Positive	27	38	65
Total	374	52	426
Validation	Ground Truth	Total
	Negative	Positive
Predicted	Negative	348	12	360
	Positive	32	42	74
Total	380	54	434

In general, it can be seen that in all the different datasets (train set, test set, and validation set), the results of precision and recall are close to each other, indicating that our logic program is stable to the change in data it is subjected to. In the train set, the classifier achieved a precision of 57%, meaning that out of all the instances predicted as flood events, 57% of them were actually flood events. It also achieved a recall of 83%, meaning that out of all the actual flood events, the classifier was able to identify 83% of them.

Similarly, in the test set, the classifier achieved a precision of 58% and a recall of 73%. In the validation set, it achieved a precision of 57% and a recall of 77%. These results suggest that the classifier is capable of identifying a good percentage of flood events (see high recall), but still generates a relatively high percentage of false positives—cases classified as flooding where a flood event does not occur (see low precision). False positives can be problematic in the context of flood monitoring, as they can lead to false alerts or, worse, the allocation of unnecessary resources. On the other hand, it is important to have a high recall, as we want to avoid missing any actual flood events. In this sense, a balancing between precision and recall is required.

In the case of this sort of classifier, the recall values are relatively high across all three data sets, but the precision values could be improved. One way to potentially achieve this in future work, is by collecting more training data or fine-tuning the logic program by introducing more sophisticated forms of NN results post-evaluation. With the use of more sophisticated forms of reasoning, in fact, in future versions the classifier performance should increase, reducing the number of false positives.

6 Weather agent

It was decided, for the purpose of implementing this agent, to simulate a future API-integrated weather service (to avoid paying for this expensive service for now). Therefore, the simulation involves a Python program that sends a random set of events to the Alert agent. The events provided by this agent are equivalent to those provided by the future API integration and include four alert levels: green, yellow, orange, and red. The purpose of these alert levels is to convey the severity and potential danger of weather events. Green indicates that no intense or dangerous events are anticipated. The yellow alert level indicates that potentially dangerous local events are anticipated. The orange alert level indicates that events are expected to be more intense than usual. Finally, the red alert level indicates that extreme events are anticipated and the situation is extremely dangerous.

7 Alert agent

As described in Section 3, this agent is in charge of alerting the relevant authorities in case of receiving warnings from Perceptor and Weather agent. In addition to performing alert andpre-alertactions, this agent also includes a mapping between the alert levels provided by the Weather agent and the corresponding hazard levels, with green, yellow, orange, and red levels corresponding to low, moderate, high, and severe hazard levels, respectively, via the very simple code shown below:

:- dynamic new_color/1.

danger_level(green, low).

danger_level(yellow, moderate).

danger_level(orange, high).

danger_level(red, severe).

danger(Color, Level):-

new_color(Color),

danger_level(Color, Level).

As said previously, pre-alert and alert refer to different stages in the process of detecting and responding to critical weather conditions. A pre-alert is triggered when one of the agents—Perceptor or Weather—detects a potential critical weather condition. In the code below, it is identified by the events weather_alert, and flood_alert. In both cases, the agent sends a message to the system indicating that a critical weather condition may be present and further investigation is needed. If another agent also detects the same potential critical weather condition, an alert is issued (see the event critical_issue. The alert is a more serious warning, indicating that critical weather conditions have been confirmed by multiple sources and authorities should be notified. The difference between pre-alert and alert is that the former is a warning signal for potential critical weather condition, while the latter is a confirmation of critical weather conditions and requires action.

:- dynamic

pre_alert_weather/0,

flood_issue/0.

weather_issue:-

danger(Level),

(Level \= low),

(Level \= moderate).

weather_alert:-

weather_issue,

retract(new_color(_)).

weather_alertI:>

assert(pre_alert_weather),

write(’Pre-alert

from Weather Agent’).

no_issue:-

+weather_issue,

retract(new_color(_)).

no_issueI:>

write(’Not critical

weather condition! ’).

flood_alertE:>

assert(flood_issue),

write(’Pre-alert

from Perceptor Agent! ’).

critical_issue:-

pre_alert_weather,

flood_issue,

retract(pre_alert_weather),

retract(flood_issue).

critical_issueI:>

write(’Alert authorities! ’).

8 Discussion and future works

The proposed hybrid system is an innovative approach to disaster management and prevention. Its ability to assign specific tasks to specialised agents distinguishes it from other traditional systems. In this architecture, the Neural agent is responsible for perceptual and image analysis tasks, relying on its NN to process large amounts of data quickly and accurately. The Perceptor agent, on the other hand, is responsible for evaluating the contents of the images and identifying the presence of flooding. Although the potential of a hybrid system is clear, below are some limitations and possible future work to improve our prototype. One enhancement of the system is the ability to perform more sophisticated filtering and evaluation of results than the simple example presented in this article. By working closely with disaster management and mitigation experts, we can develop more advanced detection and prevention strategies that take into account the probability of flooding in a given area and/or historical data on such events. The system can also be improved through defeasible reasoning, which allows us to make assumptions and draw correct conclusions about the meaning of images or other data sources classified by the DL model. With these advanced techniques, the system can become even more accurate and effective in disaster prevention. The logical agent can also help disseminate warnings to areas at risk, preventing loss of life and minimising property damage.

Besides the improvement of the Perceptor agent, the Neural agent can also be enhanced. Indeed, the datasets available for training NNs on catastrophes unfortunately have limited geographical coverage. Potentially this can lead to a problem in training the NN model, in which the model itself learns territory-specific characteristics. The portability of the approach depends on the ability to generalise these features to other geographical areas. If we want to use the agent in a specific geographical area with different characteristics than the dataset on which it was trained, it might be necessary to retrain the agent or fine-tune its parameters to ensure optimal performance in the new environment. Although it is possible to train a single agent for reuse anywhere, this approach would likely require a large and diverse dataset that includes examples from a variety of countries and regions. Alternatively, multiple agents could be trained, each specialising in a particular geographical area, such as Europe, Asia or Africa.

9 Conclusion

In this paper, we unveil a novel approach to designing and executing a MAS that seamlessly blends neural and logic modules to amalgamate data originating from diverse physical sources. Our objective is to enhance the confidence of alerts relayed to civil protection authorities. To attain this goal, we propose a propose a novel approach that integrates multiple sources of information through a perception-fusion paradigm. Our framework entails the collaboration of several agents, each tasked with processing specific data streams, such as image and weather information. During the test phase of our proposed model, we were able to achieve good results while maintaining computational efficiency. Specifically, the average time taken for the system to acquire the necessary image and weather data and determine the presence of floods in the area of interest was found to be 50 seconds. This highlights the potential of our proposed methodology to be implemented in real-world scenarios, where rapid and accurate decision-making is of utmost importance. Moreover, the sort of hybrid architecture that we have introduced holds the potential to be utilized in diverse application domains. In particular, our approach can be instrumental in aiding human operators who need assistance from AI systems or in creating Human-AI teams that can achieve synergy between humans and machines. This type of intelligent system is becoming increasingly popular as it can be used for performing complex tasks, such as initiating and coordinating rescue operations.

Footnotes

Acknowledgments

We are grateful to Professor Denise Boncioli from the Department of Physical Sciences and Chemistry for allowing us to use computing time on their deep learning workstation, which was obtained through a grant for young researchers at the University of L’Aquila in 2022.

Note that several logic-based agent-oriented programming languages and systems exist (cf., e.g., [2, 3, ] for recent surveys on such languages) whereas we adopted the language DALI, defined and implemented by our research group.

References

Bao

, Kim

, Ai

, Lai

, Wang

Urban water-log simulation and prediction based on multi-agent systems, Proceedings of the 13th International Conference on Geo-Computation, The University of Texas at Dallas, 2015, pp. 317–325.

Bordini

R.H.

, Braubach

, Dastani

, Seghrouchni

A.E.F.

, Gómez-Sanz

J.J.

, Leite

, O’Hare

G.M.P.

, Pokahr

, Ricci

A survey of programming languages and platforms for multi-agent systems, in: Informatica (Slovenia)30(1) (2006), 33–44.

Calegari

, Ciatto

, Mascardi

, Omicini

Logic-based technologies for multi-agent systems: A systematic literature review, in: Auton Agents Multi Agent Syst35(1) (2021), 1.

Costantini

, Tocchio

A logic programming language for multi-agent systems, in: S. Flesca, S. Greco, N. Leone and G. Ianni, eds.: Logics in Artificial Intelligence, European Conference, JELIA 2002, Proceedings, Vol. 2424 of Lecture Notes in Computer Science, Springer.

Costantini

, Tocchio

The DALI logic programming agentoriented language, in: J.J. Alferes and J.A. Leite, eds.: Logics in Artificial Intelligence, 9th European Conference, JELIA 2004, Proceedings, Vol. 3229 of Lecture Notes in Computer Science, Springer, pp. 685–688.

Costantini

, Tocchio

About declarative semantics of logic-based agent languages, in: Declarative Agent Languages and Technologies III, Third Intl Works, DALT 2005, Selected and Revised Papers, Vol. 3904 of LNAI, Springer, pp. 106–123.

Costantini

Ensuring trustworthy and ethical behaviour in intelligent logical agents, in: J Log Comput32(2) (2022)443–478.

De Gasperis

, Costantini

and Nazzicone

, DALI multi agent systems framework, doi 10.5281/zenodo.11042. DALI GitHub Software Repository (2014), DALI: http://github.com/AAAI-DISIM-UnivAQ/DALI.

Demir

et al., Deepglobe 2018: A challenge to parse the earth through satellite images, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018.

10.

El Mabrouk

, Sadouq

Z.A.

, Gaou

and Kassara

, An improved flood forecasting and warning system using data mining, in: International Journal of Advanced Research in Computer Science and Software Engineering3 (2013), 474–483.

11.

Field

C.B.

, Barros

, Stocker

T.F.

, Dahe

Managing the risks of extreme events and disasters to advance climate change adaptation: Special report of the intergovernmental panel on climate change, in: Cambridge University Press, 2012.

12.

Garro

, Mühlhäuser

, Tundis

, Baldoni

, Baroglio

, Bergenti

, Torroni

Intelligent agents: Multi-agent systems, in: S. Ranganathan, M. Gribskov, K. Nakai and C. Schönbach, eds. Encyclopedia of Bioinformatics and Computational Biology, Vol. 1, Elsevier2019, pp. 315–320.

13.

Gailhofer

, Herold

, Schemmel

J.P.

, Scherf

C.S.

, de Stebelski

C.U.

, Köhler

A.R.

and Braungardt

, The role of artificial intelligence in the european green deal, Luxembourg, Belgium: European Parliament, 2021.

14.

Hänsch

et al., Spacenet 8-the detection of flooded roads and buildings, 2022, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

15.

Iglovikov

, Sergey

, Vladimir

Satellite imagery feature detection using deep convolutional neural network: A kaggle competition, 2017, arXiv preprint arXiv: 1706.06169.

16.

Jehoiada

, Yussif

S.B.

, Patamia

R.A.

, Sarpong

, Qin

Flood or Non-Flooded: A Comparative Study of State-of-the-Art Models for Flood Image Classification Using the Flood-Net Dataset with Uncertainty Offset Analysis, in: Water2023, 15, p. 87https://doi.org/10.3390/w15050875

17.

Krausburg

, Chrisosthemos

, Bordini

R.H.

, Jürgen

, Disaster Response Simulation as a Testbed for Multi-Agent Systems, Multi-Agent Systems and Agreement Technologies - 17th European Conference, EUMAS 2020, and 7th International Conference, Lecture Notes in Computer Science Vol. 12520, 2020, pp. 67–81.

18.

Rafanelli

, Costantini

, De Gasperis

, A Multi-Agent-System framework for flooding events, in: WOA, 2022.

19.

Rafanelli

, De Gasperis

, Multi-Agent System for Flooding Disaster Management, Zenodo, https://doi.org/10.5281/zenodo.7705313.

20.

Rahnemoonfar

, Chowdhury

, Sarkar

, Varshney

, Yari

, Murphy

, Floodnet: A high resolution aerial imagery dataset for post flood scene understanding, in: IEEE Access2021 vol. 9, 89644–89654.

21.

Rambhad

, Dhirendra

P.S.

, Jaytrilok

Detection of Flood Events from Satellite Images Using Deep Learning, Intelligent Data Engineering and Analytics, Proceedings of the 10th International Conference on Frontiers in Intelligent Computing: Theory and Applications (FICTA 2022), Springer Nature Singapore, 2023.

22.

Rottensteiner

et al., The ISPRS benchmark on urban object classification and 3D building reconstruction, ISPRS Ann Photogramm Remote Sensing Spatial Inf Sci1(3) (2012), 293–298.

23.

Sulistiyono

, Setiawan

, Saidah

, Impact of Climate Change on Dam Safety, in: International Conference on Rehabilitation and Maintenance in Civil Engineering, Springer, 2023, pp. 157–166.

24.

Sudre

C.H.

, Li

, Vercauteren

, Ourselin

, Cardoso

M.J.

Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations, in: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, 2017, pp. 240–248

25.

Talchabhadel

, Karki

and Parajuli

, Intercomparison of precipitation measured between automatic and manual precipitation gauge in Nepal, in: Measurement106 (2017), 264–273, ISSN 0263-2241.

26.

van der Walt

, Schönberger

J.L.

, Nunez-Iglesias

, Boulogne

, Warner

J.D.

, Yager

, Gouillart

and Yu

, scikitimage: Image processing in Python, 2014, https://doi.org/10.7717/peerj.453.

27.

Wieland

, Martinis

, Kiefl

, Gstaiger

, Semantic segmentation of water bodies in very high-resolution satellite and aerial images, in: Remote Sensing of Environment, Vol. 287, 2023 ISSN 0034-4257.

28.

Yang

, Cervone

Analysis of remote sensing imagery for disaster assessment using deep learning: A case study of flooding event, in: Soft Comput23 (2019), 13393–13408.

29.

Zhao

, Shi

, Qi

, Wang

, Jia

Pyramid scene parsing network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2881–2890.

Neural-logic multi-agent system for flood event detection

Abstract

Keywords

1 Introduction

2 Related works

3 Multi-Agent-System

3.1 DALI

3.2 FloodMAS architecture

4.1.1 PSP-Net experiment

Table 1 Floodnet data distribution Dataset Before After Train 1445 1361 Test 448 426 Validation 450 434 Tot. 2343 2221

7 Alert agent

8 Discussion and future works

9 Conclusion

Footnotes

Acknowledgments

References

Table 1
Floodnet data distribution

Dataset Before After

Train 1445 1361

Test 448 426

Validation 450 434

Tot. 2343 2221