Optimized instance segmentation by super-resolution and maximal clique generation

Abstract

The rise of surveillance systems has led to exponential growth in collected data, enabling several advances in Deep Learning to exploit them and automate tasks for autonomous systems. Vehicle detection is a crucial task in the fields of Intelligent Vehicle Systems and Intelligent Transport systems, making it possible to control traffic density or detect accidents and potential risks. This paper presents an optimal meta-method that can be applied to any instant segmentation model, such as Mask R-CNN or YOLACT++. Using the initial detections obtained by these models and super-resolution, an optimized re-inference is performed, allowing the detection of elements not identified a priori and improving the quality of the rest of the detections. The direct application of super-resolution is limited because instance segmentation models process images according to a fixed dimension. Therefore, in cases where the super-resolved images exceed this fixed size, the model will rescale them again, thus losing the desired effect. The advantages of this meta-method lie mainly in the fact that it is not required to modify the model architecture or re-train it. Regardless of the size of the images given as input, super-resolved areas that fit the defined dimension of the object segmentation model will be generated. After applying our proposal, experiments show an improvement of up to 8.1% for the YOLACT++ model used in the Jena sequence of the CityScapes dataset.

Keywords

Convolutional neural networks mask R-CNN YOLACT++instance segmentation super-resolution vehicle detection

1. Introduction

In recent years, instance segmentation has emerged as one of the most popular applications in deep learning and computer vision applications. This proliferation has been caused mainly by decreasing the costs of video surveillance systems and increasing the installation of these systems leading to an expansion in the generation and collection of data. Technological advances in the field of artificial intelligence and image recognition related to visual scene understanding have led to significant progress in research for several disciplines, such as the medical field or autonomous driving. Significantly, detection and surveillance systems have benefited from these advances.

The application of convolutional neural networks is widely extended for many problems and contexts, usually image-related. Some of the uses are medical issues detection [1, 2, 3], pattern recognition [4], and scene understanding [5]. In the field of instance segmentation, CNN application has allowed the identification of the objects captured in the scene. Using these models has allowed us to locate them with a mask containing the object together with his class. Compared to classical techniques, instance segmentation requires the perception of the elements appearing in the scene to associate each image pixel with the object to which it belongs.

One of the areas in which the detection of elements in video surveillance systems is relevant is the management of road networks. Due to the growing interest in vehicle automation, there is an increasing interest in automatic road understanding applications, such as hazard detection [6], vehicle communication [7], or controlling traffic flow and density [8, 9, 10]. This task is complex because it must deal with some challenges related to the scope of the scene. In the traffic control system, it is necessary to address several problems. Data challenges include the diversity of backgrounds determined by the video surveillance systems location, orientation, angle, and distance from the vehicles. In terms of detection, elements can appear in a wide variety of positions with diverse scale sizes. In addition, there are several cases of occlusion and motion blur. These problems make it challenging to apply instance segmentation models in this field. Beyond video surveillance on road networks, image segmentation is useful to identify anomalies or defects on surfaces [11, 12, 13] and understand complex scenes [14, 15].

Several instance segmentation models have proliferated based on convolutional neural networks (CNN). In image segmentation, two models prevail mainly: Mask R-CNN [16], and Yolact++ [17].

The architecture of Mask R-CNN [16] is grouped into two main stages. The first uses a Region Proposal Network (RPN) to identify candidate regions. Subsequently, it uses a convolutional neural network as a feature extractor named backbone. The bounding box, his class, and the binary mask for each region of interest (RoI) are finally predicted in the second phase. On the other hand, Yolact++ [17] is a fully-convolutional model for real-time instance segmentation. This model first sets up a series of tentative masks and then predicts each, avoiding repooling. While these models achieve good results in datasets such as COCO (Common Objects in Context) [18], they present deficiencies in datasets that contain elements of different sizes. One of them is Cityscapes [19], where the models achieve a low mean average precision (mAP).

A novel meta-method that improves the accuracy obtained using instance segmentation models according to the presented problems is developed. This work skillfully combines several concepts, approaches, techniques and components, such as Instance segmentation models (Mask R-CNN and YOLACT++), Super-Resolution and Vehicle detection. The segmentation masks generated from the initially detected elements are used as a starting point. After applying super-resolution to the initial image, it generates an optimal set of sub-images with the objective of re-inferring on them to increase the number of pixels of the elements captured by these systems. An optimization algorithm will reduce the number of sub-images processed by the model. The re-inference makes it possible to detect elements not identified a priori.

On the other hand, those elements detected multiple times are processed according to their binary mask, obtaining much more accurate detections and with a higher degree of confidence. Unlike other proposed approaches, our method does not require modifications to the models architecture to be used. Moreover, it also does not require retraining. This feature is significant and can be considered critical if we apply a detection system to road networks using the thousands of cameras that currently exist since it is not feasible to retrain the model for each camera manually. These improvements allow the model to detect a higher number of elements more quickly. We first demonstrate the effectiveness of our approach on different sequences of the Cityscapes dataset [19]. The mean average precision of the employed instance segmentation models has been improved. For example, going from a rate of 19.4% by applying Mask R-CNN, we obtain an increase in this rate, reaching up to 26.8%. Additionally, the total number of detected elements and the number of sub-images to be processed has also been determined.

The rest of this article is organized as follows. Related work is discussed in Section 2. Section 3 details the improvements developed and explains in more detail the workflow. Section 4 includes the applied instance segmentation models, the used dataset, the employed metrics, the defined hyperparameters, and the tests performed together with their results. In Section 5, conclusions and future work are presented.

2. Related work

Given the relevance of instance segmentation, it has been an ongoing field of work since good results have been achieved on simpler problems such as image classification and object detection. The problem is commonly approached as the natural evolution of object detection, as seen in the proposal [20] based on the novel R-CNN model [21]. Since then, many methods for segmenting object instances have been proposed, such as the three-phase in cascade (from bounding box to mask and then to instances) proposal of [22]. Another is the famous Mask R-CNN [16] that extends the Faster R-CNN detection model [23] by adding a parallel branch to the bounding box extraction for the prediction of the mask of each object. There are also proposals such as [24], that base their segmentations on the shape of the objects to reduce the dependency on the quality of the bounding box to avoid failures caused by erroneous proposals inherited from object detection.

The close relationship of this problem with object detection means their solutions face similar weaknesses. To date, the general approach to solving the problem is to use supervised learning and therefore requires vast amounts of data to train neural models. This data requires pixel-level image annotation, which makes it exceptionally costly in person-hours to obtain very specific datasets such as [25]. As in other areas, alternative solutions have been searched to increase the data on which to train automatically. One example is the generation of artificial data based on cutting and pasting instances of objects in other images [26], which in this problem seems to work particularly well. Different approaches train using more easily available data, such as [27], which trains from image-level labeled data by obtaining pseudo masks before training a Mask-RCNN, or [28], which requires annotating a single pixel for each object in the image.

To improve the results of instance segmentation, some propose using other image recording technologies, such as [29], which takes advantage of information obtained from stereo cameras to perform 3D segmentation to help separate instances. However, a more popular scheme is integrating a segmentation model into a meta-algorithm to improve output quality. [30] applies polymorphic analysis to adapt a previously obtained segmentation mask to fit objects shapes better. [31] represents image regions as a quadtree to identify and correct inconsistencies between the proposed mask and the input image. There are proposals based on image scaling techniques, such as [32]. Slicing Aided Hyper Inference (SAHI) is a computer vision library for performing large-scale object detection and instance segmentation. It is based on a generic slicing-aided inference and fine-tuning pipeline. According to a dimension selected by the user, it generates windows on which to re-infer before upscaling previously.

This proposal is framed within these methods that use an existing model (Mask R-CNN and Yolact++) integrated into a meta-method. The objective is to improve instance segmentation models when dealing with objects of different sizes. Several recent proposals work on this line. [33] improves Mask R-CNN by using a multi-scale Region Proposal network structure. [34] studies the relation between the receptive field’s spatial information and the object’s size with the segmentation and proposes MR R-CNN, a network with convolutional layers hanged to the proposed semantic segmentation layer containing features pyramids. [35] proposes SCMask R-CNN, a Mask R-CNN improvement to enhance detection in high-resolution images with dense targets and complex backgrounds by modifying the ResNet101 [36] backbone to obtain more discriminative information and adding dilated convolutions. Our proposal differs from these approaches on one fundamental point: instead of proposing a new network structure to address the problems, we propose a meta-algorithm that can be used with any instance segmentation model. This is an advantage, as it will allow the proposal to improve the results as new and better instance segmentation models emerge. On the other hand, our proposal will also benefit from faster novel network approaches.

One of the critical elements in our proposal is the use of a super-resolution method, for which there is a wide range of options, such as DRCN [37], VSDR [38], or FSRCNN [39]. We have chosen to use FSRCNN due to the widespread use of the method and the good results obtained. This model extends to speed up SRCNN [40], for which the initial interpolation is removed. New convolutional layers are added with a smaller filter size. Additionally, a new deconvolutional layer is introduced at the end.

3. Methodology

Figure 1.

Architecture of the proposed system.

The proposed system for instance segmentation enhancement is composed of several modules. The system architecture is shown in Fig. 1. Next, the modules are described.

3.1 Initial module

A deep learning neural network $\mathcal{F}$ for object segmentation is considered that processes an input image $\mathbf{X}$ so that a set of detections $S$ is returned:

$\displaystyle S=\mathcal{F}\left(\mathbf{X}\right)$ (1) $\displaystyle S=\left\{(a_{i},b_{i},c_{i},d_{i},q_{i},r_{i})\mid i\in\{1,% \ldots,M\}\right\}$ (2)

where $M$ denotes the number of detections, $(a_{i},b_{i})\in\mathbb{\mathbb{R}}^{2}$ represents the coordinates of the top left corner of the $i$ -th detection within the input $\mathbf{X}$ , $(c_{i},d_{i})\in\mathbb{R}^{2}$ are the coordinates of the lower right corner of the $i$ -th detection within $\mathbf{X}$ , $q_{i}$ is the class label of the detection, and $r_{i}\in\mathbb{R}$ is the class score of the detection. It must be noted that the confidence that an object of class $q_{i}$ has been detected is measured by $r_{i}$ . We assume that the coordinate origin is placed at the center of the image.

The first step of our procedure (stage 1 in Fig. 1) consists in applying a super-resolution network $\mathcal{G}$ to the original low-resolution image $\mathbf{X}_{\textit{LR}}$ to obtain a high-resolution version $\mathbf{X}_{\textit{HR}}$ :

$\displaystyle\mathbf{X}_{\textit{HR}}=\mathcal{G}\left(\mathbf{X}_{\textit{LR}% }\right)$ (3)

The second step (stage 2 in Fig. 1) is to process the input image $\mathbf{X}_{\textit{LR}}$ , which has a low resolution, with the object segmentation network $\mathcal{F}$ to yield a set of potential detections $S_{\textit{LR}}$ :

$\displaystyle S_{\textit{LR}}=\mathcal{F}\left(\mathbf{X}_{\textit{LR}}\right)$ (4)

3.2 Graph calculation module

For each potential detection $s_{i}\in S_{\textit{LR}}$ , a window $\mathbf{X}_{i}$ with the same size as $\mathbf{X}_{\textit{LR}}$ is obtained from $\mathbf{X}_{\textit{HR}}$ (stage 3 in Fig. 1). The window $\mathbf{X}_{i}$ is centered at the center of the detection:

$\displaystyle\mathbf{y}_{i}=\left(\frac{a_{i}+c_{i}}{2},\frac{b_{i}+d_{i}}{2}\right)$ (5) $\displaystyle\hat{\mathbf{y}}_{i}=Z\mathbf{y}_{i}$ (6)

where $Z$ is the zoom factor of the super-resolution network $\mathcal{G}$ , $\mathbf{y}_{i}$ is the center of $\mathbf{X}_{i}$ expressed in coordinates of $\mathbf{X}_{\textit{LR}}$ , while $\hat{\mathbf{y}}_{i}$ is the center of $\mathbf{X}_{i}$ expressed in coordinates of $\mathbf{X}_{\textit{HR}}$ .

After that, a graph $G=\left(S_{\textit{LR}},E\right)$ is built whose set of nodes is the set of potential detections $S_{\textit{LR}}$ , and the set of edges $E$ contains the edge $\left(s_{i},s_{j}\right)\in S_{\textit{LR}}\times S_{\textit{LR}}$ if and only if the bounding box of $s_{j}$ is completely contained within the window $\mathbf{X}_{i}$ associated to detection $s_{i}$ (stages 4 and 5 in Fig. 1). This graph expresses which pairs of detections could be jointly managed by running the image segmentation network $\mathcal{F}$ on the same window of the high-resolution image.

3.3 Optimization module

Next, it is found which sets of detections could be managed with a single run of the image segmentation network $\mathcal{F}$ . These sets are cliques of $G$ , i.e. sets of nodes of $G$ which are fully connected among them. The cliques are found by executing the Bron–Kerbosch algorithm [41] on $G$ (stages 6 and 7 in Fig. 1). Then the following greedy algorithm (stage 8 in Fig. 1) is employed to select a reduced set of cliques $Q$ within $G$ :

1.
The cliques are sorted in decreasing number of nodes to yield the list $L$ .
2.
A counter is initialized to zero for each node to keep track of how many times a node appears in the cliques of $Q$ .
3.
The reduced set of cliques $Q$ is initialized to the empty set.
4.
The first clique of $L$ is added to the reduced set $Q$ . Then the counters of all the nodes of the clique are increased by one. Finally, the clique is removed from $L$ .
5.
All nodes whose counters are higher than 3 are removed from the cliques in $L$ .
6.
All cliques of $L$ that are subsets of other cliques of $L$ with more nodes are removed from $L$ .
7.
If $L$ is empty then halt and output $Q$ . Otherwise, go to step 4.

3.4 Sub-image generation module

For each clique $q_{k}\in Q$ , its center $\mathbf{q}_{k}\in\mathbb{R}^{2}$ is computed as the average of the centers of the windows associated with its nodes (stage 9 in Fig. 1). Furthermore, a window $\mathbf{X}_{k}$ is associated to $q_{k}$ around its center $\mathbf{q}_{k}$ . We also generate five fixed regions to ensure that our proposal is applied over all the image in case the model does not obtain any initial detections (stage 10 in Fig. 1). The windows associated with the cliques and the five fixed regions form the set of sub-images $\mathbf{X}_{k}$ to be further considered (stage 11 in Fig. 1).

3.5 Segmentation refinement module

Next, the image segmentation network $\mathcal{F}$ is run on each $\mathbf{X}_{k}$ in order to obtain a new list of detections (stage 12 in Fig. 1):

$\displaystyle\tilde{S}_{k}=\mathcal{F}\left(\mathbf{X}_{k}\right)$ (7) $\displaystyle\tilde{S}_{k}=\Bigl{\{}(\tilde{a}_{k,l},\tilde{b}_{k,l},\tilde{c}% _{k,l},\tilde{d}_{k,l},\tilde{q}_{k,l},\tilde{r}_{k,l})\mid\,l\in\{1,\ldots,M_% {k}\}\Bigr{\}}$ (8)

where $M_{k}$ is the number of detections for sub-image $\mathbf{X}_{k}$ .

The object detections of $\tilde{S}_{k}$ are computed in coordinates of $\mathbf{X}_{k}$ . Therefore, they must be translated into coordinates of $\mathbf{X}_{\textit{LR}}$ (stage 13 in Fig. 1). The equation to convert a point $\tilde{\mathbf{h}}$ expressed in coordinates of $\mathbf{X}_{k}$ to coordinates $\mathbf{h}$ of $\mathbf{X}_{\textit{LR}}$ is as follows:

$\displaystyle\mathbf{h}=\mathbf{y}_{k}+\frac{1}{Z}\tilde{\mathbf{h}}$ (9)

where $\mathbf{y}_{k}$ is the center of window $\mathbf{X}_{k}$ expressed in coordinates of $\mathbf{X}_{\textit{LR}}$ .

Consequently, the set of object detections for window $\mathbf{X}_{k}$ expressed in coordinates of $\mathbf{X}_{\textit{LR}}$ is:

$\displaystyle S_{k}=\bigl{\{}(a_{k,l},b_{k,l},c_{k,l},d_{k,l},q_{k,l},r_{k,l})\mid$ (10) $\displaystyle\quad l\in\{1,\ldots,M_{k}\}\bigr{\}}$ $\displaystyle(a_{k,l},b_{k,l})=\mathbf{y}_{k}+\frac{1}{Z}\left(\tilde{a}_{k,l}% ,\tilde{b}_{k,l}\right)$ (11) $\displaystyle(c_{k,l},d_{k,l})=\mathbf{y}_{k}+\frac{1}{Z}\left(\tilde{c}_{k,l}% ,\tilde{d}_{k,l}\right)$ (12) $\displaystyle q_{k,l}=\tilde{q}_{k,l}$ (13) $\displaystyle r_{k,l}=\tilde{r}_{k,l}$ (14)

The final step of our procedure (stage 14 in Fig. 1) is detailed next. A cluster $C$ is computed for each object by grouping the detections coming from the windows such that the Intersection measure computed on their associated segmentation masks is higher than 50%. After that, a unified mask is computed for each remaining cluster $C$ by computing a pixelwise weighted sum of the masks associated with the detections, where the pixels of the low-resolution image are weighted four times more than the pixels of the detections of the windows of the high-resolution image. This alleviates the effect of partial objects being detected in the windows. A pixel is marked as true in the unified mask whenever its weighted sum is greater than 20% of the maximum possible value. Then a unified bounding box is calculated from the unified mask. Finally, a unified class label is obtained by majority voting among the class labels in $C$ , and a unified class score is computed as the maximum of the class scores in $C$ .

4. Experimental results

We tested the proposed meta-method on images for instance segmentation, which is challenging according to the size range of the elements. The following subsections present the selected instance segmentation models and the super-resolution model, the experiments, the dataset, evaluation metrics, the experimental setup, and results.

4.1 Instance segmentation models

The proposed meta-algorithm aims to improve the segmentation of the elements detected by re-inference on super-resolved sub-images. One of these meta-method advantages is that it does not modify the selected instance segmentation model. On the other hand, it does not require re-training to improve the mean average precision (mAP). Two of the most commonly used models have been selected to determine the improvement obtained after applying the presented proposal.

Mask R-CNN[16]: It is an extension of Faster R-CNN [23]. This model has been selected from the Tensorflow Model Zoo repository.1

Yolact++[17]: A fully-convolutional model for real-time instance segmentation. It has several improvements over its predecessor YOLACT [42], such as incorporating deformable convolutions into the backbone network.

Both models have been trained with the COCO dataset (Common Objects in Context) [18]. This dataset is composed of a large variety of scenes. The advantage of this dataset lies mainly in the number of different classes labeled, with up to 91 different classes with diverse scales. Besides the two segmentation models selected previously, our proposal can be applied using any deep convolutional neural network instance-based segmentation model.

4.2 Super-resolution model

The Fast Super-Resolution Convolutional Neural Network (FSRCNN) is a super-resolution model that has been designed to be fast and efficient, with a processing time that is significantly faster than many other models. This makes it an excellent choice for real-time applications or for use on devices with limited processing capabilities. In addition to its speed, FSRCNN is also known for its high reconstruction accuracy, consistently producing high-quality images with good visual quality and detailed structures. It is relatively simple to implement and does not require many parameters or computational resources, making it easy to deploy in various settings. FSRCNN is also highly flexible and can enhance a wide range of image types, including natural, text, and medical images. Lastly, this model is robust and able to handle noisy or low-quality images effectively, making it a reliable choice for use in challenging situations.

Figure 2.

Differences between upscaling and the application of Super-Resolution with a scaling factor of x2 on a vehicle extracted from the CityScapes – Zurich sequence.

Figure 2 shows the differences between direct upscaling and super-resolution applications with a scaling factor of x2 according to a vehicle detected in the frame. Comparing the two extracted areas, the super-resolved one features a clear enhancement in terms of the quality and sharpness of the image.

4.3 Experiments

A comparison has been made among the following methods:

•
Original Model (RAW): The direct application of the unmodified raw instance segmentation model.
•
Super-Resolution Only (SR Only): The image is processed by applying super-resolution and is given directly as input to the instance segmentation model.
•
Slicing Aided Hyper Inference (SAHI)[32]: Re-inference from rescaled zones using Mask R-CNN R-50-FPN model.
•
Our Proposal: The presented proposal aims to generate new sub-images by applying super-resolution optimally based on a parameter denoted as window size $R$ previously defined. Thus, it is possible to improve the details that conform to the elements to subsequently re-infer these more detailed regions. See Fig. 1 for more details.

The Super-Resolution Only proposal gives as input the image after applying super-resolution to the object segmentation model. However, according to the dimensions of these images, this will not significantly impact the accuracy in some instances. The object segmentation models set a fixed input dimension. If the processed images exceed this size, the model will downscale them again. On the other hand, SAHI [32] generates several regions based on the previously defined dimensions, and then, the model performs an upscaling based on his input size dimension, which may cause a worse result in cases where the images have low resolution.
4.4 Dataset

The Cityscapes dataset [19] has been used to evaluate the presented proposal. This dataset is conformed by complex real-world urban scenes, an enabling factor for many applications. It is composed of 5000 2048 $\times$ 1024 annotated images. 2975 are for training, while 500 and 1525 are designated for testing and validation.

This dataset highlights the great variety of scenes that it contains. The images are divided according to the place where they were collected. The sequences used in this section were Zurich, Ulm, Munster, Monchengladbach, Lindau, Krefeld, Jena, Erfurt, Darmstadt, Bochum, and Frankfurt. Due to the great amounts of different objects and distances, this dataset is a challenge for typical CNN-based instance segmentation approaches. They tend to miss small objects, blend them or change their class due to the small number of pixels they represent in the whole image.

4.5 Evaluation metrics

Table 1
Selected values of the hyperparameters

Parameter	Value
Maximum number of detections per frame	300
Minimum score	0.5
Window size – $R$	0 to 100%
Number of coincidences P (greedy algorithm)	3
Evaluated classes	Person, bicycle,
	car, motorcycle
	and truck

Table 2

Mean average precision obtained for the Ulm sequence of the CityScapes Dataset according to different window sizes $R$ using the Mask R-CNN model. A detection is valid when its confidence exceeds the set threshold of 70%

	Cityscapes Dataset – Sequence Ulm – Mask R-CNN – Confidence $>$ 0.70
	Proposal	IoU $=$ 0.5:	IoU $>$	IoU $>$	IoU $=$ 0.5:	IoU $=$ 0.5:	IoU $=$ 0.5:	Sub-images
		0.95–All	0.5–All	0.75–All	0.95–Small	0.95–Medium	0.95–Large	processed per frame
Other methods	RAW	0.194	0.341	0.191	0.017	0.181	0.455	–
	SR ONLY	0.196	0.345	0.195	0.015	0.179	0.459	–
	SAHI [32] slice_size	0.242	0.466	0.221	0.105	0.268	0.432	25.000 $\pm$ 0.000
	$=$ 256 $\times$ 512
	overlap $=$ 0.2
	SAHI [32] slice_size	0.230	0.432	0.208	0.084	0.235	0.449	9.000 $\pm$ 0.000
	$=$ 512 $\times$ 1024
	overlap $=$ 0.2
	SAHI [32] slice_size	0.231	0.409	0.219	0.065	0.237	0.473	4.000 $\pm$ 0.000
	$=$ 768 $\times$ 1536
	overlap $=$ 0.2
Our proposal	$R=$ 0%	0.262	0.467	0.256	0.071	0.280	0.492	15.389 $\pm$ 6.256
with different	$R=$ 10%	0.261	0.470	0.248	0.058	0.289	0.492	12.958 $\pm$ 4.593
window-sizes	$R=$ 20%	0.265	0.484	0.247	0.082	0.285	0.493	11.116 $\pm$ 3.448
$R$	$R=$ 30%	0.268	0.479	0.253	0.068	0.293	0.495	10.189 $\pm$ 3.041
	$R=$ 40%	0.254	0.452	0.240	0.058	0.286	0.476	9.505 $\pm$ 2.566
	$R=$ 50%	0.256	0.465	0.240	0.057	0.280	0.492	8.979 $\pm$ 2.234
	$R=$ 60%	0.260	0.468	0.246	0.068	0.283	0.477	8.474 $\pm$ 2.004
	$R=$ 70%	0.258	0.465	0.246	0.068	0.281	0.475	8.189 $\pm$ 1.937
	$R=$ 80%	0.257	0.464	0.244	0.058	0.289	0.478	7.832 $\pm$ 1.626
	$R=$ 90%	0.255	0.463	0.236	0.056	0.273	0.479	7.568 $\pm$ 1.526
	$R=$ 100%	0.256	0.449	0.251	0.065	0.270	0.482	7.316 $\pm$ 1.355

A series of criteria have been selected to determine the improvement obtained after applying the presented proposal.

First, the mean average precision (mAP) obtained by the COCO evaluator is determined.2 The average accuracy is calculated over multiple IoU (Intersection over Union), which range from 0.5 to 0.95 and a step size of 0.05. The IoU selected is equivalent to the minimum area determined based on the annotation set as Ground truth (GT) and the one obtained by the instance segmentation model to define a match as positive. The mAP is also included as a function of the scale of the objects to determine the enhancement index concerning the image size that provides for it. The sizes considered are as follows:

Average Precision Small: For small objects (Area less than 32 ${}^{2}$ pixels).

Average Precision Medium: For medium objects (Area between 32 ${}^{2}$ pixels and 96 ${}^{2}$ pixels).

Average Precision Large: For large objects (Area greater than 96 ${}^{2}$ pixels).

Additionally, the total number of detected elements in the selected sequence, the average elements detected per frame, and the number of sub-images to be re-inferred for each sequence are collected to determine the quantitative improvement obtained.

4.6 Experimental setups

During the experimental testing phase, we selected a set of hyper-parameters specified in Table 1. Eleven different $R$ window sizes have been contemplated to determine the accuracy rate obtained and the average number of images on which the model must re-infer. The value of the minimum number of coincidences parameter $P$ has been set to 3. This variable represents the maximum limit over which the model will infer an image that contains the element. The selected classes to perform the evaluation have been defined according to the predominance of the elements contained in the sequences of the CityScapes dataset.

Table 3
Mean average precision obtained for the Jena sequence of the CityScapes Dataset according to different window sizes $R$ using the YOLACT++ model. A detection is valid when its confidence exceeds the set threshold of 50%

	Cityscapes Dataset – Sequence Jena – YOLACT++ – Confidence $>$ 0.50
	Proposal	IoU $=$ 0.5:	IoU $>$	IoU $>$	IoU $=$ 0.5:	IoU $=$ 0.5:	IoU $=$ 0.5:
		0.95–All	0.5–All	0.75–All	0.95–Small	0.95–Medium	0.95–Large
Other methods	RAW	0.106	0.179	0.098	0.000	0.054	0.353
	SR ONLY	0.104	0.179	0.097	0.000	0.053	0.347
Our proposal	$R=$ 0%	0.183	0.311	0.169	0.011	0.162	0.467
with different	$R=$ 10%	0.183	0.311	0.170	0.011	0.161	0.467
window-sizes	$R=$ 20%	0.179	0.300	0.174	0.012	0.154	0.470
$R$	$R=$ 30%	0.183	0.312	0.172	0.011	0.163	0.469
	$R=$ 40%	0.176	0.296	0.166	0.010	0.148	0.471
	$R=$ 50%	0.174	0.286	0.169	0.011	0.141	0.472
	$R=$ 60%	0.187	0.297	0.195	0.011	0.140	0.525
	$R=$ 70%	0.186	0.295	0.196	0.011	0.140	0.522
	$R=$ 80%	0.184	0.292	0.191	0.010	0.136	0.519
	$R=$ 90%	0.184	0.297	0.191	0.010	0.135	0.522
	$R=$ 100%	0.183	0.297	0.188	0.010	0.135	0.521

Table 4

Total number of elements detected in the sequence, average and standard deviation of the number of detected elements, and average and standard deviation of the number of sub-images processed per frame for different values of the windows size $R$ . Our proposal is compared with the RAW Mask R-CNN model

	Cityscapes Dataset – Sequence Munster – Mask R-CNN – Confidence $>$ 0.70
	Proposal	Total elements detected	Elements detected per frame	Sub-images processed per frame
Other methods	RAW	1640	9.425 $\pm$ 3.711	–
	SR ONLY	1640	9.425 $\pm$ 3.711	–
	SAHI [32] slice_size $=$ 256 $\times$ 512 overlap $=$ 0.2	3214	18.471 $\pm$ 7.869	25.000 $\pm$ 0.000
	SAHI [32] slice_size $=$ 512 $\times$ 1024 overlap $=$ 0.2	2841	16.328 $\pm$ 6.425	9.000 $\pm$ 0.000
	SAHI [32] slice_size $=$ 768 $\times$ 1536 overlap $=$ 0.2	2247	12.914 $\pm$ 4.943	4.000 $\pm$ 0.000
Our proposal	$R=$ 0%	2794	16.057 $\pm$ 6.497	22.977 $\pm$ 7.962
with different	$R=$ 10%	2774	15.943 $\pm$ 6.473	18.402 $\pm$ 6.022
window-size	$R=$ 20%	2734	15.713 $\pm$ 6.280	14.690 $\pm$ 4.327
$R$	$R=$ 30%	2716	15.609 $\pm$ 6.195	13.086 $\pm$ 3.409
	$R=$ 40%	2707	15.557 $\pm$ 6.242	11.839 $\pm$ 2.840
	$R=$ 50%	2682	15.414 $\pm$ 6.066	10.954 $\pm$ 2.286
	$R=$ 60%	2683	15.420 $\pm$ 6.109	10.259 $\pm$ 2.030
	$R=$ 70%	2665	15.316 $\pm$ 6.088	9.661 $\pm$ 1.830
	$R=$ 80%	2657	15.270 $\pm$ 6.014	9.080 $\pm$ 1.544
	$R=$ 90%	2677	15.385 $\pm$ 6.214	8.684 $\pm$ 1.438
	$R=$ 100%	2665	15.316 $\pm$ 6.113	8.466 $\pm$ 1.417

4.7 Results and analysis

In this study, the proposed methodology is evaluated to determine the improvement in the accuracy rate obtained by the selected models. The presented proposal aims to reduce the number of sub-images the model needs to re-infer to improve its accuracy. Therefore, it is necessary to previously determine the factor called window size, denoted as $R$ . To define each node’s connections, it is required to identify which elements are considered neighbors. From the center of an element, a window of size $R$ will be generated in the super-resolved image. All the detections contained in this window will be considered neighbors. Due to the scope of the scene, it will be required to determine this value previously. Tables 2 and 3 show the mean average precision (mAP) obtained by the COCO evaluator for the Mask R-CNN and Yolact++ models respectively for different sequences.

Regardless of the model on which the presented proposal has been applied, the mAP value obtained will increase compared to the direct application of the model. Referring to Table 2, we can see that for the general mAP, corresponding to the first column of the table, the model reaches 19.4%. On the other hand, applying our proposal in the less strict option, defining a window size of 30%, an increase of 7.4% is achieved, reaching a total accuracy of 26.8%. Additionally, we present the competitor, SAHI [32], with different window sizes. In the least restrictive case, in which SAHI generates 25 zones on which to re-infer, our model achieves, even in the most restrictive case ( $R=$ 100%), a higher accuracy re-inferring on a smaller number of images. The Yolact++ model is worth mentioning. Table 3 shows how the model obtains an overall mAP of 10.6%. However, applying our proposal with a window size of 60%, the model gets the best results. Therefore, we can see how the model can achieve higher accuracy by optimizing the total number of sub-images it has to re-infer.

Table 5
Total number of elements detected in the sequence, average and standard deviation of the number of detected elements, and average and standard deviation of the number of sub-images processed per frame for different values of the windows size $R$ . Our proposal is compared with the YOLACT++ model

	Cityscapes Dataset – Sequence Bochum – YOLACT++ – Confidence $>$ 0.50
	Proposal	Total elements detected	Elements detected per frame	Sub-images processed per frame
Other methods	RAW	354	3.766 $\pm$ 1.316	–
	SR ONLY	354	3.766 $\pm$ 1.292	–
Our proposal	$R=$ 0%	654	6.957 $\pm$ 2.832	9.448 $\pm$ 1.029
with different	$R=$ 10%	653	6.947 $\pm$ 2.819	9.229 $\pm$ 1.046
window-sizes	$R=$ 20%	647	6.883 $\pm$ 2.794	8.562 $\pm$ 1.019
$R$	$R=$ 30%	646	6.872 $\pm$ 2.757	8.062 $\pm$ 0.933
	$R=$ 40%	643	6.84 $\pm$ 2.792	7.646 $\pm$ 0.913
	$R=$ 50%	644	6.851 $\pm$ 2.828	7.406 $\pm$ 0.824
	$R=$ 60%	636	6.766 $\pm$ 2.742	7.177 $\pm$ 0.777
	$R=$ 70%	633	6.734 $\pm$ 2.741	7.000 $\pm$ 0.736
	$R=$ 80%	630	6.702 $\pm$ 2.697	6.781 $\pm$ 0.71
	$R=$ 90%	629	6.691 $\pm$ 2.694	6.604 $\pm$ 0.669
	$R=$ 100%	629	6.691 $\pm$ 2.682	6.552 $\pm$ 0.643

Figure 3.

Balance between precision and computational complexity. Average Sub-Images Processed are presented on the X-axis. The Y-axis represents the Mean Average Precision (mAP). Both axis coordinates have been calculated using the mean of 11 different sequences of the CityScapes Dataset (Zurich, Ulm, Munster, Monchengladbach, Lindau, Krefeld, Jena, Erfurt, Darmstadt, Bochum, and Frankfurt) using the Raw model and our proposal with different Windows Sizes $R$ .

Figure 4.

Balance between precision and number of sub-images generated on which to re-infer for the Zurich sequence. Total Sub-Images Processed are presented on the left axis (bars). The right axis (diamond markers) represents the Mean Average Precision (mAP). The X-axis represents the applied methodology (Our proposal vs SAHI). Our proposal and SAHI are represented with different window size configurations. In our case, through the percentage, and for SAHI, through its width and length. The model used was Mask R-CNN.

Figure 5.

An example applied to frame 000084_000019_leftImg8bit of the sequence is denoted as Darmstadt. The left side shows the results obtained by the raw model, while the right side shows the detections after applying our proposal with a window size of 0% using the Mask R-CNN model.

Figure 6.

An example applied to frame 000000_002083_leftImg8bit of the sequence is denoted as Krefeld. The left side shows the results obtained by the raw model, while the right side shows the detections after applying our proposal with a window size of 0% using the Yolact++ model.

Figure 7.

An example applied to frame 000001_002353_leftImg8bit of the sequence is denoted as Monchengladbach. The left side shows the results obtained by the raw model, while the right side shows the detections after applying our proposal with a window size of 50% using the Yolact++ model.

According to $R$ , the number of sub-images generated will be reduced, being inversely proportional to the increment of this value. Tables 4 and 5 show how the total sub-images are affected according to $R$ . Additionally, the total number of elements identified in the sequence, the average number of elements detected per frame, and their standard deviation are also included.

For the Mask R-CNN model, Table 4 shows the reduction of the required computing according to the selected model, going from the least restrictive case, with a total of approximately 23 sub-images per frame to 9 with $R=$ 100%. For Yolac++, we can see in Table 5 that the decrease is less notorious. This fact is due to the number of initial elements detected by the model. Since Yolact++ presents a lower performance in the CityScapes dataset, the optimization proposal will be more adjusted.

Figure 3 shows the balance between accuracy and computational complexity based on the number of total sub-images to be re-inferred. The X-axis represents the average number of sub-images processed per frame, while the Y-axis represents the mean average precision. The direct application of the model represents the worst results, obtaining the lowest mAP. We set the best solutions to proposals close to the upper left corner since this will mean that it has obtained the highest mAP by processing the least number of sub-images. According to the results, after applying the presented proposal, it is possible to considerably reduce the number of sub-images to re-infer without significantly sacrificing the mean average precision obtained.

Figure 4 shows how increasing the window size $R$ in our proposed methodology decreases the number of sub-images on which to re-infer. Considering the mAP obtained with $R$ set to 60%, it is possible to get an accuracy similar to the one obtained with $R$ at 10%, re-inferring on 712 fewer images according to the selected sequence. Besides, we can appreciate how in the more restrictive case where $R$ is 100%, the mAP is higher than SAHI, re-inferring on a total of 2087 fewer images.

To visually show the improvement obtained after the application of the presented proposal, a series of qualitative results are included, represented by Figs 5, 6, and 7. We can affirm that the number of detected elements is higher than the direct application of the model.

Therefore, according to the obtained results, our proposal allows obtaining a Pareto front in which, by altering the $R$ value, we can obtain better results at the cost of applying the base model more times and consequently increasing the execution time. Although the proposal is a meta-algorithm that could easily be applied to any instance segmentation model regardless of its inner workings and would, therefore, directly benefit from any improvements in the field, the computational complexity increases due to the number of times the detection model is applied. This would typically imply an increase in execution times. This increase could be reduced by appropriate parallelization if the computer has enough GPUs to run the base model, but we are aware that this is not generally the case. Despite the existence of this trade-off, the results show that, although the increase in performance is substantial to the base model, increasing the number of sub-images to the maximum ( $R=$ 0%) does not provide a critical performance improvement concerning a minimum number ( $R=$ 100%) or an intermediate value ( $R=$ 50% or $R=$ 60%) so the proposal obtains a desirable performance improvement even with the minimum number of sub-images (and thus a minimum increase in computational complexity).

5. Conclusions

This paper presents an optimized meta-algorithm to enhance the performance of different image segmentation models. The proposal involves three key elements. The first one consists of performing super-resolution of the input video using a suitable super-resolution neural network. As a result, the level of detail is enhanced, increasing the number of pixels for objects in the scene to facilitate subsequent object segmentation. Secondly, the original low-resolution video is processed by an image segmentation model to produce a set of potential detections. Based on these detections, a graph will be calculated where each node represents an element. Its connections will be defined according to the distance between the two elements. As a third step, a heuristic is applied to determine the optimal number of sub-images on which the model should be re-inferred, according to a previously established window size. The image segmentation network gets new detections for each optimized window. Finally, the detections for the same object are clustered together, and a unified segmentation mask is then calculated from them.

Experimental results show that our approach significantly outperforms the base image segmentation model’s precision. Specifically, our proposal detects more objects with a higher mean Average Precision in a wide variety of real-world sequences of the CityScapes dataset without modifying the layers that compose the original model or re-train it. Models such as YOLACT++ obtain a maximum gain of up to 8.1%, increasing the average number of elements detected per frame and reducing the number of sub-images where the model has to re-infer.

In future work, we plan to compare the accuracy obtained by instance segmentation models applying different algorithms and super-resolution models such as GANs or fuzzy models to determine if there is a significant improvement by using other techniques. In addition, the proposed meta-method could enhance the performance of other deep learning models recently applied to generate novel perceptual metrics [43], generate composite images [44], identify dementia using magnetic resonance images of brain asymmetries [45] based on the re-inference of several super-resolved zones by the instance segmentation model, or even for short-term scheduling road works [46].

Footnotes

https://github.com/tensorflow/models/blob/master/research/object _detection/g3doc/tf2_detection_zoo.md.

https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/ pycocotools/cocoeval.py.

Acknowledgments

This work is partially supported by the Ministry of Science, Innovation and Universities of Spain [grant number RTI2018-094645-B-I00], project name Automated detection with low-cost hardware of unusual activities in video sequences. It is partially supported by the Autonomous Government of Andalusia (Spain) under project UMA18-FEDERJA-084, project name Detection of anomalous behavior agents by deep learning in low-cost video surveillance intelligent systems. All of them include funds from the European Regional Development Fund (ERDF). It is also partially supported by the University of Málaga (Spain) under grants B1-2019_01, project name Anomaly detection on roads by moving cameras, and B1-2019_02, project name Self-Organizing Neural Systems for Non-Stationary Environments. The authors thankfully acknowledge the computer resources, technical expertise, and assistance provided by the SCBI (Supercomputing and Bioinformatics) center of the University of Málaga. The authors acknowledge the funding from the Universidad de Málaga. Iván García-Aguilar is funded by a scholarship from the Autonomous Government of Andalusia (Spain) under the Young Employment operative program [grant number SNGJ5Y6-15].

References

Acharya

Hagiwara

Tan

Adeli

Subha

. Automated EEG-based screening of depression using deep convolutional neural network. Computer Methods and Programs in Biomedicine. 2018 Jul; 161: 103-113.

Nogay

Adeli

. Detection of Epileptic Seizure Using Pretrained Deep Convolutional Neural Network and Transfer Learning. European Neurology. 2020; 83: 602-614.

Mirzaei

Adeli

. Segmentation and clustering in brain MRI imaging. Reviews in the Neurosciences. 2018 Dec; 30(1): 31-44. doi: 10.1515/revneuro-2018-0050.

Naranjo

Santos

Garmendia

. A convolution-based distance measure for fuzzy singletons and its application in a pattern recognition problem. Integrated Computer-Aided Engineering. 2020 Dec; 28: 51-63.

Jung

Jeoung

Kang

Hong

. 3D convolutional neural network-based one-stage model for real-time action detection in video of construction equipment. Computer-Aided Civil and Infrastructure Engineering. 2022 Jan; 37: 126-142.

García-González

García-Aguilar

Medina

Luque-Baena

López-Rubio

Domínguez

. Vehicle Overtaking Hazard Detection over Onboard Cameras Using Deep Convolutional Networks. 2023.

Shi

Zhou

Wang

Gong

Ran

. A deep reinforcement learning-based distributed connected automated vehicle control under communication failure. Computer-Aided Civil and Infrastructure Engineering. 2022 Dec; 37: 2033-2051.

Gouda

Mirza

WeiÃŸ

Castro

El-Basyouny

. Octree-based point cloud simulation to assess the readiness of highway infrastructure for autonomous vehicles. Computer-Aided Civil and Infrastructure Engineering. 2021 Jul; 36: 922-940.

Wang

Hou

Wang

. Reinforcement learning-based bird-view automated vehicle control to avoid crossing traffic. Computer-Aided Civil and Infrastructure Engineering. 2021 Jul; 36: 890-901.

10.

Tang

Cao

Chen

Yao

Tan

Sun

. Dynamic origin-destination flow estimation using automatic vehicle identification data: A 3D convolutional neural network approach. Computer-Aided Civil and Infrastructure Engineering. 2021 Jan; 36: 30-46.

11.

Lin

Yao

. A spectrum-domain instance segmentation model for casting defects. Integrated Computer-Aided Engineering. 2021 12; 29: 63-82.

12.

Qin

Qian

Guo

Wang

Jia

. Hybrid deep learning architecture for rail surface segmentation and surface defect detection. Computer-Aided Civil and Infrastructure Engineering. 2022 Feb; 37: 227-244.

13.

Wang

Guo

. Multi-defect segmentation from faÃ§ade images using balanced copyâ€“paste method. Computer-Aided Civil and Infrastructure Engineering. 2022 Sep; 37: 1434-1449.

14.

Wang

Zhang

Mosalam

Gao

Huang

. Deep semantic segmentation for visual understanding on construction sites. Computer-Aided Civil and Infrastructure Engineering. 2022 Feb; 37: 145-162.

15.

Shen

Yan

Xiong

. Deep learning-based object identification with instance segmentation and pseudo-LiDAR point cloud for work zone safety. Computer-Aided Civil and Infrastructure Engineering. 2021 Dec; 36: 1549-1567.

16.

Gkioxari

Dollar

Girshick

. Mask R-CNN. IEEE; 2017. pp. 2980-2988.

17.

Bolya

Zhou

Xiao

Lee

. YOLACT++ Better Real-Time Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022; 44(2): 1108-1121.

18.

Lin

Maire

Belongie

Bourdev

Girshick

Hays

, et al. Microsoft COCO: Common Objects in Context. 2015.

19.

Cordts

Omran

Ramos

Rehfeld

Enzweiler

Benenson

, et al. The Cityscapes Dataset for Semantic Urban Scene Understanding. In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.

20.

Hariharan

Arbeláez

Girshick

Malik

. Simultaneous Detection and Segmentation. 2014. pp. 297-312.

21.

Girshick

Donahue

Darrell

Malik

. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. IEEE; 2014. pp. 580-587.

22.

Dai

Sun

. Instance-Aware Semantic Segmentation via Multi-task Network Cascades. IEEE; 2016. pp. 3150-3158.

23.

Ren

Girshick

Sun

. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017 Jun; 39: 1137-1149.

24.

Hayder

Salzmann

. Boundary-Aware Instance Segmentation. IEEE; 2017. pp. 587-595.

25.

Zamir

Arora

Gupta

Khan

Sun

Khan

, et al. iSAID: A Large-scale Dataset for Instance Segmentation in AerialImages. 2019.

26.

Ghiasi

Cui

Srinivas

Qian

Lin

Cubuk

, et al. Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation. IEEE; 2021. pp. 2917-2927.

27.

Laradji

Vázquez

Schmidt

. Where are the Masks: Instance Segmentation with Image-level Supervision. 2019.

28.

Laradji

Rostamzadeh

Pinheiro

Vazquez

Schmidt

. Proposal-Based Instance Segmentation With Point Supervision. IEEE; 2020. pp. 2126-2130.

29.

Happold

Neumann

. Geometry-Aware Instance Segmentation with Disparity Maps. 2020.

30.

Liang

Homayounfar

Xiong

Urtasun

. PolyTransform: Deep Polygon Transformer for Instance Segmentation. 2020.

31.

Danelljan

Tai

Tang

. Mask Transfiner for High-Quality Instance Segmentation. 2022. pp. 4412-4421.

32.

Akyon

Altinuc

Temizel

. Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection. In: 2022 IEEE International Conference on Image Processing (ICIP). 2022. pp. 966-970.

33.

Yang

Dong

Zhu

. An Instance Segmentation Algorithm Based on Improved Mask R-CNN. IEEE; 2020. pp. 4804-4809.

34.

Zhang

Chu

Leng

Miao

. Mask-Refined R-CNN: A Network for Refining Object Details in Instance Segmentation. Sensors. 2020 Feb; 20: 1010.

35.

Feng

Cao

Zeng

Feng

, et al. Improved Mask R-CNN for Aircraft Detection in Remote Sensing Images. Sensors. 2021 Apr; 21: 2618.

36.

Zhang

Ren

Sun

. Deep Residual Learning for Image Recognition. IEEE; 2016. pp. 770-778.

37.

Kim

Lee

. Deeply-Recursive Convolutional Network for Image Super-Resolution. IEEE; 2016. pp. 1637-1645.

38.

Kim

Lee

. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. IEEE; 2016. pp. 1646-1654.

39.

Dong

Loy

Tang

. Accelerating the Super-Resolution Convolutional Neural Network. 2016. pp. 391-407.

40.

Dong

Loy

Tang

. Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016 2; 38: 295-307.

41.

Bron

Kerbosch

. Algorithm 457: finding all cliques of an undirected graph. Communications of the ACM. 1973; 16(9): 575-577.

42.

Bolya

Zhou

Xiao

Lee

. YOLACT: Real-Time Instance Segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE; 2019. doi: 10.1109/iccv.2019.00925.

43.

Duan

Yan

. Perceptual metric-guided human image generation. Integrated Computer-Aided Engineering. 2022; 29: 141-151.

44.

Tang

Liu

. YDTR: Infrared and Visible Image Fusion via Y-shape Dynamic Transformer. IEEE Transactions on Multimedia. 2022.

45.

Herzog

Magoulas

. Convolutional Neural Networks-Based Framework for Early Identification of Dementia Using MRI of Brain Asymmetry. International Journal of Neural Systems. 2022; 32.

46.

Saneii

Kazemeini

Seilabi

Miralinaghi

Labi

. A methodology for scheduling within-day roadway work zones using deep neural networks and active learning. Computer-Aided Civil and Infrastructure Engineering. 2022.

Optimized instance segmentation by super-resolution and maximal clique generation

Abstract

Keywords

1. Introduction

2. Related work

3. Methodology

3.5 Segmentation refinement module

4.1 Instance segmentation models

4.2 Super-resolution model

4.5 Evaluation metrics

Table 1 Selected values of the hyperparameters

Table 3 Mean average precision obtained for the Jena sequence of the CityScapes Dataset according to different window sizes R using the YOLACT++ model. A detection is valid when its confidence exceeds the set threshold of 50%

Footnotes

Acknowledgments

References

Table 1
Selected values of the hyperparameters

Table 3
Mean average precision obtained for the Jena sequence of the CityScapes Dataset according to different window sizes $R$ using the YOLACT++ model. A detection is valid when its confidence exceeds the set threshold of 50%