A level set approach using adaptive local pre-fitting energy for image segmentation with intensity non-uniformity

Abstract

Active contour model (ACM) is considered as one of the most frequently employed models in image segmentation due to its effectiveness and efficiency. However, the segmentation results of images with intensity non-uniformity processed by the majority of existing ACMs are possibly inaccurate or even wrong in the forms of edge leakage, long convergence time and poor robustness. In addition, they usually become unstable with the existence of different initial contours and unevenly distributed intensity. To better solve these problems and improve segmentation results, this paper puts forward an ACM approach using adaptive local pre-fitting energy (ALPF) for image segmentation with intensity non-uniformity. Firstly, the pre-fitting functions generate fitted images inside and outside contour line ahead of iteration, which significantly reduces convergence time of level set function. Next, an adaptive regularization function is designed to normalize the energy range of data-driven term, which improves robustness and stability to different initial contours and intensity non-uniformity. Lastly, an improved length constraint term is utilized to continuously smooth and shorten zero level set, which reduces the chance of edge leakage and filters out irrelevant background noise. In contrast with newly constructed ACMs, ALPF model not only improves segmentation accuracy (Intersection over union(IOU)), but also significantly reduces computation cost (CPU operating time T), while handling three types of images. Experiments also indicate that it is not only more robust to different initial contours as well as different noise, but also more competent to process images with intensity non-uniformity.

Keywords

Image segmentation partial derivative intensity non-uniformity optimization

1 Introduction

Image segmentation technology is significantly important in different kinds of applications for example object recognition [1], medical image processing [2], etc., which separates an input image into multiple areas that never intersect with different characteristics.

Active contour models (ACMs) are most frequently used methods in image segmentation. The fundamental idea of ACM is to evolve an initial curve towards the boundary of the desired target based on energy minimization method through partial derivative, which can generate an enclosed and smooth final contour that locates the desired object position. The existing ACMs are roughly composed of edge-based models, region-based models. The idea of edge-based models such as the Geodesic active contour (GAC) model in [3] used edge indicator function to instruct the evolution curve to evolve towards the edges of targets, which can segment target edges with clear gradient information. However, it cannot handle targets with unclear or discrete boundaries. In addition, the idea of region-based models was to use specific area descriptor to locate a partition on the image region, which is capable of segmenting targets with blurry or discontinuous boundaries. Nevertheless, it is not competent to handle images with intensity non-uniformity due to the hypothesis that the image intensities are statistically uniform.

To address the problem of intensity non-uniformity, a piecewise smooth (PS) model in [4] could segment images with intensity uniformities to some degree. However, it is inefficient due to complicated calculation process and complex parameter settings. See [5] for a detailed overview of ACM. Classic level set approaches in [6, 7], the level set function must be re-initialized to control its gradient, which prevents it becoming too flat or too steep. However, this reinitialization process is too time-consuming in practice. The concept of a signed distance regularization function in distance regularized level set evolution (DRLSE) model in [8] and region-scalable fitting (RSF) model in [9] was creatively proposed to solve the problem of reinitialization during evolution process in GAC model, which is a great breakthrough for traditional level set approaches. However, DRLSE and RSF models are all sensitive to different initial contours. In addition, RSF model has complex computation procedure, which causes slow segmentation speed and huge computation cost. Similarly, DRLSE model has unsatisfactory noise resistivity and slow evolution speed.

To eliminate the drawback of slow convergence in RSF model, the local image fitting (LIF) model in [10] collected local image region attribute to process images with intensity non-uniformity, which only has two convolutions in every iteration instead of at least four convolutions required by RSF model. Therefore, this model is much more efficient regarding segmentation speed than that of RSF model. Nevertheless, the issue of being susceptible to initial contours retains unresolved, which means that an improper initial contour may cause a failed segmentation due to the truth that the energy functions of most ACMs are non-convex.

The local and global Jeffreys divergence (LGJD) principle was used in [11] to measure the gap between the fitting function and input image, and drive the zero-level set function to reach the target boundary to complete the segmentation, which further improves system robustness to different initial contours as well as segmentation efficiency. However, this model is not optimal and needs further improvements. The additive bias field correction (ABC) theory was applied in [12] to effectively segment images with non-uniformity, which has strong theoretical derivation, strong system robustness, and low computation cost. Nevertheless, this model cannot effectively process some medical images with severe intensity inhomogeneity [13]. The pre-fitting bias correction (PBC) model in [14] utilized an optimized FCM algorithm to pre-calculate bias field before iteration, which can effectively segment images with unevenly distributed intensity and greatly reduces segmentation time. However, the segmentation accuracy and efficiency may be inversely affected if the FCM algorithm has bad clustering performance.

The main motivation of this paper: the traditional ACMs are often sensitive to images with noise interference and intensity non-uniformity, and their segmentation efficiency and accuracy still have space to be further improved. To solve the issues mentioned above, the authors propose an ACM driven by ALPF energy for image segmentation with intensity non-uniformity, which not only effectively segments images with noise as well as intensity non-uniformity, but also achieves high segmentation speed and accuracy. The research gaps between traditional ACMs and proposed ALPF model: the traditional ACMs such as RSF, LIF models utilize two fitting functions to update fitted images inside and outside contour line through Gaussian convolution operations during each iteration, which results in long evolution time of level set function and slow convergence speed. The proposed ALPF model employs two pre-fitting functions to generate pre-fitted images once before iteration, which makes it independent of level set evolution and significantly levels up convergence speed. To be specific, the objectives of the proposed ALPF model: the authors formulate two pre-fitting functions that are pre-calculated before iteration to decrease computation cost and improve segmentation efficiency and accuracy. In addition, the authors construct an adaptive regularization function to normalize the optimized data-driven term to improve system’s robustness while segmenting images with various initial contours, different noise and uneven intensity. Then, an activation function is defined to regularize level set function to keep its sign property, which serves as distance regularized term. Lastly, this activation function is combined with mean filtering to smooth eventual contour and remove redundant non-edge curves, which acts as length constraint term.

The contributions of this paper are enumerated as

The pre-fitting functions calculated before iteration are proposed to generate fitted images, which reduces a huge amount of operating time and improves segmentation efficiency.

An adaptive regularization function and the image standard deviation constitute an improved gradient flow, which regularizes the range of data-driven term and improves segmentation precision.

An activation function combined with mean filtering constructs an improved length constraint term, which smooths final zero level set and removes irrelevant curves from non-edge regions.

The pre-trained YOLOv5 network can be employed as a pre-processing to automatically initialize initial contours, which eliminates manually designed initial contours.

2 The proposed ACM

This section introduces local pre-fitting functions, model of adaptive local pre-fitting function, normalization of level set function, and implementation process at length respectively.

2.1 Local pre-fitting functions

Let I (x) : Ω → R be an input gray image defined on an vectorized image domain Ω ∈ R². The pre-fitting functions are designed to generate fitted images and reduce computation cost, which are defined as

${\begin{matrix} f_{med} (x) = median (I (y) ∣ y \in Ω_{x}), \\ c_{l} (x) = mean (I (y) ∣ y \in Ω_{l}), \\ c_{s} (x) = mean (I (y) ∣ y \in Ω_{s}) . \end{matrix}$ (1)

Note that Ω_x denotes a local square region centered at the point x with width (2w + 1); variable y represents all pixels in Ω_x; I (y) signifies intensity at the point y; Ω_s and Ω_l are denoted by

${\begin{matrix} Ω_{l} = {y ∣ (I (y) > f_{med} (x)} \cap Ω_{x}, \\ Ω_{s} = {y ∣ (I (y) < f_{med} (x)} \cap Ω_{x} . \end{matrix}$ (2)

Note that f_med (x) is the median intensity in Ω_x ; c_l and c_s are the average intensities in Ω_l and Ω_s respectively; Ω_s is the area, where the intensities less than the median intensities in Ω_x; Ω_l is the region, where the intensities bigger than the median intensities in Ω_x.

To illustrate the local sub-regions and results of pre-fitting functions, a square green box centered at a pixel point x = (17, 28) with a mask (2w + 1) × (2w + 1) is defined in Fig. 1, which generates a local median value f_med (x) =25, a local fitting value outside the local contour line c_s (x) =19.5 and a local fitting value inside the local contour line c_s (x) =32.2. Generally, the center point x of this box begins to move from pixel coordinate (1, 1) in the top left corner to pixel coordinate (m, n) in the bottom right corner with stride 1 pixel. Each center pixel point x generates two corresponding c_l (x) and c_s (x) locally. All the local c_l (x) and c_s (x) at different center pixel point x will be re-aligned in order to generate two eventual pre-fitted images as shown in Fig. 2. Note that m and n are the number of rows and columns of pixels respectively.

Fig. 1

Illustration of regions Ω, Ω_x, Ω_l, and Ω_s. In the left half of the figure, the region Ω represents the entire image area. In the right half of the figure, the region enclosed by square green box denotes Ω_x with center pixel point x = (17, 28) highlighted in blue. The orange curve signifies boundary between local regions Ω_l and Ω_s. Within the square green box, the above region encircled by rectangular green box and orange line describes Ω_s while the below region represents Ω_l. At this specific center pixel point x = (17, 28), c_l (x) =32.2, c_s (x) =19.5, and f_med (x) =25.

Fig. 2

Illustration of eventual pre-fitted images c_l and c_s.

To be specific, the calculation process of the local pre-fitting functions c_l and c_s is concluded as follows: firstly, an image local area Ω_x at center point x is selected to calculate its median intensity f_med (x), which divides the entire local image area into two sub-areas Ω_l, Ω_s. Secondly, the average intensities of two sub-areas are computed to generate local c_l (x), c_s (x) respectively. After that, the same computation procedure is duplicated at the subsequent pixel until all pixels in the input image area are all taken into account. Lastly, the final pre-fitted images c_l and c_s in Fig. 2 will be incorporated into the proposed energy function in Equation (3) for later computation. Dislike traditional ACMs, the pre-fitting functions are irrelevant with level set functions, which can hugely enhance the robustness to initial contours and reduce computation cost.

2.2 Model of adaptive local pre-fitting functions

Based on final pre-fitted images c_l and c_s in Fig. 2, a local pre-fitting energy (LPFE) is constructed as

$\begin{matrix} E^{LPFE} & (C, c_{s}, c_{l}) = \int_{outside (C)} | I (x) - c_{s} (x) | d x \\ + \int_{inside (C)} | I (x) - c_{l} (x) | d x, \end{matrix}$ (3) where pre-calculated fitting functions c_s and c_l are defined in Equation (1); I (x) signifies the input image; |I (x) - c_s (x) | denotes the difference between input image intensity and fitted image intensity outside the contour C; |I (x) - c_l (x) | signifies the distance between input image intensity and fitted image intensity inside the contour C. Subsequently, the contour C is substituted by zero level set of Lipschitz function φ (x, i). Hence, the energy term is translated as $\begin{matrix} E^{LPFE} & (φ (x), c_{s}, c_{l}) = \int_{Ω} | I (x) - c_{s} (x) | H_{ɛ} (φ) d x \\ + \int_{Ω} | I (x) - c_{l} (x) | (1 - H_{ɛ} (φ)) d x, \end{matrix}$ (4) where φ (x, i) is the level function function and is initialized as φ (x, 0) before iteration as follows:

$φ (x, t = 0) = {\begin{matrix} - 1, & x \in in (C), \\ 0, & x \in C, \\ 1, & x \in out (C), \end{matrix}$ (5)

After initialization, the sign of level set function φ becomes negative if the pixel point x is located inside contour C. On the contrary, its sign becomes positive when the pixel point x is places outside contour C. Particularly, the level set function φ becomes 0 (zero level set) when the pixel point x on the contour C.

In Equation (4), the improved Heaviside and Dirac functions H_ɛ (x) and δ_ɛ (x) [15] are described as

${\begin{matrix} H_{ɛ} (x) = \frac{1}{2} sin (\frac{π}{3} x) + \frac{1}{2}, \\ δ_{ɛ} (x) = H_{ɛ}^{'} (x) = \frac{π}{6} cos (\frac{π}{3} x) . \end{matrix}$ (6)

In practice, H_ɛ (x) in Equation (6) is considered as an affiliation function to differentiate the inner and outer parts of the contour line C. Specifically, H_ɛ (x) represents the part outside the contour line while (1 - H_ɛ (x)) represents the part inside the contour line. Note that δ_ɛ (x) is the improved Dirac function, which is the derivative of H_ɛ (x) and used in following gradient descent method to find out the minimum energy.

Approach of gradient descent [16] is utilized to minimize the LPFE energy in Equation (4), which obtains gradient flow equation as

$\begin{matrix} \frac{\partial φ (x, t)^{LPFE 1}}{\partial t} = & - \frac{\partial E^{LPFE}}{\partial φ (x, t)} \\ = & - δ_{ɛ} (φ (x, t)) (e_{1} - e_{2}), \end{matrix}$ (7) where e₁ = |I (x) - c_s|, e₂ = |I (x) - c_l|, and the original data-driven term is defined as e₁ - e₂; $δ_{ɛ} (x) = H_{ɛ}^{'} (x)$ is improved Dirac function defined in Equation (6). Note that Equation (7) is simplified in Proposition. 1 below.

Proposition 1. Equation (7) is further simplified as: $\frac{\partial φ (x, t)^{LPFE 2}}{\partial t} = h (2 I (x) - (c_{s} + c_{l})) δ_{ɛ} (φ (x, t)),$ (8) where h is a signed constant, which controls the evolution speed and direction of evolution curve, and the optimized data-driven term is defined as e = 2I (x) - (c_s + c_l)). For the application of two-phase gray images, if the target of interest is the white part, h = -1; if the target of interest is the black part, h = 1.

Proof. See A.

Because of the variety of image genres, the image contrast varies significantly. The values of optimized data driven item e in Equation (8) may vary hugely, which possibly increases the difficulty of subsequent experiments. Meanwhile, the sensitivity of data-driven term e in the zero-crossing region has to be improved.

To solve the issues mentioned above, an adaptive function erf (x) regularizes the optimized data-driven term e in Equation (8), which constructs an adaptive local pre-fitting (ALPF) gradient flow defined as follows:

$\frac{\partial φ (x, t)^{ALPF}}{\partial t} = δ_{ɛ} (φ (x)) \cdot α \cdot \erf (\frac{e}{τ}),$ (9) with the image standard deviation τ

$τ = std 2 (I (x)),$ (10) with the adaptive regularization function erf (x)

$\erf (x) = \frac{2}{π} \int_{0}^{x} e^{- η^{2}} d η .$ (11) where e is the optimized data-driven term in Equation (8), and η is simply an integral variable.

In Equation (9), the regularized data-driven term is defined as ex = erf (e/τ). Note that α · erf (e/τ) is computed before iteration, and independent of it. Therefore, ALPF model further reduces the computing cost greatly by only updating δ_ɛ (φ (x, t)) during iteration.

In addition, τ is the standard deviation value of the input image, which reflects the degree of dispersion of the input image; parameter α denotes a signed energy coefficient calibrating the running speed and direction of evolution curve by changing its magnitude and sign properly to eliminate issues of under-segmentation as well as over-segmentation as much as possible. For the application of double-phase images, α = -1, if the object of interest is the white part; α = 1, if the object of interest is the black part. In addition, if the input image has low contrast but complete edge, the energy ratio α shall be leveled up to accelerate the evolution speed, which prevents under-segmentation; if the input image has high contrast but incomplete edge, the energy coefficient α shall be reduced to decelerate the evolution speed, which prevents over-segmentation.

Normalization of level set function

Dislike classic ACMs, ALPF energy rules out the distance regularized part and length restraint part. However, φ (x, t) needs distance regularized part to maintain its sign property of positive outside the contour and negative inside the contour. Additionally, φ (x, t) needs length restraint part to smooth curve.

In order to make up for those flaws, firstly, an activation function softsign (x) in Equation (14) is applied to regularize φ (x, i) to make sure that it avoids being too steep or too flat and keeps its sign attribute, which serves as traditional distance regularized part. Secondly, φ_A (x, i - 1) in Equation (13) is an improved length restraint term, which constantly smooths φ (x, i - 1) and removes non-edge curves. Note that φ (x, i - 1) denotes level set function at (i-1)-th iteration while φ (x, i) signify level set function at i-th iteration.

Hence, the evolution equation of φ (x, i) is constructed to update it iteratively, which is denoted as

$φ (x, i) = φ_{A} (x, i - 1) + Δ t \cdot \frac{\partial φ^{ALPF} (x, i - 1)}{\partial t},$ (12) with the optimized length term φ_A (x, i - 1)

$φ_{A} (x, i - 1) = A_{k \times k} (softsign (6 \cdot φ (x, i - 1))),$ (13) with the activation function softsign (x)

$softsign (x) = \frac{x}{1 + | x |},$ (14)

Algorithm 1 ALPF ACM
Input:I (x), iteration N and parameters α, k, w.
Output: Final zero level set function φ (x, i).
1: Reset level set function as φ (x, 0) using Equation (12).
2: Compute fitted images c_l and c_s based on Equation (1).
3: Compute data-driven term e = (2I - (c_s + c_l)).
4: Compute standard deviation τ based on Equation (10).
5: Compute gradient flow ∂φ^ALPF/∂t using Equation (9).
6: fori ← 1 to N do
7: Update δ_ɛ (φ) using Equation (6).
8: Update evolution equation using Equations (12),(13).
9: if \|φ (x, i) - φ (x, i - 1) \| ≤ 0.001 then
Break iteration;
end
end
10: Final zero level set function φ (x, i).

where A_k×k is mean filter function with window size k × k, and k is an positive odd number greater than one. A_k×k averages all the pixels in this template, and then the original pixel value is replaced with this average value. softsign (6 · φ (x, i - 1)) guarantees that level set function φ has strong regularization properties in order to suppress slopes at the high points at both ends.

2.3 Implementation procedure

Unless otherwise stated, the parameters are initialized: α = 2, k = 8, w = 8. At length, the quantity of α will be enlarged to accelerate iteration if intensity is low; the quantity of k will be increased properly to filter out useless pixels if images contain complicated background noise; the quantity of w will be enhanced properly to include big targets with clear edges. Additionally, constant parameters are fixed: Δt = 1, ɛ = 1. The specific implementation process of ALPF model is stated in Algorithm 1. Note that it is implemented in Matlab 2022a on an AMD Ryzen 7 6800H 3.5 GHz laptop. For the tagging of segmentation outcomes, the green contours signify initial contours, the red contours denote final results, and the cyanic contours represent the whole iteration. Note that all natural images are form well-known Berkeley segmentation dataset 500 (BSDS500), which can be downloaded from website https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/. The steel plate images are referred to website https://www.kaggle.com/c/severstal-steel-defect-detection. The Magnetic Resonance Imaging (MRI) images can be downloaded from the dataset (https://www.kaggle.com/datasets/jakeshbohaju/brain-tumor).

3 Experimental verification

This section states experimental outcomes of several experiments of ALPF model including verification experiment, anti-noise experiment, robustness experiment, and companion with newly constructed ACMs in terms of natural, steel plate, and MRI images.

3.1 Validation experiment of ALPF model

To verify the validity of ALPF model, 2 images are processed by using reference variables to illustrate segmentation effects and iterative procedures, which are demonstrated in Fig. 3. The green curves in the 1st column are initial contours, the red curves from the 2nd to 4th columns are segmentation outcomes at specific iterations, and the bluish curves in the 5th column are the whole iterative procedures. In Fig. 3, the initial level set can be continuously smoothed to achieve eventual zero level set.

Fig. 3

Validation of ALPF model. The 1st column states initial contours, the 2nd column represents segmentation result at iteration number N = 3, the 3rd column describes segmentation result at iteration number N = 10, the 4th column illustrates final result, and the 5th column represents the entire iterative process.

3.2 Anti-noise experiment of ALPF model

To test the anti-noise ability of ALPF model, 6 different images with manually added Gaussian noise (fixed mean m = 0, and the variance of it can be changed) are processed. Because the targets in the last two images (e-f) are bigger than that of the first four images (a-d), the authors properly increase the value of window size w to entirely cover the larger target to obtain more target details. To be specific, the authors firstly employ the default parameter setting α = 2, k = 8, w = 8 to segment the first four images (a-d) in Fig. 4, which turns out that the value of window size w is too small to cover the whole objects in images (a-d). Then, the authors slightly increase the value of parameter w from 7 to 9, which successfully segment the first four images (a-d). Lastly, it can be observed that the objects in the last two images (e-f) are larger than that of the first four images (a-d). Therefore, the authors properly increase the value of parameter w from 9 to 12 to entirely cover larger objects in the last two images (e-f) after several attempts. Therefore, the parameter setting of the first four images (a-d) are α = 2, k = 8, w = 9, and the parameter setting of the last two images (e-f) are α = 2, k = 8, w = 12. The experimental outcomes are exhibited in Fig. 4, which signifies that ALPF model is able to process images with Gaussian noise within a certain range.

Fig. 4

Anti-noise experiment. The 1st row is the original images, and the 2nd row is results of images with Gaussian noise.

At length, the utmost Gaussian noise in percentage on images (a, c) is 2%, the utmost Gaussian noise in percentage on image (b) is 3%, the utmost Gaussian noise in percentage on image (d) is 2.5%, the utmost Gaussian noise in percentage on image (e) is 3%, and the utmost Gaussian noise in percentage on image (f) is 2.5%. In addition, if the Gaussian noise in percentage is beyond limitation, wrong segmentation may appear in the form of cracked curves.

In addition, Speckle, Salt&Pepper, Poisson noises are also added to 3 images in Fig. 5. Note that Speckle and Salt&Pepper noise variances in percentage for all 3 images in Fig. 5 are 3%, 35% respectively. In this figure, it is further proved that ALPF model is robust enough to resist different kinds of externally added noises including Gaussian, Speckle, Salt&Pepper, Poisson within limit.

Fig. 5

Anti-noise experiment. The 1st column is the segmentation results with zero noise, and the 2nd to 4th columns are segmentation results of images with Speckle, Salt&Pepper, Poisson noises.

3.3 Robustness experiment of ALPF model

To further detect the robustness of ALPF model to different initial contours, 4 different images are processed, which are described in Fig. 6. From the 1st row to 3rd row in this figure, the ALPF model is able to fully segment images with excellent accuracy wherever initial contours are assigned, which indicates ALPF model has a strong robustness to different initial contours. In addition, from the 4th row in Fig. 6, it has been further confirmed that ALPF model remains insensitive to different initial contours, which can effectively process the entire multi-object image when the initial contours are placed in each sub-object with different dimensions and shapes respectively.

Fig. 6

Robustness experiment to different initial contours. In each set of images, the left one is segmentation result with initial contour, the right one is the entire iterative process respectively.

3.4 Comparison with recently constructed ACMs on natural images

Perform the first comparison experiment to compare segmentation outcomes between ALPF model with 4 recently constructed ACMs (LPF&FCM [17], PBC [14] LGJD [11] ABC [12]) on 12 natural images (1-12) under same initial contours respectively. The visualized segmentation outcomes are illustrated in Fig. 7. In this figure, all images segmented by the proposed ALPF model are fully segmented with high accuracy while images segmented by other 4 models are possibly under-segmentation or over-segmentation, which means that ALPF model maintains its stable evolution to obtain fair segmentation results.

Fig. 7

Visual outcomes of the first comparison experiment on images 1-12. The 1st column signifies ground-truth images, the 2nd denotes initial contours, the 3rd to 7th columns are results of LPF&FCM, PBC, LGJD, ABC, and ALPF models respectively.

Table 1

Numerical analysis of outcomes (the CPU operating time T, and IOU) of the 1st comparison experiment on images 1-12 in Fig. 7

	LPF&FCM	PBC	LGJD	ABC	ALPF
Image 1 (118 * 93)	1.135/0.826	1.219/0.845	1.124/0.819	3.364/0.885	0.986/0.957
Image 2 (481 * 321)	8.162/0.621	3.014/0.586	2.634/0.876	1.897/0.815	1.532/0.912
Image 3 (481 * 321)	6.848/0.851	3.072/0.880	11.147/0.836	2.722/0.885	1.771/0.895
Image 4 (481 * 321)	10.272/0.804	1.999/0.727	3.028/0.782	1.432/0.769	1.186/0.821
Image 5 (481 * 321)	6.332/0.808	3.391/0.864	3.312/0.878	1.917/0.881	1.428/0.889
Image 6 (481 * 321)	9.852/0.674	5.333/0.779	3.283/0.776	2.554/0.847	2.169/0.866
Image 7 (300 * 203)	3.046/0.956	1.106/0.957	1.113/0.883	0.899/0.951	0.387/0.967
Image 8 (400 * 294)	1.884/0.928	4.696/0.956	0.508/0.968	4.464/0.971	0.752/0.982
Image 9 (300 * 200)	1.785/0.713	2.348/0.408	2.764/0.915	7.943/0.257	1.947/0.954
Image 10 (750 * 500)	3.952/0.931	2.254/0.935	5.143/0.765	4.895/0.783	1.219/0.962
Image 11 (200 * 200)	5.295/0.639	4.752/0.814	8.452/0.337	6.756/0.694	1.874/0.948
Image 12 (315 * 400)	7.812/0.924	2.851/0.917	2.120/0.456	10.857/0.783	0.842/0.980
Average	5.031/0.806	3.003/0.806	3.719/0.774	4.142/0.793	1.341/0.928

In addition, the numerical segmentation outcomes (the CPU operating time T, intersection over union (IOU)) between above said 4 ACMs and ALPF model are concluded in Table 1, where IOU are defined as $IOU = \frac{M_{1} \cap M_{2}}{M_{1} \cup M_{2}} .$ (15) Note that M₁ denotes the foreground region generated by different ACMs, M₂ signifies foreground region of a given ground-truth image. The IOU value is employed to gauge the similarity between foreground regions of different images. The IOU value naturally tends to 1 as the segmentation accuracy elevates.

In this table, the CPU operating time T of ALPF model is the least for images (1-7,10-12) with the highest segmentation accuracy (IOU) among all 5 models. Nevertheless, for image (8), though the CPU operating time T of the LGJD model is ranked the first, which is slightly shorter than ALPF model, but the IOU value of it are smaller than that of ALPF model. For image (9), although LPF&FCM model obtains the fastest segmentation speed, its IOU value is much smaller than that of ALPF model. Especially, for images (2,9), the final segmentation results of all other 4 ACMs are severely affected by background noise, which leads to issues of falling into false boundaries and edge leakage. Based on above analysis, ALPF model is more competent to process natural images (1-12) regarding the CPU operating time T, and IOU value than the above-said 4 recently developed ACMs on average.

3.5 Comparison with recently constructed ACMs on steel plate images

Conduct the second comparison experiment to compare the segmentation outcomes between ALPF model and 4 recently constructed ACMs (LPF&FCM [17], PBC [14] LGJD [11] ABC [12]) on 12 steel plate images (13-24) under same initial contours respectively. The visualized segmentation outcomes are illustrated in Fig. 8. In this figure, ALPF model obtains clean and sound final contours at target boundaries while other 4 ACMs frequently have issues of broken and random curves, which indicates that ALPF model is more competent to detect defects on steel plates.

Fig. 8

Visual outcomes of the second comparison experiment on images 13-24. The 1st column denotes initial contours, the 2nd states ground-truth images, the 3rd to 7th columns are results of LPF&FCM, PBC, LGJD, ABC, and ALPF models respectively.

In addition, the associated numerical segmentation outcomes (the CPU operating time T, IOU) are concluded in Table 2. In this table, the CPU operating time T, IOU of ALPF model are the best for images (13-16,18-24) among all 5 ACMs. For image (17), although the CPU operating time T of PBC model is slightly smaller than that of ALPF model, ALPF model maintains the largest IOU. Particularly, for images (13-15), LPF&FCM, PBC, and LGJD models face with severe issue of falling into false boundaries, which results in much lower IOUs. For images (15,24), ABC model undergoes issue of under-segmentation, which leads to incomplete segmentation. Based on above analysis, ALPF model can process steel plate images with uneven intensity more effectively than the above-said 4 recently developed ACMs on average.

Table 2

Numerical analysis of outcomes (the CPU operating time T, and IOU) of the 2nd comparison experiment on images 13-24 in Fig. 8.

	LPF&FCM	PBC	LGJD	ABC	ALPF
Image 13 (200 * 200)	4.149/0.629	2.335/0.567	5.298/0.093	3.259/0.811	1.315/0.872
Image 14 (200 * 200)	2.154/0.230	2.358/0.152	6.771/0.184	5.395/0.782	0.358/0.799
Image 15 (200 * 200)	1.676/0.351	1.586/0.223	6.851/0.221	4.528/0.481	0.465/0.815
Image 16 (200 * 200)	7.682/0.761	1.726/0.695	5.318/0.456	11.852/0.765	0.929/0.845
Image 17 (200 * 200)	7.551/0.675	0.748/0.781	5.341/0.264	15.258/0.766	1.571/0.804
Image 18 (200* 200)	7.581/0.753	1.679/0.278	5.337/0.649	14.952/0.854	1.117/0.867
Image 19 (200 * 200)	1.086/0.526	1.103/0.246	6.772/0.254	5.625/0.833	0.418/0.856
Image 20 (200 * 200)	1.753/0.349	1.171/0.371	5.306/0.113	8.397/0.842	0.458/0.877
Image 21 (200* 200)	2.393/0.813	1.115/0.685	5.362/0.226	10.586/0.816	0.976/0.885
Image 22 (200 * 200)	7.234/0.719	0.734/0.794	5.501/0.218	10.851/0.771	0.821/0.808
Image 23 (200 * 200)	2.362/0.657	2.983/0.341	6.832/0.278	3.341/0.812	1.518/0.839
Image 24 (200 * 200)	4.665/0.801	1.845/0.676	5.354/0.833	8.106/0.792	1.510/0.897
Average	4.191/0.605	1.615/0.484	5.837/0.316	8.512/0.777	0.955/0.847

3.6 Comparison with newly constructed ACMs on MRI images

Fig. 9

Visual outcomes of the third contrast experiment on images 25-36. The 1st column denotes initial contours, the 2nd states ground-truth images, the 3rd to 8th are results of RSF&LoG, PBFE, OLPFI, LGJD, ABC, and ALPF models respectively.

Perform the third contrast experiment to contrast the segmentation outcomes between ALPF model with 5 newly constructed ACMs (RSF&LoG [18], PBFE [19], OLPFI [20], LGJD [11], ABC [12]) on 12 Magnetic Resonance Imaging (MRI) images (25-36) under same initial contours respectively. The visualized segmentation outcomes are exhibited in Fig. 8. In this figure, ALPF model generates complete and clean eventual contours. On the contrary, other 5 ACMs have problems of over-segmentation and under-segmentation. Notably, RSF&LoG model cannot effectively process those MRI images (25-36) in the presence of random and redundant curves.

Additionally, the numerical segmentation results (the CPU operating time T, and IOU) are listed in Table 2. In this table, ALPF model achieves the best CPU operating time T and IOU for MRI images (25, 27, 29-32, 34-36). However, for MRI images (26, 28, 33), though OLPFI model remains the least CPU operating time T, its segmentation precision (IOU) is much lower than that of ALPF model respectively. Notably, RSF&LoG and LGJD models cannot effectively all MRI images (25-36) in the forms of falling into false boundaries and long convergence time, which leads to much smaller IOUs and much longer CPU operating time T. Regarding above analysis, compared with above-said 4 newly constructed ACMs, ALPF model exhibits advantages in terms of processing time and precision while segmenting MRI images (25-36) with uneven intensity on average.

4 Discussion

This section makes a series of discussions concerning optimization of data-driven term, adaptive normalization function, comparison with different gradient flows, optimization of level set function and length term, automation of initial contours to explain the reason why ALPF model is a better choice to handle images with unevenly distributed intensity. Additionally, the application of ALPF model is introduced at last.

4.1 Optimization of data-driven term

Fig. 10

Optimization of data-driven term. The most left sub-image is original image, the middle sub-image is segmentation results through original data-driven term e₁ - e₂, and the most right sub-image is segmentation results through optimized data-driven term e for each image respectively.

To discuss the optimization of data-driven term, 4 images (37-40) are processed to compare segmentation efficiency (CPU operating time T) and accuracy (IOU) between original data-driven term e₁ - e₂ and optimized one e. In other words, the gradient flow equation ∂φ^LPFE1/∂t in Equation (7) and ∂φ^LPFE2/∂t in Equation (8) are applied to segment these 3 images respectively, whose segmentation results are illustrated in Fig. 10. From this figure and Table 3, although the segmentation precision (IOU) of two data-driven terms are identical by segmenting images (37-38), the optimized data-driven term e has less CPU operating time T than that of original one e₁ - e₂.

Table 3

Numerical analysis of outcomes (the CPU operating time T, and IOU) of the 3rd comparative experiment on images 25-36 in Fig. 9

	RSF&LoG	PBFE	OLPFI	LGJD	ABC	ALPF
Image25(240* 240)	4.547/0.383	0.633/0.861	0.251/0.904	2.493/0.335	1.861/0.660	0.157/0.950
Image26(240* 240)	4.138/0.210	0.701/0.933	0.091/0.916	4.785/0.182	0.812/0.905	0.218/0.955
Image27(240* 240)	3.551/0.248	0.718/0.819	0.275/0.899	4.744/0.167	0.803/0.891	0.197/0.952
Image28(240* 240)	3.629/0.175	0.726/0.865	0.069/0.848	3.158/0.199	1.335/0.789	0.572/0.910
Image29(240* 240)	3.418/0.067	0.655/0.795	0.293/0.873	2.886/0.074	1.106/0.627	0.174/0.904
Image30(240* 240)	3.581/0.181	0.952/0.705	0.264/0.891	3.910/0.100	0.776/0.491	0.215/0.921
Image31(240* 240)	3.214/0.326	1.385/0.892	0.257/0.921	2.574/0.172	1.849/0.912	0.118/0.942
Image32(240* 240)	3.472/0.187	0.812/0.904	0.227/0.963	4.557/0.187	0.832/0.924	0.105/0.963
Image33(240* 240)	3.923/0.368	0.816/0.929	0.045/0.955	4.755/0.957	1.137/0.943	0.208/0.963
Image34(240* 240)	3.852/0.055	0.885/0.874	0.337/0.905	3.951/0.067	0.775/0.899	0.125/0.905
Image35(240* 240)	4.275/0.118	0.695/0.792	0.194/0.770	3.752/0.061	1.447/0.694	0.181/0.929
Image36(240* 240)	5.045/0.278	2.528/0.911	1.846/0.896	2.753/0.035	15.562/0.790	0.445/0.953
Average	3.887/0.216	0.959/0.857	0.346/0.895	3.693/0.211	2.358/0.794	0.226/0.937

Table 4

Comparison between original data-driven term e₁ - e₂ and optimized one e in terms of CPU operating time T and IOU in Fig. 10

	α	k	w	e₁ - e₂	e
Image 37 (117 * 154)	5.00	3.00	8.00	0.350/0.915	0.713/0.915
Image 38 (214 * 209)	3.15	8.00	10.00	1.512/0.825	0.854/0.825
Image 39 (112 * 224)	2.50	8.00	10.00	0.325/0.930	0.184/0.941
Image 40 (481 * 321)	2.00	11.00	9.00	2.482/0.870	1.532/0.912
Image 41 (103 * 131)	1.50	8.00	8.00	0.518/0.810	0.223/0.865

In addition, from Fig. 10 and Table 3, images (39-41) segmented by e₁ - e₂ are all under-segmentation while all images (39-41) segmented by e are all fully segmented. Based on above analysis, compared with e₁ - e₂, the optimized data-driven term e is capable of efficiently segment images with less CPU operating time T while maintaining a higher level of segmentation accuracy (IOU) on average.

4.2 Adaptive normalization function

Choose 4 images to make their second-order differential attributes, optimized data-driven term e, and regularized data-driven term ex visualized respectively in Fig. 11. In this figure, the second-order differential attributes of these images are listed in the first column. To explain the annotation in this column, the red curve denotes magnitude of optimized data-driven term e for every line, the dotted green line signify magnitude of e in that particular base line. Additionally, the datum above the baseline is positive while the datum under the baseline is negative, which means the intersections between base line and second-order differential attribute curve are the locations of boundary points.

Fig. 11

Visualized data-driven terms. The 1st column is the second-order differential attributes, the 2nd column is 3D visualization of the optimized data-driven term e = (2I - (c_s + c_l)), and the 3rd column is 3D visualization of adaptive regularization function ex = erf (e/τ) for each image.

Additionally, regarding the 2nd column in Fig. 11, the amplitude of ups and downs indicates the uneven distribution of energy driven by optimized data-driven term e, which may cause different segmentation issues such as under-segmentation, slow segmentation, and unstable evolution process. Hence, the adaptive regularization function erf (x) in Equation (9) effectively normalizes the energy range of e to achieve stable evolution, and increases sensitivity in the zero-crossing area. According to the 3rd column in Fig. 11, the normalized data-driven term ex guarantees system stable adequate to process iterative evolution and further improves segmentation precision.

4.3 Comparison with different gradient flows

The gradient flow function ∂φ^LPFE2/∂t in Equation (8) contains unstable data-driven term e, which may bring about wrong segmentation results during curve evolution. Hence, an adaptive regularzaiton function erf (x) is applied to normalize and improve energy distribution of e, which generates improved gradient flow function ∂φ^ALPF/∂t in Equation (9). Regarding Fig. 12, ∂φ^ALPF/∂t can effectively enhance object edges to achieve accurate and intact eventual contours and weaken interference caused by intensity non-uniformity. In contrast, ∂φ^LPFE2/∂t leads to issues of over-segmentation and broken edge. Hence, it is further confirmed that the addition of adaptive regularization function erf (x) in Equation (9) improves segmentation precision and system stability.

Fig. 12

Segmentation outcomes under different gradient flow functions. The 1st column signifies original image, the 2nd column represents 2-D visualization of data-driven terms e = (2I - (c_s + c_l)) and ex = erf (e/τ), the 3rd column illustrates 3-D mesh plots of data-driven terms e and ex, and the last column denotes final segmentation results segmented by gradient flow functions ∂φ^LPFE2/∂t and ∂φ^ALPF/∂t respectively.

4.4 Optimization of level set function and length term

On the one hand, as shown in the 2nd and 3rd columns of Fig. 13, the level set function φ (x, i - 1) is normalized by softsign (x) in Equation (14), both ends of the level set function φ (x, i - 1) are stretched and limited to (-1, 1). It ensures more stable evolution process and its strong sign properties inside and outside the evolution contour. Besides, the sensitivity of zero-crossing area is improved by elongating the data on the upper and lower sides of the zero level set.

Fig. 13

Optimization of level set function. The 1st column is original image with initial contour, the 2nd column denotes level set function without the activation function softsign (x), the 3rd column signifies level set function with softsign (x), and the last column is final segmentation result for each image respectively.

On the other hand, the optimized length term φ_A (x, i - 1) is incorporated into evolution equation in Equation (12) to conduct energy minimization. As indicated in Fig. 14, for each pair of images, the 2nd and 3rd rows indicate that it effectively smooth eventual contours at target edges and eliminates non-boundary points, which greatly improves segmentation precision.

Fig. 14

Optimization of length term. For each pair of images, the 1st row describes original images, the 2nd row signifies segmentation results with and without the improved length constraint term, and the 3rd row illustrates associated final level set function with and without the improved length constraint term respectively.

4.5 Automation of initial contours

For classic ACMs, the coordinates of initial contours are usually manually initialized, which consumes a great amount of time and improper initial contours may cause wrong segmentation results. To deal with this issue, the You Only Look Once v5 (YOLOv5) is pre-trained to automatically initialize initial contours. To be specific, the prediction frames at the output of YOLOv5 will be utilized as coordinates of initial contours, which realizes automation of initial contours.

In practice, 4 images (42-45) are selected to demonstrate the automation process of initial contours, which is illustrated in Fig. 15. In this figure, for image (42), the subject in it is accurately identified as a person with class probability 99.59% and IOU 0.955. For image (43), the target inside is accurately recognized as an aeroplane with category probability 98.93% and IOU 0.912. For image (44), the object in it is precisely identified as a bench with class probability 98.44% and IOU 0.905. For image (45), the target inside is accurately recognized as a bear with category probability 99.46% and IOU 0.930. Therefore, the pre-trained YOLOv5 network can serve as a pre-processing tool to automatically initialize initial contours, which renders ALPF model more intelligent and accelerates subsequent parameter tuning process.

Fig. 15

Combination of the YOLOv5 and ALPF model. The 1st column represents original images, the 2nd column denotes results of YOLOv5, and the 3rd column signifies results of ALPF model.

4.6 The application scopes of ALPF model

As above comparison experiments have proven, ALPF model can effectively process images with intensity-non-uniformity such as natural images, steel plate images, and MRI images. Especially, it can accurately identify and segment cracks on defective steel plate image in Fig. 8, which helps manufacturing plant to detect defective products and increase production yield; it can also precisely locate and extract lesions from MRI images in Fig. 9, which assists doctors to develop specific treatment plans.

Note that this model is mainly utilized to process double-phase images, which means this model cannot segment multi-phase images containing black and white targets at the same time such as images in Fig. 16. In the later work, the authors will attempt to employ poly-phase segmentation approach [21] to resolve this issue. Besides, the author will employ theory of machine learning [22] to get rid of manual adjustment of reference variables and achieve automatic optimization and adjustment of parameters.

Fig. 16

The application limitation of ALPF model.

5 Conclusion

This paper puts forward a level set approach in the presence of uneven intensity using adaptive local pre-fitting energy with application to natural, steel plate, and MRI images, which has superiorities listed as follows: (1) Novel pre-fitting equations are designed to be calculated ahead of iteration, which computes fitted images inside and outside the contour line and substantially decreases computation expenditure. (2) An improved gradient flow function is constructed with the combination of an adaptive regularization function and the image standard deviation, which provides a stable evolution environment and improves system performance and robustness to intensity non-uniformity. (3) An improved length constraint term is proposed by combining an activation function with mean filtering, which maintains zero level set smoothed and eliminates unrelated curves. (4) The automation of initial contours is realized through the pre-trained YOLOv5 network, which can be combined with the most optimized parameters to conduct energy minimization.

Acknowledgments

This research paper was invested by National Natural Science Foundation of China under Grant 62103293, Natural Science Foundation of Jiangsu Province under Grant BK20210709, Suzhou Municipal Science and Technology Bureau under Grant SYG202138, and Entrepreneurship and Innovation Plan of Jiangsu Province under Grant JSSCBS20210641.

Footnotes

Appendix

A. Proof of Proposition. 1

The target to be segmented is assumed to be white, and the contour position corresponds to the point I (x) on the image, which exists 3 possible conditions:

(1) The contour line is within the white target, if the point I (x) meets the criterion that

(I (x) > c_s) & (I (x) > c_l),

so - (|I (x) - c_s| - |I (x) - c_l|) = (c_s - c_l) < 0;

so h (2I - (c_s + c_l)) = -2I (x) + (c_s + c_l) < 0.

Hence, this point I (x) on the contour line is the expansion movement, and the Equation (8) takes into account the influence of the gray scale value of the image point.

(2) The contour line is within the background, if the point I (x) meets the criterion that

(I (x) < c_s) & (I (x) < c_l),

so - (|I (x) - c_s| - |I (x) - c_l|) = (c_l - c_s) > 0;

so h (2I - (c_s + c_l)) = -2I (x) + (c_s + c_l) > 0.

Hence, this point I (x) on the contour line is the shrink movement, and the Equation (8) takes into account the influence of the gray scale value of the image point.

(3) The contour line is within the target transition region, if the point I (x) meets the criterion that

(I (x) > c_s) & (I (x) < c_l),

so - (|I (x) - c_s| - |I (x) - c_l|) = -2I (x) + (c_s + c_l);

so h (2I (x) - (c_s + c_l)) = -2I (x) + (c_s + c_l).

So, the same expression is obtained, Equation (8) is proved to be right for the white target to be segmented.

In addition, Equation (8) also applies to the black segmented target following the same proving process mentioned above, which completes the proof.

References

Guillaume Cerutti , Laure Tougne , Julien Mille , Antoine Vacavant , Didier Coquin , Understanding leaves in natural images –a model-based approach for tree species identification, Computer Vision and Image Understanding 117(10) (2013), 1482–1501.

Yuxiang Sun , Ming Liu and Max Meng

Q.-H

, Active perception for foreground segmentation: An RGB-d data-based background modeling method, IEEE Transactions on Automation Science and Engineering 16(4) (2019), 1596–1609.

Vicent Caselles , Ron Kimmel , Guillermo Sapiro , Geodesic active contours, International Journal of Computer Vision 22(1) (1997), 61–79.

Tsai

, Yezzi

and Willsky

A.S.

, Curve evolution implementation of the mumford-shah functional for image segmentation, denoising, interpolation, and magnification, IEEE Transactions on Image Processing 10(8) (2001), 1169–1186.

Yiyang Chen , Pengqiang Ge , Guina Wang , Guirong Weng , Hongtian Chen , An overview of intelligent image segmentation using active contour model, Intelligence & Robotics, 2023.

Stanley Osher Ronald Fedkiw , Level Set Methods and Dynamic Implicit Surfaces, Springer New York, October 2002.

Stanley Osher and James Sethian

, Fronts propagating with curvature-dependent speed: Algorithms based on hamiltonjacobi formulations, Journal of Computational Physics 79(1) (1988), 12–49.

Chunming Li , Chenyang Xu , Changfeng Gui and Martin Fox

, Distance regularized level set evolution and its application to image segmentation, IEEE Transactions on Image Processing 19(12) (2010), 3243–3254.

Chunming Li , Chiu-Yen Kao , Gore

J.C.

, Zhaohua Ding , Minimization of region-scalable fitting energy for image segmentation , IEEE Transactions on Image Processing 17(10) (2008), 1940–1949.

10.

Kaihua Zhang , Huihui Song , Lei Zhang , Active contours driven by local image fitting energy, Pattern Recognition 43(4) (2010), 1199–1206.

11.

Bin Han , Yiquan Wu , Active contour model for inhomogenous image segmentation based on jeffreys divergence, Pattern Recognition 107 (2020), 107520.

12.

Guirong Weng , Bin Dong , Yu Lei ,A level set method based on additive bias correction for image segmentation, Expert Systems with Applications 185 (2021), 115633.

13.

Pengqiang Ge , Yiyang Chen , GuinaWang , GuirongWeng ,An active contour model driven by adaptive local pre-fitting energy function based on jeffreys divergence for image segmentation, Expert Systems with Applications 210 (2022), 118493.

14.

Ri Jin , Guirong Weng ,A robust active contour model driven by pre-fitting bias correction and optimized fuzzy cmeans algorithm for fast image segmentation, Neurocomputing 359 (2019), 408–419.

15.

Chengxin Yang , Guirong Weng , Yiyang Chen ,Active contour model based on local kullback–leibler divergence for fast image segmentation, Engineering Applications of Artificial Intelligence 123 (2023), 106472.

16.

Gilles Aubert , Pierre Kornprobst ,Mathematical Problems in Image Processing, Springer New York, 2006.

17.

Ri Jin , Guirong Weng ,Active contours driven by adaptive functions and fuzzy c-means energy for fast image segmentation, Signal Processing 163 (2019), 1–10.

18.

Keyan Ding , Linfang Xiao , Guirong Weng ,Active contours driven by region-scalable fitting and optimized laplacian of gaussian energy for image segmentation, Signal Processing 134 (2017), 224–233.

19.

Guirong Weng , Bin Dong ,A new active contour model driven by pre-fitting bias field estimation and clustering technique for image segmentation, Engineering Applications of Artificial Intelligence 104 (2021), 104299.

20.

Xin Yan , Guirong Weng ,Hybrid active contour model driven by optimized local pre-fitting image energy for fast image segmentation, Applied Mathematical Modelling 101 (2022), 586–599.

21.

Tuan Nguyen

, Vedrana Dahl

, Andreas Barentzen

,Multi-phase image segmentation with the adaptive deformable mesh, Pattern Recognition Letters 117 (2019), 97–103.

22.

Yiyang Chen , Yingwei Zhou , Yueyuan Zhang ,Machine learning-based model predictive control for collaborative Production Planning Problem with Unknown Information, Electronics 10 (2021), 1818.