Opt2Ada: an universal method for single-image low-light enhancement

Abstract

This paper proposes that the task of single-image low-light enhancement can be accomplished by a straightforward method named Opt2Ada. It contains a series of pixel-level operations, including an optimized illuminance channel decomposition, an adaptive illumination enhancement, and an adaptive global scaling. Opt2Ada is traditional and it does not rely on architecture engineering, super-parameter tuning, or specific training dataset. Its parameters are generic and it has better generalization capability than existing data-driven methods. For evaluation, both the full-reference, non-reference, and semantic metrics are calculated. Extensive experiments on real-world low-light images demonstrate the superiority of Opt2Ada over recent traditional and deep learning algorithms. Due to its flexibility and effectiveness, Opt2Ada can be deployed as a pre-processing subroutine for high-level computer vision applications.

Keywords

Low-light image enhancement Image processing Traditional method

1 Introduction

In real world, most cameras capture images from sub-optimal lighting conditions including back-lit, non-uniform illumination, weak or extremely low-lighting, color cast, and intensive noise. These images usually suffer multiple degradation that is detrimental for high-level machine vision tasks, such as object detection [1], segmentation [2], and recognition [3]. Because high quality imaging is critical to these applications, low-light image enhancement (LLIE) is one of the fundamental tasks in image processing pipelines. In this domain, methods are versatile some of these are based on the histogram equalization [4, 5] and others may depend on the Retinex theory [6]. Recent years, the deep-learning (DL) based LLIE methods are prosperous since the first seminal work [7]. Due to fact that the convolutional neural networks (CNNs) can automatically learn intricate patterns from the input images, it is a good candidate for LLIE tasks. Traditional methods usually depend on particular hand-craft priors which limit their performance and flexibility. The performances of DL based methods heavily rely on their elaborately designed CNN architectures and carefully selected paired/unpaired large-scale training data. Moreover, learning-based methods may be regarded as the fitting of data which would be inevitably suffered from a lack of interpretability. This could bring difficulties in analyzing the nature of the LLIE itself so as to find potential cue for improvement. As a result, most of these existing CNN-based methods give rise to unsatisfactory visual results, when presented with real-world images with various light intensities and intensive noises [8].

To address the above issues, this work proposes Opt2Ada, an universal method to enhance low-light images. The method is traditional and is not data-driven. By integrating a series of procedures including an optimized illuminance channel decomposition, an adaptive illuminance enhancement, and an adaptive global scaling, a robust and flexible performance is obtained. Compared with the existing model-based or learning-based methods, the present method not only adapts to specified lighting condition of the input image, but also does not biased to certain dataset. Since the method is simple and effective, this makes it convenient to be developed by any programming language and imbedded into the pre-processing subroutines for high-level computer vision tasks. Particularly, taking conventional color space transformations between RGB and CIE XYZ into consideration, we first design an optimized transformation matrix that can directly separate the illuminance channel from the original RGB channels. Then according to the maximal pixel values, the low-light image is enhanced adaptively in a pixel-wise manner. Finally, the enhanced images are scaled by an adaptive global scaling to further eliminate overexposure and noise. To make an unbiased quantitative and qualitative evaluation of the present algorithm, we collect 94 images suffer inevitable noise and poor visibility from real environments to compose a low-light image dataset and our method is also evaluated on the well-known LOL dataset. we deploy Opt2Ada on various mobile phones, and find that a native CPU implementation can reach 3-9 frames per second. Contributions of the present work can be summarized as follows:

In contrast to existing CNN-based LLIE methods, the present method dose not contain any trainable parameters. Therefore, Opt2Ada will not be biased to specific training data.

The quantitative and qualitative evaluations guarantee the universality of the present method with superior performance with DL algorithms.

We propose a low-light image dataset that contains photos captured by different mobile devices under diverse illumination conditions to evaluate the generalization of Opt2Ada. The comparisons are also made between Opt2Ada and the existing methods.

We deploy Opt2Ada on various mobile devices and test it in the wild. Although multiple pixel-wise operations are involved, the algorithm can also achieve 3 to 9 frames per second (FPS).

The remainder of this paper is organized as follows. Section 2 summarizes the recent literature on progresses of LLIE including both conventional and DL methods. Section 3 presents the overall algorithms design. After the experimental results described in Section 4, the algorithm is deployed and tested on mobile devices. The conclusions are drawn in the final section.

2 Related works

Nowadays, algorithms for LLIE can be divided into two categories. The first category which has received relatively more attention is built upon the decomposing and reassembling the input image pixels. In this aspect, the illumination component is decomposed from the input image by properly enforced priori or regularization. After improving the illuminance part and recombining it with the remaining parts, the image with enhanced brightness is obtained. Most of these decomposition regularization are based on the Retinex theory [6], which assumes that an image is composed of a reflection component with an illumination component. In this way, the proposed algorithms include traditional methods such as, HSV consistent model [9], SRIE [10], LIME [11], BIMEF [12], and LR3M [13] as well as the DL methods such as, RetinexNet [14], RUAS [8], RetinexDIP [15]. Instead of the Retinex theory, there are also certain DL methods that directly use sophisticatedly designed CNNs to separate and enhance the illumination component simultaneously. In an end-to-end manner, the input image can be decomposed, enhanced, and reconstituted by auto-encoder [7], multiple branch CNNs [16], generative adversarial networks [17], and U-net based Retinex decomposition [18]. Furthermore, efforts are also devoted to design new attention model and learning paradigm, such as signal-to-noise-ratio awareness network [19] and the self-calibrated illumination learning framework [20].

On the other hand, the second category includes methods that enhance the images in a pixel-wise manner without decomposing and reassembling operations. The most representative conventional methods, such as DHECI [21] and LDR [22], are built upon the well-known histogram equalization and they also include additional priors and constraints. Adaptive histogram equalization method intends to improve the contrast [23], and an advanced version of it, i.e. contrast limited adaptive histogram equalization method, is proposed afterward [24]. The weighted bi-histogram equalization uses the decomposed tiles based on the distributed area ratio [25]. In addition to natural image LLIE, alpha-rooting is one of the more sound histogram-based method for medical image enhancement [26]. Nevertheless, after the image has been manipulated, the gray-level of the enhanced image is reduced and certain details would be blurred. If certain images have peaks in the histogram, the contrast is unnaturally over-enhanced after the operation. Moreover, the intensity slicing algorithm is more restrictive to the lighting condition. It cannot be applied to all types of low-light images and different lighting conditions may need different procedures. There are also frequency domain enhancement methods, such as DCT [27]. Since the Fourier transformation is used, the frequency domain algorithms are complex and computationally slow. Visually, they may also lead to the loss of edges, blurring details, and creating ringing effect. As a result, both of these methods have obvious disadvantages and they are extremely sensitive to hand-craft super-parameters. For DL methods, CNNs can be designed to learn the curves for pixel value amplification. In this case, the proposed methods include ExCNet [28], Zero-DCE [29], and Zero-DCE++ [30].

3 Methodology

To provide a comprehensive understanding of our methodology, the pipeline of the proposed method can be divided into three successive procedures: an optimized illuminance channel decomposition, an adaptive illuminance enhancement, and an adaptive global scaling. These procedures will be elaborate in this section and the overall method is attribute as Opt2Ada for short. A detailed sketch of Opt2Ada is shown in Fig. 1.

Fig. 1

(Color online) Sketch of Opt2Ada. The method is composed by a series of operations which sequentially act on the input image.

3.1 Optimized illuminance channel decomposition

In our method, instead of using the Retinex theory [6] or a CNN [14], we use a linear transformation to do the illuminance channel decomposition. The input data is converted from its original format to a floating point representation of linear RGB values and normalized to [0, 1]. Since the real scene illuminance condition is usually unknown, we assume that a D₆₅ white point to convert tristimulus values between RGB and CIE XYZ format as the illuminance channel decomposition. The mapping of decomposition is denoted as $I$ whose formula is given as follows:

$(\begin{matrix} s_{ij}^{0} \\ s_{ij}^{1} \\ s_{ij}^{2} \end{matrix}) = T \cdot (\begin{matrix} r_{ij} \\ g_{ij} \\ b_{ij} \end{matrix}) .$ (1)

Here, r_ij, g_ij, and b_ij are the normalized pixel value in RGB channels with i, j being the location of the pixel, and $s_{ij}^{k}$ represents three separated XYZ channels with k = 0, 1, 2. The matrix $s_{ij}^{1}$ is attributed as the illuminance map. The transform matrix is given as,

$T = (\begin{matrix} 0.4124 & 0.3576 & 0.1805 \\ 0.2126 & 0.7152 & 0.0722 \\ 0.0193 & 0.1192 & 0.9505 \end{matrix}) .$ (2) This transformation matrix can be obtained by calculating the three primary color coordinates and the reference white point coordinates. Different reference white point coordinates gives rise to the different transformation matrix [31]. The effectiveness of the present matrix will be illustrated via experiments.

3.2 Adaptive illuminance enhancement

After the decomposition of the illuminance component according to Equation (1), the illuminance of the input image is enhanced according to an adaptive logarithmic mapping [31]: $\begin{matrix} S_{ij}^{1} = \frac{1}{{log}_{10} (s_{\max}^{1} + 1)} \frac{log (s_{ij}^{1} + 1)}{log [2 + 8 {(\frac{s_{ij}^{1}}{s_{\max}^{1}})}^{\frac{log (b)}{log (0.5)}}]}, \end{matrix}$ (3) where b is a parameter controlling the shape of the curve, and $S_{ij}^{1}$ denotes the enhanced illuminance and $s_{\max}^{1}$ is the maximum illuminance of the input image. Equation (3) is not derived from other formula but the main assumption of the present method. The principal characteristic of Equation (3) is an adaptive adjustment of logarithmic base depending on each pixel’s radiance. The Perlin and Hoffert “bias” power function is introduced for smooth interpolation among logarithmic bases [31]. As a matter of fact, bias becomes a standard tool of texture synthesis and is also used for many different tasks in computer graphics. Here, the bias function is a power function defined over the unit interval as $t^{\frac{log (b)}{log (0.5)}}$ , with b being an intuitive parameter remaps an input value to a higher or lower value.

To explore the behavior of Equation (3), we plot it for various values of b and $s_{\max}^{1}$ in the left panel and right panel in Fig. 2, respectively. As can be seen from the left panel, the shape of the curve becomes more and more closer towards the upper left corner as the b decreases, while it tends to be diagonal as the b increases. This indicates that with the decrease of b, the curve will have more room to amplify the brightness. On the other hand, the shape of the curve in the right panel of Fig. 2 will also change according to the different values of $s_{\max}^{1}$ . For changing $s_{\max}^{1}$ , the tendency of the curve is almost the same as that of changing b, that is, when $s_{\max}^{1}$ decreases, the curve has stronger ability to amplify the brightness of the input image. This is a manifestation of Equation (3) which is called adaptive. It can change the shape autonomously according to the brightness of the input image to adaptively enhance the illuminance.

Fig. 2

(Color online) Visualization of Equation (3) for various values of b in the left panel and various values of $s_{\max}^{1}$ in the right panel. Here, we fix b = 0.85 in the right panel.

The image can be transformed from the XYZ format back to the conventional RGB format via the following equation,

$(\begin{matrix} r_{ij} \\ g_{ij} \\ b_{ij} \end{matrix}) = T^{- 1} \cdot (\begin{matrix} x_{ij} \\ y_{ij} \\ z_{ij} \end{matrix})$ (4) where, x_ij, y_ij, and z_ij are defined as

$\begin{matrix} x_{ij} = & \frac{S_{ij}^{1}}{{\bar{s}}_{ij}^{0} {\bar{s}}_{ij}^{1}} \\ y_{ij} = & S_{ij}^{1} \\ z_{ij} = & \frac{S_{ij}^{1}}{{\bar{s}}_{ij}^{1} (1 - {\bar{s}}_{ij}^{0} - {\bar{s}}_{ij}^{1})} . \end{matrix}$ (5)

In Equation (5), ${\bar{s}}_{ij}^{k} = \frac{s_{ij}^{k}}{\sum_{k} s_{ij}^{k}}$ for k = 0, 1, 2 are the normalized pixel values in the XYZ channels. The outputs of the above equations are further scaled as

$y = {\begin{matrix} 2.64 x & x < 0.05 \\ 1.099 x^{\frac{0.9}{2.2}} - 0.099 & otherwise \end{matrix}$ (6) Equation (6) is based on the ITU-R BT.709 gamma correction but with certain modification in its parameters. Though Equations (6), an output image with improved brightness is obtained. It is also found from Fig. 2 that certain portion of the curve have exceed the upper boundary values allowed for image pixels. In our method, the exceed value will be truncated to one. This usually results in the overexposure for the output image. Nevertheless, this can be well solved by a following adaptive scaling operation explained in the next subsection.

3.3 Adaptive global scaling

By experiments, it is found that if we only use the operation present in the previous section to amplify the brightness, the resulting image will be suffered by inevitably overexposure and boosted noise. As shown in middle panel of Fig. 3, although the brightness is greatly improved, the visual quality of the enhanced image is not satisfied. It can also be seen from the first line in Fig. 3 that the brightness of the area in the red box is relatively high due to the presence of a street lamp. After the adaptive illuminance enhancement, although the formula is adaptive, the red box area is still enhanced. This leads to an overexposure that makes the enhanced image visually unsatisfactory. In addition, nighttime images usually contain large noise domains and strong noise intensity. From the second line of Fig. 3, we can also see that if the input image contains a large noise region, the adaptive illuminance enhancement will also amplify the noise giving rise to a large artifact block.

Fig. 3

(Color online) Outputs of adaptive illuminance enhancement procedure (middle panel) and adaptive global scaling (right panel).

To make further adjustment, we propose an adaptive global scaling operation which allows processing of the enhanced image in the previous section to eliminate overexposure and noise improving the visual quality. The essence of the adaptive global scaling is to assume that the regions with pixel value close to 1 correspond to the overexposure region, while the regions with pixel value close to 0 are regarded as the noise region. The pixels in these two regions should be cutoff, and then the remaining pixel values of the whole image are stretched to re-populate the whole range between 0 to 1. To this end, we first manually set the high and low ratios L_high∖low ∈ [0, 1] for the input image pixel truncation. For example, if we set L_high = 0.8, the upper 20% pixel values will be scaled. The selection of these ratios are empirical, which needs to be determined by experiments. When L_high and L_low are selected, the pixel values of the original image are truncated accordingly. Then the truncated image is scaled by following steps. The overall scaling is denoted by $S$ and its first step is according to the formula:

$y = x \cdot P_{1},$ (7) where x and y denote the input and output images, and $P_{1} = \frac{L_{high}}{P_{high}}$ is the global scaling factor with P_high being the pixel value corresponding to the higher ratio L_high. The second step reads

$y = \frac{x - P_{low}}{1 - L_{low}},$ (8) where P_low is the pixel value corresponding to the lower ratio L_low. A sketch of adaptive global scaling is plotted in Fig. 4. Notice that Equations (8) not only depends on the priori cutoffs but also related with the pixels itself. In this way, the method is apparently adaptive. It can be seen from Fig. 3 that the effect of adaptive global scaling is evident, which can remove the overexposure and noise giving an appropriate visuality.

Fig. 4

(Color online) Sketch of adaptive global scaling.

4 Experimental evaluation

In this section, we are in a position to verify the universality and effectiveness of Opt2Ada on real nighttime images via intensive experiments. For datasets, we chose the well known toy dataset LOL, as well as a custom real scene dataset collected and arranged by authors. For the comparison with existing methods, we select a state-of-the-art traditional algorithm LIME [11], a zero-shot-learning DL algorithm (Zero-DCE++) [30], and a non-reference DL algorithm (RetinexDIP) [15]. In the evaluation, both quantitative comparisons and qualitative visualizations are made. In quantitative comparisons, various metrics are used including full-reference metrics: SSIM (structural similarity index measurement), PSNR (peak signal-to-noise ratio), MSE (mean square error), and MAE (mean absolute error), non-reference metrics: NIQE (naturalness image quality evaluator), PI (perceptual index), and AB (average brightness), and semantic metrics: SPAQ (Smartphone Photography Attribute and Quality).

4.1 Dataset and Metrics

The datasets used in this paper are elaborated as follows:

LOL 1 . LOL [14] is the first paired low-/normal-light image dataset taken in real scenes. The low-light images are collected by changing the exposure time and ISO. LOL contains 500 pairs of low-/normal-light images of size 600×400 saved in RGB format. In our experiments, we use the LOL-test set which contains 15 paired low-/normal-light images.

Real-Scene 2 . In order to verify the universality of Opt2Ada, that is, the algorithm itself and the parameters in the algorithm are applicable to any real world scenes without fine-tuning or parameter adjustment, we collect 94 images in real nighttime environments over six different cities of China. These images cover multiple scenes, such as: indoor, country field, inside building corridor, residential square, street scene, etc. The images are taken using the mobile phone cameras with variety resolutions, ranging from 4000×3000, 1920×873 to 1368×1824 saved in RGB format. The proposed dataset is only for testing and is named as Real-Scene. During the collection, the following inclusion and exclusion criteria are used. The mobile phone should be kept fixed spatially and not shaken during shooting to avoid the image blur. The visual angle of the shooting is in normal human height, looking horizontally, up, or down at the front. The ISO is automatically adjusted by the cell phone itself and the images are native without any cell phone’s built-in processing software. We present several samples of Real-Scene in Fig. 5.

Fig. 5

(Color online) Several images sampled from the proposed Real-Scene dataset. The images are taken by different mobile devices under diverse lighting conditions and scenes.

Quantitative evaluation of a LLIE algorithm is more intricate then evaluating a high-level task and one can not get a conclusion from a single indicator such as mean average precision for object detection algorithms. Therefore, various metrics are used including full-reference metrics: SSIM, PSNR, MSE, and MAE, non-reference metrics: NIQE [33], PI [34], and AB, and semantic metrics: SPAQ [35]. The SPAQ includes three factors which stand for baseline model (SPAQ-BL), image attributes (SPAQ-IA), and scene semantics (SPAQ-SS) [35]. The details of these indicator are elaborated in a recent survey [32], and we do not intend to reproduce them here.

4.2 Implementation details

In the experimental evaluation, Opt2Ada is realized by python-3.6 with numpy-1.18.4 for matrix manipulation and opencv-4.2.0 for image reading and writing. The experiment is performed on a computer with ubuntu16.04, Intel i5-9400F CPU, and 8GB memory. We use a same set of parameters in which b = 0.25, L_high = 0.98, and L_low = 0.02 over all of our experiments. The input images keep their original resolutions. The pseudo-code for implement Opt2Ada is provided in Algorithm 1.

Algorithm 1

Pseudo code for Opt2Ada.
Require: Low-light single image in RGB color space x_RGB;
Ensure: Enhanced single image in RGB color space X_RGB;
1: Convert RGB channel to XYZ illumination channel: $I (x_{RGB}) \leftarrow x_{RGB}$ ;
2: illumination channel enhancement: $\tilde{I} (x_{RGB}) \leftarrow I (x_{RGB})$ ;
3: Adaptive global scaling: $S (\tilde{I} (x_{RGB})) \leftarrow \tilde{I} (x_{RGB})$ ;
4: Convert illumination channel to RGB channel: $X_{RGB} \leftarrow {\tilde{I}}^{- 1} (S (\tilde{I} (x_{RGB})))$ ;

4.3 Results and discussions

We provide the evaluation results of the present method on the LOL-test set in Fig. 6 and Table 1. From visualizations shown in Fig. 6, one can see that Opt2Ada gives rise to the output images closest to the ground truth (GT) images visually. The quantitative results of full-reference metrics and non-reference metrics mentioned in Section 4.1 are given in Table 1. As can be seen from full-reference metrics as provided in Table 1, Zero-DCE++ is the best of all whose MAE, MSE and PSNR outperform other methods by a wide margin. Except for Zero-DCE++, MAE, MSE and PSNR of LIME is slightly better than RetinexDIP and Opt2Ada. Whereas, Opt2Ada outperforms LIME and RetinexDIP in terms of SSIM and AB. Especially in the aspect of brightness improvement, the AB of the output images of Opt2Ada is closest to that of GT images (122.2). For the non-reference metrics: NIQE, PI, and SPAQ, a different picture is shown. The NIQE of Opt2Ada is better than other methods by 12.9%, 12.1%, and 8.3% for LIME, Zero-DCE++, and RetinexDIP, respectively. For PI, RetinexDIP gives rise to the lowest value. For SPAQ, there are three sub-indexes: SPAQ-BL, SPAQ-IA, and SPAQ-SS. Here, SPAQ-BL estimates image quality by a baseline model (residual network 50), SPAQ-IA works for by input image attributes, and SPAQ-SS accounts for input image semantic information. It is shown that Opt2Ada better than other methods in SPAQ-BL and SPAQ-SS, and achieves a nearly equal performance in SPAQ-IA.

Fig. 6

(Color online) Visual results of different methods on LOL-test.

Table 1

Quantitative comparisons on LOL-test in terms of SSIM, PSNR, MSE, MAE, NIQE, PI, SPAQ, and AB

	LIME	Zero-DCE++	RetinexDIP	Opt2Ada
MSE↓	100.81	2.41	105.64	107.08
MAE↓	61.86	12.26	87.95	118.02
PSNR↑	28.19	65.78	27.91	27.88
SSIM↑	0.62	0.99	0.36	0.69
NIQE↓	8.09	8.02	7.69	7.05
PI↓	8.95	8.84	6.46	8.83
SPAQ-BL↑	55.28	55.75	54.84	56.57
SPAQ-IA↑	54.41	54.31	54.17	54.18
SPAQ-SS↑	54.32	54.48	53.98	55.51
AB↑	71.07	121.33	30.6	122.13

In Fig. 7, and Tables 3, we provide the quantitative comparison and visualization results between various methods on the Real-Scene dataset. Different from the previous toy dataset LOL, note from Table 2 that Opt2Ada shows promoting performance in NIQE, PI, SPAQ, and AB. To be specific, NIQE of our method is reduced by 42%, 0.4%, 2.5%, 4.7% relative to the input image, LIME, Zero-DCE++, and RetinexDIP, respectively. Moreover, we also find that PI of the enhanced image obtained by both the present and the existing algorithms is higher than the input image. This means that the increase in brightness inevitably leads to a degeneration in PI. It is found that both LIME and Opt2Ada outperform the DL methods. The SPAQ-BL, SPAQ-IA, and SPAQ-SS for Opt2Ada are increased by 0.4%, 0.37%, and 0.7% with respect to RetinexDIP, while 5.3%, 8.5%, and 6.9% with respect to input images. This means that the proposed method can obtain a nearly equal SPAQ with respect to RetinexDIP, while the evidently improvement is achieved with respect to input images. Compared with LIME, indicators SPAQ-BL and SPAQ-IA have decreased, but SPAQ-SS of Opt2Ada is still the highest of all. Therefore, Opt2Ada outperforms other algorithms in restoring semantic information of images. This is also consistent with the results on LOL-test set. For the brightness, the proposed method can achieve a AB that is closest to the normal lighting scene, while the AB obtained by other algorithms can only achieve about half of the normal light AB value.

Fig. 7

(Color online) Visual results of different methods on Real-Scene dataset.

Table 2

Quantitative comparisons on Real-Scene dataset in terms of NIQE, PI, SPAQ, and AB

	Input	LIME	Zero-DCE++	RetinexDIP	Opt2Ada
NIQE↓	5.591	3.9509	4.0348	4.1226	3.9369
PI↓	6.4983	8.5224	8.3984	7.4991	8.4837
SPAQ-BL↑	50.137	52.5178	51.33	51.403	52.6519
SPAQ-IA↑	49.867	53.195	52.2165	52.3289	53.1618
SPAQ-SS↑	49.9306	52.7815	51.874	51.946	53.0076
AB↑	19.23	64.4255	65.01	52.97	112.6

Table 3

Quantitative comparisons on Real-Scene dataset in terms of EME, AME, SDME, Visibility, TDME

	Input	LIME	Zero-DCE++	RetinexDIP	Opt2Ada
EME↑	37.62	46.51	46.65	48.79	50.37
AME↑	19.45	22.52	22.39	24.49	24.65
SDME↑	47.67	59.51	60.21	63.32	63.72
Visibility↑	0.37	0.43	0.42	0.45	0.46
TDME↑	0.17	0.17	0.19	0.21	0.19

Moreover, one notes from Table 3 that Opt2Ada also shows promoting performance in EME, AME, SDME, Visibility, TDME. The definition of these metrics can be found in Ref. [36] and we would not reproduce them here. EME of our method is reduced by 33%, 8.3%, 7.9%, 3.2% relative to the input image, LIME, Zero-DCE++, and RetinexDIP, respectively. Since in Table 2 we have found that Opt2Ada outperforms in restoring semantic information of images, it may also effective in restoring image details. The AME, SDME, and Visibility for Opt2Ada are increased by 0.4%, 0.6%, and 2.2% with respect to RetinexDIP, while 26%, 33%, and 24% with respect to input images. Compared with RetinexDIP, indicators TDME have decreased, but EME, AME, SDME and Visibility is still the highest of all. This means that the proposed method can achieve almost the same detail enhancement performance as RetinexDIP, while the evidently improvement is achieved with respect to input images.

On the other hand, a similar tendency can be found from the visualization results in Fig. 7. In the Real-Scene dataset, we can see that Zero-DCE++ can achieve almost similar enhancement results to LIME, and RetinexDIP gives rise to enhancement results which are featured by lower brightness, color blocks, and artifacts. In Fig. 8, we present several test results for the high-contrast images in Real-Scene dataset. It is shown that the present method also outperforms other method on these images. Therefore, the proposed method not only enhances the brightness to an appropriate level without introducing non-existent color blocks, but also reveals certain details that exist in the original image and not clearly seen before.

Fig. 8

(Color online) Several visual results of high-contrast images for different methods.

4.4 Mobile device deployment

For evaluating the running speed of Opt2Ada and testing its performance in the wild, we implement Zero-DCE++ and Opt2Ada using C++ and deploy on mobile phones. We use Android Studio with NDK to compile C++ code, and only realized a CPU version of the code. Tencent’s NCNN library is used when decode the image reading from mobile phone camera [37]. For Zero-DCE++, we first convert the pytorch model to the ONNX (Open Neural Network Exchange) format [38], then deploy it using NCNN library. As a matter of fact, Opt2Ada includes six pixel-wise operations so that it will be very time consuming without parallel computation or non-using graphics processing unit (GPU). The results are provided in Table 4 in which FPS is measured when the application runs stably for five minutes. It can be seen that the value of FPS varies significantly from 3.60 to 9.14 with different computation capacity of CPUs on these devices. For example, the Huawei’s Nova 4e has only achieved 3.6 FPS, while the Honor 60 can nearly double the running speed of the Nova 4E. Compared with Zero-DCE++, Opt2Ada can run 2 to 4 times faster. As a result, for all types of modern mobile phone, the running speed of Opt2Ada can reach nearly real-time. A video demonstration of Opt2Ada on mobile device is provided on-line and the application will be public available soon 3 .

Table 4
FPS of Zero-DCE++ and Opt2Ada on various mobile devices

Device Zero-DCE++ Opt2Ada

Nova 4e 1.14 3.60

Honor 60 2.23 9.14

Honor play4T 2.78 5.8

Honor X10 2.83 5.11

Vivo X60 6.05 7.85

Oppo reno 1.37 3.71

Device	Zero-DCE++	Opt2Ada
Nova 4e	1.14	3.60
Honor 60	2.23	9.14
Honor play4T	2.78	5.8
Honor X10	2.83	5.11
Vivo X60	6.05	7.85
Oppo reno	1.37	3.71

Visual results of Zero-DCE++ and Opt2Ada tested on various mobile devices are provided in Fig. 9. Here, we test using images captured in real time from the cell phone’s camera, rather than using stored images. It is shown that when algorithms are tested in the real environment, the visual results are closed to those on Real-Scene dataset but quite different from those on LOL dataset. Although the qualities of image captured by different cell phones are different, the brightness enhancement of Opt2Ada is significant than that of Zero-DCE++. A stable performance across different scenario and device is achieved. These testing results not only illustrate the superior of Opt2Ada, but also tell us that toy datasets have great limitations in the evaluation of LLIE algorithms, that is, methods or models that perform well on toy datasets may not perform well in the wild.

Fig. 9

(Color online) Visual results of Zero-DCE++ and Opt2Ada on various mobile devices.

5 Conclusion

In this study, a new LLIE algorithm, Opt2Ada, is proposed by integrating an optimized illuminance channel decomposition, an adaptive illuminance enhancement, and an adaptive global scaling. Without using the Retinex theory, we first establish the illumination map by a transformation matrix which directly isolates the illuminance channel from the RGB channels. Then we adapt a pixel-dependent log shape curve that enhance the illumination map adaptively. Finally, the enhanced illumination map is reunion with the other part of the input image by an inverse transformation matrix. To eliminate overexposure, noise, and artifacts, the image is further scaled by an adaptive global scaling. Our experiments are performed on an existing benchmark dataset LOL as well as on an custom dataset. Various metrics are included and comparison are also made among traditional and DL methods. It is shown that the proposed algorithm outperforms traditional and DL algorithms in both toy dataset and real dataset resulting not only state-of-the-art quantitative results but also applicable visualizations. Additionally, the proposed algorithm is not data-driven and its parameters are universal. This has given rise to a generalization capability better than that of the data-driven methods. As a result, it can be straightforwardly integrated into the pre-processing subroutine for advanced machine vision tasks. For running speed evaluation, we deploy Opt2Ada on various mobile devices. Without parallel computation, Opt2Ada can run on CPU with 3.60 to 9.14 FPS. When tested in the wild, Opt2Ada shows a stable and effective performance better than DL models over various mobile devices and scenarios. There is also a large space for optimization. In the future, we are going to make use of Vulkan [39] to accelerate the algorithm by utilizing the GPUs of the mobile phone.

Disclosures statement

No conflict of interest exists in the submission, and it is approved by all authors for publication. We declare that the work presented is original research that has not been published previously and is not under consideration for publication elsewhere, in whole or in part.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 11604240) and the Scientific Research Project of Tianjin Municipal Education Commission under Grant (No. 2019KJ231).

Footnotes

This dataset can be downloaded from URL

This dataset can be downloaded from URL (code: d5lc)

References

Liu

, Ouyang

, Wang

, Fieguth

, Chen

, Liu

and Pietikäinen

, Deep learning for generic object detection: Asurvey, International Journal of Computer Vision 128(2020), 261–318.

, Wang

, Gao

, Yu

, Shen

and Sang

, Context Prior for Scene Segmentation, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 12416–12425.

Zhang

, Liu

and Xiong

, Two-stream action recognition-oriented video super-resolution, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, 8799–8808.

Ibrahim

and Kong

N.S.P.

, Brightness preserving dynamic histogram equalization for image contrast enhancement, IEEE Transactionson Consumer Electronics 53 (2007), 1752–1758.

Abdullah-AI-Wadud

, Kabir

M.H.

, Dewan

M.A.A.

and Chae

, Adynamic histogram equalization for image contrast enhancement, IEEE Transactions on Consumer Electronics 53 (2007), 593–600.

Land

E.H.

, The retinex theory of color vision, ScientificAmerican 237 (1977), 108–129.

Lore

K.G.

, Akintayo

and Sarkar

, Llnet: A deep autoencoder approach to natural low-light image enhancement, Pattern Recognit 61 (2017), 650–662.

Liu

, Ma

, Zhang

, Fan

and Luo

, Retinex-inspired unrolling with cooperative prior architecture search for lowlight image enhancement, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 10561–10570.

Prasath

V.B.

, Thanh

D.N.

, Thanh

L.T.

, San

N.Q.

and Dvoenko

, Humanvisual system consistent model for wireless capsule endoscopy imageenhancement and applications, Pattern Recognit. Image Anal 30 (2020), 280–287.

10.

, Zeng

, Huang

, Zhang

and Ding

, A weighted variational model for simultaneous reflectance and illumination estimation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 2782–2790.

11.

Guo

, Li

and Ling

, Lime: Low-light image enhancement via illumination map estimation, IEEE Transactions on Image Processing 26 (2017), 982–993.

12.

Ying

, Li

, Ren

, Wang

and Wang

, A new low-light image enhancement algorithm using camera response model, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, 3015–3022.

13.

Ren

, Yang

, Cheng

W.H.

and Liu

, LR3M: Robust low-light enhancement via low-rank regularized retinex model, IEEE Transactions on Image Processing 29 (2020), 5862–5876.

14.

Wei

, Wang

, Yang

and Liu

, Deep retinex decomposition for low-light enhancement, Proceedings of the Proceedings of the British Machine Vision Conference, 2018.

15.

Zhao

, Xiong

, Wang

, Ou

, Yu

and Kuang

, RetinexDIP: Aunified deep framework for low-light image enhancement, IEEE Transactions on Circuits and Systems for Video Technology 32(2021) (2021), 1076–1088.

16.

, Lu

, Wu

and Lim

, MBLLEN: Low-light image/video enhancement using CNNs, British Machine Vision Conference 220(1) (2018), 4–16.

17.

Jiang

, Gong

, Liu

, Cheng

, Fang

, Shen

, Yang

, Zhou

and Wang

, EnlightenGAN: Deep light enhancement without paired supervision, IEEE Transactions on Image Processing 30 (2021), 2340–2349.

18.

, Weng

, Zhang

, Wang

, Yang

and Jiang

, URetinex-Net: Retinex-based Deep Unfolding Network for Low-light Image Enhancement, Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022.

19.

, Wang

, Fu

C.W.

and Jia

, SNR-Aware Low-light Image Enhancement, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 17714–17724.

20.

, Ma

, Liu

, Fan

and Luo

, Toward Fast, Flexible, and Robust Low-Light Image Enhancement, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 5637–5646.

21.

Nakai

, Hoshi

and Taguchi

, Color image contrast enhancement method based on differential intensity/saturation gray-levels histograms, Proceedings of the 2013 International Symposium on Intelligent Signal Processing and Communication Systems, Naha, Japan, 2013, 445–449.

22.

Lee

and Kim

, Contrast enhancement based on layered difference representation of 2D histograms, IEEE Transactions on ImageProcessing 22 (2013), 5372–5384.

23.

Stark

J.A.

, Adaptive image contrast enhancement using generalizations of histogram equalization, IEEE Transactions onImage Processing 9(5) (2000), 889–896.

24.

Zuiderveld

, Contrast Limited Adaptive Histogram Equalization, Learning from data: P. Heckbert: Graphics Gems IV, Academic Press, 1994.

25.

Trongtirakul

and Phanthuna

, Image enhancement using weighted bi-histogram equilization, Journal of Applied Mathematics and Informatics 15 (2021), 98–101.

26.

Agaian

S.S.

, Silver

and Panetta

K.A.

, Transform coefficient histogram-based image enhancement algorithms using contrast entropy, IEEE Transactions on Image Processing 16(3) (2007), 741–758.

27.

Agaian

and Arslan.

, Two transform-based image enhancement methods. Int. Signal Processing Conf, Dallas, TX, 2003.

28.

Zhang

, Zhang

, Liu

, Shen

, Zhang

and Zhao

, Zero-shot restoration of back-lit images using deep internal learning, Proceedings of the 27th ACM International Conference on Multimedia, 2019, 1623–C1631.

29.

Guo

, Li

, Guo

, Loy

C.C.

, Hou

, Kwong

and Cong

, Zero-reference deep curve estimation for low-light image enhancement, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 1780–1789.

30.

, Guo

and Loy

C.C.

, Learning to enhance low-light image viazero-reference deep curve estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence 44(8) (2022), 4225–4238.

31.

Drago

, Myszkowski

, Annen

and Chiba

, Adaptive logarithmic mapping for displaying high contrast scenes, Eurographics 22(3) (2003), 419–426.

32.

, Guo

, Han

, Jiang

, Cheng

M.M.

, Gu

and Loy

C.C.

, Low-Light Image and Video Enhancement Using Deep Learning: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(12) (2022), 9396–9416.

33.

Mittal

, Soundararajanand

and Bovik

A.C.

, Making a ’Completely blind’ image quality analyzer. IEEE Signal Processing Letters 20 (2013), 209–212.

34.

, Yang

C.Y.

, Yang

and Yang

M.H.

, Learning a non-referenc equality metric for single-image super-resolution, Computer Vision and Image Understanding 158 (2017), 1–16.

35.

Fang

, Zhu

, Zeng

, Ma

and Wang

, Perceptual quality assessment of smartphone photography, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 3677–3686.

36.

Voronin

, Zelensky

and Sos

, 3-D block-rooting scheme with application to medical image enhancement, IEEE Access 9(2020) (2020), 3880–3893.

37.

https://github.com/Tencent/ncnn

38.

https://github.com/onnx/onnx

39.

https://www.vulkan.org

Opt2Ada: an universal method for single-image low-light enhancement

Abstract

Keywords

1 Introduction

2 Related works

3 Methodology

4.1 Dataset and Metrics

4.3 Results and discussions

Table 4 FPS of Zero-DCE++ and Opt2Ada on various mobile devices Device Zero-DCE++ Opt2Ada Nova 4e 1.14 3.60 Honor 60 2.23 9.14 Honor play4T 2.78 5.8 Honor X10 2.83 5.11 Vivo X60 6.05 7.85 Oppo reno 1.37 3.71

Disclosures statement

Acknowledgments

Footnotes

References

Table 4
FPS of Zero-DCE++ and Opt2Ada on various mobile devices

Device Zero-DCE++ Opt2Ada

Nova 4e 1.14 3.60

Honor 60 2.23 9.14

Honor play4T 2.78 5.8

Honor X10 2.83 5.11

Vivo X60 6.05 7.85

Oppo reno 1.37 3.71