A new methodology for monocular depth estimation with attention mechanisms

Abstract

Depth estimation from images is fundamental for autonomous navigation of robots, vehicles, drones, and integrating navigation aid systems for people with visual impairments. Despite the challenges of obtaining depth information from complex scenes, advancements in Deep Learning have opened new possibilities. Thus, this work introduces an approach based on recent Convolutional Neural Network architectures and attention mechanisms to enhance monocular image depth estimation, with potential applications in navigation aid systems for the visually impaired. The proposal focuses on implementing a Convolutional Neural Network model with an attention mechanism configuration that has not yet been tested in the literature, primarily integrating the Convolutional Block Attention Module and the Modified Global Context Network in the encoder and decoder, respectively. Unlike stereo camera-based systems, which require complex setups and image pairs, this model simplifies data collection and processing, although it still faces the challenge of requiring large datasets and significant computational capacity. However, it is experimentally possible to demonstrate that these limitations can be overcome by using reduced-resolution images and resizing techniques. The evaluation of the proposed model indicated satisfactory performance compared to state-of-the-art works that use images with resolutions identical to those in this work, validating the comparative tests. It presented an improvement in the Absolute Relative Error of 25.22% and 6.28% in relation to the Root Mean Squared Error. The results highlight the feasibility of conducting Deep Learning research, even with limited hardware resources.

Keywords

monodepth estimation deep learn visualy impaired people attention mechanisms convolutional block attention module global context network,convolutional neural Network

1. Introduction

Depth estimation technology is crucial for ensuring the safety and autonomy of modern autonomous vehicles¹ and drones. Beyond the automotive sector, depth maps play a vital role in various fields, including navigation assistance for visually impaired individuals, enhancing their ability to perceive their surroundings. The ability to analyze a scene in detail provides enhanced resources for decision-making.²

Depth estimation concepts date back to early computer vision theories, such as Wheatstone’s stereoscope and the exploration of motion parallax.³ Wheatstone’s groundbreaking work involved creating a stereoscope, revealing that the brain uses horizontal disparity to perceive relative depth. Another significant contribution came from Von Helmholtz,⁴ who explored depth perception through motion structure. They proposed that objects at varying distances exhibit different speeds when moving across the retinal surface, introducing the concept of motion parallax. This phenomenon allows the human brain to extract three-dimensional shapes from dynamic scenes. These fundamental theories laid the groundwork for the development of multi-view depth mapping.

While traditional methods like stereovision systems and motion parallax laid the groundwork, modern approaches have shifted towards monocular depth estimation using deep learning. This system leverages principles introduced by Wheatstone to estimate depth from multiple viewpoints. Similarly, another technique involves creating a depth map based on motion structure. In this scenario, a single camera is systematically moved with a constant baseline, resulting in the development of a depth map.⁵ These historical perspectives and methodologies paved the way for contemporary advancements in depth estimation techniques.⁶

However, the complexity of setting up stereo cameras for depth estimation has discouraged the use of these methods in real-world applications, leading to widespread adoption of monocular estimation. This simplifies the data acquisition process, eliminating the need for multiple images and the complexity associated with stereo matching.⁷

Monocular depth estimation simplifies data acquisition and processing, avoiding the complexities of stereo matching, making it a practical alternative for real-world applications. A notable work by Hoiem et al.⁸ reconstructed a three-dimensional scene in a virtual environment, using a manual approach based on features to categorize pixels based on shapes and colors. Other researchers, such as Karsch et al.⁹ and Ladicky et al.,¹⁰ applied the theories of Hoiem et al.⁸ to conduct depth analyses, referring to this process as “depth from semantic segmentation”.

Compared to parallax methods,^11,12 our neural network-based approach offers significant advantages. It is capable of generating depth maps from a single monocular image, which eliminates the need to capture multiple images from different angles. Furthermore, deep learning allows for adaptation to a wide range of scenarios and lighting conditions, potentially overcoming the limitations of traditional methods that heavily rely on the quality and alignment of stereoscopic images.

However, it is important to recognize that the neural network-based approach also has limitations, such as the need for large training datasets and the potential for limited generalization in scenarios that differ significantly from the training data. In contrast, parallax methods may be more robust in environments where capturing multiple images is feasible.

In summary, while parallax-based methods offer a traditional and straightforward solution for depth estimation, our neural network approach provides flexibility and the ability to handle monocular images, excelling in depth estimation across diverse and realistic scenarios.

Recent advancements in deep learning, such as Convolutional Neural Networks (CNNs) and attention mechanisms, have significantly improved the accuracy of monocular depth estimation,⁶ and advanced models have been studied in recent years in various ways for application in these tasks.^13–16

Finally, depth estimation is a powerful approach for the pre-training of deep networks^17,18 using image datasets.¹⁹ However, collecting large, diversified training datasets with accurate ground truth is a challenging task.^13,20

This study aims to implement a U-Net architecture integrated with attention mechanisms, specifically the Convolutional Block Attention Module (CBAM) and Global Context Network (GCNet), to enhance monocular depth estimation. The motivation is to develop a reliable method for integration into navigation assistance systems for visually impaired individuals.

Therefore, the suggested architecture is structured to incorporate, as its main components, the CBAM (Convolutional Block Attention Module) and GCNet (Global Context Network) attention mechanisms. This configuration was defined after a thorough, though not exhaustive, review of the specialized literature. Identifying the absence of U-Net architectures that simultaneously integrate these two specific attention mechanisms for the task of estimating depth maps from monocular images. This allowed for better results than the state-of-the-art cited in this work, as will be seen later.

A similar approach for feature selection, utilizing the attention mechanism, was employed by Xue et al.²¹ in their development of an external attention-based feature ranker for feature selection, highlighting the effectiveness of attention models in identifying important features across datasets. This connection further underscores the relevance of attention mechanisms in the context of our study.

The article employs an external attention mechanism inspired by concepts used in deep neural networks, particularly the Transformer model. External attention helps model the relationships between features without the need for the model to directly access the original data, which can be advantageous in terms of processing and efficiency. However, unlike the present study, the authors used Transformers originally applied in natural language processing (NLP) models and deep neural networks adapted to solve problems in other areas, such as feature selection, which is a crucial step in machine learning pipelines but requires more powerful hardware to train the model.

The paper is organized as follows: the introduction covers the key concepts of monocular depth estimation, followed by a comprehensive review of related works. In the fundamentals section, the Convolutional Block Attention Module (CBAM) and the Global Context Network (GCNet) are introduced, which form the basis of the proposed model. The methodology section details the implementation process of the model, while the experimental results section analyzes the performance metrics and outcomes. The ablation section explores the various tested configurations, providing an in-depth view of the model variations. Finally, the conclusion synthesizes the main contributions of the study.

2. Related works

This section reviews key state-of-the-art methodologies in monocular depth map estimation, highlighting significant advancements and their contributions to the field.

Eigen et al.²² pioneered the use of convolutional neural networks (CNNs) for predicting depth maps from single images. Their two-stage model, comprising a coarse and fine network, leveraged pre-trained weights from the ImageNet dataset to enhance performance.²³

Kumari et al.²⁴ introduced a residual encoder-decoder CNN with hourglass networks, enhancing depth estimation by analyzing encoded features at various scales. The inclusion of a perceptual loss function accelerated model convergence.²⁵

Alhashim and Wonka²⁶ used transfer learning with a Dense-Net encoder-decoder network, leveraging pre-trained weights for efficient feature extraction. Despite improved training time, the model faced challenges with memory usage and overfitting due to color augmentation.^23,27

Eigen and Fergus²⁸ extended the work of Eigen et al. by integrating semantic labels and surface normals into depth map estimation, using a deeper network based on VGG architecture. This approach improved performance but faced challenges with sharp transitions.^22,23,29

To address limitations in depth map sharpness, Seo³⁰ applied a signal-to-noise ratio (SNR) based method for edge and line detection, enhancing the quality of depth maps through post-processing.^31–33

3. Brief theorical foundation

The goal of monocular depth estimation is to obtain distance information from a single image of the scene. This challenge is significant due to the complex correspondence between pixels in a 2D image and infinite points in 3D space. Despite this complexity, humans are able to estimate distance by aggregating visual cues in the observed scene, even when using just one eye. This phenomenon suggests that visual cues contain distance information. Thus, monocular depth estimation can be formulated as the search for a mapping between visual cues and distance information of objects in the scene,³⁴ which can be defined as follows, equation (1):

D = f (v_{1}, v_{2}, \dots, v_{n})

(1)

where

D

is the distance,

v_{1}, v_{2}, \dots, v_{n}

are visual cues or information, and

f

is a nonlinear function. Visual cues such as scale, shadow, and texture are often incorporated into intermediate representations extracted by encoders, while decoders aggregate these features to predict distance information.

3.1. Attention mechanisms

The concept of attention, rooted in biological inspiration, plays a crucial role in tasks such as image classification³⁵ and machine translation. Prominent components, identified through a global analysis of the scene, hold greater relevance for the object category and demand more attention. In machine translation, certain words have a greater influence on the output, indicating that each component of the source impacts the target in a distinct manner, requiring differentiated treatment. The attention mechanism is not limited to modeling only the relevance between source and destination; it also has the capability to generate new representations, adjusted according to the weights assigned to each source component. From a technical standpoint, the attention model calculates coefficients through a query ( $Q$ ) and a set of keys ( $K$ ), resulting in new representations obtained by the weighted combination of values ( $V$ ) according to the coefficients,^34,36 equation (2).

A t t n (Q, K, V) = \sum_{i} p (σ (k_{i}, Q)) * υ_{i}

(2)

where

p

is the distribution function and

σ

is the alignment function.

In Convolutional Neural Networks (CNNs), the incorporation of attention mechanisms is achieved through one of two methods: post-hoc analysis of the network and the use of trainable attention mechanisms. The former technique has been predominantly employed to access the network’s intrinsic reasoning in the context of object visual recognition.^29,37 Similarly, trainable attention is subdivided into hard attention and soft attention.

Hard attention, characterized by a stochastic process, poses significant challenges to model training due to the iterative proposal of region and cropping.³⁸ These models often lack differentiability and require reinforcement learning for parameter refinement. Recently, hard attention has been extensively adopted in trainable transformers, especially due to its application in the iterative proposal of region and cropping during feature learning.⁶

Soft attention, characterized by a deterministic process, uses conventional backpropagation. These attention mechanisms eliminate the need for Monte Carlo or repeated random sampling. Soft attention is employed to highlight only the relevant activations during training, resulting in the reduction of computational resources spent on irrelevant activations and promoting more effective network generalization. Recently, additive soft attention has been applied in tasks such as image categorization and sentence-to-sentence translation.^6,39,40 Mathematically, soft attention can be formally expressed as shown in equation (3)

E_{p (S_{t} | α)} [Z_{t}] = \sum_{i = 1}^{L} α_{t},_{i} α_{i}

(3)

where

E_{p (S_{t} | α)}

denotes soft deterministic attention,

S_{t}

is the attention location,

Z_{t}

is a context vector,

α

represents the weights, and

L

corresponds to features extracted at a different image location

i

The expectation of the content vector $z_{t}$ is directly computed and deterministic attention is modeled by calculating a soft attention-weighted annotated vector $\sum_{i = 1}^{L} α_{t},_{i} α_{i}$ , as proposed by Bahdanau.⁴¹ This process introduces a softly weighted context vector $α$ , into the system. The attention mechanism is learned end-to-end through conventional backpropagation, as the entire model is differentiable.⁴²

3.1.1. GCNet attention mechanism

The non-local networks have played a crucial role in providing solid intuition and establishing the foundations for various contemporary attention mechanisms employed in deep neural network architectures geared towards computer vision.⁴³ They are essential in modeling the attention map of an individual pixel by aggregating relational information from its neighboring pixels. This process is achieved through a reduced number of permutation operations, enabling the construction of the attention map with emphasis on the query pixel.⁴⁴ From an abstract perspective, this approach demonstrates some similarity with the proposed Self-Attention Mechanism in SAGAN by Zhang et al.⁴⁵

As mentioned earlier, the development of GCNet was influenced by Non-local Networks and Squeeze-and-Excitation Networks, aiming to model an attention mechanism that enables the network to capture long-range dependencies at substantially reduced cost. This model is called the Global Context Network (GCNet). More details about non-local networks (Figure 1) and squeeze-and-excitation can be found in Cao et al.,⁴³ Hu et al.,⁴⁶ and Wang et al.⁴⁴

Figure 1.

Architecture of the non-local block (a) and its simplified version (b). Feature maps are presented by their dimensions, for example, C $\times$ H $\times$ W. $\otimes$ represents matrix multiplication, and $\oplus$ represents element-wise broadcasting addition. Source: Cao et al.⁴³

3.1.2. CBAM attention mechanism

The Convolutional Block Attention Module (CBAM) presents itself as a lightweight and versatile solution,⁴⁷ seamlessly integrable into various Convolutional Neural Network (CNN) architectures and amenable to end-to-end training alongside the base CNN.⁴⁸ As illustrated in Figures 2, from an intermediate feature map, attention weights are sequentially inferred along spatial and channel dimensions, being multiplied by the original feature map to perform adaptive adjustments. The input feature map is summed with the fully connected shared output features, followed by a sigmoid activation operation to generate the final channel attention feature map. This feature map, generated by the channel module, is then utilized as input for the subsequent module, where the spatial attention feature is generated through the sigmoid function. Finally, this feature is multiplied by the module’s input, resulting in the final generated feature.⁴⁹

Figure 2.

Representative block of the CBAM attention mechanis.

Figure 3.

Representative block of the Modified GCNet attention mechanism. Sigmoid was used instead of softmax. The GCNet has the first part NL block simplified and the second part comprises a piece of the squeeze-excitation mechanism, followed by a spatial attention mechanism.

4. Methodology

The Convolutional Block Attention Module (CBAM) attention mechanism is a sequential attention mechanism consisting of two main components: channel attention and spatial attention. The goal of CBAM is to enhance important features and suppress less useful ones both in the spatial and channel domains to improve the performance of convolutional neural networks.^34,50,51 \hyperlink{alg:cbam}{Algorithm 1} shows the implementation of the CBAM attention mechanism at a high level.

Channel Attention input: Feature map $F$ with dimensions $H \times W \times C$ , where $H$ is height, $W$ is width, and $C$ is the number of channels.

Average-Pooling: Apply average pooling along the spatial dimensions $H$ and $W$ to produce a $1 \times 1 \times C$ feature vector $F_{a v g}$ .

Max-Pooling: Apply max pooling along the spatial dimensions H and W to produce a $1 \times 1 \times C$ feature vector $F_{m a x}$ .

MLP with Weight Sharing: Apply a MultiLayer Perceptron (MLP) with a hidden layer on $F_{a v g}$ and $F_{m a x}$ . The MLP for both is shared. The MLP has the same input and output dimension, $C$ .

Sum and Activation: Sum the outputs of the MLP applied to $F_{a v g}$ and $F_{m a x}$ , and apply the sigmoid activation function to obtain the channel attention map $M_{c}$ of dimension $1 \times 1 \times C$ .

Multiply: Multiply the channel attention map $M_{c}$ by the original feature map $F$ to obtain $F^{'}$ with channel attention applied.

Spatial Attention input: Feature map $F^{'}$ with channel attention already applied.

Concatenation: Apply average pooling and max pooling along the channel dimension in $F^{'}$ to generate two 2D maps, $F_{a v g}$ and $F_{m a x}$ , both with dimensions $H \times W \times 1$ . Concatenate these two maps along the channel dimension.

Conv2D: Apply a 2D convolution with a $7 \times 7$ filter (for example) on the concatenation to produce a feature map $F_{c o n v}$ of dimension $H \times W \times 1$ .

Activation: Apply the sigmoid activation function on $F_{c o n v}$ to obtain the spatial attention map $M_{s}$ of dimension $H \times W \times 1$ .

Multiply: Multiply the spatial attention map $M_{s}$ by the feature map $F^{'}$ to obtain the final feature map $F^{″}$ with spatial attention applied.

CBAM Output: Final Feature Map $F^{″}$ with dimensions $H \times W \times C$ , where channel and spatial attentions were applied sequentially.

The GCNet (Global Context Network) is a neural network architecture that incorporates global context into local convolutions to improve the performance of computer vision tasks such as object detection and segmentation. The central component of GCNet is an attention mechanism that efficiently aggregates global context information.^53,52,44 \hyperlink{alg:gcnet}{Algorithm 2} shows the high-level implementation of the GCNet attention mechanism.

In the present study, a modified GCNet (Figure 3) mechanism was used, which combines three types of attention. Non-local attention is employed to capture long-range dependencies in the image. Squeeze-and-Excitation attention is used to recalibrate input channels according to their importance. Spatial attention focuses on specific areas of the image. Additionally, we used the sigmoid activation function instead of the softmax from the original GCNet design.

The GlobalContextAttention procedure takes as input a feature map $F$ , which is a three-dimensional matrix with dimensions $H \times W \times C$ , where $H$ is height, $W$ is width, and $C$ is the number of channels.

GlobalPooling: The first step within the procedure is to apply GlobalPooling to the feature map $F$ . This is an operation that summarizes the spatial information of $F$ , reducing its spatial dimensions $W$ and $H$ to a single value per channel, resulting in $F_{g l o b a l}$ with dimensions $1 \times 1 \times C$ . This captures the global information of the feature map.

Transform: $F_{g l o b a l}$ is transformed by a Transform operation, which can be a combination of linear operations (such as a dense layer in neural networks) and non-linear operations, to learn a more useful representation of $F_{g l o b a l}$ . The result is $F_{t r a n s f o r m e d}$ .

This excerpt defines an MLP (multilayer perceptron) for the “excitation” part in the SE block. The MLP has two dense layers: the first reduces the channels by the specified ratio, and the second returns to the original channels with a sigmoid activation.

The transformation is followed by the application of a sigmoid function $σ$ , which normalizes $F_{t r a n s f o r m e d}$ to a range between 0 and 1, generating the attention mask $M$ .

This attention mask $M$ is then used to weight the original feature map $F$ , element-wise multiplication to generate $F_{w e i g h t e d}$ , which emphasizes or attenuates specific parts of $F$ based on global context.

The weighted feature map $F_{w e i g h t e d}$ is then added to the original feature map $F$ , resulting in $F_{o u t}$ , the output feature map.

The procedure returns $F_{o u t}$ , which now incorporates global context information more effectively.

4.1. Experimental model

The experimental model described here is constructed with an encoder that uses a sequence of convolutional layers to encode image information. Each block in the encoder consists of a convolutional layer (Conv2D) followed by a max-pooling layer (MaxPooling2D), aiming to reduce the spatial dimensions of the input image. The hierarchical organization of the blocks is established by progressively increasing the number of filters, ranging from 64 to 256 in the convolutional layers as it goes deeper into the encoder, as illustrated in Figure 4.

Figure 4.

Experimental model with the appropriate CBAM and GCNet attention mechanisms.

The decoder plays a key role in reconstructing the image from the encoded representation, using up-sampling operations (UpSampling2D) to expand the spatial dimensions. Furthermore, skip connections were incorporated between the encoder and decoder layers. These connections play a crucial role in the direct transmission of low-level information to high-level layers, resulting in a substantial improvement in the accuracy of image reconstruction. In this context, the number of filters is initially set at 256 for the first layer, followed by 128 and 64.

In both stages of the process, in both the encoder and decoder, uniform parameters were adopted, including strides = 3, ‘relu’ activation function, and padding = ‘same’. However, a notable distinction is made in the decoder, where the choice was made to perform bilinear interpolation during the execution of UpSampling2D. This approach aims to enhance the resolution of the forecast, aligning it more accurately with the desired image resolution, as previously addressed in studies.^26,54

At the end of the decoder, a resizing layer (Conv2D) was incorporated with the purpose of adapting the dimensions to the desired output, employing a ’linear’ activation function.

In the transition between the encoder and the decoder, a ‘bottleneck’ layer was introduced, which serves as a compact transition region between the encoding (encoder) and decoding (decoder) phases.

The bottleneck layer emphasizes crucial and discriminative information for the task at hand, focusing the network’s attention on critical aspects of the input during encoding and subsequently on reconstruction during the decoding phase.

The addition of this layer can result in computational gains, as subsequent operations are carried out on a more compact representation, promoting efficiency in processing. The main details of the internal components for each layer of the model are shown in Figure 5.

Figure 5.

Specification of convolutional layers that integrate the models mentioned in this work.

4.2. Proposed model

In the model proposed in this study, the encoder is configured with residual convolutional layers,⁶ providing efficient learning of representational features. Each residual block is enriched with attention mechanisms, a strategic addition to enhance the model’s ability to focus on specific areas of the image. This refinement highlights crucial features during the encoding phase, improving the ability to extract and interpret relevant information.

Each block of the decoder also includes residual layers and attention mechanisms similar to those of the encoder.

The main distinction between the proposed model (Figures 6 and 7) and the model implemented for comparison (Figure 4) lies in the fact that the former was built using two consecutive layers of convolution in the encoder and two consecutive layers of up-sampling in the decoder. All other parameters were kept consistent in both configurations.

Figure 6.

Detailed representation of the proposed model, with respective indications in the legend.

Figure 7.

Proposed model with the appropriate CBAM and GCNet attention mechanisms.

For comparison purposes with the article,⁶ at the end of the model, the crop mentioned in Eigen et al.²² was used, and the images resized to $256 \times 256$ .

The proposed model essentially consists of a U-Net type convolutional neural network (CNN), which adopts an encoder-decoder architecture.

The complete model is constructed using the Model class, having the RGB image as inputs and estimated depth maps as outputs. At the end of each layer in the encoder, a CBAM attention mechanism is incorporated, while in the final layers of the decoder, a GCNet attention mechanism is introduced. Further details can be observed in Figures 6 and 7.

The algorithm 3 represents a simplified version of the proposed model for predicting depth maps. The model is constructed by sequentially applying downsampling blocks to reduce the resolution and increase the depth of channels, followed by upsampling blocks to reconstruct the resolution of the original image. Residual connections and attention mechanisms are applied at each step to improve the flow of information and the representational capacity of the model.

The input image is initially processed through a series of residual attention blocks without upsampling, where the number of filters is progressively increased through a predefined set of values (64, 128, 256, 512).

After initial processing, the image is processed by a series of residual attention blocks with upsampling, where skip connections from earlier layers are integrated. This allows the model to recover higher-resolution information lost during downsampling operations.

The model is finalized with a convolution to adjust the output dimensions as needed, and the model configuration is completed by specifying the inputs and outputs.

This architectural design, which incorporates attention and residual connections into an encoder-decoder structure, aims to enhance the model’s ability to capture and reconstruct meaningful features in image processing tasks. However, the effectiveness of this model will be empirically evaluated in specific tasks, and adjustments may be necessary based on the data and requirements of the target application.

5. Experimental results

In this section, comparative experiments will be conducted using three widely recognized datasets in the literature. The objective is to evaluate the performance of state-of-the-art architectures against the method proposed in this study, both quantitatively and qualitatively. This comprehensive analysis will help demonstrate the effectiveness of the proposed method compared to state-of-the-art solutions.

5.1. Datasets

The dataset used for comparisons with state-of-the-art works and NYU Depth v2⁵⁵ consists of video sequences of 464 indoor scenes recorded with the Microsoft Kinect. It includes 120,000 training samples and 654 test samples. For testing the proposed method, a subset of 49,000 images was utilized, divided into 7 parts of 7,000 each for training, due to hardware limitations. The model was trained on each of these subsets for 100 epochs. The maximum depth of the depth maps is 10 meters. The original size of the image and depth map in the dataset is $480 \times 640$ , but it was resized to $128 \times 128$ due to GPU memory limitations. Hence, the proposed model was trained with input images of size $128 \times 128$ , which generates the same output size of $128 \times 128$ . During testing, depth map prediction was calculated for all test images, upscaled to $256 \times 256$ , and finally evaluated using the proposed central crop by Eigen et al.²²

The labeled dataset is a subset of the raw dataset, comprising pairs of synchronized RGB and depth frames, each annotated with dense labels. In addition to the original projected depth maps, we provide a set of preprocessed depth maps where missing values have been filled using Levin et al.⁵⁶ colorization method. Unlike the raw dataset, the labeled dataset is distributed as a Matlab .mat file, from which the images used in our experiments were extracted. Beyond hardware constraints, another reason for selecting images with the specified dimensions was to ensure a fair comparison with the works cited in Jan and Seo.⁶

KITTI⁵⁷ is an outdoor dataset consisting of stereo images and 3D scans of 61 scenes captured by multiple sensors mounted on top of a moving vehicle. The dataset contains an RGB input image with a resolution of $1241 \times 375$ pixels, and the corresponding LIDAR scans. We used the training/test split defined by Eigen et al.,²² which consists of a subset of 26,000 left vision images from the official KITTI dataset for training and 697 images in the test set. In this work, the dataset was used solely for the qualitative evaluation of the estimated depth maps. For the KITTI dataset, the same resolutions and procedures used for the NYU V2 dataset were maintained, following the reasons outlined earlier when discussing the latter dataset.

In this study, the NYU v2 dataset was used for both quantitative and qualitative analyses, while the KITTI dataset was exclusively used for qualitative evaluations.

Table 1.
Quantitative comparison of different state-of-the-art methods on the NYU-depth v2 dataset.

High is better $↑$ Low is better $↓$

Methods $δ_{1}$ $δ_{2}$ $δ_{3}$ ABS Rel RMSE Log10

Eigen et al.²² 0.769 0.950 0.988 0.158 0.641 –

Laina et al.⁵⁹ 0.811 0.953 0.988 0.127 0.573 0.055

Fu et al.⁵⁸ 0.828 0.965 0.992 0.115 0.509 0.051

Alhashim and Wonka²⁶ 0.846 0.974 0.994 0.123 0.465 0.053

Jan and Seo⁶ 0.917 0.987 0.993 0.117 0.398 0.047

Ours 0.947 0.991 0.999 0.086 0.373 0.052

	High is better $↑$	Low is better $↓$
Eigen et al.²²	0.769	0.950	0.988	0.158	0.641	–
Laina et al.⁵⁹	0.811	0.953	0.988	0.127	0.573	0.055
Fu et al.⁵⁸	0.828	0.965	0.992	0.115	0.509	0.051
Alhashim and Wonka²⁶	0.846	0.974	0.994	0.123	0.465	0.053
Jan and Seo⁶	0.917	0.987	0.993	0.117	0.398	0.047
Ours	0.947	0.991	0.999	0.086	0.373	0.052

5.2. Loss function

A loss function considers the difference between the expected depth map and the map predicted by the network.²² Variations in the configuration of the loss function can have a significant impact on both the training rate and the overall performance of depth estimation. Several adaptations in the formulation of the loss function, used to optimize the neural network, are documented in the literature dedicated to depth estimation.^22,58–60

The function used in the present work is the same as described in Alhashim et al.²⁶ Where, during the training of the network, the loss $L$ between $y$ and $\hat{y}$ is defined as the weighted sum of three loss functions, equation (4).

L (y, \hat{y}) = λ L_{d e p t h} (y, \hat{y}) + L_{g r a d} (y, \hat{y}) + L_{S S I M} (y, \hat{y})

(4)

The first loss component

L_{d e p t h}

is the point-wise

L 1

loss defined on the depth values, equation (5):

L_{d e p t h} (y, \hat{y}) = \frac{1}{n} \sum_{p}^{n} | y_{p} - {\hat{y}}_{p} |

(5)

The second component

L_{g r a d}

is the

L 1

loss defined over the gradient of the depth image

g

, equation (6).

L_{g r a d} (y, \hat{y}) = \frac{1}{n} \sum_{p}^{n} | g_{x} (y_{p} - {\hat{y}}_{p}) | + | g_{y} (y_{p} - {\hat{y}}_{p}) |

(6)

where

g_{x}

and

g_{y}

compute the differences in the

x

and

y

components for the gradients of the depth image of

y

and

\hat{y}

Finally, $L S S I M$ utilizes the Structural Similarity (SSIM) term,⁶¹ which is a commonly used metric for image reconstruction tasks. Recently, it has been demonstrated to be a good loss component for depth estimation CNNs.¹⁴ Since SSIM has an upper limit of one, we define it as an $L S S I M$ loss as follows, equation (7):

L_{S S I M} (y, \hat{y}) = \frac{1 - S S I M (y, \hat{y})}{2}

(7)

It is observed that only one weight parameter

λ

was defined for the

L_{d e p t h}

loss component. Empirically, the authors found and set

λ = 0.1

as a reasonable weight for this component.

5.3. Implementation details

The U-Net was implemented using TensorFlow.⁶² For training, the Adam optimizer⁶³ was used with a learning rate of 0.001, Beta1 of 0.9, Beta2 of 0.999, and epsilon of $1 \times 10^{- 8}$ , with a batch size of 8. For the trainings and tests conducted in this work, a PC equipped with an i5 11600k processor, 16GB of RAM, and an NVIDIA GeForce RTX 3060 GPU with 12GB of VRAM was used.

The model was trained for hundred epochs on each of the seven subsets, each composed of seven thousand images, totaling forty-nine thousand images used for training. The training time for each subset was approximately 1.5 hours.

Regarding monocular depth map predictions, once the model is trained, it can estimate the depth map of a scene (image) at the specified resolution in as little as 1 millisecond. However, we cannot confirm if this processing time remains consistent for scenes with higher resolution images.

Figure 8.

Here’s a brief qualitative comparison between images from some state-of-the-art methods. In (a) are the RGB images of the scene, (b) their respective ground truth depth maps, (c) depth maps estimated by the method proposed by Fu et al.,⁵⁸ (d) depth maps estimated by the method of Alhashim and Wonka,²⁶ (e) depth maps estimated by the method of Jan and Seo,⁶ and (f) depth maps estimated by the method proposed in this study.

Figure 9.

Qualitative Evaluation: Results obtained with the NYU v2 dataset.

Figure 10.

Qualitative Evaluation: Results obtained with the KITTI dataset.

5.4. Quantitative evaluation of results

This study conducts a systematic evaluation of methods aimed at monocular depth estimation, employing various distinct metrics for quantitative analysis. This evaluation included comparing the performance of the proposed method against state-of-the-art techniques, using metrics such as Mean Relative Error (ABS Rel), Root Mean Square Error (RMSE), Mean Error Log Error (Log10), and Threshold Accuracy ( $δ_{i}$ ) with thresholds $τ_{i}^{'}$ = 1.25, 1.25 $^{2}$ , 1.25 $^{3}$ used in previous works^22,64,65 to compare the methods presented in this study with the state of the art. The metrics used for comparing the results obtained with the cited works are presented below.

- Mean relative error

r e l = \frac{1}{n} \sum_{p}^{n} \frac{y_{p} - y_{p}^{'}}{n}

(8)

- Root mean square error

r e l = \sqrt{\frac{1}{n} \sum_{p}^{n} {(y_{p} - y_{p}^{'})}^{2}}

(9)

- Mean error

l o g 10 = \frac{1}{n} \sum_{p}^{n} | l o g_{10} (y_{p}) - l o g_{10} (y_{p}^{'}) |

(10)

- Threshold accuracy

δ_{i} = % o f y_{p}, m a x (\frac{y_{p}}{y_{p}^{'}}, \frac{y_{p}^{'}}{y_{p}}) < {1.25}^{i} (i = 1, 2, 3)

(11)

In the mentioned metrics,

y_{p}

represents the pixel in the depth image

y

, while

y_{p}^{'}

denotes a pixel in the predicted image. The term

n

refers to the total number of pixels in each depth image. The results obtained with the NYU v2 dataset are detailed in Table 1, comparing them with state-of-the-art methods, where the results obtained by the proposed model are highlighted.

5.5. Result analysis

Table 1 shows a quantitative comparison of the methods studied in this work with other architectures, where the proposed model using CBAM and Modified GCNet attention mechanism achieved satisfactory results in relation to the state-of-the-art methods compared in Table 1. Specifically, the U-Net with attention demonstrated a 25.22% improvement in Absolute Relative Error (ABS Rel) compared to the lowest value presented in Fu et al.⁵⁸ Furthermore, the proposed model achieved a 6.28% improvement in terms of RMSE error compared to the method presented in Jan and Seo.⁶

The NYU Depth v2 dataset contains RGB images along with their respective depth maps, which are stored in separate files, typically organized in pairs. The images are represented in three channels, while the depth maps are monochromatic matrices which each pixel value corresponds to a depth measurement. This measurement information is available in the form of MAT (Matlab) files, which contain the RGB images, depth maps, and segmentation annotations. However, the information contained in the .mat files was not used in the model training process, as the depth maps effectively function as numerical masks that assign a depth value to each pixel. Nevertheless, in situations where it is necessary to determine the distance in meters of objects contained in the NYU V2 dataset, the values provided in the .mat files can be utilized.

In this work, we decided not to use the values contained in the available masks, focusing exclusively on the RGB images and their respective depth maps during the model training. This approach allowed us to simplify the process and concentrate our analysis on the visual features of the images and the depth information directly associated with each pixel. Although the masks can provide useful information about the reliability of the depth data, we opted for a strategy that prioritizes direct interaction with the images.

The use of this additional information, such as the values of the masks, will be addressed in a future phase of the project, in which we plan to explore how these data can contribute to improving the accuracy and robustness of the model. We believe that by integrating the masks in the next phase, we will be able to enhance the identification of problematic areas in the images and refine the analyses conducted. This strategy will allow us to leverage all the information available in the dataset, maximizing the results achieved.

5.6. Qualitative evaluation of results

The Figure 8 presents the depth maps generated from the NYU-depth v2 dataset, following the split approach proposed by Eigen.²² Figure 9 shows more results obtained with the NYU v2 dataset.

For validation purposes, the KITTI dataset⁵⁷ was also used for a qualitative analysis of the model, where Figure 10 displays the results obtained with the proposed model.

The results obtained were compared with state-of-the-art methods, highlighting, especially in the last column of Figure 8, the clarity and sharpness of the depth maps produced by the proposed model in this work. In column (f), details (highlighted by rectangles) can be observed that were preserved in the reconstruction of monocular depth maps, which do not appear in the images generated by the other compared methods present in the comparison.

Table 2.
Quantitative results of ablation tests using the NYU v2 dataset for the experimental model.

Experimental Model

High is better Low is better

Configurations $δ_{1}$ $δ_{2}$ $δ_{3}$ Rel RMS Log10

No Segmentation No Attention

Skipconnection(enc-dec) 0.847 0.946 0.968 0.219 0.692 0.107

ResConv(enc-dec) 0.883 0.955 0.971 0.175 0.447 0.092

No Segmentation With Attention

Channel-Wise(dec) 0.908 0.954 0.970 0.164 0.607 0.067

Ch-Wise_Spat-Att(enc-dec) 0.913 0.968 0.963 0.111 0.621 0.068

GCNet(enc) 0.926 0.979 0.988 0.136 0.452 0.072

CBAM(dec) 0.925 0.978 0.987 0.140 0.467 0.066

TwoDimensional(enc) 0.901 0.948 0.963 0.103 0.599 0.075

CBAM(enc)-GCNet(dec) 0.947 0.987 0.996 0.097 0.381 0.065

With Segmentation No Attenton

Seg 2 Regions - ResConv 0.826 0.894 0.929 0.316 0.687 0.089

Seg 3 Regions - ResConv 0.856 0.9120 0.941 0.295 0.479 0.082

Seg 5 Regions - ResConv 0.884 0.938 0.956 0.289 0.451 0.079

Seg 10 Regions - ResConv 0.909 0.945 0.961 0.281 0.463 0.077

With Segmentation and Attention

Channel-Wise(dec) 0.795 0.883 0.938 0.335 0.651 0.086

Ch-Wise_Spat-Att(enc) 0.876 0.925 0.957 0.290 0.459 0.078

Ch-(end)_Spat-(dec) 0.868 0.924 0.958 0.278 0.447 0.078

Spat-(Enc-Dec) 0.869 0.923 0.958 0.277 0.420 0.076

GCNet(dec)-CBAM(dec)-TwoSpatial(enc) 0.917 0.969 0.987 0.132 0.403 0.071

Experimental Model
No Segmentation No Attention
Skipconnection(enc-dec)	0.847	0.946	0.968	0.219	0.692	0.107
ResConv(enc-dec)	0.883	0.955	0.971	0.175	0.447	0.092
No Segmentation With Attention
Channel-Wise(dec)	0.908	0.954	0.970	0.164	0.607	0.067
Ch-Wise_Spat-Att(enc-dec)	0.913	0.968	0.963	0.111	0.621	0.068
GCNet(enc)	0.926	0.979	0.988	0.136	0.452	0.072
CBAM(dec)	0.925	0.978	0.987	0.140	0.467	0.066
TwoDimensional(enc)	0.901	0.948	0.963	0.103	0.599	0.075
CBAM(enc)-GCNet(dec)	0.947	0.987	0.996	0.097	0.381	0.065
With Segmentation No Attenton
Seg 2 Regions - ResConv	0.826	0.894	0.929	0.316	0.687	0.089
Seg 3 Regions - ResConv	0.856	0.9120	0.941	0.295	0.479	0.082
Seg 5 Regions - ResConv	0.884	0.938	0.956	0.289	0.451	0.079
Seg 10 Regions - ResConv	0.909	0.945	0.961	0.281	0.463	0.077
With Segmentation and Attention
Channel-Wise(dec)	0.795	0.883	0.938	0.335	0.651	0.086
Ch-Wise_Spat-Att(enc)	0.876	0.925	0.957	0.290	0.459	0.078
Ch-(end)_Spat-(dec)	0.868	0.924	0.958	0.278	0.447	0.078
Spat-(Enc-Dec)	0.869	0.923	0.958	0.277	0.420	0.076
GCNet(dec)-CBAM(dec)-TwoSpatial(enc)	0.917	0.969	0.987	0.132	0.403	0.071

6. Ablation studies

With the goal of choosing a model capable of producing competitive results in comparison with the methods presented in the state of the art, various tests were conducted with different configurations and attention mechanisms.

Two models were developed for the decision of which would be used in the tests carried out in this work, where one consists of single convolutional layers in both the encoder and the decoder (Figure 4). The second model containing two consecutive layers (Figure 7). Both are based on models present in the state of the art.

In both structures, various attention mechanisms were integrated, including channelWise Attention, spatial Attention2D, twoDimensional Attention, selfAttention, CBAM Attention (Figure 2) and also GCNet-Attention (Figure 3). However, it was not possible to train the model using the self-Attention mechanism due to hardware limitations for the defined number of heads and dimensions of keys based on the input dimensions.

Following a thorough analysis and comprehensive literature review, such as those presented in Jan and Seo,⁶ Agarwal and Arora,⁵⁴ Alhashim and Wonka,²⁶ and Li et al.,³⁴ it was identified that the configuration of two consecutive convolutional layers, maintaining the same number of filters in each layer, may prolong the time required for training. However, this approach resulted in an improvement in the final results, compared to the model that did not adopt this same structure.

Additional experiments were conducted using 1024 filters in the bottleneck layer. However, this modification did not translate into significant advances for $128 \times 128$ resolution images, observing only a slight decrease in the model’s efficiency during training. Given these results, it was decided to maintain the configuration of 512 filters in this layer, aiming to optimize the balance between performance and efficiency.

The study was directed using the NYU v2 dataset, starting with the implementation incorporating skip connections in both models. Subsequently, the analysis continued incorporating a residual convolution block. The adoption of these blocks resulted in improvements in the quantitative indicators of the tests, leading to the decision to keep them for subsequent experiments with attention mechanisms. It’s important to highlight that, in all phases of testing, RGB images with a resolution of $128 \times 128$ were used as input for training.

In a subsequent step, the tests were repeated with the same images, but segmented. For this, the k-means algorithm was employed, exploring segmentation variations in two, three, five, and ten regions. The proposed model, which incorporates residual convolution blocks and double convolution per layer, was chosen for these tests. The most promising results were observed with segmentation into ten regions, a configuration that was maintained in the following experiments with segmented images and attention mechanisms. However, the results with RGB input images surpassed those obtained with segmented images, which allowed for a careful evaluation of the most effective model for the comparative tests detailed in the next section of the work. The results of the ablation study can be seen in Tables 2 and 3, where it is possible to observe that those obtained with both models were better using the configuration with CBAM and Modified GCNet attention mechanism (highlighted in bold in the tables).

Table 3.
Quantitative results of ablation tests using the NYU v2 dataset for the proposed model.

Proposed Model

High is better Low is better

Configurations $δ_{1}$ $δ_{2}$ $δ_{3}$ Rel RMS Log10

No Segmentation No Attention

Skipconnection(enc-dec) 0.888 0.957 0.978 0.212 0.667 0.102

ResConv(enc-dec) 0.902 0.959 0.981 0.145 0.421 0.084

No Segmentation With Attention

Channel-Wise(dec) 0.918 0.962 0.970 0.157 0.601 0.060

Ch-Wise_Spat-Att(enc-dec) 0.927 0.975 0.963 0.109 0.617 0.061

GCNet(enc) 0.931 0.983 0.988 0.121 0.449 0.065

CBAM(dec) 0.930 0.987 0.987 0.128 0.452 0.059

TwoDimensional(enc) 0.913 0.950 0.963 0.101 0.585 0.057

CBAM(enc)-GCNet(dec) 0.947 0.991 0.999 0.086 0.373 0.052

With Segmentation No Attenton

Seg 2 Regions - ResConv 0.852 0.913 0.931 0.305 0.677 0.081

Seg 3 Regions - ResConv 0.877 0.919 0.947 0.241 0.465 0.075

Seg 5 Regions - ResConv 0.892 0.944 0.959 0.263 0.443 0.073

Seg 10 Regions - ResConv 0.911 0.949 0.972 0.258 0.456 0.069

With Segmentation and Attention

Channel-Wise(dec) 0.821 0.891 0.943 0.332 0.647 0.076

Ch-Wise_Spat-Att(enc) 0.894 0.937 0.959 0.283 0.453 0.071

Ch-(end)_Spat-(dec) 0.883 0.929 0.961 0.275 0.441 0.068

Spat-(Enc-Dec) 0.881 0.927 0.961 0.271 0.414 0.072

GCNet(dec)-CBAM(dec)-TwoSpatial(enc) 0.918 0.975 0.989 0.129 0.401 0.063

Proposed Model
No Segmentation No Attention
Skipconnection(enc-dec)	0.888	0.957	0.978	0.212	0.667	0.102
ResConv(enc-dec)	0.902	0.959	0.981	0.145	0.421	0.084
No Segmentation With Attention
Channel-Wise(dec)	0.918	0.962	0.970	0.157	0.601	0.060
Ch-Wise_Spat-Att(enc-dec)	0.927	0.975	0.963	0.109	0.617	0.061
GCNet(enc)	0.931	0.983	0.988	0.121	0.449	0.065
CBAM(dec)	0.930	0.987	0.987	0.128	0.452	0.059
TwoDimensional(enc)	0.913	0.950	0.963	0.101	0.585	0.057
CBAM(enc)-GCNet(dec)	0.947	0.991	0.999	0.086	0.373	0.052
With Segmentation No Attenton
Seg 2 Regions - ResConv	0.852	0.913	0.931	0.305	0.677	0.081
Seg 3 Regions - ResConv	0.877	0.919	0.947	0.241	0.465	0.075
Seg 5 Regions - ResConv	0.892	0.944	0.959	0.263	0.443	0.073
Seg 10 Regions - ResConv	0.911	0.949	0.972	0.258	0.456	0.069
With Segmentation and Attention
Channel-Wise(dec)	0.821	0.891	0.943	0.332	0.647	0.076
Ch-Wise_Spat-Att(enc)	0.894	0.937	0.959	0.283	0.453	0.071
Ch-(end)_Spat-(dec)	0.883	0.929	0.961	0.275	0.441	0.068
Spat-(Enc-Dec)	0.881	0.927	0.961	0.271	0.414	0.072
GCNet(dec)-CBAM(dec)-TwoSpatial(enc)	0.918	0.975	0.989	0.129	0.401	0.063

The tests using segmentation of the input images were conducted only with the proposed model, given that the experimental model showed less significant results.

7. Conclusions

The automation of Depth Map estimation through advanced Deep Learning techniques shows great potential for integration into navigation aids for the visually impaired and other applications. Researchers are increasingly focused on improving depth map accuracy with single-camera systems, as stereo camera setups pose significant challenges, particularly with calibration.

This study aimed to implement, train, and evaluate a encoder-decoder architecture for estimating depth maps from RGB images. The proposed model (Figure 7), incorporating CBAM attention in the encoder and Modified GCNet in the decoder, achieved accuracy comparable to state-of-the-art methods, as shown in both quantitative (Table 3) and qualitative (Figure 9) evaluations. The model’s design followed current trends in supervised depth estimation, prioritizing accuracy even with increased model complexity and computation.

This study shows that significant results in monocular depth map estimation can be achieved with a simple architecture, avoiding the need for complex, resource-intensive models. These results enable safe navigation when integrated into assistance systems for the visually impaired.

Future experiments should test the integration of transformers and Generative Adversarial Networks into the proposed model, as some studies have reported significant results in depth map estimation with these methods. However, as noted by Kim et al.,⁶⁶ integrating transformers increases computational complexity.

The main limitations of the depth map estimation system using U-Net with CBAM and GCNet include computational complexity, which, while lower than more advanced models like GANs and Transformers, can still pose a challenge for real-time applications or devices with limited resources. There are also uncertainties regarding the model’s performance on high-resolution images. Additionally, the model may struggle to generalize to real-world scenarios with environmental variations, and the requirement for large amounts of labeled training data represents a significant barrier. Another relevant limitation is the lack of evaluation in dynamic scenarios, which are crucial for assisted navigation applications, as well as the low flexibility in integrating data from multiple sources, such as LiDAR or stereo cameras, which could improve accuracy in more complex environments.

Incorporating hybrid systems could address some of these limitations. Hybrid approaches that combine deep learning models with traditional algorithms can leverage the strengths of both methods, potentially enhancing performance and robustness. For instance, integrating Generative Adversarial Networks (GANs) could improve the realism of depth estimations, while combining U-Net with classical stereo vision techniques may yield more reliable results in diverse environmental conditions. Such hybrid models can also offer greater adaptability by fusing data from multiple sources, thus improving accuracy and performance in complex and dynamic scenarios.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

References

Novotny

Liu

Morales-Alvarez

, et al. Vehicle side-slip angle estimation under snowy conditions using machine learning. Integr Comput-Aided Eng 2023; 31: 117–137.

Lore

Reddy

Giering

, et al. Generative adversarial networks for depth map estimation from rgbRGB video. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), IEEE, 2018, pp.1258–12588.

Wheatstone

. Contributions to the physiology of vision. Philosoph Trans R Soc 1838; 128: 371–394.

Von Helmholtz

. Treatise on Physiological Optics, Volume III. New York: Courier Corporation 2013.

Hartley

Zisserman

. Multiple view geometry in computer vision. Cambridge: Cambridge university press, 2003.

Jan

Seo

. Monocular depth estimation using res-unet with an attention model. Applied Sciences 2023; 13: 6319.

Zaman

SRNPN

. Single-image stereo depth estimation using gans. [Online] Available at: https://sharanramjee.github.io/files/projects/cs231a.pdf (2013, accessed 09 August 2023).

Hoiem

Efros

Hebert

. Automatic photo pop-up. In: ACM SIGGRAPH 2005 Papers, 2005, pp.577–584.

Karsch

Liu

Kang

. Depth transfer: depth extraction from video using non-parametric sampling. IEEE Trans Pattern Anal Mach Intell 2014; 36: 2144–2158.

10.

Ladicky

Shi

Pollefeys

. Pulling things out of perspective. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp.89–96.

11.

Chang

Maruyama

. Real-time high-quality stereo matching system on a GPU. In: 2018 IEEE 29th international conference on application-specific systems, architectures and processors (ASAP), 2018, pp.1–8. IEEE.

12.

Goesele

Snavely

Curless

, et al. Multi-view stereo for community photo collections. In: 2007 IEEE 11th international conference on computer vision, IEEE, 2007, pp.1–8.

13.

Eigen

Puhrsch

Fergus

. Depth map prediction from a single image using a multi-scale deep network. In: Ghahramani Z, Welling M, Cortes C, et al. (eds), Advances in neural information processing systems, volume 27. Curran Associates, Inc, 2014a.

14.

Godard

Mac Aodha

Brostow

. Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp.270–279.

15.

Groenendijk

Karaoglu

Gevers

, et al. On the benefit of adversarial training for monocular depth estimation. Comput Vis Image Underst 2020; 190: 102848.

16.

Saxena

Chung

. Learning depth from single monocular images. Adv Neural Inf Process Syst 2005; 18: 1–8.

17.

Hassanpour

Moradikia

Adeli

, et al. A novel end-to-end deep learning scheme for classifying multi-class motor imagery electroencephalography signals. Expert Systems 2019; 36: e12494.

18.

Martins

Papa

Adeli

. Deep learning techniques for recommender systems based on collaborative filtering. Expert Syst 2020; 37: e12647.

19.

Jiang

Larsson

Shakhnarovich

MMG

, et al. Self-supervised relative depth learning for urban scene understanding. In: Proceedings of the European conference on computer vision (ECCV), 2018, pp.19–35.

20.

Saxena

Sun

. Make3D: Learning 3D scene structure from a single still image. IEEE Trans Pattern Anal Mach Intell 2008; 31: 824–840.

21.

Xue

Zhang

Neri

, et al. An external attention-based feature ranker for large-scale feature selection. Knowl Based Syst 2023; 281: 111084.

22.

Eigen

Puhrsch

Fergus

. Depth map prediction from a single image using a multi-scale deep network. Adv Neural Inf Process Syst 2014b; 27: 1–9.

23.

Krizhevsky

Sutskever

Hinton

. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 2012; 25: 1–9.

24.

Kumari

Jha

Bhavsar

, et al. Autodepth: single image depth map estimation via residual CNN encoder-decoder and stacked hourglass. In: 2019 IEEE International Conference on Image Processing (ICIP), pages 340–344. IEEE, 2019.

25.

Newell

Yang

Deng

. Stacked hourglass networks for human pose estimation. In: Computer Vision–ECCV 2016: 14th European conference, Amsterdam, The Netherlands, October 11-14, 2016, proceedings, part VIII 14, Springer, 2016, pp.483–499.

26.

Alhashim

Wonka

. High quality monocular depth estimation via transfer learning. arXiv preprint arXiv:1812.11941, 2018.

27.

Huang

Liu

Van Der Maaten

, et al. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp.4700–4708.

28.

Eigen

Fergus

. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision, 2015, pp.2650–2658.

29.

Simonyan

Vedaldi

Zisserman

. Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.

30.

Seo

. SNR analysis for quantitative comparison of line detection methods. Appl Sci 2021; 11: 10088.

31.

Guariglia

Silvestrov

. Fractional-wavelet analysis of positive definite distributions and wavelets on d(c) d(c). In: Engineering mathematics II: Algebraic, stochastic and analysis structures for networks, data classification and optimization, Springer, 2016, pp.337–353.

32.

Yang

Zhong

, et al. Hyperspectral image classification using wavelet transform-based smooth ordering. Int J Wavelets Multiresolut Inf Process 2019; 17: 1950050.

33.

Zheng

Tang

Zhou

. A framework of adaptive multiscale wavelet decomposition for signals on undirected graphs. IEEE Trans Signal Process 2019; 67: 1696–1711.

34.

Wei

Fan

. Attention mechanism used in monocular depth estimation: an overview. Appl Sci 2023; 13: 9940.

35.

Rafiei

Adeli

. A new neural dynamic classification algorithm. IEEE Trans Neural Netw Learn Syst 2017; 28: 3074–3083.

36.

Chaudhari

Mithal

Polatkan

, et al. An attentive survey of attention models. ACM Trans Intell Syst Technol (TIST) 2021; 12: 1–32.

37.

Zhou

Khosla

Lapedriza

, et al. Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.2921–2929.

38.

Mnih

Heess

Graves

, et al. Recurrent models of visual attention. Adv Neural Inf Process Syst 2014; 27: 1–9.

39.

Anderson

Buehler

, et al. Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2023, pp.6077–6086.

40.

Wang

Jiang

Qian

, et al. Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp.3156–3164.

41.

Bahdanau

. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.

42.

Kiros

, et al. Show, attend and tell: neural image caption generation with visual attention. In: International conference on machine learning, PMLR, 2015, pp.2048–2057.

43.

Cao

Lin

, et al. Gcnet: non-local networks meet squeeze-excitation networks and beyond. In: Proceedings of the IEEE/CVF international conference on computer vision workshops, 2019, pp.0–0.

44.

Wang

Girshick

Gupta

, et al. Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.7794–7803.

45.

Zhang

Goodfellow

Metaxas

, et al. Self-attention generative adversarial networks. In: International conference on machine learning, PMLR, 2019, pp.7354–7363, 2019.

46.

Shen

Sun

. Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.7132–7141.

47.

Woo

Park

Lee

J-Y

, et al. Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), 2018, pp.3–19.

48.

Jiang

Song

Wang

, et al. Fusion of the YOLOv4 network model and visual attention mechanism to detect low-quality young apples in a complex environment. Precision Agriculture, 2022, pp.1–19.

49.

. Broodstock breeding behaviour recognition based on Resnet50-LSTM with CBAM attention mechanism. Comput Electro Agricult 2022; 202: 107404.

50.

Dong

Meng

, et al. Semantic segmentation and depth estimation based on residual attention mechanism. Sensors 2023; 23: 7466.

51.

Luo

Xiao

. Self-supervised coarse-to-fine monocular depth estimation using a lightweight attention module. Comput Visual Media 2022; 8: 631–647.

52.

Wang

Lai

Huang

, et al. Anytime stereo image depth estimation on mobile devices. In: 2019 international conference on robotics and automation (ICRA), IEEE, 2019a, pp.5893–5900.

53.

Wang

Yang

, et al. Unos: unified unsupervised optical-flow and stereo-depth estimation by watching videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019b, pp.8071–8081.

54.

Agarwal

Arora

. Attention attention everywhere: monocular depth prediction with skip attention. In: Proceedings of the IEEE/CVF Winter conference on applications of computer vision, 2023, pp.5861–5870.

55.

Silberman

Hoiem

Kohli

, et al. Indoor segmentation and support inference from rgbd images. In: Computer vision–ECCV 2012: 12th European conference on computer vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, Springer, 2012, pp.746–760.

56.

Levin

Lischinski

Weiss

. Colorization using optimization. In: ACM SIGGRAPH 2004 Papers, 2004, pp.689–694.

57.

Geiger

Lenz

Stiller

, et al. Vision meets robotics: the kitti dataset. Int J Rob Res 2013; 32: 1231–1237.

58.

Gong

Wang

, et al. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.2002–2011.

59.

Laina

Rupprecht

Belagiannis

, et al. Deeper depth prediction with fully convolutional residual networks. In: 2016 Fourth international conference on 3D vision (3DV), IEEE, 2016, pp.239–248.

60.

Ronneberger

Fischer

Brox

. U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer, 2015, pp.234–241.

61.

Wang

Bovik

Sheikh

, et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 2004; 13: 600–612.

62.

Abadi

Agarwal

Barham

et al. Tensorflow: large-scale machine learning on heterogeneous systems, 2015.

63.

Kingma

. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

64.

Bhat

Alhashim

Wonka

. Adabins: depth estimation using adaptive bins. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp.4009–4018.

65.

Yuan

Dai

, et al. Neural window fully-connected crfs for monocular depth estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp.3916–3925.

66.

Kim

Ahn

, et al. Global-local path networks for monocular depth estimation with vertical cutdepth. arXiv preprint arXiv:2201.07436, 2022.

	High is better $↑$			Low is better $↓$
Methods	$δ_{1}$	$δ_{2}$	$δ_{3}$	ABS Rel	RMSE	Log10
Eigen et al.²²	0.769	0.950	0.988	0.158	0.641	–
Laina et al.⁵⁹	0.811	0.953	0.988	0.127	0.573	0.055
Fu et al.⁵⁸	0.828	0.965	0.992	0.115	0.509	0.051
Alhashim and Wonka²⁶	0.846	0.974	0.994	0.123	0.465	0.053
Jan and Seo⁶	0.917	0.987	0.993	0.117	0.398	0.047
Ours	0.947	0.991	0.999	0.086	0.373	0.052

Experimental Model
	High is better			Low is better
Configurations	$δ_{1}$	$δ_{2}$	$δ_{3}$	Rel	RMS	Log10
No Segmentation No Attention
Skipconnection(enc-dec)	0.847	0.946	0.968	0.219	0.692	0.107
ResConv(enc-dec)	0.883	0.955	0.971	0.175	0.447	0.092
No Segmentation With Attention
Channel-Wise(dec)	0.908	0.954	0.970	0.164	0.607	0.067
Ch-Wise_Spat-Att(enc-dec)	0.913	0.968	0.963	0.111	0.621	0.068
GCNet(enc)	0.926	0.979	0.988	0.136	0.452	0.072
CBAM(dec)	0.925	0.978	0.987	0.140	0.467	0.066
TwoDimensional(enc)	0.901	0.948	0.963	0.103	0.599	0.075
CBAM(enc)-GCNet(dec)	0.947	0.987	0.996	0.097	0.381	0.065
With Segmentation No Attenton
Seg 2 Regions - ResConv	0.826	0.894	0.929	0.316	0.687	0.089
Seg 3 Regions - ResConv	0.856	0.9120	0.941	0.295	0.479	0.082
Seg 5 Regions - ResConv	0.884	0.938	0.956	0.289	0.451	0.079
Seg 10 Regions - ResConv	0.909	0.945	0.961	0.281	0.463	0.077
With Segmentation and Attention
Channel-Wise(dec)	0.795	0.883	0.938	0.335	0.651	0.086
Ch-Wise_Spat-Att(enc)	0.876	0.925	0.957	0.290	0.459	0.078
Ch-(end)_Spat-(dec)	0.868	0.924	0.958	0.278	0.447	0.078
Spat-(Enc-Dec)	0.869	0.923	0.958	0.277	0.420	0.076
GCNet(dec)-CBAM(dec)-TwoSpatial(enc)	0.917	0.969	0.987	0.132	0.403	0.071