Resource allocation for content distribution in IoT edge cloud computing environments using deep reinforcement learning

Abstract

This study presents a novel approach to optimize resource allocation, aiming to boost the efficiency of content distribution in Internet of Things (IoT) edge cloud computing environments. The proposed method termed the Caching-based Deep Q-Network (CbDQN) framework, dynamically allocates computational and storage resources across edge devices and cloud servers. Despite its need for increased storage capacity, the high cost of edge computing, and the inherent limitations of wireless networks connecting edge devices, the CbDQN strategy addresses these challenges. By considering constraints such as limited bandwidth and potential latency issues, it ensures efficient data transfer without compromising performance. The method focuses on mitigating inefficient resource usage, particularly crucial in cloud-based edge computing environments where resource costs are usage-based. To overcome these issues, the CbDQN method efficiently distributes limited resources, optimizing efficiency, minimizing costs, and enhancing overall performance. The approach improves content delivery, reduces latency, and minimizes network congestion. The simulation results substantiate the efficacy of the suggested method in optimizing resource utilization and enhancing system performance, showcasing its potential to address challenges associated with content spreading in IoT edge cloud calculating situations. Our proposed approach evaluated metrics achieves high values of Accuracy is 99.85%, Precision at 99.85%, specificity is 99.82%, sensitivity is 99.82%, F-score is 99.82% and AUC is 99.82%.

Keywords

Cloud computing resource allocation Internet of things deep Q network reinforcement Learning

1. Introduction

The paper delves into the critical issue of resource allocation within Internet of Things (IoT) edge cloud computing environments, addressing challenges related to complex action spaces, scalability issues, and the steep learning curve for developers [21]. With IoT devices generating significant data volumes at the network’s edge, processing this data in the cloud can lead to latency and bandwidth issues [18]. To tackle these challenges, the paper proposes employing content distribution mechanisms to optimize data processing, storage, and retrieval in edge cloud computing environments [11]. Given the rapid growth of IoT and the widespread adoption of edge computing, efficient content distribution in IoT edge cloud computing environments has become a major challenge [10]. In these numerous edge devices, smartphones, complex environments, such as sensors, and smart appliances, generate and consume data locally across various geographic locations, often with constrained resources [32]. This emphasizes the need to optimize computing resource allocation and content distribution for low-latency, high-throughput, and energy-efficient operations [34]. Traditional resource allocation techniques may fall short in addressing the dynamic and heterogeneous nature of IoT edge cloud calculating environments, leading researchers to explore advanced ML techniques, specifically Deep Reinforcement Learning (DRL), as an effective solution [14]. The paper’s proposed solution is the CbDQN approach, designed to optimize decision-making for caching and resource allocation. The novelty of the paper lies in this approach, which combines caching strategies and Deep Networks to address resource allocation challenges [20]. The integration of reinforcement learning techniques for decision-making in IoT edge cloud computing represents a unique contribution to the field. The paper establishes its background by outlining the challenges associated with resource sharing in IoT edge cloud calculating, providing context on the evolving landscape of IoT technologies, and underscoring the need for efficient resource management [16].

The results of the proposed CbDQN approach focus on optimizing latency, reducing network congestion, and enhancing resource utilization efficiency in content distribution [21]. By minimizing data transfers to the central cloud, content distribution in edge cloud environments helps optimize the use of network resources [27]. However, the key challenge remains in designing an efficient and robust DRL-based resource allocation system capable of addressing the complexity of IoT edge cloud computing environments [8]. This involves defining appropriate state and action spaces, determining suitable reward functions aligned with the system’s objectives, and developing simulation environments to train and evaluate the DRL agent effectively [1]. The paper thus contributes not only by proposing a novel solution but also by providing insights into the challenges of implementing DRL in real-world IoT edge cloud computing scenarios, thereby advancing the understanding and potential solutions in this evolving field [26].

DRL, a branch of machine learning, has been established to be highly successful in various sequential decision-making tasks [24]. It combines deep learning models and reinforcement learning principles to train agents that can learn optimal policies evolve through interactions with their surroundings and receive feedback in the form of rewards [25]. In the context of IoT edge cloud computing, DRL can enable edge devices and cloud servers to intelligently allocate resources and distribute content to optimize system performance based on real-time conditions and changing demands [15]. Various traditional methods are used such as DRL-based Resource Allocation Scheme [9], Two-Timescale (2Ts)-DRL approach [35], and DRL-based workload scheduling approach [36]. However, no proper solution is found to overcome these problems to create our proposed methodology.

The major contributions of the ongoing investigation are listed below:

Address the challenge of optimizing network delay in a scenario involving the Internet of Vehicles (IoV), which comprises Roadside Units (RSUs), Base Stations (BSs), and cloud resources.

Our approach focuses on optimizing content distribution and guaranteeing the attainment of network Quality of Service (QoS) through cross-layer collaborative content caching and invitation routing.

To tackle the issue of delay optimization, we introduce a novel policy based on a Caching-based deep Q network (CbDQN). This policy leverages historical request data and current network conditions to make intelligent decisions regarding content caching and request routing.

Our proposed solution’s effectiveness is assessed across various system conditions. Through extensive simulations using real-world data, we demonstrate that our strategy significantly reduces network latency compared to existing cloud-edge collaboration systems.

Furthermore, the introduced DQN model showcases adaptability to shifting network states and user requirements while achieving rapid convergence.

The paper is structured as surveys: Unit 2 offers an outline of the latest works in the field. In Section 2, we introduce the system model and formulate the problem statement. Our proposed methodology is outlined in Unit 4. Unit 5 offers simulation results and analyzes our method’s performance. The conclusion in Section 5 summarizes our research progress and highlights potential avenues for future advancements, enhancing the overall organization of the manuscript.

2. Related works

Some of the recent literature reviews are mentioned here,

In the IoT-based frameworks, the mist figuring permits the haze hubs to offload and handle errands mentioned from IoT-enabled gadgets in a disseminated way rather than the concentrated cloud servers to diminish the reaction delay. Nonetheless, accomplishing such an advantage is as yet testing in the frameworks with high pace of demands, which suggest long lines of errands in the haze hubs, consequently uncovering presumably a shortcoming as far as dormancy to offload the undertakings. To overcome these issues, Tran-Dang et al. [31] have developed Reinforcement Learning as a burgeoning facet of machine learning, empowering intelligent decision-making and facilitating adept responses to varying weather conditions by experts.

Edge registering expands the capacity of distributed computing to the organization’s edge to help different asset delicate also, execution of touchy IoT applications. Administration relocation is fundamental to guarantee administration progression when executing dynamic undertaking assignments. Chen et al. [4] have developed a long-term Dynamic Task Allocation and Service Migration (DTASM) edge-cloud IoT framework where clients’ processing necessities and versatility modify over the long run. The DTASM issue is formed to accomplish the long-haul execution of limiting the heap sent to the fog by satisfying the consistent relocation requirement and the inertness limitation at each season of carrying out the DTASM choice.

5G portable organization administrations have made colossal developments in the IoT system. A securities numeral of battery-fueled IoT gadgets is conveyed to serve different situations. In this unique circumstance, energy utilization became one of the most basic worries in interconnecting shrewd IoT gadgets in such situations. Besides, edge IoT hubs frequently face the greatest obstacle of performing ideal asset appropriation and accomplishing superior execution levels while adapting to the fluctuation of undertaking taking care of, energy preservation, and super dependable low-inertness. To stun these concerns, Sellami et al. [29] have proposed a Software Defined Fog-IoT Network to limit the organization dormancy while guaranteeing energy productivity by convertible mobile power under the requirements of utilization reliance.

The omnipresent IoT gadgets produce developing versatile administrations of presentations with computationally serious and idly delicate highlights, which expands the information traffic pointedly. Focused on holder innovation, microservice has arisen with flexibility and adaptability by deteriorating one help into a few free lightweight parts. Tian et al. [30] have proposed a MEC-enabled distributed compliant microservice secreting structure, named DIMA In particular, the microservice storing issue is displayed as a Marko choice interaction (MDP) to streamline the getting postponement and hit proportion.

IoT can work with plenty of information exchanges among different servers. In the IoT, mist servers are used to accomplish compelling information exchanges from dynamic gadgets. Be that as it may, load adjusting is as yet a huge errand scientist center on moderating the heap adjusting issue. A few virtual machines might be overburdened when other virtual machines are inactive because of a terrible planning strategy. Gupta et al. [13] have developed the amalgamation of Hybrid Grey Wolf Optimization (GWO) and the Reformed Moth Flame algorithm (MMFA) to enhance the enactment of the expert-based DRL method. This combined method is denoted as GMFA-DRL.

To help multi-source information transfer produced from Web of Things gadgets, edge figuring arises as a promising registering design with low inactivity and high transmission capacity contrasted with distributed computing. To upgrade the presentation of edge figuring inside restricted correspondence and calculation assets, we concentrate on a cloud-edge-end registering engineering, wherever one fog attendant and numerous computational ways can cooperatively method the figure escalated information streams that come from different sources. Wu et al. [33] have developed a Proximal Policy Optimization (PPO) that is utilized for the information brook divesting task, while the curved improvement is utilized for the asset allotment.

The trillion-overlap expansion in figuring power carries the openness of profound figuring out how to everybody. Profound learning offers exact data practically all when contrasted with other learning calculations. On the other pointer, the ubiquity of IoT has expanded in numerous areas such as Smart Cities, Petroleum Extraction, and Transference. Edge/Haze figuring climate assists with taking care of huge difficulties faced by the IoT, viz. inertness, transmission capacity utilization, and never-ending network availability. Sankaranarayanan et al. [28] have developed a DDNN-established IoT-Edge typical of the dormancy and increment exactness beginning from the information age stage.

AlQerm et al. [3] introduced the Deep Edge framework to tackle resource allocation challenges in Edge-IoT systems, aiming to optimize users’ Quality of Experience (QoE). The framework includes a novel QoE model that considers aligning diverse IoT applications’ requirements with available edge resources. Additionally, a distinctive two-stage DRL scheme is proposed, leveraging deep neural networks (DNN) for improved action exploration. This scheme maps the Edge-IoT state involves a collaborative resource-sharing strategy that encompasses both resource allocation and Quality of Service (QoS) classification. The goal is not only to maximize users’ Quality of Experience (QoE) and meet diverse application requirements but also to synchronize QoS provisions with the existing resources. Evaluation results demonstrate significant enhancements in QoE, latency, and the success ratio of application tasks facilitated by the DeepEdge framework.

Due to the partial dispensation and storage capabilities of IoT devices, handling the substantial computations required by AI applications, such as Convolutional Neural Network (CNN) processing for object detection and image classification, poses a challenge. To optimize overall system latency and reduce energy consumption by IoT devices, leveraging edge servers in the environment becomes crucial. Aghapour et al. [2] address these challenges by proposing a DRL approach that divides the offloading and resource allocation problem into two sub-problems. This algorithm continually adjusts the offloading policy in response to environmental information and, aided by the Salp Swarm Algorithm (SSA), optimizes resource allocation. Simulation results demonstrate that the algorithm put forth demonstrates the minimal cost concerning latency and power consumption, attaining an average enhancement of 92%, 17%, and 12% in comparison to full local, full offload, and joint resource allocation and computation Offloading PSO (JROPSO) methods, respectively.

The emergence of the IoT has brought about Shifts in network architectures and communication dynamics which has repercussions on the security market. This has introduced added complexity to activities such as traffic flow analysis, classification, and detection. To discourse this contest, D’Angelo et al. [6] have projected a theory centered on vibrant non-linear schemes. This theory effectively captures and comprehends the intricate dynamics of Internet traffic presented as Recurrence Plots. The approach utilizes Convolutional Autoencoders to extract significant features from the generated plots, the results were derived from an authentic dataset, showcasing the efficacy of the proposed method, and surpassing the performance of state-of-the-art classifiers.

Many companies grapple with the common task of Resource Planning Optimization (RPO) to attain various benefits, such as budget enhancements and run-time analyses. Despite the conventional use of multiple software products and tools to address this challenge, the considerable success and validity of Exploring alternative solutions for optimization problems, Artificial Intelligence-based approaches have shown substantial promise in various research fields. D’Angelo et al. [7] addressed an RPO problem related to scheduling diverse Combined Heat and power (CHP) generators by employing multiple Artificial Neural Networks (ANNs). The effectiveness of the proposed approach was affirmed through experimental results obtained from real Microgrid system data.

3. Model of the system and the problem statement

The contest of resource distribution in IoT edge cloud calculating settings is compounded by a high-dimensional and complex action space. As the amount of edge nodes, cloud servers, and IoT devices increases, ensuring the scalability of resource allocation algorithms becomes crucial. This scalability challenge emphasizes the need for resource allocation strategies that can adapt and scale seamlessly with the dynamic nature of IoT edge environments. This complexity can lead to a steeper learning curve for developers and may result in higher development and maintenance costs. To address these challenges and streamline resource allocation in IoT edge cloud computing, there is a need for innovative methodologies that enhance scalability, simplify implementation, and optimize computational efficiency. Our proposed methodology aims to bridge these gaps by providing a robust and accessible solution that considers the evolving landscape of IoT edge computing. Through careful design and integration of DRL algorithms, our approach seeks to make resource allocation in IoT edge cloud environments more manageable, even for developers with limited expertise in deep learning and reinforcement learning. By prioritizing scalability, simplicity, and efficiency, our proposed methodology aims to contribute to the broader adoption of intelligent resource allocation strategies in the rapidly evolving domain of IoT edge cloud computing. Minimize the time it takes for content requests to be fulfilled by making intelligent decisions on where to cache content and how to allocate resources. Ensure that edge resources are allocated judiciously to meet content demand, avoiding unnecessary processing at central cloud nodes when local processing is feasible.

The challenges include the limited dispensation and storing competencies of IoT edge devices, varying content demands, and the need to minimize latency. The Caching-based Deep Q-Network (CbDQN) is proposed to address these challenges learning an effective strategy for caching content at edge nodes and making decisions on resource allocation. The CbDQN method aims to address the challenge of resource sharing for content distribution within IoT edge cloud computing situations to determine how to allocate computing, caching, and communication resources to minimize latency and network congestion. The caching system determines what content to store at edge nodes based on historical access patterns. The Deep Q-Network makes decisions on resource allocation, caching, and content distribution. The DQN continuously learns from the system’s past interactions to optimize decision-making. Past experiences are stored and replayed to improve learning stability. Rewards are defined based on factors like latency reduction, efficient resource utilization, and successful content distribution.

3.1. Background

In the context of IoT edge cloud computing, resource allocation for content distribution is a crucial challenge. The increasing number of edge nodes, cloud servers, and IoT devices intensifies the complexity of managing resources effectively. The scalability of resource allocation algorithms becomes paramount as IoT environments dynamically evolve. Integrating DRL, particularly the CbDQN approach, aims to address these challenges by optimizing decision-making for caching, resource allocation, and content distribution. The goal is to minimize latency, enhance computational efficiency, and streamline resource allocation in the rapidly evolving domain of IoT edge cloud computing.

3.2. K-fold cross-validation

A recommended best practice for evaluating both ML and DL models involves employing multiple partitions of the dataset rather than a simple binary split. The K-Fold cross-validation procedure is a commonly utilized evaluation technique for this purpose. This algorithm randomly divides the dataset into k around equal-size subsets or folds. Initially, the first fold serves as the test set, and the perfect is trained on the outstanding $k - 1$ folds. Subsequently, a different fold is designated as the test set, with the remaining $k - 1$ folds utilized as the preparation set. In practical applications, K-Fold cross-validation is often conducted with k set to 5 or 10, as these values are recommended for achieving robust model validation. In our specific case, the evaluation and validation of each Artificial Neural Network (ANN) leverages the 70/30 criteria in conjunction with the K-Fold cross-validation process, using $k = 5$ . This approach ensures a comprehensive assessment by exposing each ANN to various training and testing set instances [7].

4. Proposed methodology

A novel Caching-based Deep Q Network (CbDQN) is developed in this research work regarding Distributing Content in Environments of IoT edge cloud computing. The proposed method incorporates with caching method and deep Q neural network. The caching method is used for improving system performance, reducing latency, and optimizing network usage. The DQNs are used for resource allocation tasks. Resource allocation is distributing limited resources to maximize efficiency, minimize costs, or improve performance. DQNs are a kind of RL process that can learn how to make decisions about resource allocation in complex and dynamic environments. Figure 1 shows the proposed architecture of CbDQN.

Address the challenge of optimizing network delay in a scenario involving the IoV, which comprises RSUs, BSs, and cloud resources. The Proposed approach involves enhancing content distribution and network QoS through collaborative content caching and request routing across multiple layers. To tackle the issue of delay optimization, we introduce a novel policy based on a Caching-based deep Q network (CbDQN), which addresses challenges by learning effective strategies for caching and source allocation. The method proposes to optimize decision-making by using a Deep Q-Network that learns from past interactions, considering factors like latency reduction and efficient resource utilization. Overall, the goal is to streamline resource allocation in IoT edge cloud computing, prioritizing scalability, simplicity, and efficiency. This policy leverages historical request data and current network conditions to make intelligent decisions regarding content caching and request routing. Our proposed solution’s effectiveness is assessed across various system conditions. Through extensive simulations using real-world data, we demonstrate that our strategy significantly reduces network latency compared to existing cloud-edge collaboration systems. Furthermore, the introduced DQN model showcases adaptability to shifting network states and user requirements while achieving rapid convergence. Finally, the show is evaluated in terms of sensitivity, precision, specificity, accuracy, F-score, and AUC.

Fig. 1.

The proposed architecture of CbDQN.

4.1. DRL

DRL is a specialized branch of ML that associations RL and DL approaches to enable artificial agents to learn how to make decisions and take actions in a setting to achieve specific goals. It is particularly effective in handling complex tasks with large state and action spaces, where traditional RL methods become impractical due to the expletive of dimensionality. In RL, agents engage with an environment in distinct time increments. During each of these increments, the agent observes the current public of the atmosphere and then proceeds to take action. The situation changes to a new state based on the agent’s exploit, and the cause receives a numerical reward or penalty as feedback. The objective of the agent is to acquire a plan that connects positions to actions, aiming to exploit the increasing reward over time. Traditional RL methods typically use tabular representations to store state-action values, limiting their scalability to large state spaces. DRL addresses this limitation by employing deep neural networks as function approximates to represent the policy or value function. Deep neural networks can learn hierarchical and non-linear representations from high-dimensional input data, making them suitable for complex tasks with images, audio, or other continuous data.

4.2. DRL-based supply provision scheme

In IoT edge fog calculating atmospheres, resource allocation for content distribution entails the efficient allocation and oversight of computational, storage, and network resources. This facilitates the seamless distribution of content from cloud servers to edge devices. Proper resource allocation is crucial for optimizing content delivery, minimizing latency, and ensuring the smooth operation of IoT applications. Allocate sufficient network bandwidth to accommodate the volume of content distribution while minimizing congestion and latency. Assign appropriate network resources to ensure timely and reliable content delivery, meeting the QoS supplies of IoT applications. The energy ingestion of edge devices when allocating resources, optimizing resource usage to prolong device lifetimes. Utilizing RL, particularly DRL, in tandem with a software-defined controller operating as a DRL agent, facilitates efficient management of distinct time slots. In each state, the program adeptly administers corresponding interventions as outlined below [17] $\begin{matrix} (1) & y_{r} = {φ_{r}, ψ_{r}, γ_{r}, P_{r}} \end{matrix}$

Where, $φ_{r}$ collective scheme cost at time, $ψ_{r}$ is the vectors of length l. The computing nodes’ computational capabilities are demonstrated using the following approach. $\begin{matrix} (2) & A_{k} = K - \sum_{m = 1}^{l} β_{i}, K_{j} \end{matrix}$

Where, $A_{k}$ a capacity that any computer node k.

4.3. Cloud environment

A cloud environment is a gathering of hardware, software, and socializing funds that are hosted by a cloud provider and made available to users on demand. Cloud environments can be used to store data, run applications, and provide a variety of other services. A public cloud, owned and operated by a cloud provider, is accessible to the general public. Public clouds are usually the most cost-effective choice, although they might not provide the same level of security and control found in other cloud environments [23]. On the other hand, a private cloud is exclusively owned and managed by a single organization. Private clouds are generally more secure and dependable than public ones, but they can also come with higher costs. A hybrid cloud, which combines both public and private cloud components, offers the advantages of public clouds in terms of flexibility and scalability while maintaining the security and control associated with private clouds. Cloud environments can be scaled up or down as needed, making them ideal for businesses with fluctuating workloads. Cloud environments can be more profitable than traditional on-premises explanations, especially for businesses with high IT demands. Cloud environments can be more secure than traditional on-premises solutions, as they are protected by the security measures of the cloud provider. Cloud environments can be more vulnerable to security attacks than traditional on-premises solutions [22].

4.4. Cloud calculation model

The fog center can take and process tasks from automobiles using wireless messages. It possesses the most significant computational authority among all the coats. When the vehicle decides to transfer the task to a fog center, the computation task’s postponement is represented as $B_{t}^{e}$ . This delay solely encompasses the processing time of the tasks within the cloud center [19].

Let $g_{e}$ be the fog server subtraction power, measured in CPU cycles per second. Therefore, the overall postponement in the cloud center, $B_{t}^{e}$ represented, can be expressed as follows $\begin{matrix} (3) & B_{t}^{e} = \frac{Z_{t}}{g_{e}} \end{matrix}$

The consumed energy, denoted $G_{t}^{e}$ , encompasses the computational expenses within the cloud center. $\begin{matrix} (4) & G_{t}^{e} = D_{m}^{τ} B_{t}^{e} \end{matrix}$ $D_{m}^{τ}$ is the computing power used by a cloud center per second

Additionally, the cloud center obtains task processing results from both vehicles and MEC servers. Utilizing these amalgamated processing outcomes, the cloud center can formulate informed scientific decisions.

4.5. Caching request routing

Storing content requests in IoT edge cloud computing environments for content distribution. Involves storing and retrieving frequently accessed content at the control strategies to improve the efficiency of content delivery, reduce latency, and minimize the use of network resources. Caching is a key strategy to enhance the performance and responsiveness of IoT systems by minimizing the need to fetch content from remote cloud servers. Caching involves storing copies of frequently accessed data or resources in a location that allows for quicker retrieval. When a request for a specific resource arrives, the organization checks the cache first. If the resource is created in the cache, it can be served immediately without having to go through the potentially slower process of fetching it from the source.

Request routing refers to the process of directing incoming requests to the appropriate destination based on predefined rules or algorithms. In the context of caching, request routing determines whether a request should be directed to the cache or the source, based on factors like cache availability, freshness of the cached content, and the type of content being requested.

4.6. Delay model

The time it takes for vehicles to receive their desired content is determined by two factors: the transmission latency involved in sending data over the network and the time it takes for the content to stay within service nodes to proceed with content requests.

4.6.1. Spread delay

Two transmission delay models exist, differing according to the types of links present in the network. One model concerns the round-trip delay for content retrieval between the moveable automobile i and the accessed RSU j (where “i” belongs to the set of RSU (roadside units) for content l is denoted as [5]. $\begin{matrix} (5) & H_{i, j}^{l} = \frac{A_{i, j}^{l}}{l_{i, j}} + \frac{A_{j, i}^{l}}{l_{j, i}} \end{matrix}$

Where, $l_{i, j}$ and $l_{j, i}$ are the accessible wireless connection bandwidths $c_{i, j} c_{j, i}$ and correspondingly? $A_{i, j}^{l}$ and $A_{j, i}^{l}$ are their content traffic volume l?

Likewise, the round-trip delay for content retrieval between the node k and its neighbouring node j (where both $k j$ belong to the set) in the IoV system for content l is expressed as $H_{j, k}^{l}$ [5] $\begin{matrix} (6) & H_{j, k}^{l} = \frac{A_{j, k}^{l}}{l_{j, k}} + \frac{A_{k, j}^{l}}{l_{k, j}} \end{matrix}$

Where, $l_{j, k}$ and $l_{k, j}$ are the accessible wired connection bandwidths $c_{k, j} c_{j, k}$ and respectively. $A_{k, j}^{l}$ and $A_{j, i}^{l}$ are their content traffic volume l.

4.6.2. Computation delay

Considering the limitations in computational resources for nodes at the edges, we establish a queueing system for each edge server. Let $Y_{i} (t)$ represent the count of nodes currently in the queue of a subsystem i at the start of the time slot t. As time progresses, the queue length evolution $Y_{i}$ can be described as follows [5] $\begin{matrix} (7) & Y_{i} (t + 1) = max [Y_{i} (t) + Z_{i} (t) - S_{i} (t), 0] \end{matrix}$

Here, $Z_{i} (t)$ signifies the number of incoming jobs, while $S_{i} (t)$ corresponding to the jobs being serviced within the subsystem i during the time slot t. The determination $Z_{i} (t)$ is established by the following method, $\begin{matrix} (8) & Z_{i} (t) = Z (j, k) (t) \end{matrix}$

Designate $Y m_{i} (t)$ as the workload that takes into account both the node count and the sizes in the queue of subsystems i at the outset of the time slot t. Consequently, we arrive at the following: $\begin{matrix} (9) & Y m_{i} (t + 1) = max [Y m_{i} (t) + Z m_{i} (t) - \frac{L_{i}}{λ}, 0] \end{matrix}$ $\frac{L_{i}}{λ}$ represents the processing capacity of the server in a single time slot, while $Z m_{i} (t)$ stands for the cumulative nodes that enter during the time slot t.

4.7. Energy consumption method

An energy consumption perfect is a representation or simulation of how energy is used within a system, organization, building, or any other context where energy usage is relevant. Such models are designed to help understand, predict, and optimize energy consumption patterns. They can vary in complexity, ranging from simple calculations to sophisticated simulations that take into account numerous variables and factors. There are multiple models of computation energy usage for the cloud and the edge node.

4.7.1. Energy consumption for computing in an edge node

Edge computing entails the processing of data in proximity to its source, rather than transmitting it to a centralized cloud server. This method provides reduced latency reduced network traffic, and the aptitude to method time-sensitive records in real-time. However, edge devices often have limited computational resources and are often powered by batteries or other energy-constrained sources. Optimizing energy consumption in edge nodes is crucial to prolong battery life and ensure reliable operation. The compute energy usage of an edge node as a function of workload, is a strictly convex, monotonically growing occupation. Define $H_{i} (t)$ as the energy consumption for computation at the edge node i, during the time slot t. This value is obtained through a reformulation of the following passage [12]. $\begin{matrix} (10) & H m_{i} (t) = x_{g} {(Z m_{i} (t))}^{2} + y_{g} (Z m_{i} (t)) + l_{g} \end{matrix}$

Here $x_{g} > 0$ , both $y_{g}$ a $l_{g}$ re non-negative factor parameters. $Z m_{i} (t)$ represents the combined workload during the time slot t.

4.7.2. Cloud computing energy consumption

The computation energy consumption in the cloud in the Distribution of content in IoT edge cloud computing settings is a major concern. The cloud is responsible for processing and storing the data created by IoT devices, which can lead to significant energy consumption. In addition, the cloud is often located far from the IoT devices, which can lead to additional energy consumption due to network latency. There are several ways to reduce the computation energy consumption in the cloud in the Distribution of content in IoT edge cloud computing settings. Edge computing brings computer means are brought closer to IoT devices, which can reduce the network latency and the amount of numbers that must be transmitted to the cloud. This can lead to significant energy savings. The computation energy consumption in the cloud slot t is given by [12], $\begin{matrix} (11) & G m (t) = \sum_{i \in N} G m_{(j, k)} (t) \end{matrix}$

4.7.3. Energy consumption during transmission

Energy consumption during data transmission is a critical consideration in various computing and communication systems, especially in resource-constrained environments like edge computing, wireless sensor networks, and IoT devices. Transmitting data requires energy for signal transmission, reception, and associated processing. Distribution of content in IoT edge cloud computing settings involves the efficient dissemination of data, software updates, and other content to devices situated at the authority of the link. This process requires careful consideration of various factors, including transmission energy consumption, as energy effectiveness is crucial for delaying the operational lifecycle of edge devices and reducing overall energy costs. Energy consumption of transmission in the path from i and j is given by [12], $\begin{matrix} (12) & G m_{(i, j)} (t) = x_{(i, j)} β m y_{(i, j)} (t) \end{matrix}$

Where the transmission power along this line is $x_{(i, j)} > 0$ , β is a constant.

Table 1 shows the algorithm of the proposed method of CbDQN.

Table 1
Algorithm of proposed method of CbDQN

Step 1 import the IoT dataset. This section likely involves importing the dataset related to IoT devices. IoT datasets often include information collected from various sensors and devices.

Step 2 request routing. This section focuses on routing requests efficiently, aiming to reduce latency (delay) and minimize the utilization of network resources. Efficient routing is crucial for timely and effective communication between devices.

Step 3 optimize the delay. This part seems to emphasize the need to minimize latency and ensure the timely execution of tasks in IoT edge cloud computing environments. Latency optimization is vital for real-time or near-real-time applications.

Step 4 transmission delay. The code snippet suggests the use of variables (i, j, k) and mentions a round-trip delay for content retrieval between nodes, using an equation labeled as “eqn (6).” This involved calculations related to data transmission delays.

Step 5 computation delay. This section involves computational tasks related to edge nodes, utilizing variables (Y, t, Z, S) and an equation labeled as “eqn (7).” It likely deals with optimizing computational resources for efficient processing at the edge.

Step 6 resource allocations. This part focuses on optimizing resource allocation, particularly for content delivery. Efficient resource allocation is crucial for maximizing the performance of edge computing systems.

Step 7 performance analysis. The final section involves evaluating the performance of the system. The analyzing metrics related to latency, accuracy, throughput, resource utilization, and other relevant parameters to ensure the system meets its performance goals.

5. Results and discussion

This section outlines the assessment of IoT edge cloud computing environments and examines the findings from the simulation outcomes. The proposed method is optimizing maximizing efficiency and improving performance. This method evaluates metrics used to evaluate the efficiency of the framework for the Distribution of content in IoT edge cloud computing settings.

5.1. Dataset explanation and experimental setup

The IoT constitutes a network of interconnected computing devices, mechanical and digital machines, objects, animals, or people equipped with unique identifiers (UIDs). This network enables the seamless transfer of data over a network without necessitating direct human-to-human or human-to-computer interaction. The definition of the Internet of Things has evolved due to the convergence of various technologies, including real-time analytics, machine learning, commodity sensors, and embedded systems. In the consumer market, IoT technology is most commonly associated with products that align with the concept of the “smart home.” This course comprehensively covers the fundamentals of IoT products and services. This encompasses devices designed for sensing, actuation, processing, and communication, as well as various protocols and analytics relevant to IoT. Table 2 shows the parameter description.

Table 2
Parameter description

Processor Windows 10 Pro

Version 22H2

Processor Intel(R) Core(TM) i5-3470 CPU @ 3.20 GHz 3.20 GHz

RAM 8.00 GB (7.88 GB usable)

Platform Python

5.2. Performance metrics

This unit evaluates the presentation of the projected approach and outlines the experimental setup. Various metrics such as Precision, F-score, Sensitivity, Accuracy, Specificity, and AUC are employed to enhance system performance, minimize latency, and optimize network utilization.

5.3. Performance estimation

The comparative analysis of DRL-based reserve allocation for satisfied delivery in IoT edge cloud calculating settings is more effective compared to other techniques. The performance metrics are evaluated in terms of Accuracy, precision, sensitivity, specificity F-score, and AUC. Moreover, the existing techniques like Multi-Layer Perceptron (MLP), ANN, J48 trees (J48), and Naive Bayes (NB) [7].

5.3.1. Comparison of current approaches in terms of accuracy

Figure 2 shows compare the suggested CbDQN and present methods in standings of accuracy. The planned value for the CbDQN accuracy is 99.85%. It has high accuracy compared to other existing methods. The existing methods of ANN accuracy value is 96.91% it has less compared to existing methods except MLP and NB methods. The other methods of J48 have high accuracy values compared to all other existing methods except the proposed method. The value is 99.56%. The other method of NB has an accuracy value is 90.87%. It has less value compared to all other existing methods except MLP. The MLP has a very low accuracy value of 84.99% compared to all other current earnings. Therefore our offered method attains high accuracy compared to all other methods.

Fig. 2.

Compares the suggested CbDQN and existing methods in terms of accuracy.

5.3.2. In comparison to existing methods, it differs significantly in terms of precision

Fig. 3.

Comparison of proposed CbDQN in terms of precision.

Figure 3 shows the Judgment of current techniques in standings of precision. The CbDQN model demonstrates an impressive precision of 99.85%, surpassing other established methods. Among them, ANN achieves a precision of 90.31%, falling short in comparison to most methods except MLP. Notably, the J48 method stands out with a precision of 99.56%, outperforming all other methods except the proposed one. On the other hand, the NB method lags with a precision of 92.16%, ranking lower than all other methods except MLP and ANN. Speaking of MLP, it exhibits the lowest precision at 56.83% when associated with other current systems. In summary, our planned CbDQN process excels in achieving superior precision compared to all other tested methodologies.

5.3.3. Comparison of existing methods in terms of specificity

A comparison of proposed CbDQN in terms of specificity is illustrated in Fig. 4. The CbDQN model showcases an impressive Specificity rate of 99.82%, surpassing various established methods. In comparison, ANN achieves a Specificity rate of 97.84%, high when measured against most methods except J48. Particularly noteworthy is the J48 method, which stands out with a Specificity rate of 99.56%, higher than all other methods except our proposed one. Conversely, the NB method trails behind with a Specificity rate of 90.89%, ranking lower than all other methods except MLP. Speaking of MLP, it registers the lowest Specificity rate at 84.99% when associated with other current procedures. To sum up, our suggested CbDQN method excels in achieving a superior Specificity rate when compared to all other methodologies that were tested. Figure 4 shows the Comparison of current techniques in terms of specificity.

Fig. 4.

Comparison of proposed CbDQN in terms of specificity.

5.3.4. Comparison of standing approaches in terms of sensitivity

The CbDQN model impressively attains 99.82% sensitivity, outperforming various established methods. In contrast, ANN achieves 88.09% sensitivity, falling short when compared to most methods except MLP. Notably, the J48 method stands out with a sensitivity of 99.56%, surpassing all other methods except our proposed methods. Conversely, the NB method lags with a sensitivity of 90.89%, ranking lower than all other methods except J48. Regarding MLP, it records the lowest sensitivity at 84.99% when related to other current systems. In summary, our planned CbDQN system excels in achieving superior sensitivity compared to all other methodologies that were tested. Figure 5 displays the Contrast of current methods in standings of sensitivity.

Fig. 5.

Comparison of proposed CbDQN in terms of sensitivity.

5.3.5. Comparison of remaining approaches in positions of F-score

The F-Score of the CbDQN model is an impressive 99.82%, high several established methods. In contrast, ANN achieves an F-Score of 85.98%, falling short when compared to most methods except MLP. Notably, the J48 method stands out with an F-Score of 99.56%, surpassing all other methods except our proposed one. Conversely, the NB method lags with an F-Score of 91.26%, ranking lower than all other methods except MLP and ANN. The MLP method records the lowest F-Score at 52.96% when associated with other present approaches. In summary, our suggested CbDQN method excels in achieving superior F-Score compared to all other methodologies that underwent testing. Figure 6 shows the Contrast of existing approaches in positions of F-Score.

Fig. 6.

Comparison of proposed CbDQN in terms of F-score.

5.3.6. Comparison of remaining approaches in terms of AUC

The CbDQN model demonstrates an impressive AUC of 99.82%, surpassing various established methods. In contrast, the ANN achieves an AUC of 99.05%, falling short in comparison to most methods. Notably, the J48 method stands out with an AUC of 99.56%, surpassing all other methods except our proposed ones. Conversely, the NB method lags with an AUC of 90.89%, ranking lower than all other methods except MLP. In terms of MLP, it records the lowest AUC at 84.99% compared to other existing methods. In summary, our proposed CbDQN method excels in achieving superior sensitivity compared to all other methodologies that were tested. Figure 7 shows a comparison of the existing methodologies in terms of AUC. Table 3 shows the comparison values of accuracy, precision, specificity, sensitivity, AUC, and F-score.

Fig. 7.

Comparison of proposed CbDQN in terms of AUC.

Table 3

Comparison values of precision, AUC sensitivity, accuracy, specificity, F-score

Methods	Precision	Sensitivity	Accuracy	Specificity	F-score	Area Under the ROC Curve (AUC)
ANNs	90.31	88.09	96.91	97.84	85.98	99.05
J48	99.56	99.56	99.56	99.56	99.56	99.56
NB	92.16	90.89	90.87	90.89	91.26	90.89
MLP	56.83	84.99	84.99	84.99	52.96	84.99
CbDQN (Pro)	99.85	99.82	99.85	99.82	99.82	99.82

5.4. Discussion

Resource allocation is a caching-based deep Q network for the Distribution of content in IoT edge cloud computing settings involves optimizing content delivery by intelligently managing cached data at edge nodes. This approach leverages deep Q learning to dynamically allocate resources, balancing factors like content popularity, network conditions, and storage capacity. By efficiently utilizing edge caches, this method aims to enhance content availability, reduce latency, optimize network utilization, and improve overall performance and user experience in IoT edge cloud scenarios.

Fig. 8.

Overall performance of the suggested model.

Finally, the outcomes are estimated for accuracy, precision, sensitivity, specificity, F-score, and AUC. Moreover, the performance of the presented approach is validated with a comparative analysis. Using CbDQN Allocation of resources for content dissemination in IoT edge cloud computing settings enhances content delivery, reduces latency, and minimizes network congestion. The performance of the planned typical is illustrated in Fig. 8. The comparative enactment of the designed model shows that the designed model earned better results. Table 4 shows the Overall performance analysis of proposed CbDQN.

Table 4

Overall performance analysis of proposed CbDQN

Accuracy	99.85
Precision	99.85
Sensitivity	99.82
Specificity	99.82
F-score	99.82
AUC	99.82

6. Conclusion

This paper addressed the joint resource allocation problem for content distribution in an asymmetrical IoT-edge-cloud computing environment, aiming to minimize network delay. A novel CbDQN policy was proposed to handle this problem by making gratified caching and request steering decisions based on perceived request history and network state. CbDQN achieved efficient utilization of communication computing, and caching resources by dynamically adapting to network conditions and user demand. Compared to existing approaches, CbDQN significantly reduced network delay by leveraging caching at the edge and optimizing request routing. our proposed approach evaluated metrics achieves high values of Accuracy is 99.85%, Precision at 99.85%, specificity is 99.82%, sensitivity is 99.82%, F-score is 99.82% and AUC is 99.82%. In the future, mechanisms to address security and privacy concerns related to caching sensitive content at the edge. Extending the framework to consider other network elements, such as fog nodes and content delivery networks, for further performance optimization. Extensive simulations conducted under various network scenarios confirmed the effectiveness of the proposed CbDQN policy. The results demonstrated a significant reduction in network delay compared to baseline approaches, highlighting the potential of CbDQN for optimizing resource allocation in real-world IoT-edge-cloud environments.

Conflict of interest

None to report.

References

T.A.-N.

Abdali,

Hassan,

A.H.-M.

Aman and

Q.N.

Nguyen, Fog computing advancement: Concept, architecture, applications, advantages, and open issues, IEEE Access 9 (2021), 75961–75980. doi:10.1109/ACCESS.2021.3081770.

Aghapour,

Sharifian and

Taheri, Task offloading and resource allocation algorithm based on deep reinforcement learning for distributed AI execution tasks in IoT edge computing environments, Computer Networks 223 (2023), 109577. doi:10.1016/j.comnet.2023.109577.

AlQerm and

Pan, DeepEdge: A new QoE-based resource allocation framework using deep reinforcement learning for future heterogeneous edge-IoT applications, IEEE Transactions on Network and Service Management 18(4) (2021), 3942–3954. doi:10.1109/TNSM.2021.3123959.

Chen,

Sun,

Wang and

Taleb, Dynamic task allocation and service migration in edge-cloud iot system based on deep reinforcement learning, IEEE Internet of Things Journal 9(18) (2022), 16742–16757. doi:10.1109/JIOT.2022.3164441.

Cui,

Yang,

Fang and

Yu, Deep reinforcement learning-based resource allocation for content distribution in IoT-edge-cloud computing environments, Symmetry 15(1) (2023), 217. doi:10.3390/sym15010217.

D’Angelo,

Farsimadan and

Palmieri, Recurrence plots-based network attack classification using CNN-autoencoders, in: International Conference on Computational Science and Its Applications, Springer Nature Switzerland, Cham 2023, pp. 191–209.

D’Angelo,

Palmieri and

Robustelli, Artificial neural networks for resource optimization in an energetic environment, Soft Computing 26(4) (2022), 1779–1792. doi:10.1007/s00500-022-06757-x.

Elawady,

Sarhan and

M.A.

Alshewimy, Toward a mixed reality domain model for time-sensitive applications using IoE infrastructure and edge computing (MRIoEF), The Journal of Supercomputing 78(8) (2022), 10656–10689. doi:10.1007/s11227-022-04307-8.

Fang,

Xu,

Yang,

Hu,

Tu,

Ota,

Yang,

Dong,

Han,

F.-R.

Yu and

Liu, Deep-reinforcement-learning-based resource allocation for content distribution in fog radio access networks, IEEE Internet of Things Journal 9(18) (2022), 16874–16883. doi:10.1109/JIOT.2022.3146239.

10.

T.-R.

Gadekallu,

Q.-V.

Pham,

D.-C.

Nguyen,

P.K.-R.

Maddikunta,

Deepa,

Prabadevi,

P.-N.

Pathirana,

Zhao and

W.-J.

Hwang, Blockchain for the edge of things: Applications, opportunities, and challenges, IEEE Internet of Things Journal 9(2) (2021), 964–988. doi:10.1109/JIOT.2021.3119639.

11.

Garg,

Kaur,

Kaddoum,

Garigipati and

G.-S.

Aujla, Security in IoT-driven mobile edge computing: New paradigms, challenges, and opportunities, IEEE Network 35(5) (2021), 298–305. doi:10.1109/MNET.211.2000526.

12.

Guo,

Li and

Guan, Energy-efficient and delay-guaranteed workload allocation in IoT-edge-cloud computing systems, IEEE Access 7 (2019), 78685–78697. doi:10.1109/ACCESS.2019.2922992.

13.

Gupta and

Singh, Fog-GMFA-DRL: Enhanced deep reinforcement learning with hybrid grey wolf and modified moth flame optimization to enhance the load balancing in the fog-IoT environment, Advances in Engineering Software 174 (2022), 103295. doi:10.1016/j.advengsoft.2022.103295.

14.

Javadpour,

Wang and

Rezaei, Resource management in a peer-to-peer cloud network for IoT, Wireless Personal Communications 115 (2020), 2471–2488. doi:10.1007/s11277-020-07691-7.

15.

Ji,

Alfarraj and

Tolb, Artificial intelligence-empowered edge of vehicles: Architecture, enabling technologies, and applications, IEEE Access 8 (2020), 61020–61034. doi:10.1109/ACCESS.2020.2983609.

16.

Jiang,

Sun,

Zhou,

Li,

Dong and

V.-C.

Leung, Intelligence-empowered mobile edge computing: Framework, issues, implementation, and outlook, IEEE Network 35(5) (2021), 74–82. doi:10.1109/MNET.101.2100054.

17.

Lakhan,

M.A.

Mohammed,

O.-I.

Obaid,

Chakraborty,

K.-H.

Abdulkareem and

Kadry, Efficient deep-reinforcement learning aware resource allocation in SDN-enabled fog paradigm, Automated Software Engineering 29 (2022), 1–25. doi:10.1007/s10515-021-00310-0.

18.

Liyanage,

Porambage,

A.-Y.

Ding and

Kalla, Driving forces for multi-access edge computing (MEC) IoT integration in 5G, ICT Express 7(2) (2021), 127–137. doi:10.1016/j.icte.2021.05.007.

19.

Long,

Luo,

Zhu,

Luo and

Huang, Computation offloading through mobile vehicles in IoT-edge-cloud network, EURASIP Journal on Wireless Communications and Networking 2020(1) (2020), 1–21. doi:10.1186/s13638-020-01848-5.

20.

Mekrache,

Bradai,

Moulay and

Dawaliby, Deep reinforcement learning techniques for vehicular networks: Recent advances and future trends towards 6G, Vehicular Communications 33 (2022), 100398. doi:10.1016/j.vehcom.2021.100398.

21.

Mishra and

A.-K.

Tyagi, The role of machine learning techniques in Internet of things-based cloud applications, Artificial intelligence-based Internet of Things systems (2022) 105–135.

22.

Neelakantan, A secure framework for the cloud to protect the virtual machine from malicious events, Multimedia Tools and Applications (2023), 1–24.

23.

Neelakantan and

N.S.

Yadav, An optimized load balancing strategy for an enhancement of cloud computing environment, Wireless Personal Communications (2023), 1–21.

24.

Neves and

Neto, Deep reinforcement learning applied to an assembly sequence planning problem with user preferences, The International Journal of Advanced Manufacturing Technology 122(11–12) (2022), 4235–4245. doi:10.1007/s00170-022-09877-8.

25.

T.T.

Nguyen and

V.J.

Reddi, Deep reinforcement learning for cyber security, IEEE Transactions on Neural Networks and Learning Systems (2021).

26.

Ó.

Pérez-Gil,

Barea,

López-Guillén,

L.M.

Bergasa,

Gomez-Huelamo,

Gutiérrez and

Diaz-Diaz, Deep reinforcement learning based control for autonomous vehicles in CARLA, Multimedia Tools and Applications 81(3) (2022), 3553–3576. doi:10.1007/s11042-021-11437-3.

27.

Rathore,

J.-H.

Park and

Chang, Deep learning and blockchain-empowered security framework for intelligent 5G-enabled IoT, IEEE Access 9 (2021), 90075–90083. doi:10.1109/ACCESS.2021.3077069.

28.

Sankaranarayanan,

J.J.

Rodrigues,

Sugumaran and

Kozlov, Data flow and distributed deep neural network based low latency IoT-edge computation model for big data environment, Engineering Applications of Artificial Intelligence 94 (2020), 103785. doi:10.1016/j.engappai.2020.103785.

29.

Sellami,

Hakiri,

S.-B.

Yahia and

Bertho, Energy-aware task scheduling and offloading using deep reinforcement learning in SDN-enabled IoT network, Computer Networks 210 (2022), 108957. doi:10.1016/j.comnet.2022.108957.

30.

Tian,

Xu,

Lin,

Cheng,

Qian,

Ren and

Bilal, DIMA: Distributed cooperative microservice caching for the Internet of things in edge computing by deep reinforcement learning, World Wide Web 25(5) (2022), 1769–1792. doi:10.1007/s11280-021-00939-7.

31.

Tran-Dang,

Bhardwaj,

Rahim,

Musaddiq and

D.-S.

Kim, Reinforcement learning based resource management for fog computing environment: Literature review, challenges, and open issues, Journal of Communications and Networks 24(1) (2022), 83–98. doi:10.23919/JCN.2021.000041.

32.

Wang,

Jin,

Yang,

Zhang,

Nallanathan,

Tellambura and

Jabbari, Task offloading with multi-tier computing resources in next-generation wireless networks, IEEE Journal on Selected Areas in Communications 41(2) (2022), 306–319. doi:10.1109/JSAC.2022.3227102.

33.

Wu,

Cai,

Bi,

Xia,

Gao,

Tang and

Lai, Intelligent resource allocation scheme for cloud-edge-end framework aided multi-source data stream, EURASIP Journal on Advances in Signal Processing 2023(1) (2023), 56. doi:10.1186/s13634-023-01018-x.

34.

Yu,

Chen,

Zhou,

Gong and

Wu, When deep reinforcement learning meets federated learning: Intelligent multitimescale resource management for multiaccess edge computing in 5G ultradense network, IEEE Internet of Things Journal 8(4) (2020), 2238–2251. doi:10.1109/JIOT.2020.3026589.

35.

Yu,

Chen,

Zhou,

Gong and

Wu, When deep reinforcement learning meets federated learning: Intelligent multi timescale resource management for multiaccess edge computing in 5G ultradense network, IEEE Internet of Things Journal 8(4) (2020), 2238–2251. doi:10.1109/JIOT.2020.3026589.

36.

Zheng,

Wan,

Zhang and

Jiang, Deep reinforcement learning-based workload scheduling for edge computing, Journal of Cloud Computing 11(1) (2022), 3. doi:10.1186/s13677-021-00276-0.

Processor	Windows 10 Pro
Version	22H2
Processor	Intel(R) Core(TM) i5-3470 CPU @ 3.20 GHz 3.20 GHz
RAM	8.00 GB (7.88 GB usable)
Platform	Python

Resource allocation for content distribution in IoT edge cloud computing environments using deep reinforcement learning

Abstract

Keywords

1. Introduction

2. Related works

3. Model of the system and the problem statement

3.1. Background

3.2. K-fold cross-validation

4. Proposed methodology

4.2. DRL-based supply provision scheme

4.3. Cloud environment

4.4. Cloud calculation model

4.5. Caching request routing

4.6. Delay model

4.6.1. Spread delay

4.6.2. Computation delay

4.7. Energy consumption method

4.7.1. Energy consumption for computing in an edge node

4.7.2. Cloud computing energy consumption

4.7.3. Energy consumption during transmission

Table 1 Algorithm of proposed method of CbDQN

5.1. Dataset explanation and experimental setup

Table 2 Parameter description Processor Windows 10 Pro Version 22H2 Processor Intel(R) Core(TM) i5-3470 CPU @ 3.20 GHz 3.20 GHz RAM 8.00 GB (7.88 GB usable) Platform Python

5.3. Performance estimation

5.3.1. Comparison of current approaches in terms of accuracy

Conflict of interest

References

Table 1
Algorithm of proposed method of CbDQN

Table 2
Parameter description

Processor Windows 10 Pro

Version 22H2

Processor Intel(R) Core(TM) i5-3470 CPU @ 3.20 GHz 3.20 GHz

RAM 8.00 GB (7.88 GB usable)

Platform Python