Application of deep reinforcement learning algorithms for automatic threat detection and response in dynamic network environments to improve cybersecurity

Abstract

In the present environment, as cybersecurity attacks become more sophisticated and frequent, classic protection techniques, such as rule-based firewalls and signature-based detection systems, are no longer effective. Modern cyberattacks require innovative solutions that can change and react in real time. Deep reinforcement learning (DRL) is a subfield of artificial intelligence that efficiently solves complicated decision-making issues in numerous domains, such as cybersecurity. This research aims to improve cybersecurity by establishing deep reinforcement learning algorithms for automated threat detection and response in dynamic network environments. To address these problems, this study proposed a novel Improved Mayfly Optimized Deep Deterministic Policy Gradient (IMO-DDPG) to automate threat detection and improve cybersecurity response in dynamic network environments. Collect real-time network intrusion datasets accessible to the public to train the DRL model. The data was preprocessed using normalization, for feature extraction Principal Component Analysis (PCA). To improve performance in an increasingly developing environment, the study customizes DRL algorithms with an emphasis on continuous action spaces, adversarial training, and customized reward structures. The results indicate that the IMO-DDPG method outperforms accuracy (94.5%), recall (94.0%), precision (92.5%), and F1-score (93.5%) to compare existing algorithms. The remarkable success rate, highest average reward, and superior efficiency in interpretation significantly improve threat detection and response capabilities. The results show the suggested algorithms for automated detection and response, highlighting the right decision, can significantly enhance cybersecurity defences.

Keywords

automatic threat detection cybersecurity dynamic network environments improved Mayfly Optimized Deep Deterministic Policy Gradient

Introduction

In the rapidly changing landscape of the digital world, various types of cybersecurity attacks are occurring with increasing frequency and complexity, often on an hourly basis. However, modern threats often circumvent traditional security measures such as rule-based firewalls and signature-based detection systems.¹ To address the growing sophistication of cyberspace, new solutions are required that can respond in real-time with the appropriate responses at the optimal moment—solutions not yet fully investigated or developed.² This paper focuses on the use of automated threat detection and response mechanisms that can successfully operate in dynamic network environments to enhance cybersecurity.³ As organizations increasingly rely on digital infrastructure, the urgency to identify and mitigate cyber threats swiftly becomes paramount. By employing artificial intelligence (AI) to develop systems capable of detecting security threats in networks characterized by numerous dynamic connections, these systems not only identify potential risks but can also learn from experience and improve their capabilities. Such dynamic systems must remain relevant by continuously adapting to emerging threats.⁴

A significant portion of this research is dedicated to leveraging real-time network intrusion data to construct more sophisticated models aimed at enhancing threat detection.⁵ These models are designed to improve performance in increasingly complex and evolving environments through continual learning and adaptation.⁶ The necessity for high levels of automation in decision-making processes is critical to significantly enhance cybersecurity capabilities. The results demonstrate a marked improvement in the effectiveness of automated threat detection and response techniques, affording better success rates and greater efficiency than existing solutions.⁷ The findings highlight the extent to which advanced AI solutions can redefine cybersecurity methodologies, enabling organizations to counter rapidly evolving and highly sophisticated cyber threats.⁸ Ultimately, this paper advances the ongoing discussion regarding cybersecurity, providing evidence that proactive and adaptive strategies must be integrated into the adoption of such technologies.⁹ As the cyber threat landscape continues to evolve, incorporating advanced technologies within cybersecurity frameworks will be essential for protecting digital assets and maintaining organizational resilience, representing a crucial step toward a secure digital future. An automated threat detection and response system embedded within dynamic network environments greatly reduces exposure to evolving cyber threats.¹⁰

This research aims to develop Deep Reinforcement Learning (DRL) algorithms for automated threat identification and response in dynamic network environments, with the ultimate goal of enhancing cybersecurity. The main contributions of this work are as follows:

(1) The collection of relevant data: Real-time network intrusion datasets are gathered.

(2) Pre-processing through the application of the Z-score, with Principal Component Analysis (PCA) employed to extract the most important features from the pre-processed dataset.

(3) The introduction of a novel method called the IMO-DDPG, which aims to provide enhanced detection and improve cybersecurity posture in complex networks automatically.

Related works

Shah et al. quantitatively assessed the application of machine learning (ML) algorithms in cybersecurity and concluded that these algorithms are beneficial for detecting and preventing various types of threats.¹¹ A substantial amount of data has been processed through ML algorithms and data-oriented approaches, enabling the identification of potentially suspicious activities. These algorithms continuously improve and learn from updates, acquiring knowledge from new input data and enhancing cybersecurity measures simultaneously. The general defense algorithms range from recognizing signs of well-known malware to protecting against previously unknown risks through anomaly detection.

Naseer et al. explored how businesses implement Real-Time Analysis (RTA) in the Incident Response (IR) process to enhance their cybersecurity performance.¹² By engaging a group of cybersecurity experts through interviews and applying the contingent resource-based view (RBV), the study yielded a framework illustrating how IR agility serves as a mediator between RTAC indirect effects and business cybersecurity outcomes.

Yungaicela-Naula et al. assessed recent studies on the automation of security within Software-Defined Networking (SDN) systems.¹³ The study identified and ranked various security measures based on differing levels of automation and complexity. The degree of automation was determined using four established quantitative parameters: adaptive behavior, self-healing, self-configuration, and automatic optimization. Complexity was defined by factors such as storage capacity, processing power, and implementation requirements.

Nikoloudakis et al. proposed a machine learning-based contextual awareness framework that leverages the real-time awareness feature of the SDN paradigm to identify both new and existing entities enabled by networks.¹⁴ The framework evaluated recognized threats and allocated suitable network resources for their connectivity. A dataset was utilized to train a machine learning-based intrusion detection system that continuously monitors the entities under evaluation.

Gudala et al. examined the potential for threat detection using artificial intelligence (AI) within Zero Trust Architecture (ZTA) frameworks.¹⁵ The authors discussed the foundational principles of ZTA and the inherent limitations of conventional security approaches. They emphasized the advantages of employing AI for anomaly detection, highlighting the technology’s capacity to identify minor deviations from established norms in user behavior, network traffic, and system configurations. This allows security personnel to take proactive measures to neutralize potential threats before they escalate into security incidents.

Repetto et al. presented a new methodology for managing the cybersecurity of web-based service chains.¹⁶ This innovative paradigm examines evolving norms for Information and Communication Technology (ICT) services while extending beyond traditional security perimeter models. The study emphasized the need for contemporary Security Information and Event Management (SIEM) designs to be reconfigured to accommodate dynamic topologies, multi-tenancy, and diverse administrative domains.

Mironeanu et al. described an approach for developing cybersecurity frameworks through the Experimental Cyber Attack Detection (ECAD) system, integrating various sources of information for security alerts.¹⁷ The system coordinated machine learning elements and security solutions to identify and mitigate cyberattacks. It offered an architectural model and a potential implementation strategy utilizing cutting-edge methods such as SDN and Infrastructure as Code (IaC).

Guo and Guo investigated the features of cloud computing technologies and proposed a cloud-based real-time risk assessment and mitigation strategy for intelligent ship network security.¹⁸ They developed a self-executing defense mechanism designed to spawn modules that prevent and reject attacks. Multi-sensor modules were primarily used to analyze data containing potentially harmful information.

Bringhenti et al. reviewed current methods for automating network security service configuration.¹⁹ The study identified the optimal approach for coordinating an entire service in a virtualized environment, concentrating on two distinct areas: service architecture and the actual deployment of composed functions. It evaluated prior works in each category, considering various criteria such as formal verification and the satisfaction of optimality requirements.

Rose et al. investigated the application of game theory, machine learning, and network characterization to protect the Internet of Things (IoT) against cyberattacks.²⁰ Their proposed anomaly-based intrusion detection system dynamically and actively monitored all networked devices to detect any attempts at IoT device manipulation and suspicious network transactions. Any deviation from the specified profile was treated as a threat and scrutinized further. The machine learning classifier also processed raw traffic to identify potential attacks.

Kandhro et al. introduced a novel deep learning-based method for identifying cybersecurity vulnerabilities and breaches in cyber-physical systems.²¹ The proposed framework contrasted unsupervised learning-based discriminative techniques and neural networks to detect cyber threats within IoT-driven Industrial Internet of Things (IIoT) networks, employing generative adversarial networks for enhanced detection capabilities.

Apruzzese et al. clarified the role of machine learning in cybersecurity by offering a comprehensive overview of its benefits, challenges, and future obstacles within the field.²² The focus was on the broader cybersecurity landscape, minimizing technical jargon to ensure accessibility to a wider audience. They provided a succinct description of ML applications in identifying three categories of online threats: malware, phishing, and network intrusions. The authors also highlighted additional domains within cybersecurity, including threat intelligence, alert management, cyber risk assessment, and raw data analysis, that could benefit from machine learning’s autonomy.

Gong and Lee presented a framework for analyzing threat indicators that could be generated from advanced metering infrastructure.²³ They proposed a strategy for producing cyber threat intelligence targeting the energy cloud. The research further suggested a mechanism for sharing and exchanging cyber threat intelligence between the Advanced Metering Infrastructure (AMI) and the cloud layer, facilitating the swift implementation of a security structure for large-scale energy cloud architectures.

Furdek et al. introduced a new functional block known as the Security Operation Center (SOC).²⁴ The paper detailed its architecture, defined essential specifications for its capabilities, and provided guidelines for integration with the optical layer controller. Additionally, the study incorporated unsupervised and semi-supervised neural networks, enhancing the effectiveness of machine learning-based safety diagnostic approaches to address users’ unfamiliarity with high-dimensional optical tracking data in the context of previously unidentified physical layer threats. It utilized three forms of dimensionality reduction and compared their performance against execution time complexity and machine learning accuracy.

Nespoli et al. proposed an innovative dynamic rule management system to adapt to the evolving conditions of IoT environments. Their experiments demonstrated that the system’s CPU and Random Access Memory (RAM) usage were significantly lower than those observed when employing traditional techniques.²⁵ Notably, there was a marked increase in security levels, as evidenced by a substantial rise in the number of packets processed per second.

Methodology

Real-time network intrusion datasets are collected from publicly available sources in this study and include various network attack scenarios to train and evaluate the deep reinforcement learning (DRL) algorithms and data preprocessed using Z-score normalization for feature extraction of the dataset using Principal Component Analysis (PCA). An improved Mayfly Optimized Deep Deterministic Policy Gradient method (IMO-DDPG) is proposed, which integrates improved optimization techniques, continuous action spaces, and tailored reward structures for automated detection and response of threats in dynamic network environments, rendering the performance of cybersecurity much superior to other algorithms. Figure 1 presents the methodology structure.

Figure 1.

Methodology structure.

Data collection

Data has been collected from the Kaggle source: https://www.kaggle.com/datasets/sampadab17/network-intrusion-detection. The network intrusion detection dataset on Kaggle is used to develop models for predicting network intrusions. However, the reliance on a single dataset has inherent limitations in diversity and may not fully represent the complexities of real-world scenarios. Future work should explore multiple datasets to capture a broader range of attack patterns and ensure a more comprehensive evaluation of the proposed methods.

Data preprocessing

By applying z-score normalization to network intrusion data, research computes the mean and variance for unit scaling, facilitating efficient comparison and processing of diverse data. This method standardizes the data which work on different types of intrusion data. This enables them to learn from other patterns of these threats and subsequently enhance their functionality in the detection and response to threats under different network conditions.

Z-score normalization

This technique is most widely applied. This converts the scores into a distribution with a standard deviation of 1 and a mean of 0. It is imperative that possess a firm grasp of score distribution prior to employing this methodology, as demonstrated by equation (1).

m = \frac{(T - μ)}{(σ)}

(1)

Feature extraction using PCA

When detecting threats in the new and dynamically changing data and network environments, a set of techniques is used to enhance this model’s efficiency, including Principal Component Analysis (PCA). The applicability of PCA in security systems can be viewed in the light of mitigation of complexity of network intrusion data to make the necessary processes involved in the security systems to process and analyze the data required for identification of the threats.

Principal component analysis

The PCA algorithm’s basic concept is as follows: In addition, it is composed of the centralized data sample ${w_{j}}_{j = 1}^{n}$ , where $w_{j} \in Q^{m}$ , given the input data matrix $W_{n \times m} (u s u a l l y n > m)$ is shown in equation (2).

\sum_{j = 1}^{n} w_{j} = 0

(2)

Using equation (3), PCA transforms the input vector into a new vector.

t_{j} = V^{S} w_{j}

(3)

The dataset covariance matrix $D^{'} s$ $j t h$ feature vector is represented by the column $v_{j}$ in $V$ , and an orthogonal vector of size $m \times m$ is shown in equation (4).

D = \frac{1}{m} \sum_{j = 1}^{m} w_{j} w_{j}^{S}

(4)

specifically, PCA solves the dataset covariance matrix

D^{'} s

eigenvalues and eigenvectors in equation (5).

λ_{j} v_{j} = {C u}_{j}; j = 1, 2, \dots \dots, m

(5)

where

v_{j}

represents the eigenvector that corresponds to eigenvalue

λ_{j}

and

λ_{j}

is an eigenvalue of

D

. The matrix with the formula

T = V^{S} W

is obtained by utilizing the preceding eigenvectors, which correspond to an eigenvalue of decreasing value. The new factor

t_{j} (j = 1, 2, \dots . ., o)

is referred to the primary component. The direction of the data’s highest variance distribution is represented by the eigenvector that corresponds to the largest eigenvalue. The eigenvector related to the second largest change in the data, which is orthogonal to the first eigenvector, is the one that determines the variability of the data points in the direction.

Improved Mayfly Optimized Deep Deterministic Policy Gradient (IMO-DDPG)

Automated threat recognition and response applies to dynamic network environments to include DDPG found to be preferred reinforcement learning technique due its ability to address continuous action spaces. The design employs an actor-critic framework where the actor is responsible for identifying optimal actions to mitigate cybersecurity threats, while the critic evaluates the action taken by providing feedback on the predicted benefits. The improvement upon existing DDPG methods is achieved through the integration of the Improved Mayfly Optimization, which enhances exploration capabilities and optimally balances exploitation of known strategies. This dual-pronged enhancement enables the algorithm to navigate the complex decision-making landscape of cybersecurity with greater efficacy.

Mayflies employ a method of cooperation in foraging, and this behavior is leveraged by Mayfly Optimization to enhance DDPG by aligning it with the relatively new theory of Swarm Informatics. This swarm intelligence program which gets its reference from nature is meant to optimize the search space so that the mayflies can shift their location with respect to the performance of the individual as well as the group. These models, collectively, offer a strong base to build on for advancing automated cybersecurity.

Deep deterministic policy gradient (DDPG)

A given reward function’s predicted cumulative value maximizes when a user learns to interact with a new setting, using the Reinforcement Learning (RL) approach. The environment is typically treated as a Partially Observable Markov Decision Process (PO-MDP); the user must choose an action based on the observations received from the environment at each moment. It is theoretically possible for the observation and the actual state of the system to differ. Since $t s$ is the state at which the agent builds its internal model of the surroundings, it may mistakenly equate its observation with the actual state. The user obtains a reward ${q_{s}}_{+ 1}$ and a fresh finding at the subsequent state transition of the system, from $t s$ to $t_{+ 1}$ , as a result of the computed action. Finding a strategy that optimizes the overall reward is the primary objective of the training process, which is described in equation (6).

Q_{s} = \sum_{l = 0}^{M} γ^{l} q_{s + l + 1} γ \in (0, 1)

(6)

where

γ

is the discount factor typically near 1. Additionally, the reward is calculated across several episodes, with each episode having

N

steps.

Discrete actions and observation areas are taken into consideration in traditional RL tabular approaches. The term tabular refers to the fact that the participant in such systems usually maintains a database with the expected cumulative value of rewards $Q_{s}$ (shown by the action-value equation $R (t_{s}, b_{s})$ assigned to every state-action combination. The best course of action would select a maximized $R$ for every state $s$ , or a greedy policy if the user knows the actual value of $R$ for each action-state pair. Consequently, identifying a table that appropriately depicts $R (t, b)$ is the training’s main objective.

However, tabular approaches are ineffective when dealing with continuous and high-dimensional spaces; they only have distinct areas for observation and activity. To overcome this limitation, multiple modifications have been proposed in the field of technology, primarily by utilizing neural networks’ ability to serve as general functional approximators. Deep Reinforcement Learning (DRL) is the term used to describe the fusion of DL methods and DRL algorithms. Specifically, actor-critic approaches split the RL problem into two separate issues.

• Critic: The critic determines an acceptable approximation of the action-value equation.

• $R (t, b)$ in which $s$ could adopt constant metrics.

• Actor: Using a different approximator $µ (t),$ the actor takes advantage of the critic to improve the policy.

The DDPG algorithm is an actor-critic technique that extends the DPG by utilizing deep neural networks in a model-free, off-policy manner. A critic network $R (t, b | θ^{R})$ is used for approximating the action-value $R (t, b)$ in DDPG, while an actor-network $μ (t | θ^{μ})$ is utilized for representing the current policy $R (t, b) (θ^{μ} a n d θ^{R})$ denote the associated network’s parameters). Specifically, the subsequent equation (7) is minimized by training the critic network.

K (θ^{R}) = E [{(R (t_{s}, b_{s} | θ^{R}) - z_{s})}^{2}] \approx \frac{1}{N} \sum_{j} {(R (t_{j}, b_{j} | θ^{R}) - z_{j})}^{2}

(7)

If the following state is terminal, $z_{s} = q_{s + 1} + γ R (t_{s + 1}) | θ^{R})$ (or simply $z_{s} = q_{s + 1}$ ). Usually, a mini-batch is chosen at random from the replayed buffer, and the average is computed over that batch, which holds a sizable collection of previous transitions $(t_{s}, b_{s}, q_{s + 1}, t_{s + 1})$ . The predicted return is subject to the chain principle $I$ concerning the actor parameters; updating the critic action-value differential determines the actor’s preferred direction of weighting in equation (8).

\nabla_{θ^{μ}} I = E [\nabla_{b} R {(t . b | θ^{R})}_{t = t_{s}, b = μ (t_{s})} \nabla_{θ^{μ}} {(t | θ^{μ}) |}_{t = t_{j}}

(8)

Target networks are used to increase learning stability because the $R$ updates are prone to change. The goal values were assessed using copies among the networks for actors and critics, denoted as $R (t . b | θ^{R})$ and $(t | θ^{μ})$ , respectively.

Improved Mayfly Optimized (IMO)

The Mayfly algorithm draws inspiration from the social behavior of Mayflies, particularly from their mating habits. It is believed that mayflies are considered adult as soon as when their eggs develop. In addition to how long they live, healthy mayflies typically survive. Each mayfly in the area of search occupies a position that corresponds to a method of solving an issue. The classic mayfly method creates additional variables that lead to an area’s optimal solution by utilizing RAND functions. Levy flying and the mayfly algorithm were coupled by researchers to enhance their search skills and yield the best result. According to the Levy flight concept, due to its rapid convergence and lack of dependence on differential information, it supports probabilistic random search. Enhancing local search reduction and localized trap of the optimal solution are largely dependent on Levy fly. For the recommended mayfly optimization technique to work, the following steps must be taken: Make twin mayfly groups at random, each for both genders, to symbolize the populations of men and women. Next, each mayfly which is denoted by a $d$ -dimensional vector is moved within the issue space at random as an alternative solution ${O_{h}}_{j} = (O_{h 1}, \dots ., O_{h d})$ . Next, the predetermined objective function $f (C T ({O_{h}}_{j}))$ is used to evaluate the performance.

When a mayfly’s position changes, its velocity is initialized, $u = (u_{1}, \dots, u_{d}) .$ Its orientation is determined by its interactions with social flies and human touch. More precisely, every mayfly tends to modify its course to align with its present optimal position $(o b e s t) .$ It also varies based on the highest position that another mayfly in the group has managed to reach up to this point $(h b e s t)$ . With speed ${u_{n}}_{j}$ , the male mayfly population is set up as $O_{H n j} (j = 1, 2, . . ., M H)$ . The behavior of individual mayflies, as suggested by male swarms, appears to be based on how they perceive their surroundings and their personal experiences. $o_{H j}^{s}$ is assumed to be Mayfly’s destination $j$ at a given time period $s$ in searching for an area. Applying acceleration alters the location $u_{j}^{s + 1}$ to the current role. The notation is given in equation (9):

O_{h n j}^{s + 1} = O_{h n j}^{s} + u_{j}^{s + 1}

(9)

The male mayfly is thought to be a few meters above the water’s surface, with $o_{h n j}^{0} V (O_{h n m i n}, {a n d O}_{h n m a x})$ dancing during their nuptials. Given that they are constantly in motion, one could assume that these mayflies don’t move very quickly. This leads to the following equation (10) of the rate $j$ of a male mayfly.

u_{j i}^{s + 1} = h * u_{j i}^{s} + b_{1} f^{- β q_{o}^{2}} (o {b e s t}_{j i} - o_{h n j i}^{s}) + b_{2} f^{- β q_{o}^{2}} (h {b e s t}_{i} - o_{H n j i}^{s})

(10)

In this instance, the Mayfly’s speed $I$ dimension $i = 1, . . ., n$ at every time step $s$ is represented by $u_{j i}^{s}$ . This mayfly’s $j t h$ location inside the given area $i$ at a given period $s$ was represented from $o_{H n j i}^{s}$ , and positive attraction factors that are employed to boost the interpersonal and cognitive elements’ contributions, respectively, were $b 1$ and $b 2$ . Moreover, $h {b e s t}_{i}$ indicates the Mayfly’s greatest position ever visited. The following time step $s + 1^{'} s$ personal best position, $o_{H n j i}^{s}$ , is determined according to the reduction in problems under consideration and is provided in equation (11).

o b e s t_{j} = {\begin{cases} O_{h n j}^{s + 1}, i f e (O_{h n j}^{s + 1}) < e (o b e s t_{j}) \\ i s k e p t t h e s a m e, o t h e r w i s e \end{cases}

(11)

The equation for the top position globally $(o b e s t)$ at time period $s$ is in equation (12).

h b e s t \in {{o b e s t}_{1}, {o b e s t}_{2}, \dots, {o b e s t}_{1} | e (d b e s t)} = \min (e ({o b e s t}_{1}), e ({o b e s t}_{2}), \dots, e ({o b e s t}_{M H})}

(12)

β

represents the constant presence. Its purpose is to prevent other people from seeing the mayfly. Furthermore, the Cartesian distance between

O_{H j}

and

{o b e s t}_{j}

is denoted by

q_{o}

, and the Cartesian distance among

O_{H j}

and

h b e s t

is denoted by

q_{h}

. Equation (13) was used to represent the distance as follows.

‖ O_{H n j} - W_{j} ‖ = \sqrt{\sum_{i = 1}^{m} {(O_{H n j i} - W_{j i})}^{2}}

(13)

where

W_{j}

indicates the

{o b e s t}_{j}

h b e s t

and

O_{H n j}

relates to the

i^{t h}

component of mayfly

j

. If the method is to function as intended, the best mayfly in the swarm must continuously perform their upward and downward nuptial dance. Consequently, it’s important to maintain the flying speed of these top mayflies, which is determined as follows in equation (14).

u_{j i}^{s + 1} = u_{j i}^{s} + c \times q

(14)

here,

q

depicts the interval’s randomized element

[- 1, 1]

, while

c

stands for the coefficient of nuptial performance.

The initial velocities of the female can vary depending on species $u_{e j}$ in $O_{H e j} (j = 1, 2, \dots, M H) .$ It is rare for female mayflies to form a swarm, unlike males. When mating, it typically follows the males in the area. $o_{H e j}^{s}$ is said to be the female mayfly’s site $j$ at time step $s$ within the search area, and its location is altered when velocity $u_{j}^{s + 1}$ is added to the present position in equation (15).

O_{H e j}^{s + 1} = O_{H e j}^{s} + u_{j}^{s + 1}

(15)

The attracting method used in this instance is unable to be randomized due to $O_{H e j}^{0} V (O_{H e m i n}, O_{H e m a x})$ . It follows that determinism will be the default for the model. Thus, these velocities are computed as follows in equation (16) in response to the occurrence of reduction issues.

u_{j i}^{s + 1} = {\begin{cases} h * u_{j i}^{s} + b_{2} f^{- {β q}_{n e}^{2}} (O_{H n j i}^{s} - O_{H e j i}^{s}), i f e (O_{H e j}) > e (O_{H n j}) \\ h * u_{j i}^{s} + e 1 \times q, i f e (O_{H e j}) \leq e (O_{H n j}) \end{cases}

(16)

in the present instance,

u_{j i}^{s}

represents the velocity

j

of the female mayfly at time step

s

in dimension

i = 1, . . ., n

. The position of the female mayfly

(j)

in dimension

i

at time step

s

is indicated by

O_{H e j i}^{s}

, and a positive attracting constant

(b 2)

is indicated while the visibility coefficient is fixed. Additionally,

h

represents the gravitational coefficient, and

q_{n e} .

The Cartesian separation among both genders allows them to fly,

q

symbolizes the interval’s randomized number

[- 1, 1],

while

e 1

representing the random walk frequency.

This phase gives the speed of a mayfly candidate response, which is determined using the Levy flying technique. Equation (17) is utilized to ascertain the mayfly candidate solution’s velocity.

u_{j i}^{s + 1} = {\begin{cases} u_{\max}, i f u_{j i}^{s + 1} > U_{\max} \\ - U_{\max}, i f u_{j i}^{s + 1} < - U_{\max} \end{cases}

(17)

in this process, the Levy fly method is used to change the position of the global smallest component. The Levy fly technique is associated with a specific search, even though it has been employed in exploration at this point. In this instance,

U_{\max}

is computed as follows in equation (18).

U_{\max} = L e v y (λ) * (O_{H n m a x} - O_{H n m i n})

(18)

thus,

δ

indicates a factor of scale that should be compatible with the element of the search field. It is given as

1,

as shown in equation (19).

L e v y (λ) = 0.01 \frac{q_{5} σ}{{| q_{6} |}^{\frac{1}{β}}}

(19)

Moreover, the calculation of $σ$ is as follows in equation (20).

σ = [\frac{Γ (1 + λ) \sin (π (\frac{λ}{2}))}{(Γ (\frac{1 + λ}{2}) λ [2^{\frac{λ - 1}{2}}])}]

(20)

here,

Γ (w) = (w - 1)! q_{5}

and

q 6

are indiscriminate numbers that fall between [0, 1] and

1 < β \leq 2

, where

β

is replaced with a constant value. Levy represents the measure of step size (

λ

), that has mean values for Levy distributions with limitless variance

1 < λ < 3

. The allocation component is shown as

λ

, whereas the gamma function range is shown by

r (.)

. Calculating the gravity coefficient result, the gravitational coefficient’s value, or

h

, is a set quantity that lies in

0

and

1

as shown in equation (21).

h = h_{\max} - \frac{h_{\max} - h_{\min}}{{I t e r}_{m a}} \times i t e r

(21)

where

{I t e r}_{m a}

indicates the highest possible number of iterations, and

h_{\max}

and

h_{\min}

show the highest and lowest possible values for the gravitational factor, respectively. Iteration indicates the algorithm’s current iteration.

In the format given below, the crossover operators describe the mating procedure for mayflies. In the same selection process, females’ attraction to males is used to pick each parent from the male and female populations. To be more precise, parents may be selected at random or according to their level of fitness. When it comes to fitness functioning, males and talented females’ pair together, the second-best male and female pair with each other, and so on. Equation (22) contains the formula for the two offsprings produced by this crossover.

{\begin{cases} o f f s p r i n g 1 = L \times m a l e + (1 - K) \times f e m a l e \\ o f f s p r i n g 2 = K \times f e a m l e + (1 - K) \times m a l e \end{cases}

(22)

thus,

L

is a random integer that falls in two predefined ranges; the male is represented by the male parent, while the female is represented by the female parent. The initial speed of the offspring is zero.

Result

Operations on Windows 11 were performed using python 3.12 and the RAM included is 32 GB, Intel Core i7 processor of 12th generation. The device is being tested in a modern laptop configuration engineered to create reasonable loads of two demanding multitasking and development workloads. The proposed approach in this work presents the use of Deep Reinforcement Learning algorithms to automatically find threats in a dynamic network environment covering a higher level of accuracy compared to traditional approaches like Random Forest (RF)²⁶ and convolutional neural network (CNN).²⁷ It gives critical metrics such as accuracy (%), recall (%), precision (%), and F1-score (%) required for much better cybersecurity outcomes in the case of threat detection assessments.

Performances of threat detection

In terms of duration, the high condition required efficiency of 40% to complete with a 70% success rate and an average reward of 60%, while the medium condition performed well with an 85% success rate and 100% average reward in 60% efficiency. With just 85% efficiency, TCP achieved an 80% average payout and 90% success rate, making it the most effective protocol type. By comparison, UDP took 50% efficiency and had a 75% success rate, with an average return of 53.33%. In terms of service, HTTP stood out because it processed requests under 90% efficiency and had a 93.33% average reward and 95% success rate. FTP, on the other hand, performed poorly, requiring 25% efficiency and having an average reward of 46.67% with a 60% success rate. Figure 2 shows the performances of thread detection.

Figure 2.

Performance Metrics of Threat Detection (a) Duration, (b) Success Rates, (c) Service Efficiency.

Accuracy

To assess how well the proposed IMO-DDPG model detects and reduces network threats in dynamic environments, it is compared with other approaches. The new IMO-DDPG technique with an accuracy score of (94.5%) is significantly better than traditional methods. This is higher than CNN (75.0%) and RF (92.5%). Results demonstrate that the IMO-DDPG model enables a practical approach to enhancing cybersecurity and boosting system protection by automating threat detection and response in dynamic network environments. The accuracy result is presented in Figure 3 and Table 1.

Figure 3.

Accuracy result.

Table 1.

Overall performance.

Methods	Accuracy (%)	Recall (%)	Precision (%)	F1-score (%)
RF (Khanza et al., 2024)²⁶	92.5	93	91.8	92.4
CNN (Al-Rubaye and Türkben, 2024)²⁷	75	60	80	68
IMO-DDPG (proposed)	94.5	94	92.5	93.5

Recall

Recall is the system’s capacity to identify potential network threats. To utilize the IMO-DDPG, which is recommended for automatic threat identification and response, the system achieved a recall rate of 94.0%. In terms of scores, both the current approaches (CNN and RF) achieved 60.0% and 93.0%, respectively. This shows that the proposed strategy significantly enhances existing techniques for exploring cybersecurity risks in a dynamic network environment, consequently increasing the system’s ability to identify and counteract attacks. The result of the recall is shown in Figure 4 and Table 1.

Figure 4.

Recall result.

Precision

Precision is also defined as the measure of how a system detects important threat signals across a dynamic network. When compared to the current techniques, RF and CNN achieve precision values of 91.8% and 80.0%, respectively. As shown by its precision value of 92.5%, the suggested IMO-DDPG algorithm performs better than any of the existing methods in the proper detection of cybersecurity threats. The result of precision is presented in Figure 5 and Table 1.

Figure 5.

Precision result.

F1-score

F1-score is an accurate means of system evaluation of the ability to automatically identify and respond to threats in dynamic network environments. This enables to compare other methods such as CNN and RF that yielded F1 scores of (68.0%) and (92.4%), respectively. Other approaches could not outperform the suggested IMO-DDPG strategy with an F1-score of (93.5%) to solve the problem of recognizing potential threats. This indicates effective deep reinforcement learning is compared to traditional approaches when it comes to cybersecurity. The result of F1-score is presented in Figure 6 and Table 1.

Figure 6.

F1-score result.

Discussion

Random Forest (RF) and Convolutional Neural Networks (CNNs) are limited in automatic threat detection and response in dynamic network environments due to the necessity for high-quality, labeled training data that can be hard to obtain because threats are inherently dynamic. For high dimensional data, RF may find it hard to generalize, so CNNs have difficulty with both large models and slower training times. Furthermore, CNNs can be less interpretable, such that cybersecurity analysis attempting to understand the factors influencing decision-making will find them more difficult to comprehend. In addition, both methods might struggle in responding to new threats that are substantially removed from patterns previously encountered. To overcome the shortcomings of random forests and convolutional neural networks in automatic threat detection, an improved Mayfly Optimized Deep Deterministic Policy Gradient (IMO-DDPG) approach is presented. This approach makes use of reinforcement learning’s strengths for adaptive online optimization of decision-making modalities. IMO-DDPG integrates an improved mayfly optimization algorithm for improving exploration and exploitation capabilities, which aid in better dealing with high-dimensional data and dynamic threat landscapes. Additionally, this approach contributes to better interpretability and flexibility to new threats, resulting in more robust and efficient cybersecurity responses.

Conclusion

Dynamic network environments are targeted for automated threat detection and response through deep reinforcement learning (DRL) in this study. The research proposes the method of the Improved Mayfly Optimized Deep Deterministic Policy Gradient (IMO-DDPG) algorithm to tackle the issue of traditional cybersecurity methods. The IMO-DDPG model is trained on real-time network intrusion datasets and enhances performance by focusing on continuous action spaces, adversarial training, and customized reward structures. The results demonstrate accuracy (94.5%), recall (94.0%), precision (92.5%), and F1-score (93.5%). It shows that the IMO-DDPG outperforms existing algorithms. Based on the results from the study, we conclude the proposed approach that provides a strong defines against cybersecurity vulnerabilities through real-time, adaptive decision-making. Despite the promising results, the IMO-DDPG algorithm presents several limitations. It may be vulnerable to advanced adversarial attacks, highlighting the need for robust testing against such threats. The model also exhibits high computational complexity, which could hinder real-time application in resource-constrained environments. Furthermore, its generalizability to unseen environments is limited, as the training heavily relies on specific datasets. To address these challenges, future research could explore hybrid models, enhance adversarial training techniques, and incorporate adaptive learning strategies that allow for efficient scaling in larger networks.

Statements and declarations

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ya Ma

References

Chen

Cam

. Quantifying cybersecurity effectiveness of dynamic network diversity. IEEE Trans Dependable Secure Comput 2021; 19(6): 3804–3821.

Capuano

Fenza

Loia

, et al. Explainable artificial intelligence in cybersecurity: A survey. IEEE Access 2022; 10: 93575–93600.

Sewak

Sahay

Rathore

. Deep reinforcement learning in the advanced cybersecurity threat detection and protection. Inf Syst Front 2023; 25(2): 589–611.

Keyes

McLaughlin

Barner

, et al. An ecological network approach to predict ecosystem service vulnerability to species losses. Nat Commun 2021; 12(1): 1586.

Shahin

Maghanaki

Hosseinzadeh

, et al. Advancing network security in industrial IoT: A deep dive into AI-enabled intrusion detection systems. Adv Eng Inf 2024; 62: 102685.

Ruan

Zhou

, et al. A unified cognitive learning framework for adapting to dynamic environments and tasks. IEEE Wireless Commun 2021; 28(6): 208–216.

Bécue

Praça

Gama

. Artificial intelligence, cyber-threats and Industry 4.0: Challenges and opportunities. Artif Intell Rev 2021; 54(5): 3849–3886.

Trim

Lee

. Combining sociocultural intelligence with Artificial Intelligence to increase organizational cyber security provision through enhanced resilience. Big Data Cogn Comput 2022; 6(4): 110.

Melaku

. Context-based and adaptive cybersecurity risk management framework. Risks 2023; 11(6): 101.

10.

Kaur

Gabrijelčič

Klobučar

. Artificial intelligence for cybersecurity: Literature review and future research directions. Inf Fusion 2023; 97: 101804.

11.

Shah

. Machine learning algorithms for cybersecurity: Detecting and preventing threats. Rev Española Doc Científica 2021; 15(4): 42–66.

12.

Naseer

Ahmad

, et al. Real-time analytics, incident response process agility and enterprise cybersecurity performance: A contingent resource-based analysis. Int J Inf Manag 2021; 59: 102334.

13.

Yungaicela-Naula

Vargas-Rosales

Pérez-Díaz

, et al. Towards security automation in software defined networks. Comput Commun 2022; 183: 64–82.

14.

Nikoloudakis

Kefaloukos

Klados

, et al. Towards a machine learning based situational awareness framework for cybersecurity: an SDN implementation. Sensors 2021; 21(14): 4939.

15.

Gudala

Shaik

Venkataramanan

. Leveraging machine learning for enhanced threat detection and response in zero trust security frameworks: An Exploration of Real-Time Anomaly Identification and Adaptive Mitigation Strategies. J Artif Intell Res 2021; 1(2): 19–45.

16.

Repetto

Striccoli

Piro

, et al. An autonomous cybersecurity framework for next-generation digital service chains. J Netw Syst Manage 2021; 29(4): 37.

17.

Mironeanu

Archip

Amarandei

, et al. Experimental cyber attack detection framework. Electronics 2021; 10(14): 1682.

18.

Guo

. Real-time risk detection method and protection strategy for intelligent ship network security based on cloud computing. Symmetry 2023; 15(5): 988.

19.

Bringhenti

Marchetto

Sisto

, et al. Automation for network security configuration: state of the art and research trends. ACM Comput Surv 2023; 56(3): 1–37.

20.

Rose

Swann

Grammatikakis

, et al. IDERES: intrusion detection and response system using machine learning and attack graphs. J Syst Architect 2022; 131: 102722.

21.

Kandhro

Alanazi

Ali

, et al. Detection of real-time malicious intrusions and attacks in IoT empowered cybersecurity infrastructures. IEEE Access 2023; 11: 9136–9148.

22.

Apruzzese

Laskov

Montes de Oca

, et al. The role of machine learning in cybersecurity. Digital Threats 2023; 4(1): 1–38.

23.

Gong

Lee

. Cyber threat intelligence framework for incident response in an energy cloud platform. Electronics 2021; 10(3): 239.

24.

Furdek

Natalino

Di Giglio

, et al. Optical network security management: requirements, architecture, and efficient machine learning models for detection of evolving threats. J Opt Commun Netw 2021; 13(2): A144–A155.

25.

Nespoli

Díaz-López

Mármol

. Cyberprotection in IoT environments: A dynamic rule-based solution to defend smart devices. J Inf Secur Appl 2021; 60: 102878.

26.

Al-Rubaye

Türkben

. Using artificial intelligence to evaluating detection of cybersecurity threats in ad hoc networks. BJN 2024; 2024: 45–56.

27.

Khanza

Yulian

Khairunnisa

, et al. Evaluating the effectiveness of machine learning in cyber threat detection. Corisinta 2024; 1(2): 172–179.