Deep Reinforcement Learning Approach for Automated Vehicle Mandatory Lane Changing

Model Formulation

Double Deep Q Network

The double deep Q network (DDQN) ( 25 ) is an advancement on the original deep Q network (DQN) algorithm ( 26 ). The DQN combines Q-learning with a deep neural network to perform predictions and make decisions. A DQN agent can learn successful policies directly from high-dimensional sensory inputs using end-to-end RL ( 26 ). In the DQN algorithm, two neural networks exist: a main network and a target network. The two networks are initialized with random weights, where the input to the networks are the states of the environment and the outputs are the set of actions that can be chosen by the agent. The main network weights are updated according to the Bellman equation ( 27 ) and the Q values associated with the actions. Figure 1 shows a simple illustration of a DQN. The target network is identical in architecture to the main network but is updated less frequently than the main network (i.e., every N steps, the weights of the main network are copied to the target network). This is done to improve the stability of the learning process and helps the algorithm converge faster by learning more efficiently. The DQN also utilizes a tool called “experience replay” to improve performance. In experience replay, the agent’s experiences at each time step are stored in a replay memory, which we then sample from randomly for the Q-learning process instead of just using the current state/action pairs that occur during simulation. On the other hand, the DDQN was proposed by Van Hasselt et al. ( 25 ) to address some over-estimations that occur in the original DQN algorithm, while also improving its performance. More details on the structure of the DDQN can be found in Van Hasselt et al. ( 25 ).

OPEN IN VIEWER

Figure 1.

Simple illustration of the deep Q network ( 27 ).

Model Parameters

The network architecture utilized in this work is a simple fully connected deep neural network. The details of the network architecture are shown in Figure 2. The proposed network was sufficient to achieve favorable results for the task at hand and was chosen over more complex architectures such as convolutional neural networks because of its computational efficiency. Note that a considerable time has been spent on identifying a suitable network structure for the lane-changing problem. Table 1 lists the hyperparameter settings for the formulated DDQN model. We utilize a discount factor, γ , of 0.999, ensuring that the RL agent would strongly consider future rewards when making a decision ( γ = 1 . 0 means that the agent considers no difference between the current reward and future reward, i.e., the agent becomes more farsighted [ 27 ]). We do this to give a strong account to the final reward of finishing the task successfully, which we will discuss in detail in later sections. A replay memory size of 100,000 is chosen to stay within computational capacity and is shown to yield good results. Note that larger replay memory size can significantly increase the calibration time and delay the model convergence. As discussed previously, smaller replay memory can result in an undesirable memory loss about effective past actions. Moreover, the agent is set to train for 10 million steps to ensure convergence. The policy selected for learning is the linear-annealed epsilon-greedy policy, where the ε value decreases linearly with the number of steps from 1.0 to 0.05. This ensures the agent explores for an adequate amount of time before starting to follow the greedy action choices, and thus guarantees optimal/near-optimal performance. Note that the minimum ε is set to 0.05 to ensure some level of exploration throughout the calibration process. This is critical to ensure that the system does not stay within a local minimum.

OPEN IN VIEWER

Figure 2.

This study’s deep Q network architecture.

OPEN IN VIEWER

Table 1.

Deep Q Network Hyperparameter Settings

Hyperparameter	Value
Number of layers	2
Number of hidden units	512, 512
Learning rate	0.001
Policy	Epsilon-greedy
ε	1 . 0 → 0.05
Discount factor γ	0.999
Replay memory size	50,000
Number of episodes	10,000–50,000
Batch size	64
Activation functions	ReLU, linear
Optimizer	Adam

Simulation Setup

Simulation Environment

Our problem is defined within a two-lane environment: a merge lane and a main/target lane. Each lane is 4 m wide, and the merge lane is 200 m long with a 100-m taper section. Figure 3 illustrates the road environment designed for the simulation experiments. No vehicles other than the automated vehicle are present on the merge lane, while other vehicles in the main lane are designed according to the intelligent driver model (IDM) ( 28 ) to govern their longitudinal motion along the road segment. Main lane vehicles are also modeled to respond to the attempts of the automated vehicle to merge into the target lane. This is done by following a sigmoid cumulative distribution function that controls the probability of a trailing vehicle to switch its leading vehicle from the IDM vehicle ahead (old leader) to the automated vehicle (new leader), depending on how close the automated vehicle is to the main lane.

OPEN IN VIEWER

Figure 3.

Road environment.Note: AV = Automated Vehicle, i.e., our ego vehicle.

Reinforcement Learning Environment

The automated vehicle’s task is to merge into the target lane safely and efficiently, and continue driving along the target lane until a goal point is reached. The goal point is designed to be 100 m beyond the end of the merge lane. Safety is defined as the ability of the automated vehicle to merge and navigate without crossing the outer borders of the two lanes or colliding with a neighboring vehicle, while efficiency is defined based on traffic state and shockwave formation. A successful episode is achieved if the automated vehicle merges successfully with minimal disturbance, and proceeds to drive safely until reaching the set goal point.

To perform the aforementioned tasks successfully, several components need to be defined appropriately. To begin with, a proper definition of state and action spaces is required in order for the RL agent to be able to learn important features and corresponding best actions. In our study, the state space consists of the x–y coordinates of the RL vehicle, speed and heading of the RL vehicle, and the locations of the leading and trailing vehicles (with respect to the automated vehicle). On the other hand, we define the action space in a more complex manner (compared with existing studies); the action space in our environment is pairs of acceleration/deceleration values and steering angles. The acceleration/deceleration values range among −1, 0 and 1  m / s 2 , while the steering angles are either −30, 0, or 30 degrees. The RL agent is responsible for choosing the acceleration/deceleration value as well as the steering angle for the vehicle at each time step (t), which is 0.1 s in this study. This makes the task of lane changing a continuous one, as opposed to simpler definitions that perform the lane changing as a one-step “turn right” or “turn left” command. In addition, we need to create a meaningful reward system that guides the RL agent into eventually performing the task successfully. Thus, a multi-part reward system is proposed to tackle this problem. The three main elements of this reward system are a lane-cross negative reward that the RL vehicle incurs whenever it deviates beyond the boundaries of the two lanes, a similar negative reward that is given for any collision that occurs during the merging procedure, and a sizeable positive reward that is awarded at task completion. In addition to the main reward elements, small continuous rewards (which occur every time step) are defined to account for additional requirements concerning efficiency. Those incremental rewards include a small negative reward for every time step the RL vehicle does not finish the task, positive rewards that are added every time step whenever the RL vehicle is within the borders of the target lane and is centered in the target lane, and finally a negative reward that the RL vehicle accrues if it deviates above or below desirable speeds. The first three main rewards ensure a safe lane-changing maneuver, and a well-defined task for the RL agent, while the remaining reward elements ensure an efficient and timely completion of the task. Table 2 presents a detailed overview of the RL and road environment parameters.

OPEN IN VIEWER

Table 2.

Reinforcement Learning and Road Environment Settings

Parameter	Value
Number of states	6
Number of actions	9
Lane-cross reward	−3000
Collision reward	−3000
Merge reward	+50/step
Lane centering reward	+50/step
Undesirable speed reward	−200/step
Noncompletion reward	−20/step
Task completion reward	+10,000
Acceleration/deceleration values (m/ s 2 )	−1, 0, 1
Steering angle values (degrees)	−30, 0, 30
Time headway (s)	5, 2.5, 1
Desired velocity (m/s)	30
Time step length (s)	0.1

We note that the different reward values were updated in an iterative manner as we experimented with a range of values. For example, in earlier stages of training, we only started with the three main reward elements, but as we observed the behavior of the RL agent, we added several other elements. For instance, the negative undesirable speed reward was added after we observed that the RL agent was performing the merging task successfully but then proceeded to slow down heavily to avoid collision. Similarly, multiple other reward elements were incorporated. On the other hand, the specific values of each reward element were chosen using trial and error. It cannot be claimed that this is the optimal reward structure; however, this specific combination of reward values worked for our specific problem. Other reward values may result in comparable and potentially better results. In addition, different scenarios may require some tweaking of the reward structure to meet the objectives of those respective scenarios.

Results and Discussion

We start by running our model in a trivial environment that contains no vehicles on the entire roadway segment except for the automated vehicle. This was done as a baseline run to verify the ability of the RL agent to learn and perform the lane-changing task successfully. Figure 4 shows the path of the RL vehicle on the two-lane road section. It can be seen that the RL agent learns to make the lane change as soon as possible to avoid a negative reward and then proceeds to drive approximately in the middle of the target lane until reaching the goal point. Note that the deviation from the center of the lane is about ± 0 . 5 m. Random lateral oscillations in the vehicle’s movement can also be seen, which is a product of the inherent randomness of RL (minimum ε is set to 0.05, thus at least 5% of vehicle actions are random throughout the training process).

OPEN IN VIEWER

Figure 4.

Reinforcement learning vehicle path—empty environment.

After verifying the ability of the RL agent to learn through the trivial case, we explore three main cases with IDM vehicles in the main lane: (1) 5-s time headway; (2) 2.5-s time headway; and (3) 1-s time headway. To make our problem more challenging, and to test the scalability of the model proposed, we change the initialization location of the RL vehicle in each scenario. To that end, the RL vehicle is initialized at 450 m in the 5-s headway scenario, and at 225 and 90 m for the 2.5- and 1-s scenarios, respectively. The general setup and length of segments remain unchanged. Starting with the 5 s time headway simulation, we only needed to train the model for approximately 10,000 episodes in order for the model to converge and learn optimal policy for this problem. Figure 5a presents the reward values during the training process. It can be noted that the curve reaches a near-plateau state toward the end of the training process, while at earlier steps, mostly negative rewards are observed because of high randomness at that phase (exploration phase). Figure 5, b–e , shows the ability of the RL vehicle to make the lane-changing maneuver without too much effort and without causing disturbances in the target lane. Similar to the empty environment scenario, we can see that the RL vehicle does not strictly remain in the middle of the target lane, which is expected since a slight randomness is inherent in the system and in the way the problem is set up. However, the deviation from the center of the lane is about ± 0 . 5  m. It should also be noted that scale variance in Figure 5e makes the lateral movements of the automated vehicle after making the lane change seem unrealistic. It is noted that slight smoothing was applied to the speed profiles of the RL vehicle in the three scenarios to account for the variability in acceleration/deceleration choices taken by the RL agent, which cause the speed profiles to be somewhat non-smooth and would possibly result in an uncomfortable experience for the passenger. We argue that such smoothing may be advantageous to apply to the model to improve the ride experience. However, of course, we note that further investigation of the effects of such smoothing on the actual operations needs to be considered.

OPEN IN VIEWER

Figure 5.

The 5-s headway scenario: (a) training reward; (b) vehicle trajectories; (c) speed profiles*; (d) acceleration profiles; (e) reinforcement learning (RL) vehicle path.

Subsequently, we train our model for the 2.5-s time headway scenario. It can be seen in Figure 6a that the training is converging to near-optimal reward values. However, while the 5-s scenario required around 10,000 episodes to train, it is shown here that more than 35,000 episodes were required. On the other hand, as observed in the rest of the figures, the RL vehicle can be seen to be performing the task of lane changing in this scenario quite well. Specifically, the speed and acceleration profiles in Figure 6, b and c , illustrates that the RL vehicle had minimal impact on main lane traffic and that even the vehicle directly behind it (i.e., IDM-2) was almost unaffected by the merging maneuver. Note that the abrupt changes in the acceleration of this vehicle are in response to the abrupt changes in RL vehicles’ acceleration (because of discrete action–acceleration value–space). We also show here, as a representative example, the steering profile for the RL vehicle in Figure 6f. We can see that in the early steps of the task, the RL vehicle mostly chooses a steering angle of +30 degrees, which corresponds to turning left and quickly completing the merge maneuver. This is followed by −30 degrees which corresponds to correcting its trajectory to follow the vehicles in the main lane after completing the merge maneuver. Finally, the steering angle profile can be seen to nearly converge to 0 degrees. The steering angle profile was smoothed because the discrete nature of the action space forces the RL agent to jump between −30, 0, +30, while a smoothed profile shows the trend of actions taken by the agent. Note that the steering angle profile is consistent with the RL vehicle path shown in Figure 6e and is similar in other scenarios.

OPEN IN VIEWER

Figure 6.

The 2.5-s headway scenario: (a) training reward; (b) vehicle trajectories; (c) speed profiles*; (d) acceleration profiles; (e) reinforcement learning (RL) vehicle path; (f) steering angle profile*.

Finally, we explore the case of 1-s time headway between the vehicles and test the ability of the RL agent to learn the proper actions needed to perform the merging task in this challenging scenario. Figure 7a shows the trend of the reward gained during the training process. In this case, more steps are required until the reward per episode starts stabilizing and converging to the optimal value after yielding a negative reward for a significant number of episodes. In order for the RL agent to learn to perform the task successfully within this more challenging scenario, we trained the model for approximately 50,000 episodes, which explains the steadier trend when compared with the previous scenario. Figure 7 also shows the vehicle trajectories, speed profiles, acceleration profiles, and the RL vehicle path in the 1-s time headway scenario. It can be observed that because of the small time headway between all the vehicles, the IDM directs all the vehicles (except the leader—IDM-5) to slow down for a portion of time at the beginning of the simulation until an equilibrium headway starts forming between vehicles. This happens because the desirable time headway used for the IDM was set to 1.5 s. In addition, it can be seen that the RL vehicle learns to conduct the lane-changing maneuver with minimal impact on the traffic stream, which can be observed from the speed profiles (Figure 7b). It can also be seen that the RL vehicle decelerates more gradually relative to neighboring vehicles, and maintains a higher speed that results in controlling the shockwave propagation. However, despite a controlled deceleration of the RL vehicle, because of the very small time headway in the target lane, the vehicles directly affected by the lane changing show significant deceleration values for a very short period of time. Note that such a large deceleration can be avoided by replacing IDM vehicles with a more robust platooning algorithm (e.g., the MPC-based model of An and Talebpour [ 15 ]).

OPEN IN VIEWER

Figure 7.

The 1-s headway scenario: (a) training reward; (b) vehicle trajectories; (c) speed profiles*; (d) acceleration profiles; (e) reinforcement learning (RL) vehicle path.

Further simulation runs and exploration of this scenario reveals an interesting behavior that the RL agent occasionally learns to perform on its own. Figure 8a shows that because of the small time window the RL vehicle has to complete the lane-changing movement within, it learns to navigate longer through the merge lane before taking the decision to overtake its leading vehicle and making the lane-changing maneuver in a slot ahead of its starting position (with respect to the platoon). The overlap in the trajectories between the RL vehicle and the vehicle ahead (IDM-3) is not a sign of collision. In fact, each vehicle is moving in its own respective lane (only the longitudinal position is plotted here) and it is an overtaking maneuver, which can be verified by Figure 8b. This figure shows that the RL vehicle actually remains within the merge lane well beyond the overlap point of approximately 200 m.

OPEN IN VIEWER

Figure 8.

Alternative 1-s headway scenario: (a) reinforcement learning (RL) vehicle path; (b) vehicle trajectories.

While the presented results seem promising and can certainly be improved by fine-tuning the reward function and hyper parameters, the outcome of calibration is not always satisfactory (mainly because of the randomness involved in the action selection mechanism). Accordingly, we also show hereafter that some disadvantages exist with applying a RL approach to model the lane-changing behavior. Figure 9 shows one of the simulation runs for the RL vehicle. It can be seen that after completing a successful lane change, the agent randomly decides to start turning right and almost leaves the target lane before finally swaying back up again and finishing the task. While the RL agent did perform the task successfully and managed to gain a favorable reward for its actions, it is apparent that such behavior is not desirable and has no reason to occur in a realistic setting. Alternatively, while most runs on the 5-s scenario showed no impact of the lane-changing maneuver on the main lane, in some cases, the RL vehicle makes the decision to slow down, which ultimately slows down the whole stream of traffic, even though by looking at the vehicle path alone, one would expect the RL vehicle to be performing better. Figure 10, a and b , illustrates the aforementioned case.

OPEN IN VIEWER

Figure 9.

Case 1: unsatisfactory vehicle path.

OPEN IN VIEWER

Figure 10.

Case 2: unsatisfactory vehicle trajectories: (a) reinforcement learning (RL) vehicle path; (b) vehicle trajectories.

In addition to the aforementioned challenges, several other issues were faced during the training and testing phases of the RL model. For instance, some runs resulted in falling into local minima that at first glance gave the impression that the model was converging. However, on further examination and visualization of the vehicle path and trajectory, it was found that the RL vehicle was learning a sub-optimal policy and, in reality, was not achieving the task originally described. In addition, substantial computational capacity is required to run the simulation and train the model for a sufficient amount of time. While we sometimes selected the number of training steps to be in excess of 50,000 episodes, earlier stages of training were run with much fewer steps. That resulted in a wider range of randomness and variance in the results, which required increasing the number of steps until a more stable model was produced. In other words, most of the above challenges were solved by providing the right balance between exploring the environment and exploiting the findings from previous experiments.

On the other hand, another critical aspect to deep RL models is training initialization and the randomness in some parameter choices. While most parameters can have a solid mathematical reasoning behind their selection, there can still exist some trial and error. For instance, our final choice of 0.999 for the discount factor, γ , was made after several trials of values ranging between 0.70 and 1.0. Our choice of 0.999 ultimately provided the most favorable results among the different scenarios explored. This may be the case since we explicitly define a “task success” reward, which needs to be accounted for somewhat significantly to guide the agent toward finishing the task successfully. Note that in other scenarios and problem definitions, such an end-of-task reward might not exist, and thus, a lower discount factor might become a better choice. Alternatively, the decay method for ε can also result in different outcomes. We have selected a linearly decaying epsilon; however, other methods such as step decay may have different outcomes, such as faster or slower convergence of the model. This is essential, as we have discussed earlier, where the model could converge to undesirable local minima. It is also worth mentioning that for some scenarios, it may be infeasible (a non-convex setting) or the deep RL model may be unable to find the global optimal solution for the problem. While it may reach near-optimal results, in some real-world situations, some near-optimal solutions may not perform reasonably. This was apparent in many instances of our training and testing procedure, where the RL agent, for example, would perform the task of lane changing successfully but would continue to collide into nearby vehicles. Accordingly, it is critical to capture such instances that may lead to catastrophic outcomes in a real-world setting. Note that the reward setup also plays an important role since, for instance, the RL agent may be achieving a higher reward for completing the lane change while it may not care about the penalty of colliding.

Conclusion and Future Work

We proposed a deep RL-based algorithm utilizing the DDQN for automated vehicle merging in a decentralized non-cooperative manner. The proposed approach utilizes a continuous state space and more realistic action space compared with previous studies. For the RL agent, we defined a six-element state space consisting of the x–y coordinates of the RL vehicle, the speed and heading of the RL vehicle, and the locations of the leading and trailing vehicles on the target lane. We also defined the action space in a more realistic manner than before where the model has to choose between nine different action pairs at each time step. Each action pair consists of a combination of an acceleration/deceleration value (−1, 0, or 1  m / s 2 ) and a steering angle value (−30, 0, or 30 degrees). Therefore, we take the task of lane changing one step closer to realism by letting the RL agent choose its action every time step (i.e., 0.1 s) and move along the lanes continuously, while monitoring its location and the locations of neighboring vehicles. We demonstrate the model’s ability to learn the proper actions it requires to perform the lane-changing maneuver under different circumstances and scenarios. Finally, we propose a reward system that allows the RL agent to perform the task in a timely and efficient manner.

Incorporating additional components to the reward structure to fine-tune the RL vehicle behavior and including the impacts of the RL vehicle on the entire traffic stream as part of the decision-making have been left for future research.