Multi-Intersection Control with Deep Reinforcement Learning and Ring-and-Barrier Controllers

Abstract

This paper discusses a machine-learning traffic signal control method. A full-scale corridor is analyzed and the transferability of using a model pre-trained on a single intersection is examined. Two controller designs are explored, a simple two-phase design and a full ring-and-barrier style controller. The full ring-and-barrier controller adapts many of the key features present in traditional controllers, such as protected-permissive left turns, so that they can be used in the reinforcement learning (RL) paradigm. This study is the first to propose a method that uses deep reinforcement learning (DRL) to implement a full ring-and-barrier style controller. The study also examines the feasibility of using transfer learning to pre-train a model on a single intersection and then fine-tune it for application in a complete environment. Training is done on a simple four lane intersection and the pre-trained model is then transferred for fine-tuning to six controllers operating on a corridor modeled with field data obtained for University Avenue in Waterloo, Ontario, Canada. The performance of the fully trained model is then compared with the existing signal plans in relation to the average delay and average queue length. Application of the ring-and-barrier design to this corridor was found to reduce delays by at least 5% and average queue lengths at intersections by 27%.

In recent years, significant advancements in machine learning have resulted in the development of highly intelligent algorithms. In fact, many tasks that were previously thought to be restricted to human intelligence and cognition can now be performed much more efficiently by computers. One key example of this is pattern recognition, which has led to computers exceeding the performance of humans in complex strategy games ( 1 – 3 ). These advancements have been made possible through the use of deep neural network models, and new applications for these techniques continue to be found each day. Recently, some developments have been made applying these techniques to signalized intersections, but many current implementations are limited in their scope and applicability ( 4 – 6 ).

As a repetitive decision-making process, traffic control lends itself well to reinforcement learning (RL) and deep reinforcement learning (DRL). These techniques work by allowing an agent to explore their world in an unsupervised fashion and learn from experience ( 7 ). Applications of RL and DRL are typically framed as Markov decision processes (MDP), which model the agent-world interaction in relation to states (S) and actions (A). When an agent chooses a particular action, its choice influences the world, and the world transitions according to a stochastic process to a new state (S’). A key assumption of MDPs is that the transition to successor states is independent of past actions and states. While previous actions and states may have brought about the current state, the agent’s past decisions do not affect the optimality of its choices in the current state. To motivate agents during learning, rewards or penalties are often generated from the state the agent is in according to a specified reward function. RL seeks to map out the expected rewards from choosing particular actions in a given state ( 7 ).

One of the most successful techniques used to teach agents an optimal strategy is called Q-learning. Q-learning operates by using a state-value function that predicts the expected reward of choosing a particular action when faced with a specific state. The value considers the long-term consequences by discounting the expected value of future rewards according to some rate, and the optimal policy is one that chooses actions to maximize this reward. Traditional approaches to learning are divided into two methods: on-policy and off-policy. In off-policy learning, the agent learns an optimal policy regardless of the current policy being used to choose actions, though some policies may not lead to the actual optimal policy (or may take a lot of training to do so). Off-policy methods typically focus on exploring the possible states and actions as best as possible, such as ε-greedy training or softmax action selection ( 8 ). On-policy methods update the Q-function with the expected value for continuing to choose actions according to the current policy. On-policy approaches can be useful where the optimal policy is not the goal of the algorithm or when comparisons between a particular policy and the optimal one are desired.

In practice, learning the Q-function can be complicated if the range of possible and actions states is vast, and, in practice, the Q-function is typically approximated using techniques such as linear approximation or neural networks ( 9 ). When RL is used with deep Q-networks (DQN), a deep neural network is often used in tandem with other techniques commonly applied in deep learning, such as convolution and pooling.

A neural network is made of large groups of weights that generate output based on their input, activation function, and any bias terms ( 10 ). Input may be from numeric representations of the input (e.g., color mapping of a picture, positions of vehicles) or it may be from other neurons. Neurons are grouped into layers, and neurons within the same layer are typically connected to the same input and output sources and have the same activation function. When training a model, typically the weights that govern the output of individual neurons are adjusted based the error between the prediction of the model and the actual error. These errors are propagated backwards through the model using the well-known backpropagation algorithm that determines the contribution of each individual weight to the error ( 11 ).

In models with complex inputs and multiple actions, the number of neurons required to process the input can be very high, which can lead to difficulty in training. This problem is further exacerbated in deep neural networks where many layers are typically used. To reduce this effect, convolutional layers and pooling layers are used to filter and identify important features. In a convolutional layer, a smaller collection of weights progressively scans and receives input from a smaller area of the total input. The advantage of this approach is easier identification of the relevant features of a particular situation ( 12 ).

One important concept in machine learning tasks is the concept of “transfer learning,” which is an important tool that can increase the applicability of a model. In transfer learning, a dataset is used to train the model initially. The dataset required to train the model initially may be very large but may not cover all application cases. If the model needs to be applied to a domain outside the initial task and dataset, then transfer learning can be used to reduce the amount of data required to achieve the same level of performance ( 13 ). In image classification problems, this is the idea that a model can be pre-trained on a dataset of labeled images, and then fine-tuned for application on other unrelated images. In traffic, this is akin to training the model on a specific intersection configuration and driver behavior regime and applying it on another ( 14 ).

This paper discusses the application of a DRL algorithm to control a network of traffic signals. In particular, the idea of transfer learning is evaluated by first pre-training the model on a simple single intersection and then applying it to multiple intersections in a corridor. This paper also explores the performance of deep learning relative to traditional methods on a corridor of intersections.

Literature Review

Machine Learning

Artificial neural networks contain nodes that emulate biological neurons. Each node has a set of weights w₀, w₁, … w_n that receives input from other nodes (including the input layer). Each neuron also has an activation function $o_{j} = f (\sum_{i} w_{ji} v_{i})$ that generates the outputs using the weights and input. The most common activation function in use today for deep neural nets is the rectified linear function (ReLU) shown below:

g (z) = \max (0, z)

(1)

where z is the aggregate sum of the input received by the neuron. Qualitatively, the function truncates negative inputs (outputting zero) and is linear on its output, and has shown better performance than linear functions. Another common activation is the softmax function, especially on the final layer, as it can be used to generate a probability distribution to sample actions from ( 15 ).

Weights in the individual layers are updated using backpropagation, which uses stochastic gradient descent to update the weights based on the following Equation ( 11 ):

w_{i} \leftarrow w_{i} - \frac{η (δ Error (E, \bar{w}))}{δ w_{i}}

(2)

where $w_{i}$ is the partial derivative representing the contribution of each weight to the total error observed. $η$ is the learning rate and determines the magnitude of the weight update. While, intuitively, high learning rates should lead to faster weight updates, this is not always the case as large rates may cause the model to diverge. Often variable learning rates are used where initial updates use a larger rate, and smaller rates are used to fine-tune the model ( 16 ).

In DQN, the algorithm uses RL to learn the Q-function. The problems suited to RL must be framed as MDPs ( 17 ). These processes all have four basic elements: a set of states S_t the agent can be in at time t; a set of possible actions A_t they can take at that time; a transition function that defines how the system changes state from time t to time t+ 1; and rewards R(S_t, A_t, S_t₊₁) that can be determined based on the states and actions that measure the agent’s performance ( 17 , 18 ). If the agent cannot observe the true state S_t and instead is limited by its observations O_t, then the situation is called a partially observable Markov decision process (POMDP) ( 19 , 20 ). In RL problems, rewards are usually assigned at every time step (though they may be sparse), and the agent’s goal is to learn an optimal policy $π (S)$ that maximizes these rewards. Future rewards are usually discounted by some factor $γ \in [0, 1]$ to give the value of being in a particular state V as shown below:

V = R_{1} + γ R_{2} + γ^{2} R_{3} \dots = \sum_{j = 0}^{T} γ^{j} r_{j + 1}

(3)

The value function $V^{π}$ is a function that quantifies the value of rewards expected when following a particular policy $π$ . While the summation is infinite, if $γ < 1$ then the value V will have a finite value. Further, if $γ$ is closer to 1, more value is assigned from future rewards, but if $γ$ is 0 then the agent will only consider the immediate reward R₁. When an agent is faced with a choice in a particular state S, the function $Q^{π} (S, A)$ can also be defined as the expected value of choosing action A and then following policy $π$ for subsequent actions. Formally, the definition for these functions is ( 3 , 17 , 18 ):

Q^{π} (S, A) = \sum_{S'} P (S' | A, S) (R (S, A, S') + γ V^{π} (S'))

(4)

V^{π} (S) = Q^{π} (S, π (S))

(5)

where S’ is any successor state to S. When discussing Q-learning and DQN, typically the goal is to learn the optimal policy $π^{*}$ that maximizes these functions. In traditional Q-learning, value iteration is typically used, using the Bellman equation shown below ( 17 ):

Q (S, A) \leftarrow Q (S, A) + α (R + γ max_{A' \in A} Q (S', A') - Q (S, A))

(6)

which is an iterative update that adds the value of the previous value of Q with an error calculated based on the expectations from the best successor state. $α$ is the learning rate and determines how fast this updating takes place. The net result of the process is a mapping of the state-action pairs that can be used to guide optimal action selection.

In well-defined problems, Q-learning will converge to an optimal policy (if the agent is able to visit each possible combination of states and actions enough times) ( 7 ). However, in many situations, the state-action space is very large and complex, and approximations of the Q-function provide better performance. In DQN, this approximation is achieved using a neural network that estimates the value of Q that would be provided by Equation 6 and is trained using the same error ( 3 ).

Machine Learning and Traffic Signal Control

In recent years, many different approaches to machine learning and traffic signal control have been explored by researchers. One of the earliest works to use ordinary Q-learning was done by Abdulhai et al. ( 21 ). Their agent was provided with queue lengths on the four approaches, elapsed phased time, and a reward based on the total delay. The agent was trained using a $ϵ$ -greedy approach and achieved substantial improvements in relation to delay when compared with pre-timed signals. Their work was extended by El Tantawy et al. to consider a multi-agent paradigm using input from adjacent controllers to the same model ( 22 ). Multi-agent approaches have also been explored by other researchers such as Arel et al. where a neural network was used to approximate the Q-function ( 23 ).

Recently, some studies have also begun to consider the deep-learning based implementations. For example, Casas et al. proposed a deep deterministic policy gradient (DDPG) algorithm that used an actor-critic structure with two separate outputs ( 6 ). In the actor-critic design, the critic provides estimates of the Q-value while the actor chooses actions. Their algorithm was designed to function with loop detectors and used a “speed score” calculated based on their output as both the input for the agent and as the basis for the rewarding system. Chu et al. also proposed a scalable multi-agent RL agent based on Advantage Actor Critic (A2C) ( 5 ). Their algorithm used the cumulative delay of the first vehicle together with the number of approaching vehicles as the state input. This design allowed compatibility with existing loop-detectors. Li et al. used a DQN structure to control traffic signals on eight approaches ( 24 ). Their model had a three-layer structure that used sigmoidal activation functions and evaluated it in a simulation environment. They compared their approach with traditional Q-learning and found improvements in the agent’s performance. Tan et al. proposed a cooperative learning framework and used transfer learning to scale up their model to many intersections ( 25 ). Their implementation, however, was limited to two-phase signal controllers. It is important to note that currently no existing products use deep learning techniques to re-time signals; however, some research has also been done on proposing systems to integrate new machine learning techniques with existing signal re-timing processes through engineer-in-the-loop approaches ( 26 ).

While some research has already been done exploring the potential of deep learning, current research is still in the early stages. In particular, few studies have been done exploring the use of deep learning in complex multi-agent environments, and many proposed systems are limited to isolated intersections with two-phase movements. Furthermore, few studies address practical issues surrounding the use of machine learning on intersections, including the issue of the transferability of pre-trained models to other intersection arrangements. This paper’s study is designed to fill in this gap and explores how much additional fine-tuning would be required for a model trained on a simple intersection for application to a multi-intersection corridor. This paper also proposes a novel approach to adapting the ring-and-barrier paradigm commonly used by traditional traffic controllers for use in RL configurations.

Model Design

This paper builds on previous work conducted by the authors exploring the use of deep learning in traffic signal control ( 4 ). A model previously developed by the authors for single intersections is made use of, and it is evaluated on a multi-intersection corridor.

Model Design

The basic structure of the model is adapted from work presented by Google Deep Mind that showed high success when applied to play computer games ( 3 ). The model design uses experience replay that stores past experiences e in a queue $D = {e_{1}, e_{2} \dots e_{t}}$ including their state, the action taken, the reward, and the subsequent state reached. To stabilize training, a target network $θ^{-}$ is maintained. The target network is a copy of the current weights of the neural network that is held constant and synchronized periodically with the network that is constantly updated. At each iteration, the DQN model’s weights are updated during training with samples of experience randomly drawn from D and the current experience e_i and using the target network to calculate the loss according to the following function ( 3 ):

L_{i} (θ_{i} | e_{i}) = {(R + γ \times {max}_{A}^{'} Q (S', A'; θ_{i}^{-}) - Q (S, A'; θ_{i}))}^{2}

(7)

where $θ_{i}$ are the parameters of the Q-network at iteration i, and $θ_{i}^{-}$ are the parameters of the target network used to computer Q at iteration i. The target network’s parameters are updated at fixed intervals. The loss function used is effectively the squared difference between the actual value of Q resulting from the experiences and rewards received, and the expected value calculated by the model based on its beliefs. Training proceeds using a $ϵ$ -greedy approach where the agent initially selects most actions randomly but progressively begins to rely more on and exploit the learned Q values as the model’s training progresses.

State Definition

States are defined as described by Muresan et al. ( 4 ). This is adapted from the work done by Mnih et al. for a traffic system ( 3 ). Four frames representing the transition of the state space across the past four time stamps are used to build the basic components of the input. Vectors with queue length information ( $q_{i})$ , signal timing information (P), and time of day ( $T_{h}, T_{d}$ ) data are used as input. In real-world implementations, these data could be obtained from video detection, LiDAR, or through surrogate measures from other sensors (such as magnetic detectors). These vectors are arranged to fit in an 84 × 84 grid and are coded with binary values. The queue vector $q_{i} = {c_{1}, c_{2}, \dots c_{N}}$ contains values $c_{j}$ which are 1 if there is a vehicle queued in lane group i and 0 otherwise up to a maximum of N vehicles. In this implementation N = 80 and includes eight vectors, one for each through and left movement. Similarly, phase information $P = {p_{1}, p_{2}, \dots p_{k}}$ with values $p_{j}$ that are 1 if a phase is timing and 0 otherwise. k is the maximum number of phases, which is defined as eight in this implementation corresponding to a common arrangement with protected/permissive green and through movements. Finally, the time-of-day vectors $T_{h} = {h_{1}, h_{2}, \dots h_{24}}$ and $T_{d} = {d_{1}, d_{2}, \dots d_{7}}$ encode the time of day and day of the week the agent is operating in. All vectors are then arranged into $S = {q_{1}, \dots q_{8}, P, T_{h}, T_{d}}$ and scaled such that they fit within the 84 x 84 input matrix by scaling each individual vector’s width and length.

Action Definition

There are many different ways to define the actions an agent can take with respect to traffic signal control. The most straightforward method is called phase switching, and describes the situation where agents directly control the phasing decisions similar to how pre-timed signals switch phases after fixed time intervals. In phase switching, the agent has near full control over when to switch the signal to the next phase, but no control over the phase order. In contrast, an agent can be given full control over the phase structure itself or can be restricted to making incremental adjustments to specific timing plan values (such as the split time, cycle time, or offset time) similar to systems like SCOOT ( 27 ). Phase switching provides additional flexibility to respond to fast-changing environments while still allowing engineers to specify which phases should or should not occur.

This paper examines two model structures. In the first, a transferability study is conducted using a two-phase signal based on the authors’ previous work ( 4 ). Action filtering is used to enforce ring-barrier rules on the controller’s decision as well as other rules such as minimum greens and maximum greens. Restrictions on maximum and minimum greens can also be done to enforce other restrictions such as pedestrian signal timing requirements. The agent continuously decides at each timestep $Δ t$ whether to advance to the next phase, or keep the current phase timing. Filtered actions are not selectable by the agent. In this study, $Δ t$ is set to 1 s at all intersections except when the agent can only do the “do nothing” action. This situation arises when timing minimum greens after it has just changed the signal, and, in these cases, $Δ t$ is equal to the minimum green time.

Finally, a novel extension is proposed that emulates the basic operation of a ring-barrier controller with some modifications to reduce the number of actions required and fit the paradigm of RL and action selection. In the proposed method, phases are grouped into two groups, “optional” and “mandatory,” and five actions are defined for the controller to choose from that correspond to advancing to the next phase or skipping the next optional phase on a two-ring controller. Figure 1 highlights the actions available to the controller for a typical protected-permissive design with a callable protected left-turn phase. Actions 1, 2, and 3 correspond to advancing to the next phase in sequence for Ring 1, 2, or both, respectively. For Actions 1 and 2, if the phase change would create an incompatibility with the other ring, that ring’s phase is advanced to the next compatible phase within the barrier group as highlighted in Figure 1. In the figure, Phases 1, 3, 5, and 7 are protected left-turn phases and are defined as “callable” and do not need to be served if no vehicles are waiting. Therefore, if the controller is currently serving Phases 2 and 6 (WB, EB) and the controller does Action 1 (top left scenario of Figure 1), then Ring 1 will advance from EB to NBL and Ring 2 will advance from WB permissive (Phase 2) to the next compatible phase (NB) skipping the optional protected SBL phase to give priority to the NB movement. In contrast, if the controller is currently serving Phases 1 and 5 (EBL and WBL) and Action 1 is selected (bottom left scenario of Figure 1), then the controller will only advance Ring 1 from WBL to EB, as EB is compatible with EBL. Certain combinations of phases will prevent certain actions from being chosen. For example, if the controller is timing Phases 4 and 7 (priority for all NB movements), its only legal actions are the “do nothing” Action 0 or Action 1 to advance Ring 1 from NBL to SB. It cannot choose Action 2 to advance Ring 2 as the barrier after Phase 4 requires Phase 8 to be timing.

Figure 1.

Actions and action filtering for a ring-barrier controller with two rings.

Action 4 is a special action that allows the controller to skip all optional phases and move to a mandatory phase. This is a simplification from what would typically occur in a traditional ring-and-barrier controller, as phases are normally “called” by detectors and can be skipped if no call is present but can be forced to serve through virtual calls to the controller (recall). In this sense, Action 4 can be understood as instructing the controller to ignore all calls from vehicles and skip to the next phase that has a recall option enabled. This simplification is suitable for most configurations but may not be appropriate in certain configurations where multiple sequential phases could be skipped.

Reward Definition

While many performance measures can be considered when defining a system that quantifies “optimal performance,” fundamentally the goal of traffic systems is to maximize the throughput of an intersection. In this case, the simplest reward function is one that provides a fixed reward for each vehicle discharged from the approach. Since rewards are provided at each timestep, the maximum reward is achieved when the controller discharges vehicles immediately (future rewards are discounted, so there is no incentive to delay vehicles). To improve training speed, an additional penalty is also defined that penalizes the agent for causing vehicles to stop and be delayed. The reward function can be formally defined as follows:

r = v_{d} \times x - v_{w} \times y

(8)

where $v_{d}$ is the total vehicles discharging, $v_{w}$ is the total vehicles waiting, x is the reward per discharged vehicle, and y is the penalty per queued vehicle. The ratio between x and y is the reward-to-penalty ratio. While, theoretically, any penalty can be applied, since penalties are applied every time step and rewards are applied only on discharge, the selection of an appropriate value is important to ensure that the agent is not overwhelmed with negative rewards.

Reward clipping is a process that normalizes the reward values, and has been shown to stabilize learning and produce better results ( 28 ). In this study, reward values are clipped to $(- 1, 1)$ using the sigmoid function below:

r_{c} = \frac{\frac{r}{s_{q}}}{\sqrt{1 + \frac{r^{2}}{s_{q}}}}

(9)

where $r_{c}$ is the clipped reward, and r is the reward value calculated using the previous equation. When $r = q_{n}$ , the resulting reward is approximately 0.6. The response of the sigmoid function is more sensitive when $r < r_{c}$ , thus $r_{c}$ is set to a value expected from a moderately busy intersection. This research uses values of $x = 5$ and $y = 0.4$ for Equation 8 and $s_{q} = 250$ for Equation 9 in the two-phase controller, and values of $x = 5, y = 0.1$ for Equation 8 and $s_{q} = 250$ for Equation 9 in the ring-and-barrier controller. The reward values are fed into Equation 7 according to the process described previously, and a value of 0.99 was selected for $γ$ . These values were chosen after some experimentation, including past work done in the authors’ previous paper for single intersections ( 4 ).

Neural Network Design

A deep convolutional neural network (CNN) modeled after Google Deep Mind’s proposed structure is used in the model ( 3 ). The structure of this network is shown in Figure 2. This model has three layers with an input layer of 84 × 84 × 4 and is used to input the last four states to the model. Following the first layer has 32 filters of 8 × 8 with a stride of 4, the second has 64 filters of 4 × 4 with a stride of 2, and the final has 64 filters with a stride of 1. The output of the last convolutional layer is fed into a fully connected layer with 512 units and is then reduced to give Q values for each possible action. This model uses experience replay to sample 10,000 past experiences that are stored in a queue.

Figure 2.

Convolutional neural network (CNN) network structure.

Simulation Network

This study makes use of two simulation scenarios to design and test the model outlined. The intersection is coded in SUMO, which is an open-source microsimulation platform ( 9 ). A single intersection is used to develop and train the model (Figure 3 left) and afterwards it is then applied and fine-tuned on six intersections (Figure 3 right) of a corridor modeled after University Avenue in Waterloo, ON. On the single intersection, a ramping volume scenario is used to expose the controller to varying levels of demand on both the north-south and east-west roadway. The volume changes linearly over 3 h and starts at 2200 vph on the east-west approaches and 0 vph on the north-south approaches and ends at 0 vph on the east-west corridor and 2200 vph on the north-south corridor. The corridor network has two major intersections with similar demands on all approaches, and four minor intersections where University Avenue has higher demand. Volumes on the corridor are derived from field observations from the AM peak period, and signal timing plans are pre-timed with actuated protected left turns at some intersections. The average degree of saturation (x) and peak 15-min degree of saturation for the intersections of this corridor are summarized below in Table 1.

Figure 3.

(Left and right)—SUMO model of individual intersection (left) and corridor (right).

Table 1.

Average and Maximum Total Intersection Volume for Corridor Scenario

Name (East/West and North/South)	Scenario average (vehicles per hour)	Peak 15-min^a (vehicles per hour)
University/Westmount	2,504	3,380
East leg	831	1,180
West leg	393	460
North leg	690	876
South leg	589	864
University/Seagram	2,082	2,880
East leg	1,121	1,640
West leg	615	756
North leg	96	336
South leg	250	148
University/Phillip	1,461	1,860
East leg	676	828
West leg	648	808
North leg	129	208
South leg	8	16
University/Albert	1,934	2,828
East leg	570	776
West leg	608	816
North leg	521	916
South leg	234	320
University/Hazel	1,377	1,924
East leg	592	744
West leg	676	952
North leg	35	60
South leg	73	168
University/King	1,865	2,420
East leg	525	632
West leg	697	892
North leg	290	408
South leg	353	488

Estimated by using the maximum total 15-min volume from the data.

Model Training and Transfer Learning Process

Model training was undertaken in two stages for the transfer learning scenarios tested. In the first phase, the model is pre-trained on the isolated intersection layout for 1 million simulation frames. This duration was chosen by examining the performance of the model and determining where performance gains begin to level off.

During training, the simulation’s volumes are modeled as a Poisson process by converting the average volume to a randomized headway. At any time, if the model creates a situation where more than 150 vehicles are queued and waiting, a terminal state is triggered, and the model is re-started. This prevents the model from spending too much time in congested situations caused by its sub-optimal control decisions. Actions must be selected at each timestep. At each timestep, the selected action is implemented, and the consequences of the action are observed in the subsequent timestep.

A $ϵ$ -greedy approach to training is used, and actions are chosen randomly for approximately 50,000 simulation frames ( $ϵ = 1$ ) as an observation period. Then, following the observation period, $ϵ$ is annealed to a final value of 0.005 over approximately 250,000 simulation frames. These values were selected after some trial and error along conducted during the authors’ previous work ( 4 ). $ϵ$ controls the tendency of the algorithm to exploit its beliefs about the environment and when not acting randomly the agent will select the action with the highest predicted Q value. However, when selecting actions randomly, the design of the controller means that the “do nothing” action is equally as likely to be chosen as any other action. This results in the controller typically only showing the minimum green time when $ϵ$ is high. To provide varied experience and improve training times, if the controller randomly selects the “do nothing” action, a random number $X \in (0, 50)$ is selected and the “do nothing” action is repeated X times adding X seconds of green to the current phase.

In the second phase, the individual model’s state after 1 million simulation frames is then transferred to the model on the evaluation network, giving each intersection the same starting model. These models are then fine-tuned simultaneously using a shorter training process that capitalizes on the model’s pre-existing learned patterns. During fine-tuning, $ϵ$ is set to 0.2, and the observation time is reduced to 5,000 simulation frames and the exploring time to 125,000 simulation frames, while all other parameters remain the same.

Results

Following the completion of the training process, the model is evaluated for a period of 25 simulated peak periods. For the transfer learning scenarios, evaluation is conducted after the model re-converges from the fine-tuning.

Two-Phase Signal: Benefits of Transfer Learning

Transfer learning is evaluated by comparing the training speed in relation to the network’s performance of the model to training on all six intersections from an empty model training on all six intersections simultaneously. For simplicity, and to reduce training speed, this portion of the analysis was conducted using a two-phase signal. As part of the modification, turning volumes were removed and Synchro was used to estimated new signal timing plans for each intersection to use as a baseline comparison with a fixed cycle length of 60 s. Offsets used by the field timing plan were maintained. A minimum green of 6 s and a maximum green of 100 s were used to restrict which actions the controller could choose. No coordinated cycle length was imposed on any of the DRL controllers. All signals were coded to use 5 s of amber and 2 s of all red. Terminal states are triggered if any intersection has more than 150 vehicles waiting.

Average cumulative delay and total counts of stopped vehicles for all vehicles on the network were collected using SUMO’s data collection tool. The data collection tool used in SUMO does not record delay until the vehicle exits the simulation, thus some inaccuracies arise if terminal states are triggered because of excess queues (only occurs before convergence). These results are summarized in Figure 4 below for both the full training and the fine-tuning portion of the transfer learning scenarios. In addition, average cumulative delay values for vehicles travelling on specific key routes are also tabulated and shown in Figure 4.

Figure 4.

Average delays (top), queues (middle), and per route delays (bottom) for vehicles in the two-phase controller.

While the fine-tuning process creates congestion and delays on the network, at their worst these delays are half the delays caused by training the model from scratch. This suggests that the model is able to exploit some of the learned patterns from the pre-training process. It should be noted that the model applied during the transfer learning did not create terminal states because of high queue lengths. The use of transfer learning allows the model to reach convergence in about half the time it takes to train a complete model from initial conditions. In relation to computation time, on average approximately 500,000 simulation frames can be generated per 24 h period on the hardware evaluated (Intel i7-4790) and during training each intersection’s model uses 2gb of RAM. Training speed is bottlenecked by data extraction from SUMO and speed is reduced during highly congested periods.

In relation to the delay, when compared with the Synchro plan, the DRL model provides higher delays on the corridor, but lower delays on the side streets. However, this optimization led to similar delays overall but average queue lengths at intersections were lower. On average for all vehicles the Synchro plan produced delays of 53 s and at any given time approximately 64 vehicles were waiting at a stoplight. In contrast, the DRL model produces average delays of 54 s and an average of 26 queued vehicles at any given time after convergence in the transfer learning scenario and 53 s and 25 vehicles queued when the full model is re-trained.

Full Ring Barrier Controller: Transfer Learning

The transfer learning approach discussed previously was then used to train and fine-tune a model using the full ring-barrier controller with five actions. This controller was trained on the same volume scenario used to train the two-phase signal, but with 10% less volume and with 5% left-turning and 5% right-turning traffic. A total of 3 million simulation frames were required to achieve convergence. Comparison was done to the currently implemented traffic signal timings and counts were obtained from the Region of Waterloo and are based on field studies conducted during the AM peak period. Signal plans were modified to increase their amber time to 5 s (from 4 s) after the through movements had timed green as amber times of 4 s were found to cause lock-ups and priority issues with turning movements in SUMO because of inadequate clearing time. The existing signal controllers have actuated protected left-turn phases at all intersections except Hazel St, including green extension and a maximum green time for the protected phase. This design was replicated in SUMO using the built-in signal controller.

After training a single intersection model, the same transfer learning process discussed previously was applied to evaluate the model on the corridor. For simplicity, the same ring-and-barrier model design and structure discussed in the previous sections were used for all intersections (even though the intersection at Hazel does not have protected left phasing, the DRL controller was permitted to use it). The results are highlighted in Figures 5 and 6 which show the overall performance for the simulated period and the performance on a per 5-minute basis respectively.

Figure 5.

Average delays (top left), queues (top right), and per route delays (bottom) for the full ring barrier design.

Training time increases substantially when compared with the case with a two-phase controller. A total of nearly 5 million frames of simulation were required before convergence was reached (3 million for the single intersection, and 2 million for the for fine-tuning) and was done over 10 days on an Intel i7-4790.

In relation to delay, the fully trained model provided better performance than the field implemented times at all intersections. In particular, the field times currently in use do not provide adequate service to the left turns at Albert and Westmount during the peak hour period, and queue-spillback blocking the main street occurs for part of the simulation. This spillback is better handled by the DRL model. This is clearly visible in Figure 6 where the spike in demand on the minor streets leads to higher delays in the field plan that are mitigated by the DRL model. On average under the field plan, vehicles experience a total delay of 83 s and at any given time an average total of 91 vehicles are waiting behind a stoplight. In contrast, in the DRL model vehicles experience an average delay of 78 s and at any given time an average of 66 vehicles are waiting behind a stoplight. It should be noted that, for some time periods during the peak hour, the queues overflow such that vehicles can no longer enter the simulation. This delay is not accounted for, as only in-network travel-time is considered in the delay calculation. Thus, the delay under the field conditions is higher than what is reported here. Based on these numbers, the DRL model improves on the field timing plan’s performance by at least 5% in relation to average delay.

Figure 6.

Average delays for minor (left) and corridor streets (right) by time interval for last 10 runs.

Conclusion

The performance of RL is ultimately tied to the rewarding system used. In the proposed framework, a rewarding system based on queue lengths and discharging vehicles was used and compared against traditional methods and the existing signal timing plan. In general, this resulted in a controller that sought to lower queue lengths as much as possible, and when compared with traditional actuated control queues were lower by 60% in the two-phase design and by 27% in the full ring-barrier design.

Although the DRL controller reduced the average amounts of queued vehicles in the system significantly, only modest reductions in delay were observed. For the two-phase design, performance was similar between the DRL controller and an optimized Synchro-developed timing plan. In the full ring-barrier design, delays were reduced by approximately 5% when compared with currently implemented field timings. However, these reductions did not take into account queue-spillback on some side streets that prevented vehicles from entering the simulation during the peak hour period. The DRL controller was able to react to oversaturated conditions on side streets and prevent queue-spillback.

Limitations and Future Research

The proposed platform discussed here is one of the first examinations of DRL to replicate a fully featured ring-and-barrier style controller. While the proposed controller accommodates many of the important aspects of a traditional controller, some simplifications were made to adapt the design to better suit the paradigm that RL operates in. Future research will explore other approaches to action filtering.

The full ring-and-barrier design increases the action count in many situations from two actions to five. This results in higher training times. Initial training on the single intersection required approximately 3×as much experience in the full ring-and-barrier implementation, and just over 2×the amount of experience in fine-tuning. In relation to real-world time, the full ring-and-barrier design required 10 days of computation time to train to convergence compared with approximately 3 days for the two-phase design. The proposed platform does not use any GPU acceleration, and training was done only using the CPU, with the major bottleneck being the interface between the simulation platform and the controller.

This research focused on a feasibility assessment of using transfer learning to reduce the amount of training required, and demonstrates the potential for DRL to be used as an intelligent traffic controller. While some aspects are explored, many additional questions need further exploration. In particular, this study does not test varying levels of demand or compare the performance of other adaptive systems. Additionally, while some elements of coordination are discussed, this research did not focus on evaluating green progression along the corridor, and, in some cases, the proposed model provided higher delays on the evaluated corridor to maintain better levels of service on the side streets. Furthermore, in the platform evaluated, all signals were controlled by the DRL system, and no co-operation with other coordinated intersections was studied. Future research will further explore the effectiveness of DRL to work in wider coordinated systems as well as focus on improving the platform to better consider the actions of adjacent intersections.

Footnotes

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: M. Muresan, G. Pan, L. Fu; data collection: M. Muresan; analysis and interpretation of results: M. Muresan, G. Pan, L. Fu; draft manuscript preparation: M. Muresan, G. Pan, L. Fu. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded in part by the National Science and Engineering Council (Canada) (PGS-D award, reference #503970), Mitacs Canada (Globalink Research Award [Application #IT09579]), and Ontario Research Fund—Research Excellence (ORF-RE) (Intelligent Systems for Sustainable Urban Mobility [ISSUM] project).

References

Vinyals

Babuschkin

Chung

Mathieu

Jaderberg

Czarnecki

W. M.

Dudzik

Huang

Georgiev

Powell

Ewalds

AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. DeepMind Blog, 2019.

Silver

Schrittwieser

Simonyan

Antonoglou

Huang

Guez

Hubert

Baker

Lai

Bolton

Chen

Mastering the Game of Go without Human Knowledge. Nature, Vol. 550, No. 7676, 2017, p. 354.

Mnih

Kavukcuoglu

Silver

Rusu

A. A.

Veness

Bellemare

M. G.

Graves

Riedmiller

Fidjeland

A. K.

Ostrovski

Petersen

Human-Level Control through Deep Reinforcement Learning. Nature, Vol. 518, No. 7540, 2015, p. 529.

Muresan

Pan

Adaptive Traffic Signal Control with Deep Reinforcement Learning an Exploratory Investigation. Presented at 97th Annual Meeting of the Transportation Research Board, Washington, D.C., 2018.

Chu

Wang

Codecà

Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, Vol. 21, No. 3, 2019, pp. 1086–1095.

Casas

Deep Deterministic Policy Gradient for Urban Traffic Light Control. arXiv Preprint arXiv:1703.09035, 2017.

Sutton

R. S.

Barto

A. G.

Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, Vol. 135, 1998.

Tokic

Palm

Value-Difference Based Exploration: Adaptive Control between Epsilon-Greedy and Softmax. Proc., Annual Conference on Artificial Intelligence, Springer, Berlin, Heidelberg, 2011, pp. 335–346.

Baird

Residual Algorithms: Reinforcement Learning with Function Approximation. Proc., 12th International Conference on Machine Learning, Tahoe City, CA, 1995, pp. 30–37.

10.

Basheer

I. A.

Hajmeer

Artificial Neural Networks: Fundamentals, Computing, Design, and Application. Journal of Microbiological Methods, Vol. 43, No. 1, 2000, pp. 3–31.

11.

Hecht-Nielsen

Theory of the Backpropagation Neural Network. In Neural Networks for Perception, Academic Press, Cambridge, MA, 1992, pp. 65–93.

12.

LeCun

Bottou

Bengio

Haffner

Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, Vol. 86, No. 11, 1998, pp. 2278–2324.

13.

Pan

S. J.

Yang

A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, Vol. 22, No. 10, 2009, pp. 1345–1359.

14.

Pan

Muresan

M. I.

Winter Road Surface Condition Recognition using a Pre-Trained Deep Convolutional Neural Network. Presented at 97th Annual Meeting of the Transportation Research Board, Washington, D.C., 2018.

15.

Agarap

A. F.

Deep Learning using Rectified Linear Units (Relu). arXiv Preprint arXiv:1803.08375, 2018.

16.

Krizhevsky

Sutskever

Hinton

G. E.

Imagenet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, Massachusetts Institute of Technology Press, Cambridge, MA, 2012, pp. 1097–1105.

17.

Szepesvári

Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, Vol. 4, No. 1, 2010, pp. 1–103.

18.

Watkins

C. J.

Dayan

Q-Learning. Machine Learning, Vol. 8, No. 3–4, 1992, pp. 279–292.

19.

Hoey

Poupart

Solving POMDPs with Continuous or Large Discrete Observation Spaces. IJCAI, 2005, pp. 1332–1338.

20.

Astrom

K. J.

Optimal Control of Markov Processes with Incomplete State Information. Journal of Mathematical Analysis and Applications, Vol. 10, No. 1, 1965, pp. 174–205.

21.

Abdulhai

Pringle

Karakoulas

G. J.

Reinforcement Learning for True Adaptive Traffic Signal Control. Journal of Transportation Engineering, Vol. 129, No. 3, 2003, pp. 278–285.

22.

El-Tantawy

Abdulhai

Abdelgawad

Multiagent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): Methodology and Large-Scale Application on Downtown Toronto. IEEE Transactions on Intelligent Transportation Systems, Vol. 14, No. 3, 2013, pp. 1140–1150.

23.

Arel

Liu

Urbanik

Kohls

Reinforcement Learning-Based Multi-Agent System for Network Traffic Signal Control. IET Intelligent Transport Systems, Vol. 4, No. 2, 2010, pp. 128–135.

24.

Wang

F.-Y.

Traffic Signal Timing via Deep Reinforcement Learning. IEEE/CAA Journal of Automatica Sinica, Vol. 3, No. 3, 2016, pp. 247–254.

25.

Tan

Bao

Deng

Jin

Dai

Wang

Cooperative Deep Reinforcement Learning for Large-Scale Traffic Grid Signal Control. IEEE Transactions on Cybernetics, Vol. 50, No. 6, 2019, pp. 2687–2700.

26.

Jin

Guo

Wang

F.-Y.

An End-to-End Recommendation System for Urban Traffic Controls and Management under a Parallel Learning Framework. IEEE Transactions on Intelligent Transportation Systems, Vol. 99, 2020, pp. 1–11.

27.

Hunt

P. B.

Robertson

D. I.

Bretherton

R. D.

Royle

M. C.

The SCOOT On-Line Traffic Signal Optimisation Technique. Traffic Engineering & Control, Vol. 23, No. 4, 1982, pp. 190–192.

28.

Mnih

Badia

A. P.

Mirza

Graves

Lillicrap

Harley

Silver

Kavukcuoglu

Asynchronous Methods for Deep Reinforcement Learning. Proc., International Conference on Machine Learning, New York, NY, 2016, pp. 1928–1937.