A machine learning pipeline for extracting decision-support features from traffic scenes 1

In order to train and test our pipeline, we first had to search a dataset featuring traffic footage with aerial angulation and appropriately annotated vehicles. Meeting these requirements provide sufficient conditions to perform object detection and tracking tasks using CNN architectures.

We consider the UAVDT dataset [10]. The UAVDT dataset features traffic scenes with diverse heights (i.e., low, medium, high) and angulations (i.e., front, side, and bird). This dataset includes 50 different traffic scenes, totaling over 80,000 frames. However, in order to ensure that the whole intersection is captured in the scenes, we selected only a subset of scenes that are high enough to enable all incoming and outcoming lanes to be observed. This includes scenes with front or side views with at least medium altitude as well as bird view at any altitude. The final list of scenes is presented in Table 1. These scenes are essential to train the vehicle detector method, which we describe in detail in the next section.

At this stage of the pipeline, the main objective is to feed the frames provided by the dataset as input to the detector and identify the vehicles in the scene. The detector takes the input data and produces bounding boxes around the vehicles present in the scene, thus providing their locations.

In order to perform object detection, we employed the YOLOv4 network [31], which is considered the state-of-the-art architecture for real-time object detection. The YOLOv4 network was pre-trained with the COCO dataset. However, for the particular case of traffic scenes, this was not enough to yield satisfactory performance. To this end, we fine-tuned the YOLOv4 network using the UAVDT dataset.

We started with the Darknet implementation of YOLOv4.2

We further customized the input shape of the network to 832 × 832 pixels (the original network had an input of 512 × 512 pixels) to improve the performance of the network on distant scenes. The batch size and number of subdivisions were both set to 64. The number of channels was set to 3. The saturation and exposure values were both set to 1.5. The momentum value was set to 0.949, weight decay to 0.0005, learning rate to 0.001, and max batches to 6000 samples (the original network uses 2000 samples) in order to take into account the RAM limitations of our setup. Figure 2 illustrates the output of this phase. Our model was trained using Google Colab Pro, featuring a NVIDIA Tesla P100 GPU, providing significant computing power to our training process. The model was trained for a period of 30627 seconds (approximately 9 hours) and has an average inference cost of 0.07 seconds per frame.

OPEN IN VIEWER

Fig. 2.

Output of the vehicle detection phase based on a frame from the M0603 scene.

In the object tracking phase, each vehicle detected in the previous phase is tracked throughout a sequence of frames in order to enable the identification of its trajectory. This step is essential to enable the interpretation of the origins of destinations of the vehicles, as performed by the phase described in the next section.

We highlight that, although the object detection phase is the one responsible for identifying the vehicles, it is not able to tell whether an object appearing in subsequent frames represent the same vehicle. This is precisely the objective of the tracking phase, which needs to compare the position of vehicles in subsequent frames in order to have a unique correspondence.

To perform the task at hand, we utilized the tracker proposed in [6]. This method operates under the assumption that the detector produces a single detection per frame for each vehicle to be tracked. Essentially, for a given vehicle, the distance between its positions in consecutive frames must be either minimal or non-existent. Additionally, the method relies on the bounding boxes representing the vehicle in two successive frames having a substantial overlap, as measured by the Intersect Over Union (IoU) metric. We set the threshold for the IoU metric at 50%, meaning that to be considered the same object, the previous and current bounding boxes must have at least a 50% overlap.

OPEN IN VIEWER

Fig. 3.

Object tracking phase, where (a) shows a sketch of the method proposed by [6], and (b) presents the output of the tracking phase based on a frame from the M0603 scene.

Figure 3a illustrates how the tracking technique works. In the figure, two objects are shown along a sequence of four frames. Each object is delimited by a bounding box and the arrow denotes its direction. Position are represented as x and y coordinates. As seen, for any object, the bounding boxes on subsequent frames present a high overlap. Building upon that observation, the trajectory of an object can be inferred by observing the position of its bounding box along frames.

In this sense, after receiving the vehicle detection coordinates, the tracking algorithm starts its verification to infer whether the vehicle is new on the scene or not. If the vehicle is new, an ID is assigned to it. Otherwise, the newly perceived coordinates are stored in an array containing all coordinates previously detected for that particular vehicle. Figure 3b presents the result of the tracking method, where an ID is assigned to each vehicle identified in the previous stage.

At this stage of the pipeline, the objective is to detect the points where a vehicle enters and exits the intersection. The idea is that, by detecting such points, one can infer the origin and destination of each vehicle. Recall that, for RL-based traffic light control, this information is of uttermost importance, as it allows the agent to properly model the flow of vehicles and make its decisions [23]. To this end, we define virtual lines in the frames to represent the entrance and exit points of the intersection, where each virtual line is composed of a pair of x and y coordinates. The sequence of steps of this phase are depicted in Fig. 4a.

OPEN IN VIEWER

Fig. 4.

Origin and destination identification phase: (a) shows how the virtual lines are positioned in the scene; (b) illustrates the detection of a single vehicle in a single frame, and (c) its center; (d) presents the same vehicle being detected along multiple frames and their corresponding detection centers (e) shows the detection centers being joined by a line, thus representing a trajectory; (f) represents the moment that the trajectory touches the virtual line A; (g) illustrates the moment that the trajectory touches virtual line F. The tracking shown in (d) makes use of an IoU threshold lower than the one described in Section 3.3. This was done only for illustration purposes, in order avoid too close frames and thus enhance presentation.

In order to enable the identification of origins and destinations, virtual lines are placed at the entrances and exits of the intersections. Then, through the data provided by the tracking phase, it is possible to discover the entire trajectory of the vehicle along the scene based on the returned array of coordinates. For each coordinate, its center is calculated to generate the trajectory, as depicted in Fig. 4d. In this way, the trajectory is a set of centroids obtained from the array of coordinates, as seen in Fig. 4e.

To be able to know where a vehicle entered and exited the intersection, we calculate the intersection between the vehicle’s trajectory and the annotated virtual line, as shown in Figs 4f and 4g. If the intersection is positive, we store the vehicle ID in an array associated with the respective straight segment. In addition, we increment a counter that shows how many vehicles have passed through that line.

OPEN IN VIEWER

Fig. 5.

Identification of the origins and destinations of the vehicles based on frames from the M0603 (Fig. 6a) and M0403 (Fig. 6b) scenes.

In Fig. 5, we can observe the operation of the pipeline. In the figure, once vehicles and their trajectories are detected, it is possible to compute through which entrance and exit lines each vehicle goes through. Such information can then be used to count the number of vehicles on each origin and destination pair within the given intersection.

As mentioned in Section 3.3, the object detection phase is not able to distinguish between object instances; it only tells where each object is placed at the scene. Our tracking algorithm is then responsible for discerning between them by a certain degree of confidence. Unfortunately, there are cases in which we lost track in the middle of the experiment. For instance, consider the bus tracked in Fig. 6a with id 920 (the one turning right). In the next frame (Fig. 6b), when the bus passes beneath the tree, the detector fails to detect the given bus. Once the bus crosses the tree (Fig. 6c), the detector is again able to detect the bus. However, as the connection with the previous frames was lost, our algorithm assigns it a new id, namely 1520, and after that the bus crosses an end-point. So, as shown in this example, the object with id 1252 seems to only crosses an end-point, but without ever crossing any starting point.

In order for a vehicle’s track to be counted as finished, it must cross both the start point and end point. If the detector loses track of a vehicle in the middle of its journey, a new ID is assigned and its track begins again. This can occur multiple times for the same vehicle, resulting in its ID being reassigned and its track being restarted. To address this issue, our approach is straightforward: we identify all tracks that have crossed the end point but not the start point (a.k.a output tracks) and attempt to match them with tracks that have crossed the start point but not the end point (a.k.a. input tracks). Our aim is to find the shortest distance between the last frame of the input track and the first frame of the output track, thereby determining if the two tracks correspond to the same vehicle.

In the example shown in Fig. 7, the detector loses track of a bus multiple times, resulting in the creation of several tracks for the same vehicle, including both input and output tracks. After the simulation is completed, our algorithm attempts to match each output track with an input track by using the shortest distance criteria between the last frame of the input track and the first frame of the output track. In this particular case, the algorithm matches output track 1272 with input track 748, combining them both into a single track with ID 748, thus establishing an origin-destination pair. It is worth noting that track 1182, shown in Fig. 7d, is neither an input track (as it has not crossed a start point) nor an output track (as it has not crossed an end point), and therefore it is not included in the matching algorithm.

OPEN IN VIEWER

Fig. 6.

Tracker losing identification on vehicle. As the bus from Fig. 6a passes beneath the tree, the tracker loses its track, as shown in Fig. 6b. Later, in Fig. 6c, when the bus became once again detectable, the tracker associates that object as a new one; thus, assigning it a new id.

OPEN IN VIEWER

Fig. 7.

This example demonstrates the method described in Section 3.4. In Fig. 7a, the center of the bus’ bounding box crosses imaginary line A, marking this track (known as track 748) as a new input track. The detector continues to track the bus without issue until Fig. 7b, where it fails to detect the bus in the next frame (Fig. 7c), resulting in the loss of track 748. In Fig. 7d, the detector detects the bus again, but assigns it a new track (track 1182) since track 748 was lost. As mentioned in Section 3.4, this loss of detection can occur multiple times for the same object, as is evident in Fig. 7e, where the center of the bus’ bounding box crosses imaginary line F with a track ID of 1272 (indicating that the detector failed to detect the bus again). Track 1272 is an output track. After completing the detection run, our algorithm matches the closest pairs between input and output tracks, combining them to create a single track. In this case, tracks 748 and 1272 are matched, resulting in a single track with an ID of 748 (using the ID of the input track).

4.1. Methodology

In order to test our pipeline, we selected the scenes M0403 and M0603 from the UAVDT dataset. To evaluate the pipeline, we adopted two complementary strategies. The first one is to evaluate the pipeline’s task, which is multi-object tracking. Measuring the performance of multi-object trackers requires some care. Although a variety of assessment metrics have been proposed in the literature [24], there has yet to be a general consensus on the best set of metrics to use. In this sense, we decided to employ the following metrics. In the list, the arrows in parentheses indicate whether each metric should be maximized (↑) or minimized (↓).

IDF1 is a metric that combines precision and recall to evaluate the performance of multi-object tracking algorithms. It is based on optimal assignment between ground truth objects and detected objects. The IDP and IDR metrics are calculated based on optimal assignment between ground truth objects and detected objects. The recall metric assesses the algorithm’s ability to detect all objects in the scene, while precision measures its ability to avoid false positives. False positives refer to objects falsely detected as present, while false negatives (FN) refer to objects present in the scene but not detected by the algorithm. The IDs metric provides an indication of tracking stability, with a lower value indicating better tracking. MOTA calculates the percentage of errors made by the algorithm in tracking all objects, while MOTP measures tracking accuracy in terms of object positions.

The strategy to evaluate the complete pipeline, on the other hand, requires metric able to compare the number of vehicles on each origin-destination pair as output by our pipeline in comparison to the ground truth counting. To this end, we propose a new metric called OD Counting Error, as shown below: (1) OD Counting Error = 1 GT ∑ i = 0 n | Detection i − GT i | , where GT is the total number of vehicles that completed their routes, n represents the number of OD pairs in the scene (which in this case is the same as the number of routes), Detection i and GT i denote the number of vehicles from OD pair i that completed their routes as detected by our pipeline and represented as ground truth, respectively. An intuitive description of the newly proposed metric is provided in Fig. 8, where an illustrative traffic intersection with 20 vehicles is explained and the resulting OD Counting Error is computed. This metric is employed to assess our pipaline in the next section by means of Tables 4 and 5.

OPEN IN VIEWER

Fig. 8.

An example of the newly proposed metric (Eq.(1)) can be illustrated as follows. Consider two OD pairs (AC and AB). Figure (a) shows the ground truth, indicating that GT AB = 10 , GT AC = 10 , and GT = 20 . Figure (b) shows our detection results, where we intentionally made a mistake in identifying the AC pair, resulting in Detection AB = 10 and Detection AC = 9 . When the OD Counting Error metric is computed using Eq. (1), we obtain a value of OD Counting Error = 0.05 , indicating that we lost tracking of 5% of the OD pairs.

In order to better assess the performance of our approach, we adopted [37] as a baseline method for object detection and tracking, which we consider to be the state of the art in the context of traffic. As for the origin-destination identification part, we include no comparison given the lack of works on the topic.

Our experiments were executed in two different environments. The object detector was trained using Google Colab Pro, featuring a NVIDIA Tesla P100 GPU. The remaining experiments, in turn, were performed on a standard computer with Intel Core i7 CPU with 16GB of RAM and a NVIDIA GeForce RTX 2060 GPU.

We start the results section by analyzing the performance of our pipeline in its first two phases, the vehicle detector and the vehicle tracker. Table 2 presents the results obtained by our method and the baseline for the M0603 and M0403 scenes of the UAVDT dataset. As seen, the average results obtained by our pipeline outperform the current state-of-the-art approach by a good margin for most metrics. These results indicate that our pipeline is able to properly detect and track vehicles. This was made possible because we used a fine-tuned version of the YOLOv4 network, which yielded superior results than those obtained by the siamese network used in [37]. Moreover, since the our method yields better detections, the tracking phase ends up outperforming [37] as well.

The detector achieved a precision of 79.5%. This result directly influence the accuracy of the tracker, which ended up with an MOTP 74.4%. In order to improve the accuracy of the detector, it is necessary to train the model with more instances of the truck and bus classes, whose number was not sufficient in the UAVDT dataset. Throughout the experiments we observed that these specific classes end up failing more than cars. In addition, the use bird view images hinders the identification of smaller objects in the image. To improve this aspect, one could increase the size of the network at the time of training, which could enhance the accuracy at the cost of increasing the complexity of the object detection network.

Table 3 presents the results related to the number of vehicles that passed through each virtual line drawn in the scenes. Observe that the number of virtual lines is seven and nine for the M0603 and M0403 scenes, respectively. Results are compared to the annotated ground truth values. As seen, our method correctly detected all cases in the first scene and mistakenly detected only 3.38% of the cases in the second scene. We remark that the errors occurred because the detector failed to identify these vehicles. These errors could be reduced by improving the accuracy of the detector.

Table 4 presents the overall results of our pipeline on the M0603 scene by means of the OD Error metric. Recall that this metric shows for how many vehicles the pipeline failed to identify their complete trajectories. As seen in the table, our pipeline missed a single vehicle, which goes from E to D. In that case, the error happened because part of the scene is occluded, which prevented the detector from detecting the vehicles on that part of the scene. This situation is illustrated in Fig. 6, where the vehicle with ID 920 becomes undetectable when passing close to a tree and then becomes detectable again, thus receiving a new ID (namely 1252). In addiction to that, the detector was not well trained to deal with different classes of vehicles (such as buses). That fact increases the difficulty of finding a continuous track, as well as its origin and destination.

Finally, Table 5 presents the overall results of our pipeline on the M0403 scene by means of the OD Error metric. As seen, only a single vehicle was not properly identified, which represents a percentage error of 4.35% in the identification of the complete trajectory of the vehicles. In this scene, no occlusion points exist. However, the error also happened due to a failure in the detector. After the vehicle started its trajectory entering C, the detector failed, as such the vehicle ended up receiving a new ID before ending its trajectory.