Automatic detection system of speed violations in a traffic based on deep learning technique

Abstract

With the growth in vehicular traffic, there is a greater risk of road accidents. Over speeding, intoxicated driving, driver distractions, red-light runners, ignoring safety equipment such as seat belts and helmets, non-adherence to lane driving, and improper overtaking are the leading causes of accidents. Speed violation, in particular, has a significant influence on today’s transportation. Also, detecting this speed violation and punishing this violator are more time-consuming tasks. For that reason, a novel automatic speed violation detection in traffic based on Deep learning is proposed in this paper. This proposed method is separated into two working modules: object detection and license plate recognition. The object detection module uses the most efficient PP YOLO neural networks. It utilizes open ALPR (Automatic License Plate Recognition) for the vehicle’s number plate identification, which passes the traffic above maximum speed. With the number plate details, the authorities can take action against the rule violator with less time and effort. The simulation results show that the proposed automatic speed violation detection system also has an accuracy rate of 98.8% for speed violation detection and 99.3% for license plate number identification, demonstrating that the approach described in this work has a higher performance in terms of accuracy. Furthermore, the proposed technique was compared to recent existing results.

Keywords

Speed violation intoxicated driving deep learning PP YOLO object detection license plate recognition

1 Introduction

The world’s population is growing at a rate comparable to automobile production. As a result of this occurrence, road congestion, violations, and accidents occur [1]. The number of road traffic accidents and deaths is increasing year after year, according to the World Health Organization’s (WHO) 2018 global status report on road safety [2], and road accidents are indeed the top cause of mortality for children as well as young people.

The problem is exacerbated in low- and middle-income countries, with more excellent poverty rates. The extent to which traffic laws are followed is lagging behind the performance of autos and transportation networks. In India, only road accidents have claimed the lives of 20.42 lakh people, with 82.30 persons injured. Two-wheelers were the most often engaged in traffic incidents in 2017. Four two-wheeler riders die per hour because they don’t wear a helmet [3], and the road’s capacity is insufficient to meet the growing number of automobiles, resulting in an imbalance. Traffic offences have become a severe concern in today’s globe. The increased number of cars and other vehicles on the road is to blame for the high level of traffic. Individuals disregard traffic regulations, and there is a pressing need to address this [4]. Due to weak and challenging human memory, a manual vehicle inspection is time-consuming and prone to mistakes. As a result, to undertake this duty, a traffic violation detection system that can recognize criminal traffic offences [5], such as signal jump, over-speeding, and vehicle count] is needed. Officers utilize CCTV video to detect cars breaching the law when it comes to traffic cops. Investigating each camera and discovering the vehicle that violated the traffic law takes time [6].

There are hundreds, if not thousands, of traffic offences every day, and unique cameras capture the offending vehicles. If all of these images are inspected by people and the plate numbers are manually entered, time and effort will be wasted. As a result, automatic licence plate identification and high-precision number plate detection are necessary [7]. Machine Learning will be an excellent solution to this issue. This technique starts with capturing a picture, which is subsequently subtracted from the reference image to reveal the violation [8]. It appears that installing various cameras at high-traffic junctions and streets and providing effective technology for the automatic detection of unsafe driving behaviours are essential. The information provided by video security cameras may be considered big data since they generate large volumes of data in a short period—they detect semantic items and classes in digital photos and videos [9]. Automated images and video systems for capturing traffic rule infractions help avoid traffic offences. Around 11.2 thousand images and video detection systems already detect speeding violations. In a prior study, a traffic infraction detection application enabled by the Internet of Things was developed, which sees an object of a vehicle’s license plate number. As a result, the tool’s accuracy established before is still less than 70%. We can remedy this by utilizing the OPEN ALPR API (application programming interface). The open APLR is an image processing API that converts the picture of letters and digits on a vehicle’s license plate number into a string. Open ALPR has already been tested in image processing accuracy, with a score of more than 70% [10].

Although it will provide information regarding the vehicle’s license plate, it will not assist in the detection of speed violations. As a result, we’ve suggested a novel Automatic Detection System of Speed Violations in Traffic based on Deep Learning Technique. This proposed method focuses on speeding violations because over-speeding is currently the leading cause of most traffic accidents. In this approach, a speed estimation is utilized for training the machine learning algorithm that will aid in identifying vehicles that cross the road at speeds beyond the speed limit in a shorter amount of time and effort. The vehicle’s license plate number is then determined, and it will aid authorities to take action against the traffic violators as per the traffic regulations.

The main contributions of this work are presented below:

By using the proposed method, authorities can take action against the violator with less time and effort.

To achieve the fastest and more accurate object prediction most efficient YOLO based neural network is utilized in this work

By using the open ALPR, the license plate number of the violator can be detected without any blur.

This paper’s remaining structure is as follows: Section 2 discusses the literature review; Section 3 describes the proposed approach; Section 4 presents the summary of the work; Section 5 presents the work’s results; Section 6 provides the conclusion of this work, Section 7 provides the proposed method’s references.

2 Literature survey

Many authors have developed speed violation detection algorithms and models based on deep learning techniques to reduce the work and time used by authorities. Some of these approaches are discussed briefly below.

Wong et al. [11] proposed an approach for recognizing moving objects (vehicles) and detecting the RFID tag’s sensitivity. Unfortunately, the problems and limits in the camera, which must be adjusted, as well as differences in brightness and unfavourable circumstances, will cause the testing findings to be misinterpreted in some cases.

Sushmitha et al. [12] proposed a method for detecting the stated violations, including rejecting traffic signals, parking in no-parking zones, and driving in the wrong direction. The detection convergence for the three types of traffic infractions stated differs since each has a distinct threshold requirement. Furthermore, the application runtime is slightly sluggish, which may be improved by utilizing a high-speed CPU or GPU computer.

Li, Yunxua et al. [13] presented a Spatio-temporal kernel density estimation (STKDE) model. It was used to uncover geographical and temporal trends of traffic infraction behaviour at metropolitan junctions. To illustrate the density of traffic infractions, the STKDE model constructs a space-time cube with various voxel colours. An intriguing result was that traffic signs and marking violations were concentrated mainly outside typical peak hours. Furthermore, while it is widely known that traffic infractions are one of the significant causes of traffic collisions at junctions, a quantitative analysis of the mechanisms behind violations and crashes is required.

Umar et al. [14] proposed utilizing KNN to develop an automatic license number identification system for red-light traffic offenders. The system was implemented as planned. The system correctly identifies the correct number of characters. A rate of 92 % in detecting license plates was attained. The results achieved are encouraging and can be improved for a more significant road with additional lanes. Future research will develop a novel method for using adaptive techniques in image processing to accommodate variable plate dimensions.

Bhat et al. [15] concentrate on identifying traffic violations, which has grown important in intelligent traffic control research. The main goal is to develop an algorithm that improves the input algorithm by using evolutionary principles to produce the “fittest” algorithm. The fitness value is calculated repeatedly in some situations, which might be computationally costly. There are no assurances that the answer is optimum.

Saeed et al. [16] presented a method for measuring vehicle speed depending on motion towards an intelligent transport system. The proposed approach is made up of three main modules: vehicle identification, tracing, as well as speed estimation. Each moving object is detected when it enters the ROI using the Mixture-of-Gaussian background subtraction method. The various components of these objects are then transformed into uniform filled shapes using morphological transforms, and certain predetermined filtration functions leave just the items with the highest probability of becoming a vehicle intact. Vehicles that are detected are then tracked using the blob tracking method, and their displacement between successive frames is determined for the ultimate speed assessment module. The vehicle’s image, associated speed, and detection limit are among the system’s outputs. Compared to current rate measuring methods, experimental findings suggest that the proposed methodology offers adequate accuracy.

Alperen et al. [17] suggested an innovative license plate identification system for primary roadside surveillance cameras that uses deep learning-based object detection methods as proposed segmentation inside the character detection and recognition process. The proposed method was tested using 2000 photographs taken along a route. Unlike prior work on deep learning-based character recognition, the models in this research were developed with purposefully generated datasets. Once the license plate region has been discovered, the character detection techniques and license plate identification rules are applied to the plate detection technique’s outputs.

Nikhil et al. [18] devised a de convolutional machine-learning YOLO technique to detect the Triple Riding. This program counts the number of people riding a bike and determines whether the vehicle is a rule-breaking vehicle or not. The junctions collect the data, which operate as a data-gathering centre. The picture of the vehicle categorized as a rule breach is saved with information such as the car manufacturer ID and the vehicle speed at that particular frame. The information is sent using the Gsm technology and the NodeMCU installed on the automobile. The transport bureau will double-check the vehicle number. The technology is being provided with a GSM module to withstand the lack of internet access or low internet access; otherwise, the data connected to the vehicle may be retrieved by the development boards put at the intersections, functioning as the primary portion of the public internetwork installed. This public internetwork acts as a channel for data sent from the car to the centralized computer. In NodeMCU, this is accomplished by utilizing dynamic network configuration. The system is significantly more practical, available, and stable thanks to Node MCU and the global network system. As a result, motorcyclists will be more likely to observe the regulations and less likely to drive irresponsibly.

Siddhartha et al. [19] suggested a deep learning-based solution for detecting and identifying Indian license plate numbers automatically. It used OpenCV to segment the chartered and constructed the recognition system by training a personally gathered automobile number plate dataset utilizing a Deep Convolutional Neural network. The algorithm was tested on over 500 photographs of various colours and qualities. In Number Plate Recognition (NPR) technology, an image of the vehicle plate number is captured using an image capturing device. The license number is interpreted to identify the vehicle owner’s details quickly. The primary objective of this research is to increase the accuracy of picture number plate identification. This method can detect a number plate even if it is obscured. The suggested method’s fundamental restriction is that it can only segment a single line of a license plate. It is unable to segment two lines of a number plate. Some Indian license plates have two lines; in this situation, the method will not operate since it cannot segment both two lines.

Aman et al. [20] suggested an intelligent traffic infraction detection method for the Indian traffic situation. In its sophisticated and intelligent form, visual computing will aid in identifying and pruning various actions/alerts following the classifications of categories for infractions. This study uses an innovative visual computing-based system to create a road traffic violation detection and warning system that is smarter, challenge-free and similarly aids human-based traffic management systems with a similar feel to a human being. The system model assists them in type-of-violation detection, identification, and real-time notifications based on interest-class detection and scenario. The system’s framework is based on various open-source tools, accounting for its low cost.

Baratsanjeevi et al. [21] created a prototype for detecting traffic signs and controlling violations. This prototype is structured to use a Deep Neural Network (CNN) to recognize traffic signs and then respond in real-time to deny the user’s control if he tries to break the laws in the traffic sign. If the CNN approach detects traffic indicators, the observed signals are encrypted into unique binary codes and logged in IoT cloud data. A NodeMCU collects the logged data and provides it to the Arduino UNO, operating the motors using the control signal given to IC L293D. The system’s primary goal is to improve automation, which will reduce human error while also increasing dependability, efficiency, and speed.

Raj et al. [22] proposed employing Artificial Intelligence and Deep Learning to recognize traffic signal violations. For traffic infraction detection, including signal jump, speed limit, and vehicle number, the suggested system was built utilizing YOLOV3 object detection. Furthermore, the system has been tuned for precision. Signal jump is determined by using the Region of Interest and the vehicle’s location in the length of frames. This approach has a vehicle count detection accuracy of 97.67 percent and a speed violation detection accuracy of 89.24 percent.

Heng et al. [23] developed a YOLO detector with a CRNN-12-based end-to-end license plate identification system. In this paper, the YOLO technique is used to identify license plates. The Convolutional Recurrent Neural Network (CRNN) is used to recognize them, resulting in state-of-the-art recognition rates. The detection performance of YOLOv2 and YOLOv3 was evaluated after they were trained for license plate detection. The CRNN-12 is a license plate identification system that includes a DCNN for extracting features and a 2-layer bidirectional Gated Recurrent Unit (GRU) decoding feature patterns. To jointly train DCNN and RNN, the Connectionist Temporal Classification (CTC) loss function is employed. This suggested method can do end-to-end detection without partitioning, and GRU can make greater use of relevant information in license plate photos, resulting in enhanced recognition accuracy.

Saifa et al. [24] offer a video-based parking infringement detection system in this paper. The proposed approach takes and analyses photos from a parking infraction area and determines whether or not there are any automobiles in the area. In this case, a camera is utilized as a detector, and movies are taken to determine the infraction. Vehicles in motion are also identified. The background subtraction approach is used in this operation. The suggested method is being tested at Prince of Songkla University’s Faculty of Engineering. This proposed approach helps detect automobiles that break parking zone rules.

Udaya et al. [25] propose a technique for detecting speed violations and identifying license plates using a GPU. The maximum speed for an automobile is assessed within the camera’s coverage range depending on the traffic density existing at the time, and vehicles that surpass the prescribed limit are marked as a violation. The suggested system’s main component is license plate recognition. Another issue that must be addressed is correctly removing the license plate with the best perspective to proceed with the identification procedure. For the goal mentioned above, this suggested technique used deep neural networks differently. Because these neural networks have many parameters, the GPU was used for processing to achieve stability in real-time. The speed violation and its perpetrator are captured using this unique segmentation-free license plate recognition approach that employs the object detection concept.

Bhat et al. [26] examined the identification of violations using a Genetic algorithm, which is used to optimize input supplied to the computer, which might be a set of records acquired from CCTV video. Wankhede et al. [27] concentrated on creating a behavioural detection model using a matching technique. Particle swarm optimization was used to create the same model (PSO). The module has been tested on several video clips of Nagpur traffic to detect traffic violations. Zhong et al. [28] developed AutoFuzz, a grammar-based fuzzing approach for detecting traffic violations directed by a restricted Neural Network (NN) evolutionary search over the API language to build scenarios that strive to identify unique traffic infractions.

Based on the literature review, we can infer that contemporary deep learning and image processing approaches have many benefits, even though improving the model’s accuracy is still a problematic portion of finding the speed violation in traffic. A new unique detection model was developed with the shortcomings of the prior works. The following part describes the most efficient newly proposed automatic speed violation detection system, its primary processes, and its working strategies.

3 Automatic detection system of speed violations in a traffic based on Pp Yolo

Rapid urbanization has led to widespread migration to cities, and it, including population growth, has increased the number of automobiles owned by people. Because of the large number of automobiles on the road, many people disregard traffic restrictions with human error, which has resulted in a slew of mishaps that have caused pain and misery in society. Traffic accidents happen for a variety of causes. While difficulties with roads and safety facilities cause some incidents, others are caused by drivers’ inability to follow regulations regarding pedestrians and recognize unsafe behaviours. Managing traffic violations has now become a time-consuming task. Even if there are several automated systems to reduce traffic violations, dealing with those scenarios is incredibly challenging. Although accidents can occur for various causes, excessive speeding on two or four wheels leads to severe mishaps. Managing this speed infraction is a significant issue. As a result, the solution to this problem is to develop a system connected to various factors, such as speed estimation and recognizing the vehicle’s license plate, to determine how frequently a driver violates traffic regulations. A unique Automatic Speed Violation Detection System is created in this research to reduce accident levels by identifying speed violations and the flow diagram of the proposed method is shown in Fig. 1. This proposed speed violation technique is separated into two working modules: objection detection and number plate recognition. For object detection, PP YOLO neural network is utilized in this work.

Fig. 1

Flow diagram of proposed automatic Speed Violations Detection.

3.1 Speed estimation to find violation

Vehicle speed is estimated by manipulating image pixels and determining the bounding box of a tracked vehicle. Vehicle IDs were used to label tracked cars in this proposed system. A-Frame Counter variable is also set to keep track of the number of frames to correlate to the presence of a vehicle within the ROI. Every vehicle’s displacement in two consecutive frames is estimated in pixels. The significant aspects should be combined into a calibration parameter to provide accurate displacement computation. As a result, a regression thresholding parameter P_Th is used, which reacts differently in different sections of the ROI. Lastly, the displacement average dp_Average As well as the number of frames Frame counter transfer to the speed measurement phase. The displacement estimate in pixels is shown in Equation (1). ${dp}_{x} = | {pos}_{x} - {pos}_{x - 1} | \times CC$ (1)

Where dp_x is the amount of displacement in frame x relative to the preceding frame, and CC is the calibration constant. Furthermore pos_x and pos_x-1, the bounding box centroids in frames x and x-1 are repeated as long as the vehicle is within the ROI. By departing the zone, the average of displacements is determined as stated in Equation (2): ${dp}_{Average} = \sum_{frame = x}^{n + x} {dp}_{x} / n$ (2)

In this stage, the vehicle’s speed is calculated using two crucial parameters: displacement and elapsed time. Each displacement vector represents the instantaneous speed in pixels during a specific period. The displacement is converted into a real-world scale, such as km/h. Two independent mapping functions must be developed for converting pixels to metres and frames to seconds. Because the ROI size is specified in pixels and the constant calibration CC is given, the pixel density is: $A_{Average} = {dp}_{Average} / {ROI}_{height}$ (3)

Where A_Average denotes the average area of each ROI pixel in metres. As a result, the displacement is computed as follows: $DP (m) = {dp}_{Average} \times A_{Average}$ (4)

However, to compute time in seconds, time-related factors such as Frame Counter and fps are required. As a result, the elapsed time is: $time (s) = Framecounter / fps$ (5)

Finally, Equation (6) computes the vehicle’s speed in km/h. $Speed (km / h) = (DP \times 3600) / (time \times 1000)$ (6)

The final estimated speed is, in reality, the average of the vehicle’s several instantaneous speeds within the ROI. Because vehicle movement inside the ROI is just a few metres, the variation between standard and rapid speeds is insignificant. Because different types of roads have varied speed limits, the final determining speed of vehicles must be compared to the legal speed limit. Consequently, when a speed infringement is detected, a photo of the vehicle, its speed, and duration of presence in ROI should be gathered as proof. We have established two types of speed restrictions in our, such as 20Km and 40Km. It is deemed a speed violation when the vehicle’s running speed exceeds the speed limit, chosen based on the location.

The object is detected, which is run above the speed limit, and an object detection algorithm termed PP YOLO is utilized in this work. Rather than swiping over an image numerous times, YOLO looks at it once and correctly detects everything in it.

3.2 Speed violation object detection

Numerous methods have been developed to detect the objects for a speed violations in traffic. From neural networks, YOLO (You Only Look Once) architecture, which is used to distinguish numerous objects in a picture, is now trendy. The primary difference between this architecture and others is that almost all systems implement neural networks to distinct parts of the view numerous times; in YOLO, the neural network is performed once to the entire image in one go. The deep learning-based neural network splits the picture into a grid and then predicts bounding boxes and each segment’s chance to contain the desired object. YOLO is also 1000 times faster than R-CNN and approximately 100 times quicker than Fast R-CNN. The speed and intricacy of the system are the primary reasons for employing YOLO. YOLO presents the detection issue as a regression problem and trains on features from a complete picture. Although YOLO V3 has many advantages compared to the other neural networks, it also has limitations in finding the objects accurately. As a result, a modified YOLO V3 technique known as PP YOLO is used in this research. It is the latest and most efficient object detecting approach. PP YOLO is used in this work to detect objects in image frames.

3.2.1 Working Principle of PP YOLO

PP YOLO is a deep learning system for object detection based on the YOLO4 architecture. This approach aims to make object detection easier in the creation, training, optimization, and deployment of all these models in a much more efficient and effective manner. Also, it includes numerous standard techniques to improve modularity and data augmentation techniques, loss function, and so on, which aids in lowering platform size and enabling high-performance implementation. The architecture of Paddle Detection is primarily composed of three categories:

➢Backbone: The convolution neural network used to produce features is located in this part. It does include a pre-trained categorization model. It is ResNet50-v3 in this instance.

➢Detection Neck: The Feature Pyramid Network (FPN) is built to construct a feature pyramid by merging and mixing the ConvNet outputs.

➢Detection Head: This portion determines the object’s prediction and bounding box.

Most notably, because of its quick object identification and processing speed, PP YOLO is suited for traffic violation detection. The modified YOLO V3 architecture is shown in Fig. 2.

Fig. 2

YOLOv3 network architecture and inject points for PP-YOLO [29].

The proposed automated speed violation detection system’s whole procedure is described below and the hyper-parameters used in the proposed method has been described in Table 1. The input acquisition or data gathering stage is the initial step in this proposed automated speed violation detection system. It is concerned with acquiring video data via CCTV services, followed by remote transmission to the computer on which the proposed application is run. This information contains the various vehicles and their speeds. The collected videos were sent into the PP YOLO’s input for object recognition. The fundamental operation in the PP YOLO algorithm is pre-processing, which extracts frames from videos. The PP YOLO backbone of CNN, which subsequently acquired the frames, is pre-processed to pool image pixels to create features at various granularities.

Table 1

Hyperparameter used in the proposed model. This work uses a learning rate of 10^-5

Type	Filters	Size	Output
Conv	32	3×3	256×256
Conv	64	3×3/2	128×128
Conv	32	1×1	–
Conv	64	3×3	–
Residual			128×128
Conv	128	3×3/2	64×64
Conv	64	1×1
Conv	128	3×3
Residual			64×64
Conv	256	3×3/2	32×32
Conv	128	1×1	–
Conv	256	3×3	–
Residual			32×32
Conv	256	3×3/2	16×16
Conv	256	1×1
Conv	256	3×3	–
Residual			16×16
Conv	1024	3×3/2	8×8
Conv	512	1×1
Conv	256	3×3
Residual			8×8

ResNet is a more widely used backbone architecture, with more frameworks tuned for its implementation and fewer parameters than Darknet53. PP YOLO uses the ResNet-50 network architecture, which comprises 50 convolution layers. ResNet-50 includes 16 residual bottleneck blocks with convolution sizes of 1×1, 3×3, and 1×1 and feature mappings (64, 128, 256, 512, 1024). ResNet introduced the concept of skip connections to assist activations in propagating across deeper layers avoiding gradient degradation. FPN (Feature Pyramid Network) is used in the ResNet50-V3 backbone architecture to extract the feature maps at various sizes. FPN is a popular choice for such a feature extractor addition that works by piling features at various sizes in a bottom-up approach. A feature extractor converts the model’s input into a specific feature representation, which assists in learning and detecting patterns connected with the desired items. This procedure is not affected by the backbone convolutional architectures. To get high-level features, the researchers first apply a convolutional feature extractor to the input picture.

Fig. 3

Bounding box prediction [30].

Each lateral link connects the feature maps from the bottom-up to the top-down pathways, resulting in distinct pyramid levels. In FPN, the preceding pyramid level is up-sampled by a factor of 2x before combining the feature maps with the same spatial size. This classification/regression network (the head) is then used at each stage of the pyramid, assisting in detecting objects of various sizes. Anchor boxes are preconfigured boxes that encompass an entire image and are used to identify several items of variable dimension or size. To detect many things in a scene, an object detection model employs the idea of anchor boxes. Each grid cell has B bounding boxes, including a confidence score. Every bounding box in the picture has the following attributes: width (B_w), height (B_h), Class (C), and bounding box centre (B_x, B_y). Using a single bounding box regression, PP YOLO predicts objects’ height, breadth, center, and class. The grid sensitivity made it easier for such a model to anticipate the exact location of the bounding box centre on the grid border.

•Each feature map layer uses the same set of default boxes centred at the relevant cell. On the other hand, Distinct layers employ separate sets of default boxes to tailor object detections at various resolutions. C class scores and four offsets compared to the original default bounding box form are determined for each bounding box. As a result, there are (number of classes+4 offsets) * bounding boxes * layer output size. The confidence for each detection is retrieved and compared to the confidence threshold. If the confidence level exceeds the minimum, the detection will be considered affirmative and legitimate, and processing will continue. If the class does not represent a vehicle, it is disregarded. Intersection Over Union (IOU) is an object detection phenomenon that defines how boxes overlap. PP YOLO uses IOU to create an output box that properly surrounds the items, implying that any vehicle travelling at less than the maximum speed gets excluded depending on the speed estimation condition. For the small object detection, we can increase the number of anchor boxes and decrease the threshold for IoU like the shown Fig. 2. We must gather information about the violator after locating the violated automobiles. As a result, the next part details the number plate identification in the violated vehicle.

3.3 Number plate recognition

The offenders’ license plate numbers will be detected, and a newly developed licence plate recognition is named open ALPR and the model of the OpenALPR shown in Fig. 4. The Open ALPR software is very accurate, analyzing pictures and video feeds to identify license plates and vehicles using cutting-edge machine learning and artificial intelligence.

Fig. 4

Automatic license plate recognition [31].

This article sends the object-predicted images from PP YOLO to the open ALPR. Once the open ALPR receives the image, it is processed, and the result in the text representation of any license plate of the vehicle is visibly displayed. Open ALPR is an open-source Automatic License Plate Recognition library built in C++ with C#, Java, Node.js, and Python interfaces. To recognize license plates, the library analyses images and video streams. The number plate of a vehicle that exceeds the maximum speed limit on the road will be identified by Open ALPR. The required information received from the Open ALPR is stored in a database which will be helpful to detect the details of the time and the person who violates the rule. Authorities can take action from these details quickly with less effort.

Consequently, the first module does object detection using the most accurate and fast PP YOLO neural network. In contrast, the second module performs license plate identification using the newly designed open ALPR. The traffic footage was converted to frames after being analyzed in PP YOLO. PP YOLO is taught utilizing vehicle speed estimation developed with OpenCV by manipulating image pixels and computing the bounding box of a tracked vehicle. When a vehicle passes at or over the limit speed, the PP YOLO recognizes it, and the vehicle’s licence plate is identified by open ALPR. Officials can take action against the violation with less time and effort if they access the open ALPR. The simulation result of the proposed model is explained in the next section with a detailed description.

4 Results and discussion

This section will describe the results of the proposed automatic speed violation detection system in traffic. Also, this section compares the proposed speed violation detection method with various existing violation detection techniques.

Software: Python

OS: Windows 7(64 bit)

Processor: Intel premium

RAM: 8GB

Highway Traffic Videos Dataset: The video was shot over two days from a fixed camera overlooking I-5 in Seattle, Washington. The video was manually categorized as light, medium, and heavy traffic, which correlate to free-flowing traffic, reduced-speed traffic, and halted or extremely slow-speed traffic. Every frame retrieved from the video is processed to object identification. The vehicles and their speeds were among the different items subjected to our prototype system’s training procedure. Because this PP YOLO algorithm accurately detects vehicles in each frame, it has been upgraded to identify the vehicle speed in a video. It recognizes the vehicles as well as their relative speeds within the frame. Both are performed by utilizing pixel calculation, the area of a border-box, and anchor points.

4.1 Speed Violation Detection

A few key concepts determine this speed: the variation of the object within the video frames provides us with speed, but it isn’t the actual speed, and because the object in the video doesn’t move linearly, the area of a boundary frame by frame provides us with the actual distance covered by the vehicle in the video. Also, the frame by frame detected vehicles are shown in Fig. 5. In this fig, the red line denotes the centre line where the vehicle should completely halt during the red traffic light. The green boxes are the bounding boxes created to detect vehicles by the PP YOLO algorithm with vehicle IDs.

Fig. 5

Frame by frame vehicle detection using boundary boxes.

After the object detection, the speed estimation was conducted using the distance formula. Two-speed limits applied for this research are 20Km and 40Km.

Case (1) Speed limit 20Km

When the speed limit is set to 20 km/h, cars that move more than 20 km/h are speeding. PP YOLO also provides the speed limits of the vehicles and their car IDs to detect whether the vehicle runs at over-speed or not. The detected Car IDs and their corresponding speeds are given in Fig. 6.

Fig. 6

Speed estimation when the speed limit is set to 20Km.

Figure 7 shows a graphical representation of a vehicle travelling at regular speed. This fig demonstrates that 22 cars were travelling at regular speeds when the maximum speed restriction was set at 20 kilometres. Vehicle IDs are used to represent vehicles in this work.

Fig. 7

Graphical representation of the vehicle passing the traffic at normal speed (<20Km).

The graphical representation of the vehicles running above the maximum speed limit is shown in Fig. 8. It shows that 37 vehicles ran above the maximum speed limit, indicating a speed violation.

Fig. 8

Graphical representation of the vehicle passes the traffic below maximum speed limit (>20Km).

Case (2) Speed limit 40Km

When the speed limit is set to 40Km, the vehicle’s speed is compared. A vehicle that passes the traffic above 40Km is considered an over-speeding vehicle. Figure 9 shows the over speeding vehicle’s ID and their speed ranges.

Fig. 9

Vehicle IDs and their corresponding speed limit when the maximum speed limit fixed to 40Km.

Figure 10 shows a graphical representation of vehicle IDs with their corresponding travelling speed when the maximum speed limit is set to 40Km. It demonstrates that 54 cars were travelling at regular speeds when the maximum speed restriction was 40 kilometres.

Fig. 10

Graphical representation of the vehicles which pass with normal speed (<40Km).

The graphical representation of the vehicles running exceeding the speed limit is shown in Fig. 11. It shows that only five vehicles were travelling above the speed limit when the maximum speed limit was fixed to 40Km. The open ALPR recognized those vehicles’ number plates.

Fig. 11

Graphical representation of the vehicle that passes the traffic with over speed when the speed limit is fixed at 40Km.

4.2 Number Plate Recognition

The vehicle’s images that travel above maximum speed are given as an input to the open ALPR shown in fig. These license plate numbers will be blurry, and it will be hard to recognize the characters. Also, the pixel size won’t be enough to process. For that reason, the images are converted into greyscale images, which helps recognize the vehicles’ numbers. Then, the converted greyscale image was filtered to increase visibility. The predicted violators’ vehicle is shown in Fig. 12(a).

This work transformed these resized colour images into greyscale images, and it simplifies the technique and lowers the computing requirements. Figure 12(b) shows the greyscale image of the input. The noise in the greyscale images was removed using the filters. In this research, the Gaussian Blur filter is used to remove the noise in the images Fig. 12(c).

Fig. 12

(a). Input image for Open ALPR, (b). Grayscale image of the vehicle and (c). Filtered greyscale image.

Whereas the whole frame plus automobile coordinates are being utilized to train the vehicle recognition neural network, the vehicle patches (with a margin) and its LP coordinates are utilized for training the LP detection circuit. Figure 13 shows the reduced margin size. We calculated the needed margin in the validation data once all LPs were totally inside the bounding boxes of the cars detected by the vehicle recognition. It is performed to alleviate losing LPs when the vehicles aren’t discovered well.

Fig. 13

Reduced margin size of the vehicle.

We adopt a threshold of 0 for LP detection since there may be occasions where the LP is recognized with extremely low confidence (e.g., 0.1). In situations where more than one LP is identified, we maintain just the detection with the highest confidence because each vehicle only has one LP. The proposed approach detects all license plate digits without any blur. The detected licence plate number is shown in Fig. 14, which will be helpful for the authorities to take action against the violators.

Fig. 14

Detected license plate of the vehicle.

4.3 Performance Analysis

This section analyzes the various performance parameters of the proposed model.

The training and testing loss of the proposed model is depicted in Fig. 15. It demonstrates that the model provides 0.1 percent loss in both the training and testing phases in the eighth epochs, and it indicates that the suggested model provides less loss as the number of epochs grows.

Fig. 15

Training and Testing Loss of the Proposed Model.

Figure 16 depicts the proposed model’s training and testing accuracy, demonstrating that the suggested model achieves 99 percent accuracy in both the training and testing phases at the 8th epochs. Based on the graph, we may deduce that as the number of epochs grows, so does the accuracy.

Fig. 16

Training and Testing Accuracy of the proposed Model.

Figure 17 depicts the performance analysis of the proposed model, which assessed Accuracy, Recall, F1 score, and Precision. Because of the successful identification of objects utilizing the PP-YOLO architecture, which is based on YOLO v3, the proposed model achieves 96.5 percent accuracy, 97.2 percent precision, Recall, and F1 score.

Fig. 17

Performance analysis of the proposed model.

4.4 Comparative analysis for speed violation detection

The proposed automatic speed violation detection method’s accuracy is compared to various current techniques such as YOLO V3, image processing, robotic models, etc. The detection accuracy of the YOLO V3 [22] is 89.24 percent, and the frame counting [33] technique provides an accuracy rate of 93.3 percent. The nave Bayes model [32] provides an accuracy rate of 88.73 percent, the robotic model [34] has an accuracy rate of 85.75 percent, the image processing technique [35] provides an accuracy rate of 91.7 percent; the GMM [36] provides an accuracy rate of 97.52 percent, the IoT technology [37] gives 80%, Blob tracking has an accuracy of 94.8%. MNN [38] and CCTV camera [39] provided an accuracy of 95.7% and 97.01% Fig. 18 depicts a graphical depiction of the different approaches’ performance in detecting speed violations. It demonstrates that the proposed method has an accuracy rate is 98.8 percent in speed violation detection.

Fig. 18

Comparative analysis of speed violation detection.

As a result, Fig. 18 demonstrates that our proposed automatic speed violation detection algorithm has a higher accuracy rate than other violation detection algorithms due to a bottom-up fashion. It makes the network more robust and efficient, and also the proposed YOLO-based model uses boundary boxes to detect the objects in the given frames accurately.

4.5 Comparative analysis of license plate detection

Researchers have developed many methods for detecting license plates and identifying offenders. This work performs a performance analysis to demonstrate the open ALPR system’s licence plate detection accuracy.

Figure 19 depicts a graphical depiction of the current licence plate identification techniques and their accuracy rates. This graph demonstrates that the detection accuracies of the Single Stage Detector [17], CRNN [23], and YOLO [40] approaches are 73.3 percent, 98.86 percent, and 78.33 percent, respectively. The sliding window [41] and CNN [42] techniques have 78 and 98.1 percent, respectively. The accuracy rate is enhanced to 99.3 percent by employing open ALPR, demonstrating that our suggested speed violation detection approach is more accurate and efficient.

Fig. 19

Comparative analysis of licence plate detection.

As a result, the proposed YOLO-based traffic violation detection model better detects the objects from the traffic video to find out the speed violation. The violated vehicle numbers have been recognized using OpenALPR very clearly, which helps to take necessary action to prevent speed violations and accidents.

5 Conclusion

A traffic violation is growing increasingly severe in several countries, and it is challenging for the police service to punish the criminal due to the overwhelming traffic condition. One of the most prevalent and serious traffic offences is overspeeding. This paper proposed an autonomous speed violation detection system based on deep learning to manage this speeding. The recently created PP YOLO neural network is used in this proposed approach for speed violation detection. The open APLR (Automatic License Plate Recognition) identifies the licence plate numbers of vehicles that exceed the speed limit. Once the licence plate number data are provided to the regulatory authority, the offender may be penalized for excessive speeding. The primary goal of this work is to eliminate the time and labour required by the present methods. Finally, the proposed model gives 96.5% accuracy for detecting and recognizing the speed violated vehicles in the traffic.

References

Agarwal

, Jain

and Karabasoglu

, Smart vehicle monitoring and assistance using cloud computing in vehicular ad hoc networks, International Journal of Transportation Science and Technology (2018).

Global Status Report on Road Safety 2018, World Health Organization: Geneva, Switzerland (2018).

https://www.indiatoday.in/diu/story/two-wheeler-death-road-accidents-helmets-states-india-1602794-2019-09-24.

A.C.P.

, Quiros

A.R.F.

, Bedruz

R.A.

, Abad

, Bandala

, Sybingco

and Dadios

E.P.

, Automated traffic violation apprehension system using genetic algorithm and artificial neural network, In 2016 IEEE Region 10 Conference (TENCON), IEEE (2016, November), 2094–2099.

Poddar

, Giridhar

M.K.

, Prabhu

A.S.

and Umadevi

, Automated traffic monitoring system using computer vision, In 2016 International Conference on ICT in Business Industry & Government (ICTBIG), IEEE (2016), 1–5.

A.C.P.

, Bedruz

R.A.

, Quiros

A.R.

, Bandala

and Dadios

E.P.

, Machine vision for traffic violation detection system through the genetic algorithm, In 2015 International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM), IEEE (2015, December), 1–7.

Firouzabadi

M.H.H.

and Azizi

, Automatic identification of personal automobiles plates of Iran using genetic algorithm, Computer Engineering and Intelligent Systems (2019).

Dey

and Praveen

, moving object detection using genetic algorithm for traffic surveillance, 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT), Chennai (2016).

Manocha

, et al., Korean Traffic Sign Detection Using Deep Learning, International SoC Design Conference (ISOCC), Korea (2018).

10.

Hirawan

, Hadiana

and Abdurakhim

, “the prototype of traffic violation detection system based on internet of things, IOP Conf Series: Materials Science and Engineering (2019).

11.

Wong

S.F.

, Mak

H.C.

, Ku

C.H.

and Ho

W.I.

, Developing advanced traffic violation detection system with RFID technology for smart city, In 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), IEEE (2017), 334–338.

12.

Ms Sushmitha

, Subhasree

, Thushara

S.S.

and Mrs Rajalakshmi

, Traffic Rules Violation Detection System (2020).

13.

, Abdel-Aty

, Yuan

, Cheng

and Lu

, Analyzing traffic violation behaviour at urban intersections: A Spatio-temporal kernel density estimation approach using automated enforcement system data, Accident Analysis & Prevention 141 (2020), 105509.

14.

Umar

B.U.

, Olaniyi

O.M.

, Agajo

and Isah

O.R.

, Traffic Violation Detection System Using Image Processing, Computer Engineering and Applications Journal 10(2) (2021), 81–92.

15.

Bhat

T.A.

Mahima Rao

and Deepthi Pai

, Traffic Violation Detection in India Using Genetic Algorithm, Global Transitions Proceedings (2021).

16.

Tourani

, Shahbahrami

, Akoushideh

, Khazaee

and Suen

C.Y.

, “Motion-based vehicle speed measurement for intelligent transportation systems, International Journal of Image, Graphics and Signal Processing 11(4) (2019), 42.

17.

Elihos

, Balci

, Alkan

and Artan

, Deep learning based segmentation free license plate recognition using roadway surveillance camera images (2019). arXiv preprint arXiv:1912.02441.

18.

Mallela

N.C.

, Volety

and Nadesh

R.K.

, Detection of the triple riding and speed violation on two-wheelers using deep learning algorithms, Multimedia Tools and Applications 80(6) (2021), 8175–8187.

19.

Roy

, Automatics Number Plate Recognition Using Convolution Neural Network (2020).

20.

Kumar

, Kundu

, Kumar

, Tiwari

U.K.

and Kalra

, S-TVDS: Smart Traffic Violation Detection System for Indian Traffic Scenario, International Journal of Innovative Technology and Exploring Engineering (IJITEE) 8(4S3) (2019), 6–10.

21.

Baratsanjeevi

, Deepakkrishna

, Harrine

M.P.

, Sharan

and Prabhu

, IoT based Traffic Sign Detection and Violation Control, In 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), IEEE (2020, July), 333–339.

22.

Anand

S.R.

, Kilari

and Kumar

D.U.S.R.

, Traffic signal violation detection using artificial intelligence and deep learning, International Journal of Advanced Research in Engineering and Technology 12(2) (2021), 207–217.

23.

Sun

, Fu

, Abdussalam

, Huang

, Sun

and Wang

, License plate detection and recognition based on the YOLO detector and CRNN-12, In International Conference On Signal and Information Processing, Networking and Computer, Springer, Singapore (2018, May), 66–74.

24.

Khantasak

, Jindapetch

, Hoyingcharoen

, Chetpattananondh

, Ikura

and Chumpol

, Parking violation detection system based on video processing, In 2018 IEEE 5th International Conference on Smart Instrumentation, Measurement and Application (ICSIMA), IEEE (2018, November), 1–5.

25.

Dampage

, Hasantha

K.K.N.

, Gimhana

H.A.D.S.

, Bandara

H.M.T.Y.N.

and Maddumage

S.V.

, Gimhana, H.M.T.Y.N. Bandara and S.V.Maddumage, Novel and Real-Time GPU Accelerated Speed Violation Detection and License Plate Identification System (2021).

26.

Bhat

T.A.

, Mahima Rao

and Deepthi Pai

, Traffic violation detection in India using genetic algorithm, Global Transitions Proceedings 2(2) (2021), 309–314.

27.

Wankhede

S.S.

and Bajaj

, Traffic Violation Detection Model Using Soft Computing Tools, In 2021 6th International Conference for Convergence in Technology (I2CT), IEEE (2021), 1–5.

28.

Zhong

, Kaiser

and Ray

, Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles, arXiv preprint arXiv:2109.06126 (2021).

29.

Long

, Deng

, Wang

, Zhang

, Dang

, Gao

, Shen

, Ren

, Han

, Ding

and Wen

, PP-YOLO: An effective and efficient implementation of object detector. arXiv preprint arXiv:2007.12099 (2020).

30.

Franklin

R.J.

, Traffic Signal Violation Detection using Artificial Intelligence and Deep Learning, In 2020 5th International Conference on Communication and Electronics Systems (ICCES), IEEE (2020, June), 839–844.

31.

https://www.pyimagesearch.com/2020/09/21/opencv-automatic-license-number-plate-recognition-anpr-with-python/

32.

Kuşkapan

, Çodur

M.Y.

and Atalay

, Speed violation analysis of heavy vehicles on highways using spatial analysis and machine learning algorithms, Accident Analysis & Prevention 155 (2021), 106098.

33.

Kumar

and Kushwaha

D.S.

, Traffic surveillance and speed limit violation detection system, Journal of Intelligent & Fuzzy Systems 32(5) (2017), 3761–3773.

34.

Bedruz

R.A.

, Uy

A.C.P.

, Quiros

A.R.

, Billones

R.K.

, Sybingco

, Bandala

and Dadios

E.P.

, A Robotic Model Approach of an Automated Traffic Violation Detection System with Apprehension, In 2018 IEEE 10th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM), IEEE (2018), 1–4.

35.

Vismaya

and Malar

R.K.J.

, Novel Vision-Based Traffic Violation Detection. In 2019 International Journal of Information and Computing Science 6(4) (April 2019), 32–34.

36.

Setiyono

, Dr Sulistyaningrum , Fajriyah

and Wicaksono

D.W.

, Vehicle speed detection based on gaussian mixture model using sequential images, In Journal of Physics: Conference Series, IOP Publishing 890(1) (2017, September), 012144.

37.

Khan

M.A.

and Khan

S.F.

, IOT based framework for Vehicle Over-speed detection, In 2018 1st International Conference on Computer Applications & Information Security (ICCAIS), IEEE (2018, April), 1–4.

38.

Choy

J.L.C.

, Wu

, Long

and Lin

Y.B.

, Ubiquitous and low power vehicles speed monitoring for intelligent transport systems, IEEE Sensors Journal 20(11) (2020), 5656–5665.

39.

Kurniawan

, Ramadlan

and Yuniarno

E.M.

, Speed monitoring for multiple vehicles using closed-circuit television (CCTV) camera, In 2018 International Conference on Computer Engineering, Network and Intelligent Multimedia (CENIM), IEEE (2018, November), 88–93.

40.

Laroca

, Severo

, Zanlorensi

L.A.

, Oliveira

L.S.

, Gonçalves

G.R.

, Schwartz

W.R.

and Menotti

, a robust realtime automatic license plate recognition based on theYOLO detector, In 2018 International Joint Conference on Neural Networks (IJCNN), IEEE (2018, July), 1–10.

41.

Chen

R.C.

, Automatic License Plate Recognition via sliding-window Darknet-YOLO deep learning, Image and Vision Computing 87 (2019), 47–56.

42.

Pustokhina

I.V.

, Pustokhin

D.A.

, Rodrigues

J.J.

, Gupta

, Khanna

, Shankar

, Seo

and Joshi

G.P.

, Automatic vehicle license plate recognition using optimal K-means with convolutional neural network for intelligent transportation systems, IEEE Access 8 (2020), 92907–92917.