Design of autonomous driving system based on IoT and visual cognitive technology

Abstract

The traditional automatic driving system has the problems of low perception accuracy and low driving intelligence. To solve this problem, the research proposes an autonomous driving system based on the Internet of Things and visual cognition. Based on the traditional system, a symmetrical visual cognition sensor is added and the internal module of the sensor is designed. Then, the road information is obtained by combining visual cognition technology, and the driving route is planned by the sparrow optimization algorithm. In the process of automatic driving, dynamic sensing optimization is realized through dynamic detection and path control navigation. The experimental results showed that the highest automatic driving perception accuracy obtained by the proposed automatic driving system was 100%, and the perception time was within 7.62 seconds. Compared with other automatic driving methods, it had higher perceptual accuracy and faster response speed. This shows that the proposed system can improve the comfort and safety of autonomous driving, providing a positive research path for autonomous driving technology in the automotive industry.

Keywords

visual cognitive technology autonomous driving IoT sensors path planning

Introduction

The automobile is one of the most frequently used means of transportation in people’s daily lives. According to the purpose of the automobile, an automobile can be divided into truck, off-road vehicle, towing vehicle, bus, car, etc. The advent of science and technology has precipitated the proliferation of autopilot sensing systems in a multitude of automobiles, thereby rendering them increasingly intelligent.¹ In the design of an autopilot sensing system, the overall design of a multi-channel display and control system constitutes the core element. This is because the system incorporates complex ergonomic interface elements, and the display and control system must present a category of complex information.² In the context of the ongoing innovation and development of embedded systems, the information that must be processed in its entirety for multi-channel display and automobile control is becoming increasingly diversified. Drivers should identify more information while driving automatically, which will bring certain safety hazards to the driving of automobiles.³ Consequently, the development of automotive driving perception systems that enhance human-computer interaction while ensuring vehicular safety has emerged as a prominent research area in the domain of vehicle design.

When a person sees something and accepts it, integrating it into their brain to understand it, the cognitive component of visual perception occurs. Concentration, acquisition, vision, remembering, and categorization are five critical abilities that should be trained to improve visual perception. Executive functioning, data processing, sensory perception, and remembrance are just a few of the cognitive abilities needed for driving.^4,5 Such cognitive abilities typically decline with age, rendering driving increasingly challenging and hazardous over time. A cognitive driver assessment determines the risk level in both healthier and much more challenged people with various illnesses, as well as the competence of the driving behavior. The ability to interpret visual stimuli, perceive spatial relations among objects and envisage various scenarios or representations is generally referred to as visual spatial processing. Road detection and as object recognition are typically part of the visual system used in autonomous vehicle environment perception. When driving in various weather conditions, visual cognitive awareness is mainly used to identify obstacles and the environment. The cameras, radar, and Lidar are the three main detectors used in automated driving.^6,7

At present, many scholars around the world have put forward some methods for autopilot perception, which have been widely used. Li et al. proposed an automatic driving motion target detection system based on YOLOv3.⁸ Based on YOLOv3, a more reasonable loss function was designed, and a K-means clustering algorithm was used to cluster the target frame in the data set. To achieve accurate and documented autonomous driving perception, Yao et al. proposed a perception method that integrates radar and camera sensors. Based on multi-sensor fusion, this method introduced object detection and semantic segmentation algorithms to further improve automatic driving perception accuracy. This method enabled the system to discern the environmental context of automated driving, with a detection accuracy rate of 92.64%.⁹ Li et al. proposed an innovative collaborative perception framework MKD-Cooper to solve the problem of three-dimensional object detection in automatic driving systems. This framework realized object detection in the case of occlusion by means of multi-teacher knowledge distillation. The results showed that the detection accuracy of this method reached more than 90%.¹⁰ Wang et al. proposed a new anchor-free detection network to assist autonomous vehicles to achieve high-precision and high-efficiency detection of the surrounding environment. Feature pyramid and variable convolutional structure were introduced to optimize the network model. Experimental results showed that the accuracy of this method reached 55.6% and the frame rate reached 30 frames/s.¹¹

Large-volume structures in big vehicles commonly result in rollover incidents. To address this, a semi-active adaptable suspension system based on the AdaBoost algorithm¹² was created to significantly increase the vehicle’s rollover safety in risky driving situations. Additionally, the Lyapunov stability concept was used to create the model reference control algorithms and develop the adaptive controller. Vehicle lateral dynamics evaluated simulations in the time domain and primarily concentrated on one or more particular working practices to assess the effectiveness of the steered control scheme, but they were unable to evaluate the controller’s efficiency in its entirety. The effectiveness of three alternative automatic steering methods could be logically assessed from the phase plane when integrated with the bifurcation theory. Using this theory,¹³ the four-wheel steering control game’s safe driving zone was greatly extended, and the direct yaw-moment control system could achieve the best stability.

The primary structure of a chassis synchronized control strategy is the recognition of driving conditions, such as the vehicle’s operating intentions, crucial vehicle stages, and road stickiness conditions, as well as integrated control of X-by-wire chassis subsystems. In this work,¹⁴ various strategies for identifying driving conditions during steering and braking operations were given. A modular three-layer trajectory tracking approach was presented in this research¹⁵ to enable real-time obstacle detection in challenging driving situations. This was mostly attributed to the creation of a grouping of collision-free trajectories depending on path re-planning and quickness. The upper-level controller must generate a standard quadratic nonlinear path based on sequential minimum programming, assuming slight fluctuations in the speed and acceleration of adjacent vehicles. An integrated vehicle-following control method for FWID with Vehicle-to-Everything communication capacity was suggested in this study¹⁶ to consider non-ideal communication characteristics such as time-varying latency and packet failures. To address the issue of V2X information loss, a packet dropout compensation was developed. Then, to provide vehicle-following management, a lateral trajectory tracking regulator was combined with a longitudinal controller that had a delay compensator.

In terms of improving the perception accuracy and intelligence of the auto drive system, the Internet of Things (IoT) technology enables vehicles to acquire and process the data of the surrounding environment in real-time. Visual cognitive technology further enhances vehicles’ understanding and processing capabilities of road information. By combining these two technologies, a more intelligent and efficient autonomous driving system can be built. To improve the perception ability of autonomous vehicles to complex scenes and improve the safety and reliability of autonomous driving, an autonomous driving perception system based on visual cognition is designed. The system is designed for hardware through symmetrical visual cognitive sensors, circuit optimization transformation modules, memory expansion modules, and human-computer interaction unit modules. It collects road information through visual cognitive technology and plans autonomous driving routes based on the information collection results. Compared with the existing research, the innovation and contribution of this research is that it not only improves the perception accuracy and intelligence of the automatic driving system but also shortens the time spent on perception. By introducing symmetrical visual cognitive sensors, the system can obtain more comprehensive road information and improve the response speed to environmental changes. Concurrently, the implementation of the sparrow optimization algorithm enhances the efficiency and precision of route planning while effectively mitigating potential safety hazards. Furthermore, the research endeavors to enhance the human-computer interaction experience by optimizing the hardware design and software interface. This objective is pursued to enable the driver to more easily control the automatic driving system.

Hardware design of autopilot awareness system

There are some problems in traditional autopilot sensing system, such as small storage capacity and unreasonable circuit connection.^17,18 Therefore, based on the traditional sensing systems, the hardware system of autopilot sensing is optimized from many aspects such as memory devices and circuits. The optimization framework for the autopilot dynamic awareness hardware system is shown in Figure 1.

Figure 1.

Frame diagram of autopilot aware hardware optimization system.

As shown in Figure 1, this study adds a symmetrical visual cognitive sensor to the traditional hardware system and makes adjustments to the circuit. The designed autonomous driving perception hardware system includes visual cognitive sensors, circuit optimization and conversion models, memory expansion modules, and human-computer interaction unit modules.

Addition of symmetrical visual cognitive sensor

Based on the principle of symmetric visual cognitive imaging, a visual cognitive sensor based on a symmetric visual cognitive camera is added to the traditional perception hardware system. Symmetrical visual perception sensors are installed at the left and right ends of the front end of the autopilot to collect road information from two angles, thereby improving the autopilot’s perception accuracy. Therefore, the positions of two symmetrical binocular sensors need to maintain a certain distance.¹⁹ Symmetrical visual cognitive sensor is mainly divided into two parts: camera equipment and information sensing equipment. The camera consists of a lens, film, viewfinder, focusing device, aperture, and other equipment. Since the human eye has the auto zoom function, the auto zoom lens is selected in the lens. The camera has more than 9 million pixels, and the size of Charge-Coupled Device (CCD) sensitive device is 10.7 times that of the camera. The camera is equipped with two aspheric optical zoom lenses. A light-sensitive embedded device known as a CCD captures an image by transforming light into an electron. The image components are divided into pixels by a CCD sensor. Each pixel is converted into charge, and its intensity depends on the amount of light it can collect. To provide superior resolution than earlier technology, CCDs are utilized in camcorders. Applications that necessitate accuracy and a high level of sensitivity are still using CCD sensors. CCDs are silicon-based sensors made up of an epitaxial layer that has been formed on top of a silicon substrate. The silicon substrate is engraved with an embedded system to create a pattern of pixels that track the number of incident photons and translate them into emitted electrons.^20,21

Circuit optimization and transformation module

The symmetrical visual cognitive sensor designed above mainly consists of a reset circuit, a real-time clock circuit, a status acquisition interface circuit, a terminal display interface circuit, a UART circuit, and a JTAG circuit. By reconnecting the circuit, the acquisition interface circuit, the terminal display interface circuit, and the power supply circuit, some circuits in the symmetrical visual cognitive sensor are optimized.^22,23 During the implementation process, the reset circuit must ensure that only the circuit and the operating state are reset, without affecting the microprocessor and storage devices in the system.²⁴ The connection mode of the modified reset circuit is shown in Figure 2.

Figure 2.

Schematic diagram of reset circuit.

To enhance the output signal of the circuit, the study adds an acquisition signal amplification circuit on the basis of the traditional acquisition interface circuit. It incorporates the Lm358 dual operational amplifier module into the pulse signal acquisition circuit. The specific circuit connection is shown in Figure 3.

Figure 3.

Circuit diagram of pulse signal acquisition interface.

The acquisition interface is connected according to the circuit diagram shown in Figure 3. After the acquisition signal is input, it is filtered and amplified, and finally rectified to output the acquisition signal. To improve the display resolution, the adjustment of the terminal display interface circuit is studied to increase the number of pins of the display interface to ensure the output form of the sub-circuit.^25,26 The interface pin connection settings are detailed in Table 1.

Table 1.

Interface pin signal description.

The serial number	The name of the pin	Pin type	Signal description	Level of output
1	RTK-LED	Output	RTK lamp	Power for low level and light for high level
2	POWER-OFF	In	The power switch	High power usually cut off the power supply and low level for the suspension of the power supply
3	VCC-Input DC	In	1pps seconds	3.3 V LV TTL level
4	Event	In	External events	High level trigger
5	COMC-Rx	In	DC power input	LVTTL level
6	RESET-RTS	Output	RESET-RTS interface	LVTTL level
7	COMA-RTA	Output	COMA-RTA interface	RS232 level
8	USB-D	Input/output	USB data port	—
9	GND	GND	To	—

The power supply circuit in the autonomous driving dynamic perception hardware system needs to provide stable power support for each module in the system, ensuring stable voltage and enabling each module to work normally at rated voltage.²⁷ To improve the conversion efficiency of the power supply circuit, a direct current to direct current (DC-DC) switching regulator is added to the circuit design.

Memory expansion module

Due to the addition of symmetrical visual cognitive sensors, a large number of analysis programs and sensing data are generated during operation.^28,29 All sensing data (including historical search data) will be stored in the system memory. As the number of data increases, the memory’s capacity gradually increases, resulting in the reduction of the remaining idle capacity, which indirectly affects the perceived rate.³⁰ Therefore, it is necessary to expand the memory. In addition to traditional RAM and ROM, external memory needs to be connected to increase the memory capacity of the system from 5 GB to 20 GB.

Human-computer interaction unit module

To get a better optimization effect of the perception system, a human-computer interaction function is added to the system, that is, task instructions are issued to the perception system through voice input, and the system can also output the perception task state and running state in real-time by voice. Therefore, by installing a voice input device and voice playback device in the hardware of the automatic driving perception system, and realizing the rotation, translation, and scaling of driving road image through a human-computer interaction unit, the road condition information can be observed more intuitively. It is essential to develop an exceptionally active, approachable, and alluring interface since human-computer interaction is point-to-point communication between the system and the driver. Drivers in autonomous vehicles sometimes become preoccupied with other activities because of control systems, but HMI must help drivers become more situationally aware.³¹ These components have several features that can be used in the following situations.

The system’s operations should be transparent to the user. Users must be informed of current circumstances when necessary; nevertheless, the driver is always kept informed of current scenarios.

The user experience should be improved and made more comfortable through the design. This becomes difficult to constantly monitor the screen on long flights. Therefore, voice instructions that may quickly grab users’ attention during difficult situations should be incorporated with the display system.

Reduce memory demands on the user. Given the integration of audio signals within the architectural framework, the driver is relieved of the burden of discerning the purpose of the screen’s intermittent blinking elements.

By displaying a warning message that may appear, errors can be avoided.

Keep the driver informed of any system failures throughout lengthy, busy, and monotonous trips with voice activation and clever graphical representations.

The user should not find the error message unexpected.

Software optimization of automatic driving perception system

Based on the completion of hardware optimization of the automatic driving perception system, the software system of automatic driving dynamic perception is optimized to realize the accurate positioning and planning of the perception route and realize a more intelligent perception function.^32–34 The functional framework of the software optimization system is shown in Figure 4.

Figure 4.

Framework of automatic driving dynamic perception software optimization system.

As shown in Figure 4, the components of the autonomous driving framework include perception, planning, decision-making, and management. The perception subsystem utilizes the data collected by visual perception sensors. This system consists of multiple key modules, including object recognition, monitoring, simultaneous localization, and map construction. The object recognition function identifies the positions and classifications of all relevant items from data such as images or photogrammetry. Path planning achieves intelligent obstacle avoidance through intelligent algorithms to prevent collisions between nearby vehicles and pedestrians. Object detection is a key method for locating and identifying objects by analyzing visual images and point cloud data.

Symmetrical binocular vision technology for obtaining road information

The symmetrical binocular vision sensor is used to establish the road model and collect the road information. The sensing information collection and processing flow is shown in Figure 5.

Figure 5.

Flow chart of perception information collection and processing.

As shown in Figure 5, the study first initializes and sets the serial port, and then preprocesses the information. The processed data are read. During the reading process, it is necessary to determine whether the data is empty. If the data are empty, it needs to be preprocessed again, otherwise, the next step will be executed. The data are passed to the processing module for partition processing and identification of valid information.

Dynamic calibration of visual cognitive parameters

Using the imaging principle of the visual cognitive acquisition device, the parameters of the driving road information are calibrated, and the stereo data acquisition results are obtained. The stereo-imaging principle of visual cognitive sensors is shown in Figure 6. Stereo vision is the way of transforming numerous 2D perspectives of a scene into 3D information. A computer vision system called the stereo vision system uses stereoscopic range methods to determine distance. To create the perception of depth, this solution incorporates multiple lenses into one, and it uses the objects’ dispersion to precisely estimate the distance. Stereoscopic vision is a well-known method that retrieves depth maps from vision using horizontally and vertically offset camera modules, obtaining two simultaneous perspectives of the same image, just like a person’s own eyes. The main concept is to capture a scene from two distinct angles and then use the contrast to show the position, relationship, and organization of the scene’s objects. Depth is created by the variation in implications for the design across two images. When the object is in the two sensors’ overlapping fields of vision, their distances are calculated.

Figure 6.

Schematic diagram of stereoscopic imaging of visual cognition. (a) Left camera imaging and (b) right camera imaging.

Figure 6 shows the situation when visual cognition is to observe the same scene with left and right camera devices. The point to be measured is P and the imaging point is $(x_{n}, y_{n})$ . In the ideal case, the optical axes of the two cameras are parallel. If the distance between the imaging point P in the left and right visual image planes and the coordinate origin is $x_{1}$ and $x_{2}$ , the position difference of point P in the imaging results of the two cameras is the visual difference, which is recorded as $| x_{1} - x_{2} |$ . When the geometric position of the camera is fixed, the visual difference is only related to the z value in Figure 6, and has nothing to do with the distance between P point and the optical axis of the camera, so the camera calibration result will not produce error due to the vehicle driving.³⁵

Building road information model

Autonomous driving perception needs to strictly abide by traffic rules in the process of driving, so it is necessary to build a road information model. The model needs to include road boundary line, road sign line, traffic signal, and road information. The specific layer structure in the model is shown in Table 2.

Table 2.

Layer structure of road information model.

Road model element	Primitive objects	Layer
Road	CMapRoundEdge	Road boundary layer
Road	CMapRoundArea	Road area layer
Flower bed	CMapParterre	Flower bed layer
Bay bus station	CMapBusStop	Bay bus stop layer
Direction sign	CMapDirectional	Directions flag layer
Indication sign	CMapGuideSign	Indicator mark layer
The traffic light	CMapTraffic	Traffic light layer
Street lamp	CMapStreetlamp	Street light layer
Camera	CMapCamera	Camera layer

As shown in Table 2, each layer of elements constructs a complete road information model through accurate spatial positioning and attribute information to ensure that the auto drive system can accurately identify and respond to various road information and improve driving safety. Through this multi-level and refined model construction, autonomous vehicles can not only accurately perceive the surrounding environment but also make intelligent decisions in real-time, effectively avoiding potential risks and ensuring driving safety. The layers work in coordination with each other. The data are updated in real-time to ensure that the model dynamically adapts to complex road conditions and enhances the stability and reliability of the autonomous driving system.³⁶

Automatic driving route planning

Location of perception points

Automatic driving route planning first needs to input the starting point and end point of the perceived route. Generally, the starting point of perception is the current position. With the help of visual cognitive sensor and positioning system, the current position is expressed in the form of longitude and latitude. Since there is no spatial displacement in the driving process, there is no need to set spatial coordinates. The coordinates of the perception starting point are defined as $(x_{0}, y_{0})$ , and then the location information of the end point is input to be reached in the road information model.³⁷ According to the input name, the corresponding target location in the road model needs to be found, and the terminal information in the form of coordinates needs to be output. The output result is $(x_{n}, y_{n})$ . In addition, in the process of automatic driving, it is necessary to locate the driving position in real-time. When the positioning result deviates from the planned route seriously, it is necessary to adjust the sensing route in time, or re-plan the sensing route from the current position.

Generating the optimal route

The optimal perception route of autopilot has different definitions for different users. Therefore, in the process of perception route, it is divided into two modes, namely, the shortest form path and the least time-consuming path. According to the positioning results of the starting point and the endpoint, an evaluation of the distance between the top point and the target terminal is provided, so that the search algorithm can give priority to the nearest target vertex. The search algorithm adopted in this study is the Sparrow Search Algorithm (SSA), which simulates the sparrow’s foraging behavior and seeks the optimal solution by iteratively updating the search location. The SSA is an optimization algorithm based on swarm intelligence. By simulating the foraging and evasion behaviors of sparrows, it effectively balances global search and local search, improving the efficiency and accuracy of path planning. The research first initializes the sparrow population, sets foraging and evasion parameters, iteratively updates the position, evaluates the quality of the path, and finally outputs the optimal path. Compared with the A* algorithm, SSA avoids the local optimal trap. Compared with the Dijkstra algorithm, SSA has more advantages in computational efficiency. Especially in a dynamic traffic environment, it can adjust the path in real-time to ensure driving safety and efficiency. Furthermore, when dealing with large-scale data, the SSA demonstrates higher stability and robustness, effectively enhancing the accuracy and real-time performance of path planning. Many vertices will appear in the optimized path. Vertices are key nodes in the road information model, including intersections, entrances and exits of expressways, and important landmarks. The calculation of vertex values is shown in formula (1).³⁸

F_{N} (x, y) = H (x, y) + P (x, y)

(1)

In the formula, $H (x, y)$ represents the straight-line distance from the starting point to the end point regardless of the middle building. $P (x, y)$ represents the evaluation function from the current position to the target end point. $F_{N} (x, y)$ is the evaluation function of the vertex. To determine the optimal path, the definition of the optimal path is expressed in the research so that the SSA can search for the optimal path, as shown in formula (2).³⁹

R_{s t} = {V_{1} = s, E_{1}, V_{2}, E_{2}, \dots, V_{p}, E_{p}, V_{p + 1} = t}

(2)

In the formula (2), $V_{1}$ is the vertex passed by the route. $E_{1}$ is the arc connecting the vertex when there is a curve on the route. The final optimal form route is shown in Figure 7.

Figure 7.

Schematic diagram of optimal driving route.

As shown in Figure 7, multiple parallel paths may occur in the optimal path planning. The system will dynamically select the optimal branch based on the real-time traffic conditions to ensure driving efficiency.

Lane changing trajectory planning

In the process of automatic driving, to improve the restoration degree of human driving simulation, lane changing processing will be carried out according to the actual road conditions and traffic management requirements. The commonly used lane changing tracks include arc lane changing track, constant velocity offset lane changing track, and sine function lane changing track. Referring to the vehicle lane change on the straight road, assuming that the center of the instantaneous turning of the inner lane and the outer lane coincides, denoted as $O_{R}$ , the curvature radius of the outer lane center line is constant, defined as $R$ . The motion state of automatic driving lane changing process is shown in Figure 8.

Figure 8.

Schematic diagram of lane change trajectory planning.

As shown in Figure 8, the longitudinal driving speed is one of the key control parameters for the vehicle when changing lanes $v_{m} (x)$ . To ensure the safety and smoothness of the lane changing process, it is necessary to precisely adjust the longitudinal driving speed according to the current state of the vehicle, the surrounding environment, and the road conditions. Aiming at this, the longitudinal acceleration of car lane change is set. The details are shown in formula (3).⁴⁰

v_{m} (x) = {\begin{cases} 0, 0 < x \leq x_{1} \\ 0.4 (x - x_{1}), x_{1} < x \leq x_{2} \\ 0.2, x_{2} < x \leq x_{3} \\ - 0.4 (x - x_{4}), x_{3} < x \leq x_{4} \\ 0, x_{4} < x \leq x_{5} \end{cases}

(3)

In formula (3), $x_{i}$ is the start time of lane change, the time when acceleration increases to the maximum, the time when acceleration decreases, the time when acceleration increases to the minimum, and the time when lane change ends. The starting time of lane change is set as 0, 0.4, 0.2, −0.4, and 0, respectively. This is because in the early stage of lane change, the vehicle needs to accelerate gradually to smoothly enter the lane change process, so the acceleration is set to be positive. When approaching the target lane, to smoothly complete the lane change and avoid the safety hazards caused by excessive speed, the acceleration gradually decreases to a negative value to achieve deceleration. This setting is designed to ensure that the dynamic performance of the vehicle throughout the lane change process is both safe and efficient. Assuming that the displacement of the vehicle to the lane at the time of lane change is expressed by $W_{e} (x)$ , the angular velocity of the vehicle around the turning center can be expressed as formula (4).⁴¹

S_{f} = \frac{W_{e} (x)}{R - y_{e} (x)}

(4)

According to the optimal speed control parameters in the formula, the lane changing process of autonomous driving can be automatically controlled to achieve lane changing path planning perception and complete software optimization of the autonomous driving perception system. Considering that in the actual application scenario, there are more complex traffic conditions and weather changes, these external factors will affect the effect of dynamic obstacle avoidance. A variety of complex traffic scenarios and weather change data are introduced into the model training process to improve the stability and reliability of the automatic driving system in various situations. To further improve the dynamic obstacle avoidance ability of the automatic driving system, the You Look Only Once v5 (YOLOv5) algorithm is introduced to detect road targets, and the detected target information is fed back to the automatic driving system in real-time so that it can make more accurate decisions. At the same time, the Long Short-Term Memory (LSTM) network is used to predict and optimize the vehicle trajectory during the automatic driving process, thereby improving the safety and reliability of the automatic driving system.

IoT technology enables real-time data interaction between vehicles and their surroundings, which is essential for autonomous driving systems. Compared with other sensors, visual cognitive sensors have higher perceptual accuracy and stronger environmental adaptability, and can better cope with complex and changeable driving scenes. In addition, visual cognition technology can realize 3D reconstruction of road information and obstacle detection, providing a more comprehensive environmental perception ability for automatic driving. In terms of efficiency, the combination of IoT and visual cognitive technology can significantly improve the response speed and decision-making ability of autonomous driving systems. From a financial perspective, while the preliminary expenditure on IoT and visual cognitive technology is substantial, it is justifiable when considering the long-term advantages and enhanced security they provide. To sum up, the choice of autonomous driving system design based on the IoT and visual cognitive technology is due to many considerations such as efficiency, cost, and technical feasibility.

Experimental analysis

To prove the effectiveness of the proposed AVS, an experiment platform is built in Matlab to extract the AVS features. The experimental data used in the study originate from the test dataset of Google AVS, which contains 1200 documents and 800 information features. To get closer to real-life scenarios, the research team expands the dataset by installing sensors on the ground to collect traffic flow and weather change data at different times, ensuring coverage of various driving environments. The traffic participants in the collected data include pedestrians, bicycles, and motorcycles. The experiment first preprocesses the original dataset to eliminate noise and outliers to ensure data quality. Subsequently, multiple simulation tests are conducted using the expanded dataset to record the performance of the system in different scenarios. Ultimately, the test dataset contains 4500 documents and 2614 information features. Each test is conducted 345 times to ensure the accuracy and reliability of the data. The experiment sets the same initial position and inputs the same navigation target position to activate the navigation system to navigate and control the car to the designated destination. During the experiment, traffic signal obstacles, lane change tasks, dynamic vehicle obstacles, and other navigation interference are set up, and the realization of each navigation function is recorded.

During the experiment, the driver’s seat is adjusted to the last position, and the driver’s posture is arranged according to the driving motion. The driver’s posture is shown in Figure 9.

Figure 9.

Driver’s posture.

The driver’s multi-channel display and control system in this position is analyzed by the comfort analysis tool. Many comfort evaluation standards are predetermined in the comfort analysis tool. In the course of the experiment, the ANSUR standard is used as a reference to analyze the comfort of the neck, shoulder, elbow, knee, and other related parts of the driver. The experimental results of the designed system are shown in Figure 10.

Figure 10.

Experimental results of autopilot sensing system.

As shown in Figure 10, the reason why the comfort level of the autonomous driving system is high is that the improved method takes the driver’s visual cognition as the starting point and designs the overall position, size, shape, and color of the car’s multi-channel display control system. During the design process, full consideration is given to the matching issues with other related components of the car, thereby providing drivers with better comfort and better human-machine interaction effects.

The main frame specifications of the self-driving automobile dashboard shall be measured, and the main frame specifications of the self-driving automobile dashboard shall be described in Figure 4 as follows: 1500 mm in length, 530.6 mm in width, and 241.8 mm in height. Clamping force, also known as clamping force, is one of the key parameters in the production of self-driving dashboards, which is usually used to reflect the performance of the dashboard. When the clamping force is too small, the dashboard will not be up to the standard. When the clamping force is too big, the raw material will be consumed excessively.

The clamping force is calculated through the vertical projection area of the self-driving dashboard on the sample board during the production process. The formula is as follows⁴²:

F = K_{p} \times S

(5)

In the formula (6),

K_{p}

represents the clamping force constant.

S

represents the vertical projection area of the automatic driving dashboard on the template. The stiffness of the automobile instrument side cover plate is analyzed, and the analysis results are shown in Table 3. A major issue with a body panel’s consumer perception is stiffness. It influences how well the panels feel to a client, together with oil canning. Dent resistance is crucial to prevent panel damage during manufacturing and to reduce scratches and knocks on exterior parts while in use. Poor panel quality in pre-owned vehicles will typically lower residual values and may users to assess’ brand preferences. The extremely low and high resonant frequencies are dominated by stiffness and damping, while the medium frequency range is dominated by mass. Automobile body panels are more intricate than straightforward flat panels, and they can behave very differently from flat panels in regard to sound signal attenuation. The high-frequency sound emission of a body panel can generally be increased by strengthening beads and other structural elements of the panel. Based on this supervision and criteria, the stiffness of the side cover of the instrument panel is about to be set.

Table 3.

Analysis results of stiffness of side cover of instrument panel of autonomous driving vehicle.

Design scheme	Stress/Mpa under 200 N load	Displacement of loading point/mm
Common structure	32.574	23.285
Stiffener structure	27.546	16.864
Honeycomb construction	16.564	14.634

Table 3 shows that when the load is 200 N, the maximum stress of the side cover is 32.574 MPa and the displacement of the loading point is 23.285 mm, both of which are greater than the maximum compression degree of PP material and cannot be processed. The maximum stress of the honeycomb structure and stiffener structure is less than 30 MPa, and the maximum displacement is less than 20 mm, which accords with the processing and manufacturing conditions. Comparing these two methods, the maximum stress and displacement of the loading point of a honeycomb structure automobile dashboard is lower than that of a reinforcing rib structure automobile dashboard. This means that the rigidity of a honeycomb structure automobile dashboard is the best and it is most suitable for processing.

To further verify the effectiveness of the system, a comparative analysis of the perception accuracy of autonomous driving is conducted using the systems in references 8 and.⁹ The system in reference 8 is an autonomous driving system that conducts target detection for autonomous driving based on the YOLOv3 algorithm and combines the SSD algorithm for multi-scale feature fusion. The system in reference 9 is a multi-sensor perception system that integrates radar and cameras. It adopts target detection and semantic segmentation technologies to achieve precise navigation for autonomous driving. The comparison results are shown in Figure 11.

Figure 11.

Comparison results of automatic driving perception accuracy.

Figure 11 shows that the research system achieves an autonomous driving perception accuracy of 98.45%. Compared with the other two systems, its perception accuracy has always been higher. In this paper, a visual cognitive sensor based on symmetrical visual cognitive cameras is added to the traditional sensing hardware system. These sensors are installed on the left and right sides of the front end of the autonomous driving system, collecting road information from two angles, thereby significantly enhancing the perception accuracy of the autonomous driving system.

To further verify the effectiveness of this system, reference 8 system and reference 9 system are used to compare and analyze the perception time of automatic driving. The comparison results are shown in Table 4.

Table 4.

Comparison results of autopilot perception time of three systems.

Number of experiments/time	Research system	Reference 8 system	Reference 9 system
10	5.26 ± 0.48*^￥	10.62 ± 1.41	18.02 ± 1.63
20	5.38 ± 0.51*^￥	10.95 ± 1.36	18.62 ± 1.60
30	5.86 ± 0.50*^￥	11.02 ± 1.49	19.02 ± 1.62
40	6.05 ± 0.50*^￥	11.65 ± 1.37	19.62 ± 1.63
50	6.93 ± 0.52*^￥	12.32 ± 1.41	20.25 ± 1.65
60	7.02 ± 0.53*^￥	12.65 ± 1.40	20.65 ± 1.64
70	7.62 ± 0.56*^￥	13.52 ± 1.45	21.62 ± 1.67

Note: In Table 4, * and ￥ indicate that there is a significant difference between the system results of this paper and those of the reference 8 and reference 9 systems (p < 0.05).

The data in Table 4 shows that the perception time of the research system for autonomous driving is within 7.62 seconds, the time in reference 8 is within 13.52 seconds, and the time in reference 9 is within 21.62 seconds. The time taken by the research system to perceive the automatic driving is the shortest and the sensing efficiency is the highest.

To further verify the effectiveness of the proposed autonomous driving system, the study aims to apply it to actual autonomous driving scenarios. The experiment lasts for 6 months and covers different weather conditions and road scenarios. The actual test scenarios include urban roads, expressways, and complex and changeable rural roads. The study classifies the weather conditions during the test process into three types: rainy days, sunny days, and storms, comprehensively evaluating the ability of the autonomous driving system in path planning. The evaluation indicators include recall rate, mean average precision (mAP), distance estimation error, speed estimation error, conflict avoidance rate, and response delay time. The specific results are shown in Table 5.

Table 5.

Automatic driving performance comparison of each system.

Project		mAP (%)	Recall (%)	Distance estimation error (%)	Velocity estimation error (%)	Conflict avoidance success rate (%)	Response delay time (s)
Urban road	Research system	98.54	95.34	1.24	1.31	100.00	0.42
	Reference 8 system	90.12	85.22	3.67	3.92	88.73	1.23
	Reference 9 system	92.16	89.10	2.89	3.14	90.15	0.96
Rural road	Research system	98.57	95.40	1.18	1.36	99.82	0.44
	Reference 8 system	90.32	85.32	3.67	4.05	88.94	1.28
	Reference 9 system	92.14	89.25	3.03	3.12	90.34	1.06
Expressway	Research system	98.28	95.68	1.25	1.30	100.00	0.42
	Reference 8 system	90.36	85.50	3.73	3.88	88.67	1.25
	Reference 9 system	92.27	89.72	2.95	3.12	90.41	0.93

As shown in Table 5, the research system performs well under three different road conditions: urban roads, expressways, and rural roads. The mAP values of the research system on urban roads, rural roads, and expressways have reached 98.54%, 98.57%, and 98.28%, respectively, which are significantly higher than those of the reference system. Especially, it has obvious advantages in the success rate of conflict avoidance and the response delay time. Furthermore, the recall rate and distance estimation error of the research system are significantly better than others, and its corresponding delay time is only 0.42 s. Compared with other systems, the research system has a higher response efficiency.

Conclusion and prospect

Conclusion

The autopilot perception system based on visual cognition improves the driver’s comfort and has good human-computer interaction effects.

The designed automatic driving perception system makes the important buttons and information on the dashboard almost not blocked, which can effectively improve the safety of car driving and verify the effectiveness of the system designed in this paper.

The stiffness of the side cover plate of the automobile instrument panel of the autopilot sensing system designed is the best, which is most suitable for processing and production, and is worth popularizing and applying in practice. There are still some limitations in the autonomous driving system studied at present. Although in the experimental analysis, the study introduced a variety of weather conditions and road types to evaluate the performance of the system, the actual road conditions are far more complex than the experimental environment. Future studies should consider further enhancing the system’s ability to respond to emergencies and its stability under extreme weather conditions.

Discussion

The design of the automatic driving perception system, which is based on visual cognition, is a complex and extensive system. This study focuses on a specific aspect of the system; however, the overall integration of the system is inadequate. Due to the use of a mobile computing platform inside the vehicle, its computing capacity is inherently limited. The real-time execution of object detection models and feasible driving area models on these mobile platforms faces significant challenges. Further research is needed to explore and develop models that can maintain or improve the detection accuracy while reducing the computing requirements. When weighing the perception accuracy against the computing/hardware cost, the system performance and economy need to be comprehensively considered. By optimizing the algorithm structure and adopting lightweight models, the detection accuracy can be improved without significantly increasing the cost. At the same time, by exploring solutions and distributed processing methods that combine edge computing and cloud computing to reduce the burden of mobile platforms, efficient and low-cost system optimization can be achieved. Furthermore, the research also has adjustment sensitivity and insufficient adaptability to environmental changes, and the robustness of the algorithm needs to be enhanced. Future work can further optimize data fusion strategies, improve the efficiency of multi-sensor information fusion, and cope with complex road conditions. By exploring the combination of deep learning and reinforcement learning, the self-learning and adaptive capabilities of the system can be enhanced to ensure stable performance in various driving scenarios.

In comparison, deep learning performs better in complex scene recognition, while research system has advantages in real-time performance and computational efficiency. The hybrid MPPT strategy has a slight edge in energy utilization rate. However, in terms of system integration and cost control, the research method is of greater practical value. Future research should combine the advantages of each method to explore more comprehensive perception solutions for autonomous driving.

Footnotes

ORCID iD

Ruiping Kong

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declarations of conflicting interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.*

References

Peng

Zhang

, et al. A new safe lane-change trajectory model and collision avoidance control method for automatic driving vehicles. Expert Syst Appl 2020; 141: 112953.

Pan

Ogai

. Fast lane detection based on deep convolutional neural network and automatic training data labeling, IEICE Transactions on Fundamentals of Electronics. Communications and Computer Sciences 2019; 102(3): 566–575.

Wang

You

Chu

, et al. Research on customer marketing acceptance for future automatic driving - a case study in China city. IEEE Access 2019; 12(99): 1–10.

Kagita

Kumar

. Machine learning techniques for multi-media communications in business marketing. J Mult-Valued Log Soft Comput 2021; 36(1-3): 135–150.

Lee

. Fixed-time-synchronized consensus control of multiagent systems. IEEE Trans Control Netw Syst 2021; 8(1): 89–98.

Yuan

Yang

. System dynamics approach for evaluating the interconnection performance of cross-border transport infrastructure. J Manag Eng. 2022; 38(3): 04022008.

Jiang

Zhao

Zhu

, et al. A practical and economical ultra-wideband base station placement approach for indoor autonomous driving systems. J Adv Transp 2022; 2022: 1–12.

Hou

Wang

. Moving objects detection in automatic driving based on YOLOv3. Comput Enginee Des 2019; 040(004): 1139–1144.

Yao

Guan

Huang

, et al. Radar-camera fusion for object detection and semantic segmentation in autonomous driving: a comprehensive review. IEEE Trans Intell Veh 2023; 9(1): 2094–2128.

10.

Liang

Wang

, et al. MKD-Cooper: cooperative 3D object detection for autonomous driving via multi-teacher knowledge distillation. IEEE Trans Intell Veh 2023; 9(1): 1490–1500.

11.

Wang

, et al. Centernet-auto: a multi-object visual detection algorithm for autonomous driving scenes based on improved centernet. IEEE Trans Emerg Top Comput Intell 2023; 7(3): 742–752.

12.

Tianjun

Wan

Wang

, et al. Model reference adaptive control of semi-active suspension model based on ad boost algorithm for rollover prediction. SAE Int J Veh Dyn Stab NVH-V131-10EJ 2022; 6(1): 71–86.

13.

Lai

Huang

Jiang

. Comparative study on bifurcation and stability control of vehicle lateral dynamics. SAE Int J Veh Dyn Stab NVH 2022; 6(1): 35–52.

14.

Zhang

Wang

, et al. Chassis coordinated control for full X-by-wire vehicles-A review. Chin J Mech Eng 2021; 34: 42.

15.

Zhang

Deng

, et al. An enabling trajectory planning scheme for lane change collision avoidance on highways. In: IEEE transactions on intelligent vehicles, 2021, pp. 1.

16.

Liu

Wang

Zhang

. Integrated vehicle-following control for four-wheel-independent-drive electric vehicles against non-ideal V2X communication. IEEE Trans Veh Technol 2022; 71(4): 3648–3659.

17.

Luo

Zhang

Yuan

, et al. EST Net: embedded spatial-temporal network for modeling traffic flow dynamics. IEEE Trans Intell Transport Syst 2022; 23: 1–12.

18.

Sun

Zhang

Wang

, et al. SES-YOLOv8n: automatic driving object detection algorithm based on improved YOLOv8. Signal Image Video Process 2024; 18(5): 3983–3992.

19.

Shao

Chen

, et al. Semi-automatic geographic information system framework for creating photo-realistic digital twin cities to support autonomous driving research. Transp Res Rec 2024; 2678(6): 1068–1084.

20.

Ban

Zhang

, et al. Adaptive control of time delay teleoperation system with uncertain dynamics. Front Neurorob 2022; 16: 928863.

21.

Han

Wang

Guan

, et al. Research on road environmental sense method of intelligent vehicle based on tracking check. IEEE Trans Intell Transport Syst 2022; 24: 1–15.

22.

Rigon

Haus

Mercorelli

, et al. Automatic tuning of extended Kalman filter in synchronous reluctance motor drives with a master-slave configuration. IEEE Open J Pow Elect 2025; 99: 1–14.

23.

Hou

Simos

Famelis

. Neural network solution of pantograph type differential equations. Math Methods Appl Sci 2020; 43: 3369–3374.

24.

Badue

Guidolini

Carneiro

, et al. Self-driving cars: a survey. Expert Syst Appl 2020; 165(1): 113–126.

25.

Shekar

Liang

Liu

, et al. Label-free robustness estimation of object detection CNNs for autonomous driving applications. Int J Comput Vis 2021; 129(4): 56–63.

26.

Chen

Zuo

. The influence of incentives on drivers’ aggressive behaviors toward automated vehicles. Int J Hum Comput Interact 2025; 41(6): 3659–3670.

27.

Istiyanto

Syah

ADM

Najwa

, et al. Design and build pedestrian lane automatic vehicle restraint system: solutions to make it easier for road crossers. Fidelity: Jurnal Teknik Elektro 2025; 7(1): 11–21.

28.

Yuan

Gao

. Analysis of fractional factor system for data transmission in SDN. App Mathe Nonline Sci 2019; 4: 191–196.

29.

Wang

Zhu

, et al. An adaptive neural sliding mode control with ESO for uncertain nonlinear systems. Int J Control Autom Syst 2021; 19: 687–697.

30.

Chen

Jian

Huang

, et al. Autonomous driving: cognitive construction and situation understanding. Sci China Inf Sci 2019; 62(8): 1–27.

31.

Chen

Pei

, et al. A deep reinforcement learning based method for real-time path planning and dynamic obstacle avoidance. Neurocomputing 2022; 497: 64–75.

32.

Maqsood

Tahir

Fatima

, et al. Interpreting rescue vehicle patterns using geovisual analytics for spatiotemporal resource allocation. Arabian J Geosci 2020; 13: 660.

33.

Wang

Shi

, et al. Research on binocular visual feature points positioning based on improved SURF matching algorithm. Software Eng 2019; 22(3): 5–8.

34.

Yang

. Automatic detection system for threaded hole verticality based on dual-camera symmetrical measurement method. J Nanchang Aero Uni: Nat Sci Edi 2020; 1: 82–88.

35.

Karatas

Tamura

Fushiki

, et al. Improving human-autonomous car interaction through gaze following behaviors of driving agents. Trans Jpn Soc Artif Intell 2019; 34(2): 1–11.

36.

Wang

Tian

Zhang

, et al. Control of time delay force feedback teleoperation system with finite time convergence. Front Neurorob 2022.

37.

Arnold

Dianati

Temple

, et al. Cooperative perception for 3D object detection in driving scenarios using infrastructure sensors. IEEE Trans Intell Transport Syst 2020; 15(99): 1–13.

38.

Bernardin

Schwitzer

Angioi-Duprez

, et al. Retinal ganglion cell dysfunction is correlated with disturbed visual cognition in schizophrenia patients with visual hallucinations. Psychiatry Res 2021; 298(20): 113–140.

39.

Zou

Sun

Zhou

, et al. Limited sensing and deep data mining: a new exploration of developing city-wide parking guidance systems. IEEE Intell Transport Syst Mag 2022; 14(1): 198–215.

40.

Perry

. Here comes driverless ride sharing: cruise unveils the origin, a fully autonomous SUV designed for app-controlled urban transportation - [Spectral Lines]. IEEE Spectr 2020; 57(3): 4–10.

41.

Zhang

, et al. Object traversing by monocular UAV in outdoor environment. Asian J Control 2020.

42.

Zhang

Jiang

Wang

, et al. Research on management system of automatic driver decision-making knowledge base for unmanned vehicle. Int J Pattern Recogn Artif Intell 2019; 33(4): 1–19.