Evaluation Methodology of Leader-Follower Autonomous Vehicle System for Work Zone Maintenance

Times and Locations of Tests

The tests were conducted from March 26, 2019 through March 30, 2019 at Fort Walton Beach, Florida, and from April 22, 2019 through April 25, 2019 at Sedalia, Missouri.

Test Cases

A total of 31 cases were planned to test the system’s performance. Among them, 23 of the test cases were quantifiable and are documented, including five tests on communication loss, seven on following distance and accuracy, three on obstacle detection, and eight on emergency situations. Each test was repeated three times to assure statistical accuracy. The other eight test cases were simple yes/no testing, such as visual inspection of the system and the trucks, data logging, turn signals, and the functionality of the user interface, and are thus not documented in this paper.

Vehicle Operating Mode

The transition of each vehicle operating mode, from the beginning of a test scenario to the end, was recorded in a log. The log files included three modes for each ATMA vehicle: IDLE, ROLLOUT, and RUN. IDLE mode is when the safety operator of an ATMA vehicle is in control of the vehicle (instead of the autonomous system having control), and it is the mode in which the ATMA vehicle starts. The FT will transition to the ROLLOUT mode and then to the RUN mode. ROLLOUT mode describes the initial state of the ATMA system when operations begin. The initial state is the distance the ATMA vehicle travels toward the LT before transitioning to the eCrumb navigation method. The travel distance needed to transition to an eCrumb path ranges between 20 ft and 30 ft. ROLLOUT mode should be performed at an approximate speed of 4–8 mph. RUN mode is where the FT is operating autonomously.

Log File Format

The FT log file is formatted as a comma separated value (CSV) file format. Figure 2 shows a sample of the LT and FT log files.

OPEN IN VIEWER

Figure 2.

Screenshot of autonomous truck-mounted attenuator (ATMA) vehicles’ log files: (a) leader truck and (b) follower truck.

Data Processing Procedure

The log files generated by the ATMA system were collected for each test scenario and, during testing, the logged data were exported from the ATMA system once every 24 h. The vehicle log file was formatted in a CSV format. In some cases, the test data were converted to Keyhole Markup Language (KML) for optional plotting with Google Earth. The log file can be analyzed to gain insights into the performance and behavior of the system, especially in relation to the FT. Data were plotted in Excel, and the statistical characteristics and hypothesis testing were analyzed in Python.

System Performance Statistical Characteristics

Among the 31 defined test cases, some results could be directly observed, such as visual inspection and emergency stops, so that it could be determined if the test had been passed. However, some test results could not be directly observed, such as lane accuracy, so that further analyses were needed to determine whether that test was passed.

General statistical characteristics of a CTE were analyzed and presented to illustrate the system’s overall performance. CTE is the horizontal deviation from intended path and could be positive or negative. The mean value indicated the center of distribution, and standard deviation and quantiles (including minimum value, 25th percentiles, 50th percentiles, 75th percentiles, and maximum value) illustrated the dispersion of the distribution. Together, they provided an overview of the data statistics for the system’s performance that was tested. Thus, by comparing the statistical characteristics of three repetitive test results with predefined criteria from MoDOT, it could be determined whether the system met requirements.

System Performance Probability Distribution

Based on the data collected for each test case, the probability distribution of CTE frequency was analyzed to determine the probability that the CTE fell within a certain range. This was done by dividing the CTE into several groups, and the frequency of each group was counted. Based on these data, the probability distribution of the system performance could be obtained. The derived probability distribution can help engineers give a simple “pass” or “fail” answer and, more important, learn more about the system so that a more quantified understanding of its performance can be obtained.

Hypothesis Testing

While it was expected that the developed ATMA system could function reliably each time the same test was performed, its function consistency needed to be tested. A hypothesis test was defined and performed to achieve this goal.

The Friedman test is a nonparametric statistical procedure for comparing more than two samples that are related. It is commonly used to detect differences in treatments across multiple test attempts ( 28 ). Assumptions of the Friedman test include:

The Friedman test statistic is determined by Equation 1

F r = 12 Nk ( k + 1 ) ∑ i = 1 k R i 2 − 3 N ( k + 1 )

(1)

where N is the number of rows, or subjects, k is the number of columns, or conditions, and R_i is the sum of the ranks from columns, or condition, i .

The degree of freedom for the Friedman test is determined by Equation 2

df = k − 1

(2)

In the field tests, each test case was repeated three times. So, for each test case, three sets of data were obtained (e.g., simulated radio frequency loss test) and, sometimes, there were even six sets of data (e.g., the following accuracy test was performed on both tangents and curves). The CTE was extracted from repeated tests in which no condition was changed, so the data were related. The three (or six) sets of data were grouped, with the same interval between them, and the frequency was counted. The Friedman test was then used to verify that those three (or six) sets of the CTE had no statistically significant difference, which meant the results were stable and the system had performed consistently.

For testing purposes, the following null and alternative hypotheses were used:

The confidence level was set at α = 0 . 05 , which means that if the p value is less than the α , then the null hypothesis should be rejected. Otherwise, the null hypothesis should be accepted, and the system performed consistently.

Communication Loss

A total of five test cases were designed to test the system’s performance when communication is lost:

Table 1 demonstrates the general statistical characteristics of CTE for RF loss and single V2V radio loss tests. It was found that, for each test, the number of valid data points ranged from 200 to over 1,000, thus ensuring sufficient data size to draw statistically significant conclusions. Another observation was that the CTE was mostly very low, when compared with the defined criteria of ± 6 inches, meaning that the system performed well in all tests. In a simulated RF loss test case, it was observed that the characteristics from tests 2 and 3 were more similar, as the values of mean value and standard deviation of these two datasets were closer to each other. On the contrary, the values of mean and standard deviations for test 1 seemed to be slightly higher than the other two tests. While for the single V2V radio loss tests, however, it was observed that the characteristics for tests 1 and 3 were similar. The values of the mean and standard deviations were closer to each other. The absolute values of the mean and standard deviations for test 2 seemed to differ slightly from those of the other two tests.

OPEN IN VIEWER

Table 1.

Statistical Characteristics of Cross Track Error in Communication Loss Tests (Inches)

Tests		Count	Mean	SD	Min.	25th%	50th%	75th%	Max.
RF loss	Test 1	899	–2.98	2.07	–6.77	–4.69	–3.62	–0.79	2.17
	Test 2	1117	–2.75	1.77	–8.58	–3.82	–2.99	–1.30	1.14
	Test 3	989	–2.79	1.90	–6.30	–4.29	–3.46	–0.87	1.02
Single vehicle-to-vehicle radio loss	Test 1	363	0.17	1.61	–5.90	–0.08	0.43	1.06	3.98
	Test 2	340	0.98	1.13	–1.34	0.00	0.91	1.69	4.96
	Test 3	230	0.45	1.27	–4.21	–0.08	0.24	1.26	2.83

Note: RF = radio frequency; SD = standard deviation; Min. = minimum; Max. = maximum.

Next, probability distribution and hypothesis tests were conducted to evaluate the system’s performance more quantitatively. Figure 3 demonstrates the probability distribution. For RF loss, − 8 . 58 inches was set as the minimum and 2 . 17 inches as the maximum of CTE, then the CTEs were grouped with 0 . 90 in. as an interval, and the number of each test counted. In three tests, the percentages of CTE that were within ± 6 in. were 98.2%, 96.5%, and 99.1%. For single V2V radio loss tests, − 5 . 90 in. was set as the minimum and 4 . 96 in. as the maximum of the CTE, then the CTEs were grouped with 0 . 90 in. as an interval, and the number of each test counted. In three tests, all CTEs were within ± 6 in. Those numbers indicated that the system’s performance in these two aspects was satisfactory.

OPEN IN VIEWER

Figure 3.

Frequency distribution histogram of cross track error in communication loss tests: (a) simulation radio frequency (RF) loss test and (b) single vehicle-to-vehicle (V2V) radio loss test.

In relation to hypothesis testing, for RF loss, the result showed that the p-value was 0.843 (>0.05), which meant that the null hypothesis should be accepted. For the single V2V radio loss test, the result showed that the p-value was 0.230 (>0.05), which meant that the null hypothesis should also be accepted. This meant that the CTEs for both the RF loss test and single V2V radio loss test were verified as belonging to the same population, which indicated that the system’s performance in different tests runs was stable.

For the other three communication loss tests (both V2V radios, loss of sensors, and GPS-denied environment), the results showed that the FTs all came to an A-Stop after communication was lost. When both V2V radio communications were lost, the time from loss of communications until an A-Stop was initiated was 1.9 s for three repetitive tests, and the distances from the points of communication loss until the FT stopped were 55.2 ft, 48.8 ft, and 56.2 ft. When the loss of sensors happened, the average and maximum times to stop were 6.0 s and 7.7 s, and the average and maximum distances to stop were 34.3 ft and 37.5 ft. In the case of a GPS-denied environment, the dead reckoning assembly (DRA) accuracies were ± 4.5 in., ± 5.1 in., and ± 4.7 in. on tangents, and ± 3.0 in., ± 3.4 in., and ± 5.3 in. on curves. All DRA accuracies were within ± 6 in. The DRA timeout was 45 s for three tests. These data indicated that the system performance in all three tests was satisfactory. Since the last three cases were yes/no tests, analysis of probability distribution and hypothesis testing was not necessary and thus was not performed.

Following Distance and Accuracy

A total of seven cases were designed to test the system’s performance in following distance and accuracy:

Table 2 demonstrates the general statistical characteristics of the following distance for CTE in the seven tests performed. It was found that, for each test, the number of valid data points ranged from a few hundred to over 1,000, thereby sufficient data size was guaranteed to allow statistically significant conclusions to be drawn. Another observation was that the following distance error was mostly very low, when compared with defined criteria.

OPEN IN VIEWER

Table 2.

Statistical Characteristics of the Cross Track Error in Following Distance and Accuracy Tests (Inches)

Tests			Count	Mean	SD	Min.	25th%	50th%	75th%	Max.
Following accuracy test	Tangents	Test 1	521	–1.61	1.14	–7.32	–1.97	–1.42	–1.02	0.00
		Test 2	456	–1.85	1.02	–6.18	–2.32	–1.89	–1.38	0.75
		Test 3	472	–1.73	1.14	–7.20	–2.17	–1.73	–1.22	1.02
	Curves	Test 1	552	–0.75	1.14	–4.76	–1.34	–0.91	–0.43	6.69
		Test 2	541	–1.34	1.50	–8.86	–1.93	–1.22	–0.67	4.69
		Test 3	557	–1.06	1.34	–6.89	–1.73	–1.14	–0.31	4.06
Lane-changing test	Left to right	Test 1	422	–1.85	1.73	–6.14	–2.91	–2.17	–1.06	4.21
		Test 2	500	–0.87	1.26	–3.07	–1.81	–1.18	0.00	3.31
		Test 3	511	–1.22	1.54	–5.12	–2.17	–1.46	–0.51	3.15
	Right to left	Test 1	489	–1.42	1.69	–5.67	–2.48	–1.57	–0.24	3.03
		Test 2	487	–1.54	1.81	–8.03	–2.28	–1.38	–0.31	2.13
		Test 3	488	–1.57	1.81	–8.78	–2.20	–1.42	–0.43	2.24
Bump test		Test 1	1272	–0.016	–0.63	2.40	–6.97	–2.13	–1.54	0.55
		Test 2	1172	–0.020	–0.79	2.01	–6.38	–2.01	–1.46	0.00
		Test 3	1231	–0.073	–2.87	2.52	–10.24	–4.61	–3.46	–1.50
Roundabout test		Test 1	1103	–0.098	–3.86	2.52	–9.69	–5.91	–4.61	–1.42
		Test 2	988	–0.088	–3.46	2.76	–8.23	–5.75	–3.74	–1.22
		Test 3	981	–0.086	–3.39	2.56	–8.62	–5.59	–3.39	–1.30
Minimum turn radius test	Left turn	Test 1	722	–3.31	2.24	–8.82	–4.84	–3.07	–1.85	4.37
		Test 2	674	–3.15	1.93	–8.31	–4.84	–3.15	–2.28	0.00
		Test 3	609	–3.58	1.85	–8.27	–4.92	–3.11	–2.40	0.00
	Right turn	Test 1	856	–1.93	2.17	–7.83	–3.23	–2.60	0.00	2.95
		Test 2	818	–1.65	2.05	–5.91	–3.15	–2.32	0.16	3.15
		Test 3	782	–1.73	1.89	–5.24	–3.03	–2.52	0.00	2.56
U-turn test	Left U-turn	Test 1	945	–3.74	2.68	–10.08	–6.22	–2.91	–1.81	3.82
		Test 2	805	–4.06	2.56	–10.08	–6.54	–3.35	–2.01	0.59
		Test 3	761	–3.90	2.80	–10.39	–6.54	–3.03	–1.77	3.82
	Right U-turn	Test 1	875	–0.79	2.20	–5.00	–2.60	–1.73	1.54	4.25
		Test 2	782	–0.55	2.20	–4.69	–2.52	–1.06	1.57	3.78
		Test 3	838	–1.10	2.48	–6.10	–3.11	–2.13	1.42	4.69

Note: SD = standard deviation; Min. = minimum; Max. = maximum.

Figure 4, a–f, show the CTE probability distribution. Figure 4, a, b, e, and f , illustrate that all of the CTE were within ± 6 in. in the tangent following accuracy tests, lane-changing tests (from left to right), minimum radius right-turn tests, and right U-turn tests. In curve following accuracy tests, lane-changing tests (from right to left), and bump tests, the percentages of CTEs within ± 6 in. ranged from 96.5% to 100%, as shown in Figure 4, a–c . In the roundabouts tests and minimum radius left turn tests presented in Figure 4, d and e , the percentage of CTEs within ±6 in. ranged from 77.8% to 85.1% and more than 99% of the CTEs were within ±8 in. In left U-turn tests, the percentages of CTEs within ±6 in. were more than 70.0%, and all the CTE were within 10 in. These numbers indicated that the system performance for these tests was satisfactory.

OPEN IN VIEWER

Figure 4.

Frequency distribution histogram of the cross track error in following distance and accuracy tests: (a) following accuracy test, (b) lane changing test, (c) bump test, (d) roundabout test, (e) minimum turn radius test, (f) U-turn test.

In relation to hypothesis testing, the p-values of the six tests were 0.753, 0.535, 0.808, 0.746, 0.484, and 0.612. All were higher than 0.05, which means the null hypothesis should be accepted for all seven tests. In other words, the CTE in all tests was verified as belonging to the same population, which indicates that the system performed consistently in following accuracy in all tests performed.

Obstacle Detection

The obstacle detection test was conducted to determine if a FT could detect an obstacle that had been placed in its path, and execute an A-Stop in time, after the LT passed. Each test was repeated three times for statistical accuracy. A total of three cases were designed and tests were conducted.

The results showed that, in all repeated tests, the radar/LiDAR of the FT detected the barrel in the center of the lane, or the vehicle in the adjacent lane, and executed an A-Stop. Since this was a yes/no test, analysis of probability distribution and hypothesis testing was unnecessary and thus not performed. This test was successfully passed in all three test cases.

Emergency Situations

Tests of emergency situations were designed to determine the system’s performance under emergency conditions, where special operations were needed. A total of eight test cases were defined and conducted.

When the LT temporarily dropped the ATMA vehicle, the FT caught up to the LT and stabilized at 100 ft in the end. The maximum speed did not exceed 12 mph. The gap distance dropped from 200 ft to 100 ft while the speed increased to 12 mph. The final stabilized gap distances were 109.45 ft, 113.68 ft, and 109.45 ft. Compared with the desired gap distance (100 ft), actual distance errors were within ± 15 ft. The test was successfully passed.

The system also performed well in the other four tests. During all four emergency stop tests, the FT successfully stopped and the tests were successfully passed. When the LT initiated use of the brake, the recorded data showed that, for the three repetitive tests, the actual gap distances were 72 ft, 72 ft, and 80 ft, and the gap distances after stopping were 68 ft, 67 ft, and 76 ft. The gap deltas were 4 ft, 5 ft, and 4 ft, all within ± 15 ft allowable range. In the test when a human driver took control of the FT, the FT quickly disengaged from the system and allowed the human driver to take control when the FT was released from the autonomous system to idle. Finally, when a rear impact took place, the FT released the throttle, applied full brakes, and turned on hazard lights. These tests were all successfully passed.