Evaluation of Network-Level Data Collection Variability and its Influence on Pavement Evaluation Utilizing Random Forest Method

Abstract

The use of pavement condition data to support maintenance and resurfacing strategies and justify budget needs becomes more crucial as more data-driven approaches are being used by the state highway agencies (SHAs). Therefore, it is important to understand and thus evaluate the influence of data variability on pavement management activities. However, owing to a huge amount of data collected annually, it is a challenge for SHAs to evaluate the influence of data collection variability on network-level pavement evaluation. In this paper, network-level parallel tests were employed to evaluate data collection variability. Based on the data sets from the parallel tests, classification models were constructed to identify the segments that were subject to inconsistent rating resulting from data collection variability. These models were then used to evaluate the influence of data variability on pavement evaluation. The results indicated that the variability of longitudinal cracks was influenced by longitudinal lane joints, lateral wandering, and lane measurement zones. The influence of data variability on condition evaluation for state routes was more significant than that for interstates. However, high variability of individual metrics may not necessarily lead to high variability of combined metrics.

Pavement condition evaluation is a key element in a network-level pavement management system ( 1 ) in that it provides a basic evaluation of overall network condition and is used for identifying needs of maintenance, reconstruction, and rehabilitation. Since huge amounts of pavement condition data are collected and processed annually, data collection variability has become a major challenge for highway management agencies.

Several factors influence data collection variability. Even with well-calibrated equipment and well-trained operators, variability of pavement condition data collected by automated collection equipment is unavoidable because uncertainty is a part of measurement ( 2 ). From the perspective of highway agencies, variability of pavement condition data becomes a concern when these data are used for budget/maintenance needs analysis ( 3 , 4 ) and performance target setting ( 5 ). However, efficient ways to evaluate the influence of pavement condition data variability on a network-level performance evaluation are lacking. It is time-consuming and sometimes even impossible to evaluate data variability by the same method used for project-level performance evaluation for which data variability could be determined by multiple test runs.

As indicated in a previous study ( 6 ), a large total error may occur in any single measurement. Therefore, to obtain a clear picture of the road condition for the entire network, it is of importance for state highway agencies (SHAs) to understand and evaluate the influence of data variability on network-level pavement evaluation. For example, when SHAs set pavement performance targets, the variability of pavement condition data could potentially affect the final distribution of pavement condition which is used to determine the baseline performance. If those segments which are subject to collection variability can be identified and considered in budget/performance analyses, pavement engineers will be able to set more reasonable performance targets and make more reliable budget and resurfacing plans.

Some efforts have been made to evaluate data variability of manual and automated rating methods. Landers et al. introduced Cohen’s weighted kappa statistics into quality assurance/quality control processes to assess the level of agreement beyond change among raters for the British Columbia Ministry of Transportation ( 7 ). The index served as an overall estimate of level of agreement between a manual benchmark survey and contractors’ ratings. Ong et al. developed a set of performance measures to quantify the quality of data collected and processed by automated technique. These measures can also be used for assessing the effects of sampling on overall condition ratings ( 8 ). To improve the Northern New Mexico Pavement Evaluation Program, Bogus et al. proposed a framework consisting of assessment of consistency over time and assessment of agreement between evaluators ( 9 ). By investigating data from the Long-Term Pavement Performance (LTPP) program’s distress rater accreditation workshop, Rada et al. found that there was a large variability in individual rater with increases in distress quantity ( 10 , 11 ). Ong and Sinha evaluated the quality of pavement roughness data collected at project- and network level ( 12 ). They found the International Roughness Index (IRI) data quality at project level was affected by the individual run, the wheel path, and the lane in which the profile was measured, whereas IRI quality at network level was affected by the wheel path profile. Siabil et al. proposed a computational technique for detecting errors in network-level pavement condition data sets ( 13 ). They also developed an error detection method that considered all the data properties and consistency among multiple performance indicators ( 14 ).

There are also studies focusing on the influence of data variability on network-level pavement management activities. By investigating longitudinal profile data from LTPP, Yin et al. developed measures of the variability between routine network-level profile visits to a site. They also provided recommendations for quantifying and controlling the variability of longitudinal profile data ( 15 ). Saliminejad and Gharaibeh quantified the impact of systematic and random error in pavement condition data on the results of Maintenance and Rehabilitation (M&R) analysis. They concluded that, in practice, both systematic and random errors can highly distort the analysis results even within acceptable error ranges ( 16 ). To understand and quantify the influence of IRI measurement variability on pavement evaluation, Jia et al. constructed a probabilistic relationship to consider the influence of run-to-run variability of IRI for network-level pavement evaluation. The uncertainty of pavement evaluation was introduced in the process ( 17 ). Jia et al. also developed methods to consider the influence of variability of surface distress data on pavement condition evaluation ( 18 ).

In this study, network-level data collection variability was evaluated using parallel testing that was conducted by two testing vehicles equipped with the identical testing device, Laser Crack Measurement System (LCMS), which was developed by Pavemetrics System Inc. in Canada. The results of the analysis provided understanding of network-level data variability and its influences on pavement condition evaluation. The method used in this study is based on state-of-the-art data mining technology and could be used by agencies in their data quality management plans to identify potential segments that could be subject to data variability and to quantify the influence of data variability on network pavement evaluation. A demonstration of how to implement the proposed method into pavement management activities is also provided by this study.

Objective and Scope

The objective of this study was to evaluate the variability of network-level data collection and its influence on pavement condition evaluation. Classification models were constructed by random forest (RF) method to identify road segments subject to data variability. The pavement condition data used in this study were from Tennessee Department of Transportation’s (TDOT) pavement management system (PMS).

Parallel Data Collection

Network-level data collection variability was evaluated through parallel tests performed on selected road segments using two vehicles. One vehicle is owned by the data collection vendor for annual network-level data collection. The other is owned by TDOT for agency's data collection. Both vehicles are installed with the identical data collection/processing system. The equipment consisted of a 3-D imaging system that was used to identify surface characteristics, which were then used for extracting data on surface roughness and distresses by the automated distress identification method. The two testing vehicles were validated at same test site using the same validation procedure. To decrease systematic errors caused by vehicle operators, testing conditions, and surface properties, the parallel tests were performed within 2 weeks under same environmental conditions by the operators who are certified by the validation test. Test routes were selected based on the type and extent of the surface distress. A route comprising 180 centerline miles was selected for parallel testing. It should be noted that owing to the large amount of efforts, it is difficult to establish ground-truth for evaluating network-level data accuracy. Instead, in this study the parallel tests evaluated data repeatability and its influence on pavement evaluation.

The pavement condition data used in this study consisted of roughness and surface defects. The roughness data included the IRI and rut depth, which characterize surface profiles in longitudinal and transversal directions of the pavement, respectively. The surface defects were cracking-type distresses, which are the main methods of pavement evaluation in Tennessee. There were five major types of cracks from TDOT’s PMS: fatigue cracks, longitudinal wheel-path cracks, longitudinal nonwheel-path cracks, block cracks, and transverse cracks; there was also a national measure: crack percent. All data were reported in 0.1-mi segments, which were used for further analysis in this study.

Roughness and individual surface distresses are generally combined in indices for evaluating pavement condition. Three indices were used in this study: the pavement smoothness index (PSI), pavement distress index (PDI), and the pavement quality index (PQI). PSI is a function of IRI, whereas PDI is a function of individual distress and rut depth. By combining PSI and PDI, PQI can be calculated using the following equation:

P Q I = P S I^{0.3} \times P D I^{0.7} = {[f_{1} (I R I)]}^{0.3} \times {[f_{2} (r u t d e p t h, s u r f a c e d i s t r e s s e s)]}^{0.7}

Three categories are employed to evaluate the condition of a pavement. The thresholds are defined in relation to PQI, which are listed in Table 1.

Table 1.

Definition of TDOT Performance Measures

Ratings	Interstate	State route
Good	PQI > 4.0	PQI > 3.5
Fair	4.0 ≥ PQI ≥ 2.0	3.5 ≥ PQI ≥ 2.0
Poor	PQI < 2.0	PQI < 2.0

Note: TDOT = Tennessee Department of Transportation; PQI = pavement quality index.

As pavement evaluation methods vary between states, national performance measures were proposed to provide each SHA with a standard, facilitating the reporting of pavement condition in the same manner. Four performance metrics are used for pavement condition evaluation: IRI, crack percent, rut depth (for asphalt pavement only), and faulting (for concrete pavement only). In this study, a segment was initially rated according to IRI, rut depth, and cracking percent. For each pavement type, three metrics were used to determine performance measures, comprising three categorial conditions (Good, Fair, and Poor). The thresholds for asphalt pavements are listed in Table 2. The final rating was the combination of these three metrics. With all three metrics rated as “Good,” the overall rating of a segment is Good, whereas if two or more metrics are rated “Poor,” the overall rating is Poor. All other combinations are considered to be “Fair.”

Table 2.

Definition of National Performance Measures ( 19 )

Metric	Individual rating			Overall rating
	Good	Fair	Poor	Good	Poor	Fair
IRI	<95	95–170	>170	All three metrics rated “Good”	Two or more metrics rated “Poor”	All other combinations
Rut depth	<0.2	0.2–0.4	>0.4
Crack percent	<5	5–20	>20

Note: IRI = international roughness index.

Table 3 lists the statistical summary of pavement condition data from the parallel tests. Surface cracks of differing types and severity levels were observed on the parallel test road segments. Cracks were classified according to three severity levels: low, moderate, and high. IRI from both wheel paths and half car were collected and included in the models. IRI results were then averaged to calculate PSI. Rut depths from both wheel paths were also included. Crack percent was collected and used for determining national performance measures. All other types of surface cracks along with average rut depth were then used to calculate PDI. Therefore, in Table 3, PSI, PDI, and PQI calculated from other pavement condition data are presented. Figure 1 illustrates the histogram of pavement condition data collected from the parallel tests.

Table 3.

Summary of Pavement Condition Data

Pavement condition data	Description	Statistics (Data set 1/Data set 2)
		Minimum	Median	Mean	Maximum
IRI, in./mile	Left side	21.30/21.40	63.60/63.30	71.98/72.16	358.50/401.60
	Right side	22.90/23.70	68.55/69.80	79.64/80.61	521.00/503.90
	Half car	18.9/19.30	53.6/54.90	61.82/62.96	352.70/312.10
Rutting, in.	Left side	0.018/0.021	0.117/0.127	0.127/0.140	0.643/0.675
	Right side	0.016/0.024	0.08/0.095	0.095/0.110	0.577/0.541
Fatigue cracking, %	Low severity level	0/0	1/1	14.98/15.16	100/100
	Moderate severity level	0/0	0/0	0.63/1.36	100/100
	High severity level	0/0	0/0	1.28/0.69	100/100
Longitudinal wheel-path cracking, %	Low severity level	0/0	1/0	1.28/1.44	57/65
	Moderate severity level	0/0	3/4	5.95/6.83	49/62
	High severity level	0/0	0/0	0.265/0.47	15/36
Longitudinal nonwheel-path cracking, %	Low severity level	0/0	1/0	1.77/1.60	60/64
	Moderate severity level	0/0	3/4	5.306/5.82	53/53
	High severity level	0/0	0/0	0.546/1.06	12/27
Block cracking, %	Low severity level	0/0	0/0	15.99/17.75	100/100
	Moderate severity level	0/0	0/0	6.64/5.93	96/91
	High severity level	0/0	0/0	0.35/0.22	46/31
Transverse cracking, count	Low severity level	0/0	1/1	3.36/3.40	100/100
	Moderate severity level	0/0	7/7	11.79/12.12	100/91
	High severity level	0/0	0/0	0.88/0.79	32/29
Cracking percent, %	Defined per HPMS field manual	0/0	6/7	12.7/13.62	62/63
PSI	Pavement smoothness index	0.477/0.601	3.466/3.457	3.357/3.348	4.413/4.401
PDI	Pavement distress index	0/0	3.345/3.235	3.454/3.342	4.999/4.949
PQI	Pavement quality index	0/0	3.254/3.255	3.39/3.313	4.692/4.658

Note: HPMS = highway performance monitoring system; IRI = international roughness index; PSI = pavement smoothness index; PDI = pavement distress index; PQI = pavement quality index.

Figure 1.

Histogram of pavement condition data from two testing devices.

Variability of Network-Level Data Collection

Comparison of Distributions of Pavement Condition Data

The purpose of network-level pavement evaluation is to provide a global view of pavement conditions over the entire network instead of for individual segments. The pavement condition distribution is of interest and importance as it may be used for budget allocation, strategic investment planning, performance target setting, and public reporting. Therefore, distributions of pavement condition data were evaluated using the Kolmogorov–Smirnov test (K-S test).

The K-S test is used to compare two data sets to determine whether they are significantly different. The K-S statistic, D_n,m, developed from the K-S test quantitively evaluated the distance between the cumulative distribution functions of the two data sets collected in the parallel tests. The K-S statistic is a supremum function, which is defined as

D_{n, m} = \sup | F_{1, n} (x) - F_{2, m} (x) |

where $F_{1, n} (x), F_{2, m} (x)$ are the distribution functions of the two data sets. The null hypothesis is that the two distributions are identical. A significance level of 0.05 was employed.

Table 4 lists the summary of the K-S tests for each data item collected from the parallel tests. A p-value < 0.05 indicated the difference of distributions of the longitudinal wheel-path cracks, nonwheel-path cracks, and rut depths between the two data sets were significant.

Table 4.

Results of the Kolmogorov–Smirnov Tests for the Parallel Tests

Pavement condition data		D statistics	p-value
Fatigue cracks	Low severity	0.039701	0.1292
	Moderate severity	0.039701	0.1292
	High severity	0.011507	0.9998
	Sum of low, moderate and high severity level	0.044879	0.06036
Block cracks	Low severity	0.035673	0.2187
	Moderate severity	0.024166	0.6905
	High severity	0.010357	1
	Sum of low, moderate and high severity level	0.023015	0.7467
Longitudinal wheel-path cracks	Low severity	0.44822	p < 0.05 ^*
	Moderate severity	0.19563	p < 0.05 ^*
	High severity	0.63464	p < 0.05 ^*
	Sum of low, moderate and high severity level	0.18182	p < 0.05 ^*
Longitudinal nonwheel-path cracks	Low severity	0.4183	p < 0.05 ^*
	Moderate severity	0.12025	p < 0.05 ^*
	High severity	0.60069	p < 0.05 ^*
	Sum of low, moderate and high severity level	0.078251	p < 0.05 ^*
Transverse cracks	Low severity	0.021289	0.8258
	Moderate severity	0.025316	0.6334
	High severity	0.014384	0.9938
	Sum of low, moderate and high severity level	0.044304	0.06599
Crack percent per HPMS field manual		0.032221	0.3277
IRI	Left wheel-path	0.044304	0.06599
	Right wheel-path	0.025892	0.6049
	Half car	0.029919	0.4181
Rut depth	Left wheel-path	0.06847	p < 0.05 ^*
	Right wheel-path	0.13636	p < 0.05 ^*

Note: HPMS = highway performance monitoring system; IRI = international roughness index. Bold in “p-value” column indicates a significant difference between two data sets.

Evaluation of Data Collection Variability

As the distributions of rut depth and longitudinal cracks between the two data sets were significantly different, further investigations were conducted to find out what might have caused this difference.

Rut Depth

Figure 2 illustrates the cumulative distributions of rut depth from the two data sets. As can be seen, the cumulative distribution curves of rut depth collected by the vendor were slightly higher than those collected by TDOT. However, the cumulative distribution curves from the two data sets were slightly closer to each other on the left than on the right wheel-path.

Figure 2.

Comparison of distribution of rut depth from two wheel paths (in.): (a) left rut depth, and (b) right rut depth.

It should be noted that the rut depth in this study was reported in 0.1-mi intervals. For each 0.1-mi segment, the rut depth was measured at 0.005-mi intervals and averaged for reporting. A total of 20 different cross-sections were used for calculating rut depth within 0.1-mi segments. For network-level data collection, it is difficult to ensure individual segment termini matching between different testing vehicles (i.e., 0.1-mi segment) because of the potential cumulative error from distance measuring instruments (DMIs; used to measure and record elapsed distance traveled by a vehicle). Further investigation may be needed to evaluate the influence of DMI measurements on the variability of rutting. As the evaluation of DMI measurements is beyond the scope of this study, no more discussion is provided in this paper.

Longitudinal Cracks

Figure 3 illustrates the cumulative distributions of longitudinal wheel-path and nonwheel-path cracks from the two data sets. As the cumulative distributions of cracks at different severity levels exhibited similar patterns, only the distributions of total cracking are illustrated. It was found that the cumulative distribution curves of both longitudinal wheel-path and nonwheel-path cracks collected by the vendor were slightly higher than those collected by TDOT.

Figure 3.

Comparison of distributions of longitudinal cracks (%): (a) total longitudinal wheel-path cracking, and (b) total longitudinal nonwheel-path cracking.

Potential contributors to differing distributions of longitudinal wheel-path and -nonwheel-path cracking are summarized in the following.

Longitudinal lane joints were rated as longitudinal nonwheel-path cracks

In Tennessee, longitudinal lane joints are rated separately, not factored into calculations of pavement condition indices. As illustrated in Figure 4, the longitudinal lane joints might have been rated as longitudinal nonwheel-path cracks as the location of these lane joints were generally close to nonwheel-path zones, or Zone 1 or Zone 5 as per AASHTO R85 “Standard Practice for Quantifying Cracks in Asphalt Pavement Surfaces from Collected Pavement Images Utilizing Automated Methods.” Once covered by nonwheel-path zones, longitudinal lane joints are generally rated as nonwheel-path cracking by automated imaging process systems. Currently, there is no other way of identifying this type of error than by reviewing the downward pavement images manually.

Figure 4.

Longitudinal lane joints rated as longitudinal nonwheel-path cracks.

Lateral wandering during data collection

Lateral wandering, which results in incomplete coverage of lane width, may contribute to data variability. Figure 5 illustrates downward images collected from two testing vehicles. The surface distresses for this segment indicated that there was a significant difference in fatigue cracking and longitudinal wheel-path and -nonwheel-path cracking between the two testing vehicles. As illustrated in Figure 5a, owing to lateral wandering, the full lane was not captured in the downward image. Some surface information was also missing on the left side of the lane where excessive distresses can be observed (Figure 5b).

Figure 5.

Incomplete coverage of lane width because of lateral wandering: (a) downward image from Vehicle #1, and (b)downward image from Vehicle #2.

Variations of AASHTO measurement zones

As per AASHTO PP67, five zones across a pavement section are defined to determine nonwheel-path and wheel-path areas. Zone 1 in Figure 6a includes areas where pattern cracks were observed, whereas Zone 1 in Figure 6b excluded the pattern cracks. As per AASHTO PP67, widths and locations are only specified for Zones 2, 3, and 4. The widths for Zones 1 and 5 mainly depend on the lane boundary. For automated methods, the lane boundaries are generally determined based on the pavement markings or surface features. Lane boundaries may be subject to change where there is a change in surface features. Figure 6 provides an example in which the lane boundaries were different according to the two testing vehicles. Zone 1 in Figure 6a appears to be wider than in Figure 6b. As a result, the distress quantities in Figure 6a were higher than in Figure 6b.

Figure 6.

Variation of AASHTO measurement zone: (a) downward image from Vehicle #1, and (b) downward image from Vehicle #2.

Classification Models

Owing to collection variability, the final performance ratings based on the pavement condition data from the two data sets could be different. Therefore, RF classification models were constructed to identify the segments whose ratings may be subject to data collection variability.

Random Forest Classification Model

As one of the most powerful machine learning technologies, the RF method has been used in a wide range of applications. In recent years, RF modeling has been successfully used in pavement management activities. The RF method is based on classification and a regression tree algorithm that is used for growing decision trees for the “forest.” The term “random” means the samples used for growing a tree are randomly selected. A subset of samples is randomly generated—also called “bootstrapping process” or “row subsampling.” The number of independent variables is also randomly selected using “column subsampling.” Through row and column subsampling methods, the risk of over fitting can be reduced and thus model performance can be improved.

The RF algorithm is designed to grow a series of decision trees by selecting several splits such that the training data set is split into several subsets. To grow a single decision tree, a series of nodes are determined by minimizing the difference within each subset and maximizing the difference among subsets. Impurity measures are typically used for quantitatively determining the degree of impurity which describes the heterogeneity of a dataset. For categorical dependent variables, the Gini index is commonly employed. The Gini index was determined by Equation 1.

G (t) = 1 - \sum_{i = 1}^{n} p^{2} (i | t)

(1)

where

i = target categorical variable,

p(.) = proportion of observations in the leaf, and

t = split.

For a certain split, a data set can be divided into left node, t_L, and right node, t_R. Then the decrease in the Gini index, which describes the impurity of a data set, can be written as Equation 2. The split of a node in a decision tree is determined by finding the splitting value that can maximize D(s,t).

D (s, t) = G (t) - P_{L} G (t_{L}) - P_{R} G (t_{R})

(2)

where

s = split on decision node,

$G (t_{L})$ = Gini index from the left child node,

$G (t_{R})$ = Gini index from the right child node,

P_{L} = \frac{number of samples in left node}{number of samples in dataset},

and

P_{R} = \frac{number of samples in right node}{number of samples in data set} .

Consistency of Parallel Data Collection

A segment in the road network may be incorrectly rated because of measurement variability. Figure 7 provides an example of determining the probability of a segment being incorrectly rated owing to data variability. For example, Segment #1 was initially rated Good in relation to the threshold of “Good/Fair.” However, the gray area under the curve indicates the probability that this segment may be rated as Fair. Similarly, there was a probability that Segment #2 might be incorrectly rated as Fair as indicated in the gray area under the Segment #2curve. As constructing probabilistic curves is very costly and time-consuming, parallel testing was employed to evaluate network-level data collection variability. The same road segments were tested and rated separately by two vehicles under the same testing and surface conditions. Generally, ratings from the two vehicles were the same for the same segment. However, there were also some segments for which ratings from the two vehicles were different. The segments with inconsistent ratings from the two testing vehicles were of interest because these segments could potentially influence network-level performance ratings and annual maintenance work plans. On identification of these segments, one might be able to focus on those that are subject to data variability. For example, segments initially rated Good may actually be in Fair condition because of data variability. However, if these segments are still in the early stages of Fair condition, pavement preservation may be an option. Conversely, there might be a few segments that are in or close to Poor condition rated as Fair. The application of pavement preservation to those segments would not be a cost-effective solution.

Figure 7.

Variability of performance ratings.

In the parallel tests, individual segments were classified into three consistent and two inconsistent groups. The three consistent groups (GG, FF, and PP) included all the segments with the same ratings from the two testing vehicles. GG means the segment was rated as Good by both vehicles; FF means the segment was rated as Fair by both vehicles; and PP means the segment was rated Poor by both vehicles. The two inconsistent groups, meaning the segment may have been subject to collection variability, were GF and FP: the segment was rated as Good and Fair or as Fair and Poor, respectively, by the two vehicles.

Model Input Variables and Parameters

The input variables for the classification models are listed in Table 5. The dependent (categorical) variables mirrored the five aforementioned groups: three levels indicating consistent ratings between testing vehicles (GG, FF, and PP) and two levels indicating inconsistent ratings (GF and FP).

Table 5.

Sampling of Imbalanced Data Sets

Performance measures	Sample rates for each group rate before resampling (rates after resampling) (%)
	FF	FP	GF	GG	PP
IRI	14.52 (20.01)	0.69 (20.01)	3.74 (19.95)	78.92 (20.01)	2.13 (20.01)
Rut depth	5.01 (25.01)	NA	2.25 (24.96)	92.34 (25.01)	0.40 (25.01)
Crack percent	22.72 (20.00)	3.83 (20.00)	3.59 (20.00)	43.42 (20.00)	26.43 (20.00)
Overall rating	55.70 (20.03)	2.88 (19.98)	4.89 (19.98)	37.92 (20.03)	1.21 (19.98)
PQI (interstate)	63.69 (20.01)	2.65 (20.01)	6.28 (19.95)	25.07 (20.01)	2.31 (20.01)
PQI (state routes)	51.41 (20.01)	2.65 (20.01)	4.21 (19.95)	39.42 (20.01)	2.31 (20.01)

Note: IRI = International Roughness Index; PQI = pavement quality index; FF = the segment was rated as “Fair” by both vehicles; FP = the segment was rated as "Fair" by one vehicle and “Poor” by another; GF = the segment was rated as “Good” by one vehicle and “Fair” by anthor; GG = the segment was rated as “Good” by both vehicle; PP = the segment was rated as “Poor” by both vehicles; NA = not available.

The model input variables (predictors) included roughness data (IRI and rut depth); surface distresses (fatigue cracks, longitudinal wheel-path and nonwheel-path cracks, transverse cracks, block cracks, and cracking percent); and pavement performance indices (PSI, PDI and PQI).

The two parameters for the RF model were number of trees and number of variables in the model. Generally, as the number of trees in a model increases, the accuracy of the prediction increases. However, many trees may result in a significant increase in computation time without improvements in model accuracy. When developing an RF model, the errors in each group need to be balanced. Based on the results from trial tests, the number of trees for this study was determined to be 500. The number of variables was used to specify how many variables (serving as predictors) were to be used for splitting for each “tree” in the “forest.” The default value for the number of variables in a classification tree is the square root of the total number input variables. Based on the results from trial tests, the number of variables was determined to be five. The variables in the tree were randomly selected by the algorithm.

Resampling of Imbalanced Data Sets

A data set is considered to be imbalanced when there are disproportionate ratios of observations in each class ( 20 ), that is, the number of examples representing certain classes is much lower than others. In this study, the numbers of segments in group GF, FP, and FF were significantly lower than those in group GG and FF. It should be noted that models established on imbalanced classes may result in a bias toward the majority class. Like other machine learning models, the RF algorithm is designed to maximize accuracy and minimize error. Therefore, the final model may increase the number of false negatives, which means the model may not perform well in identifying segments with inconsistent ratings.

It was necessary to resample the data set for the number of false negatives in the models to be reduced. The resampling method used in this study was synthetic minority over-sampling technique (SMOTE). SMOTE generates duplicates for existing minority groups through the “k nearest neighbors” method. Duplicates are interpolated in the nearest neighbors, determined by the Euclidean distance between the data points. Table 5 presents the results of the resampling of the data sets used for the RF models. Note that there were only four classes for rut depth with no samples for the FP class. This was because rut depth for the majority of segments was in Good condition, and the proportion of Poor condition segments was less than 0.5%. Therefore, the average rate after resampling for rut depth was 25%. The average rates for other groups after resampling were 20%.

Training and Validation of Classification Models

The performance of classification models was evaluated by model statistics, such as “sensitivity,”“specificity,” and “accuracy.” In binary classification, the metric “sensitivity” is used to evaluate the model’s ability to predict true positives, whereas “specificity” is used to evaluate the model’s ability to predict true negatives in each category. “Accuracy” is the ratio of correct predictions to the total number of predictions. Generally, the higher the accuracy, the better the model performs. In this study, multiclass classification models were constructed. These model statistics were calculated by a one-to-all approach. For example, when the metrics for GG are calculated, true positive means all the GG group is classified as GG; true negative means all non-GG cases in the group are classified as non-GG; false positive means all non-GG are classified as GG; and false negative means all GG are classified as non-GG.

To construct classification models, the resampled data set was split into two subsets for each model with 70% of the samples used as the training set and 30% as the testing set. Table 6 presents the summary statistics for each model. From the results in Table 6, it can be observed that the overall accuracy for all classification models was greater than 0.9, which means the models generally performed well and may be used to preliminarily identify those segments subject to variability of performance ratings. For national performance measures, it was found that the overall accuracy of the classification model for rut depth was the highest, followed by IRI and overall rating, whereas the overall accuracy of the crack percent model was lowest (0.92). For the TDOT indices, PQI (state route) model accuracy was slightly higher than that of PQI (interstate). The positive predictions for FP and GF groups were of interest because the two groups were used to identify those segments subject to inconsistent ratings. As can be seen from Table 6, the models generally performed better in classifying the FP than the GF group, except for the classification model for crack percent. It should also be noted that, owing to the limited FP group data, the classification model for rut depth was not capable of identifying segments for this group. However, this model may be improved when more data are available in the future.

Table 6.

Summary of Model Statistics

Rating systems	Model types	Model statistics	By class
			FF	FP	GF	GG	PP
National performance measures	IRI	Sensitivity	0.9444	1.0000	0.9192	0.9348	1.0000
		Specificity	0.9833	1.0000	0.9744	0.9954	1.0000
		Positive prediction value	0.9358	1.0000	0.8922	0.9773	1.0000
		Negative prediction value	0.9857	1.0000	0.9812	0.9864	1.0000
		Balanced accuracy	0.9639	1.0000	0.9468	0.9651	1.0000
		Overall accuracy	0.9621
		95% CI	(0.9421, 0.9767)
	Rut depth	Sensitivity	0.9931	NA	0.9847	0.9474	1.0000
		Specificity	0.9922	NA	0.9874	0.9976	1.0000
		Positive prediction value	0.9795	NA	0.9627	0.9908	1.0000
		Negative prediction value	0.9974	NA	0.9949	0.9857	1.0000
		Balanced accuracy	0.9926	NA	0.9861	0.9725	1.0000
		Overall accuracy	0.983
		95% CI	(0.9679, 0.9922)
	Crack percent	Sensitivity	0.9121	0.8889	0.9292	0.9434	0.9231
		Specificity	0.9724	0.9742	0.9806	0.9881	0.9853
		Positive prediction value	0.8737	0.8889	0.9292	0.9524	0.9474
		Negative prediction value	0.9814	0.9742	0.9806	0.9857	0.9782
		Balanced accuracy	0.9423	0.9316	0.9549	0.9657	0.9542
		Overall accuracy	0.9202
		95% CI	(0.8936, 0.9418)
	Overall rating	Sensitivity	0.9785	1.0000	0.91	0.8505	0.9911
		Specificity	0.9931	1.0000	0.958	0.9834	1.0000
		Positive prediction value	0.9681	1.0000	0.8349	0.9286	1.0000
		Negative prediction value	0.9954	1.0000	0.9786	0.9629	0.9976
		Balanced accuracy	0.9858	1.0000	0.934	0.9169	0.9955
		Overall accuracy	0.9471
		95% CI	(0.9244, 0.9645)
TDOT indices	PQI (interstate)	Sensitivity	0.9348	0.9655	0.8889	0.8426	1.0000
		Specificity	0.9931	0.9903	0.9557	0.9738	0.9976
		Positive prediction value	0.9663	0.9655	0.8224	0.8922	0.9912
		Negative prediction value	0.9863	0.9903	0.9739	0.9601	1.0000
		Balanced accuracy	0.9640	0.9779	0.9223	0.9082	0.9988
		Overall accuracy	0.9280
		95% CI	(0.9026, 0.9486)
	PQI (state routes)	Sensitivity	0.8913	0.9741	0.8990	0.9352	0.9912
		Specificity	0.9908	0.9854	0.9720	0.9786	1.0000
		Positive prediction value	0.9535	0.9496	0.8812	0.9182	1.0000
		Negative prediction value	0.9774	0.9927	0.9766	0.9833	0.9976
		Balanced accuracy	0.9411	0.9798	0.9355	0.9569	0.9956
		Overall accuracy	0.9413
		95% CI	(0.9177, 0.9598)

Note: TDOT = Tennessee Department of Transport; IRI = International Roughness Index; PQI = pavement quality index; 95% CI = 95% confidence interval; FF = the segment was rated as “Fair” by both vehicles; FP = the segment was rated as “Fair” by one vehicle and “Poor” by another; GF = the segment was rated as “Good” by one vehicle and “Fair” by anthor; GG = the segment was rated as “Good” by both vehicle; PP = the segment was rated as “Poor” by both vehicles; NA = not available.

Influence of Data Variability on Performance Measures

The classification models developed were applied to 2019 pavement condition data to identify segments whose performance rating was subject to change owing to data variability. To investigate the influence of data variability on performance measures, adjustments to performance ratings were applied, as illustrated in Figure 8. For each segment, the ratings adjustment was made in relation to the original rating and the class determined based on classification models. There are different ways to adjust the performance ratings when considering the influence of data variability on pavement evaluation. In this study, adjustments were made to ensure that no segment missed the optimal timing for pavement preservation. For example, if preventive maintenance is recommended on segments in the early stages of a Fair condition, segments close to Fair may be excluded from preventive maintenance programs if they are rated as Good owing to data variability.

Figure 8.

Adjustment of performance ratings by class.

Figure 8 illustrates how performance ratings were adjusted in this study.

For segments with an original rating of Good, there were three possible classes: GG, GF, and FF. Both GF and FF were adjusted to Fair given they may be close to a Fair condition.

There were five classes for segments originally rated as Fair. Segments in Group GG were adjusted to Good because they were close to a Good condition. Similarly, those in Group PP were adjusted to Poor.

There were three classes for segments originally rated as Poor. Segments in Groups FF and PF were adjusted to Fair condition.

In Figure 8, one might find that there are some scenarios for which the original rating was different from the class, for example, an original rating of Good with a class of FF, an original rating of Fair with class of PP, and so forth. These segments were also subject to data collection variability. For example, a segment with an IRI of 93 in./mi, a rut depth of 0.18 in., and a crack percentage of 4% will be rated as Good as per the definition in Table 2. However, based on the RF model, the condition of this segment was closer to Fair. Therefore, the original rating could have been incorrectly determined.

Figure 9 illustrates the comparison of original and adjusted performance ratings. Figure 9, a–d, illustrate the changes of performance ratings by national metrics, whereas Figure 9e illustrates the changes by PQI-based TDOT measures.

Figure 9.

Influence of rating adjustments on final ratings: (a) ratings by IRI, (b) ratings by rut depth, (c) rating by cracking percent, (d) overall ratings, and (e) PQI ratings.

For national performance measures, Figure 9a illustrates the change in percentage of performance measures by IRI. There was a slight decrease in percentage of Good and Poor conditions for both interstates and state routes, whereas the percentage of Fair condition increased. In general, the influence of rating adjustment on performance measures for state routes was more significant than for interstate. Figure 9b illustrates the percentage change in performance measures by rut depth. There was little change of percentage in the Poor condition from rating adjustments. The main reason was because the rut depths for the majority of the segments were significantly lower than the threshold of the “Fair/Poor” condition. In addition, the percentage change of the “Good/Fair” condition for state routes was higher than for the interstates. Figure 9c illustrates the change of percentage for performance measures by cracking percent. There was a slight change in percentage of Poor condition by rating adjustments. Figure 9d illustrates the change in percentage of performance measures by overall rating: change of “Good/Fair/Poor” for overall ratings was generally lower than individual measures, which means data variability of a single metric (i.e., crack percent) may not necessarily increase the variability of overall ratings.

For PQI, there was little influence of rating adjustments on percentage of Poor segments for interstate. There was a significant increase in the percentage of Fair condition for state routes (10%), whereas there was a slight decrease in the percentage of Poor. From the perspective of a budget allocation, if the segments in Poor condition are considered candidates for major rehabilitation and those in Fair condition for preservation and minor rehabilitation, adjustments on performance ratings could slightly shift budgetary priorities from major rehabilitation to a preservation/minor rehabilitation program.

It should be noted that depending on the rule applied (see Figure 8), the adjusted rating may be subject to change. The rules illustrated in Figure 8 were established to ensure that no segments missed the window of opportunity for preservation. Therefore, segments that may be inconsistently rated as Good or Poor were adjusted to Fair, because “False Good” segments may miss the timing for preservation, whereas “False Poor” segments may be subject to unnecessary rehabilitation. If the data are used for other purposes, such as performance target setting and long-term prediction, other rules may be applied. For example, to identify all the potential segments in Good condition, one could keep all the segments with an original rating of Good unchanged and adjust those in the GF class with an original rating of Fair to Good. Similarly, the FP class with an original rating of Fair may be changed to Poor with all the Poor segments unchanged. The segments in GF/FG and FP/PF classes are subject to change because of collection variability. Evaluating potential data variability is crucial for state agencies to set reasonable performance targets and interpret the performance measures by identifying these segments. It should also be noted that adjusting the definitions of Good, Fair, and Poor could also affect the results.

Conclusions and Recommendations

This paper presented a study in which the variability of network-level pavement condition data were evaluated. Network-level parallel tests were conducted on selected routes to gather raw data that were then used for training RF classification models to identify segments with inconsistent performance measures. The models were applied to a road network to quantify the influence of data variability on pavement condition evaluation. The case examples provided in this paper will be of assistance to SHAs to evaluate the influence of data collection variability on network-level pavement condition ratings and decision-making. Some conclusions can therefore be made as follows.

From the parallel tests it was observed that the distributions of longitudinal wheel-path cracks, nonwheel-path cracks, and rut depth from the two data sets were significantly different from each other.

The reasons contributing to the variability in longitudinal cracks were summarized as (1) longitudinal lane joints rated as longitudinal nonwheel-path cracks; (2) lateral wandering; and (3) variations in lane measurement zones.

The influence of data variability on condition evaluation for state routes was more significant than that for interstates.

The variability of an individual metric (i.e., crack percent) may not necessarily increase the variability of overall ratings, which are the combination of all three metrics (cracking percent, rut depth, and IRI).

Some recommendations are as follows:

Further investigations are needed to evaluate the influence of DMI variability on the distribution of rut depth. Certification of transverse profiling systems is also recommended to reduce rut depth measurement uncertainty.

Methods for adjusting the performance ratings could vary depending on the agencies’ priorities when undertaking pavement evaluation. In this study, adjustments were made to ensure that no segment missed optimal timing for preservation and no segments were subject to unnecessary rehabilitation. However, were the purpose of pavement evaluation to set performance goals, different rating adjustment rules would be considered and applied in the stage depicted in Figure 8. Accordingly, the results illustrated in Figure 9 could be quite different.

Footnotes

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: X. Jia, M. Woods, B. Huang; data collection: X. Jia, D. Zhu; analysis and interpretation of results: X. Jia, H. Gong, W. Hu; draft manuscript preparation: X. Jia, M. Woods, D. Zhu. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

The contents of this paper reflect the views of the authors, who are responsible for the facts and the accuracy of the data presented here, and do not reflect the views of Tennessee Department of Transportation. The contents do not constitute a standard, specification, or regulation.

References

Huang

Pavement Analysis and Design, 2nd ed. Pearson Prentice Hall, Upper Saddle River, NJ, 2004.

International Organization for Standardization. International Vocabulary of Metrology. International Organization for Standardization, Geneva, Switzerland, 2008.

Livneh

Repeatability and Reproducibility of Manual Pavement Distress Survey Methods. Proc., 3rd International Conference on Managing Pavements, San Antonio, TX, 1994.

Flintsch

Ghee

Mc K.

NCHRP Synthesis 401: Quality Management of Pavement Condition Data Collection. Transportation Research Board, Washington, D.C., 2009.

Jia

Woods

Gong

Huang

Faraj

Utilization of State Performance Indices to Correlate National Performance Measures for Asphalt Pavements in Tennessee. Transportation Research Record: Journal of the Transportation Research Record, 2019. 2673: 379–388.

Pierce

McGovern

Zimmerman

. Practical Guide for Quality Management of Pavement Condition Data Collection. Report No. FHWA-HIF-14-006. Federal Highway Administration, Washington, D.C., 2013.

Landers

Bekheet

Falls

Cohen’s Weighted Kappa Statistic in Quality Control–Quality Assurance Procedures: Application to Network-Level Contract Pavement Surface Condition Surveys in British Columbia, Canada. Transportation Research Record: Journal of the Transportation Research Board, 2003. 1860: 103–108.

Ong

Noureldin

Sinha

Methodology to Evaluate Quality of Pavement Surface Distress Data Collected by Automated Techniques. Transportation Research Record: Journal of the Transportation Research Board, 2009. 2093: 3–11.

Bogus

Migliaccio

Cordova

Assessment of Data quality for Evaluations of Manual Pavement Distress. Transportation Research Record: Journal of the Transportation Research Board, 2010. 2170: 1–8.

10.

Rada

G. R.

Bhandari

Elkins

Bellinger

Assessment of Long-Term Pavement Performance Program Manual Distress Data Variability: Bias and Precision. Transportation Research Record: Journal of the Transportation Research Board, 1997. 1592: 151–168.

11.

Rada

Elkins

Bhandari

Bellinger

Update of Long-Term Pavement Performance Manual Distress Data Variability: Bias and Precision. Transportation Research Record: Journal of the Transportation Research Board, 1998. 1643: 71–79.

12.

Ong

Sinha

Evaluating International Roughness Index Data Quality at Project and Network Levels. Proc., Airfield and Highway Pavements: Efficient Pavements Supporting Transportation’s Future, Bellevue, Washington, D.C., 2008, pp. 385–396.

13.

Siabil

Gharaibeh

Assessing the Effect of Considering Multiple Data Properties on Detecting Potential Errors in Pavement Condition Data. Transportation Research Record: Journal of the Transportation Research Board, 2017. 2639: 39–45.

14.

Siabil

Gharaibeh

Computational Technique for Detecting Errors in Network-Level Pavement Condition Data. Transportation Research Record: Journal of the Transportation Research Board, 2016. 2589: 14–19.

15.

Yin

Stoffels

Antle

Profile Data Variability in Pavement Management: Findings and Tools from LTPP. Proc., Airfield and Highway Pavement: Meeting Today’s Challenges with Emerging Technologies, Atlanta, GA, 2006, pp. 984-995.

16.

Saliminejad

Gharaibeh

N. G.

Impact of Error in Pavement Condition Data on the Output of Network-Level Pavement Management Systems. Transportation Research Record: Journal of the Transportation Research Board, 2013. 2366: 110–119.

17.

Jia

Huang

Zhu

Dong

Woods

Influence of Measurement Variability of International Roughness Index on Uncertainty of Network-Level Pavement Evaluation. Journal of Transportation Engineering, Part B: Pavements, Vol. 144, No. 2, 2018, p. 04018007.

18.

Jia

Huang

Dong

Zhu

Maxwell

Influence of Pavement Condition Data Variability on Network-Level Maintenance Decisions. Transportation Research Record: Journal of the Transportation Research Board, 2016. 2589: 20–31.

19.

Calculation of Performance Management Measures. Code of Federal Regulations, Title 23 § 490.313. https://ecfr.federalregister.gov/.

20.

López

Fernández

García

Palade

Herrera

An Insight into Classification with Imbalanced Data: Empirical Results and Current Trends on Using Data Intrinsic Characteristics. Information Sciences, Vol. 250, 2013, pp. 113–141. https://doi.org/10.1016/j.ins.2013.07.007.