State Transportation Planning: Linking Quantifiable Performance Measures and Infrastructure Condition Outcomes

Abstract

This study focuses on whether state Departments of Transportations (DOTs) with quantifiable performance targets have better road repair and maintenance outcomes than DOTs without quantifiable performance targets. We also examine state DOT use of quantifiable pavement condition targets to examine differences across states in terms of agency and governance/legislative factors. While the results are mixed, they suggest that state agency capacity may be related to the use of concrete and specific performance measures. Furthermore, these concrete and specific performance measures may be related to the use of external accountability bodies and citizen participation. This suggests that quantifiable performance targets may be a useful tool for decision making, planning, and resource allocation in pursuit of desired performance outcomes.

Keywords

infrastructure capital budgeting condition performance and demand maintenance strategies resources management practices performance

Introduction

Deteriorating transportation infrastructure has been a particular challenge for the states as demand on the transportation system has outpaced resources. The American Society of Civil Engineers (ASCE) in its 2013 report card assigned a grade of D+ for America’s infrastructure, specifically assigning grades of D for roads and transit, and C+ for bridges (ASCE, 2013). States have consistently invested disproportionately in road expansion while underfunding repair and preservation (Smart Growth America and Taxpayers for Common Sense, 2011). For example, between 2004 and 2008, the states spent US$22 billion to add 23,300 lane-miles of major roads, but spent only US$16 billion for repair and preservation of the 1.9 million lane-miles of existing major roads (Smart Growth America and Taxpayers for Common Sense, 2011). This decision to defer repair and preservation has resulted in infrastructure deterioration, which has become a large and growing financial liability. According to projections, states would need to spend US$45 billion annually beginning in 2013 and for the next 20 years to improve and maintain road conditions, which is up from US$43 billion estimate in 2010 (Smart Growth America and Taxpayers for Common Sense, 2011, 2014). In contrast, states only spent US$17 billion annually for repair and preservation between 2009 and 2011.

Adding to the complexity of the environment is the federal transportation legislation Moving Ahead for Progress in the 21st-Century Act (MAP-21), which was enacted in 2012. One of the purposes of MAP-21 is to move the surface transportation programs toward a performance outcomes approach (U.S. Government Accountability Office, 2015). Prior to this legislation, federal funding of state transportation was not linked to performance-based outcomes, but based instead on funding formulas that were largely not performance-related. Under MAP-21, states are required to collect data on and compare outcomes with established targets. However, states reportedly are challenged by the new requirements due to the availability of data and the resources needed to collect and analyze the data.

According to goal setting theory, performance measurement and management requires agreement on goals and strategies, involves development of a measurement system to document performance and support decision making, and ends with the use of performance information for managing programs and agencies, improving accountability to stakeholders, demonstrating performance, and supporting resource allocation and decision making (Behn, 2003; Hatry, 2006; Heinrich, 2002; Poister, 2008; U.S. General Accounting Office, 1996; Wholey, 1999). According to Yusuf and Leavitt (2014), “Nowhere is performance measurement and management more relevant than in the areas of public works and public infrastructure.” (p. 206) The utility of performance measurement and management is further underscored by Plant (2007), who asks “How best can transportation systems be managed? How is performance defined, made operational, and measured in transportation?” (p. 5). Performance measurement and management are particularly relevant to transportation planning, as transportation projects are visible and tangible to citizens, have long lives and long-term impacts, often require significant financial investments or commitments, and can span multiple jurisdictions and affect a large service delivery area (Yusuf & Leavitt, 2014).

Given the unique challenges that states face in transportation planning and the issues related to performance measurement and management for such planning, this study asks two research questions.

Research Question 1: As a test of goal setting theory, does use of quantifiable performance targets make a difference in transportation outcomes as measured by pavement condition and repair and maintenance spending?

Then, we focus on the agency, legislative, and governance factors that may be associated with the use of quantifiable performance targets, which could subsequently influence outcomes. Therefore, our second question is as follows:

Research Question 2: How are state Departments of Transportations (DOTs) that use quantifiable performance targets different from those DOTs that do not?

In the next section, we provide a discussion of goal setting theory in general and specifically applied to transportation planning. Then, we discuss the factors that may relate to the use of quantifiable performance measurement. Finally, we review the methods of analysis, present results, and discuss implications in the concluding remarks.

Goal Setting Theory: Linking Performance to Outcomes

As a structural policy¹ to address accountability, performance measurement and management systems are expected to improve transportation outcomes. Performance measurement provides managers and decision makers with information regarding performance that can be used to increase effectiveness by redirecting resources and adjusting operations and service delivery to produce better results (Poister, 2008). If considered in the context of goal setting theory, performance measurement provides information useful for setting goals and priorities. Goal setting theory assumes a direct linkage between having specific, measurable goals, on one hand, and performance on the other (Locke & Latham, 2002; Rodgers & Hunter, 1992; Verbeeten, 2008). Through performance measurement and management, agencies clarify and agree on goals and outcomes, systematically monitor outcomes to generate information for use in managing and decision making, and ultimately improve performance (Behn, 2003; Hatry, 2006; Marr, 2009; Poister, 2008; Poister, Pasha, & Edwards, 2013; Walker, Damanpour, & Devece, 2011; Wholey & Hatry, 1992). By emphasizing goal setting, the performance measurement and management system focuses resources away from goal-ambiguous activities toward goal-relevant efforts (Latham, 2004), keeps the agency focused on outcomes and results, and thus leads to improved performance (Ammons & Rivenbark, 2008; Kelly, 2003; Poister, 2008; Van Dooren, Bouckaert, & Halligan, 2015).

Empirical research provides support for the goal setting theory and, particularly, the importance of having quantifiable measures that can be held against a specific target. For example, Verbeeten (2008) found empirical support that specifying clear and measurable goals provides a managerial and operational focus for decision making, and improves performance. In another example, Boyne and Chen (2007) found use of objective measures of performance coupled with performance targets significantly increased educational outcomes. In the case of transit, Poister et al. (2013) found a positive relationship between performance measurement and outcomes such as passenger trips.

Linking Transportation Performance to Transportation Outcomes

Performance measurement and management requires having standards or targets as “some kind of desired result with which to compare . . . and thus judge performance” (Behn, 2003, p. 598). Performance measurement and management “are being used increasingly to establish the criteria for transportation system plans, as well as for subsequent decisions about preserving existing assets and programming projects to advance those plans” (Poister, 2007, p. 498). The last decade has seen several studies of and reports on performance measurement and management in the transportation area.² Three recent studies (Pew Center on the States and The Rockefeller Foundation, 2011; Smart Growth America and Taxpayers for Common Sense, 2011; U.S. Government Accountability Office, 2010) are particularly illuminating for understanding performance measurement and management practices of state DOTs.

The Pew Center on the States and the Rockefeller Foundation (2011) conducted an assessment of whether the states and their DOTs have goals, performance measures, and data that are useful for policy makers to make decisions on transportation investments. The assessment focused on whether the states “have the essential tools in place to help them understand if and to what degree they are making progress” (p. 86). Accordingly, the Pew/Rockefeller research team rated the states according to three groups (leading the way, mixed results, trailing behind) based on meeting 10 criteria that reflect key elements of performance measurement and components of a good transportation management system.³ Overall, 26% of the states were rated as “leading the way,” while 36% and 38% were rated as “mixed results” and “trailing behind,” respectively.

In a survey of transportation planning processes, the U.S. Government Accountability Office (2010) assessed state DOT long-range transportation planning practices. The study found that almost all state DOTs include goals and objectives in their long-range statewide transportation plans. In addition, it found that only 18 states reported including quantitative performance targets in their transportation plans. In response to this finding, the U.S. Government Accountability Office noted that it recommends agency performance goals and measures to include a quantifiable, numerical target, and that “performance targets within long-range statewide transportation plans could provide a performance standard by which the state DOT can demonstrate to the public what effect its investment decisions are having on achieving the goals established in the plan” (p. 15).

In a similar study, Smart Growth America and Taxpayers for Common Sense (2011) jointly reviewed states’ use of performance measurement related to pavement condition to examine whether state DOTs have performance goals for maintaining their road assets in certain conditions and to assess the stringency of the targets the states have established. They found that 40 states have quantifiable pavement condition targets, and four states (Alabama, Connecticut, Oklahoma, and South Carolina) have targets that are neither specific nor quantifiable. The remaining states (Arkansas, Maine, Mississippi, New Hampshire, Rhode Island, and West Virginia) do not have targets or benchmarks for pavement conditions.

State governments have used various performance measures and indicators, depending on the different functions and activities and the multiple interests of stakeholders (National Conference of State Legislatures [NCSL], 2008). Even within state DOTs, the variety of functions and activities require a wide range of performance measures. A multitude of different performance categories and measures have been recommended for and used by state DOTs. For example, the Transportation Research Board (National Cooperative Highway Research Program, 2006) suggested asset management-related performance measures in several categories such as asset preservation, operations and maintenance, mobility, and accessibility. The Pew Center on the States and the Rockefeller Foundation (2011) focused on six categories of performance: safety, jobs and commerce, mobility, access, environmental stewardship, and infrastructure preservation. The recent federal legislation, MAP-21, emphasized the following national transportation goals: safety, infrastructure condition, congestion reduction, system reliability, freight movement and economic vitality, environmental sustainability, and reduced project delivery days. Infrastructure condition and related maintenance and preservation issues are consistently identified as performance categories for transportation agencies. Table 1 lists infrastructure condition performance measures that have been identified as relevant for state DOTs.

Table 1.

Performance Measures for Infrastructure Condition.

International Roughness Index measure of surface bumpiness
Pavement Serviceability Rating (subjective evaluation of ride quality)
Rideability index for pavements
Bridge health index
% of deficient bridges
Lane-miles in good condition (or lane-miles in poor condition)
Lane-miles in good condition as a % of total (or lane-miles in poor condition as a % of total)
% of rural roads with narrow lanes
Deferred maintenance costs.
Lane-miles per centerline-mile

Source. American Association of State Highway and Transportation Officials Center for Excellence in Project Finance (2010), National Cooperative Highway Research Program (2006), Pew Center on the States and The Rockefeller Foundation (2011), Hartgen et al. (2014), Smart Growth America and Taxpayers for Common Sense (2011).

In this study, we focus on state DOT use of quantifiable performance targets because standards or targets are critical elements of performance measurement and management (Behn, 2003; Boyne & Chen, 2007; U.S. General Accounting Office, 1996; Wholey, 1999; Yusuf & Leavitt, 2014). Recent reports by Smart Growth America and Taxpayers for Common Sense (2011, 2014) have emphasized the need to focus on investing in road repair and preservation projects because states have disproportionately focused on expansion projects at the cost of regular maintenance and repair. On average, 52% of state-owned roads were in poor or fair condition in 2008, compared with 61% in 2011 (Smart Growth America and Taxpayers for Common Sense, 2011, 2014). Therefore, we focus on outcomes related to the condition of transportation infrastructure and meeting repair and preservation spending needs. For the former, we focus on the percent of roads categorized as being in poor condition. For the latter, we measure maintenance spending by the state DOT. States with more extensive performance measurement and management systems are expected to have higher quality road infrastructure (in terms of pavement conditions) and spend more on maintenance to meet repair and preservation needs.

Factors Associated With Performance Measurement and Management

State DOTs have used performance measurements “to help forecast and track the impacts of program investments, maintenance, and operations improvements; monitor the condition of system assets; and gauge the management and service delivery of the agency” (National Cooperative Highway Research Program, 2006, p. 10). The emphasis on performance has been driven both by DOT management and legislative pressures. As noted by Rall, Wheet, Farber, and Reed (2011), some state DOTs have been encouraged or required to move toward performance management by legislative directive. In Washington state, legislation was enacted in 2005 that required all state agencies, not just the state DOT, to develop and implement a performance-based system to address quality and accountability (Rall et al., 2011). In other states, such as Maryland, Minnesota, and Nevada, legislation have been introduced that specifically targeted state DOT performance (Rall et al., 2011).

Therefore, we would expect both agency and legislative factors to be associated with state DOT use of performance measurement and management. This categorization of factors is fairly consistent with what de Lancer Julnes and Holzer (2001) structure as rational/technocratic factors versus political/cultural factors. Specifically, we posit that two categories of factors are related to state DOT use of performance measurement and management for transportation planning.

The first are agency factors, such as agency size, responsibilities, and infrastructure management capacity. These agency factors correspond to the resources component of rational/technocratic factors defined by de Lancer Julnes and Holzer (2001) in their study of factors explaining adoption and implementation of performance measurement. They found that resources play a key role in both adoption and implementation. The greater the agency’s size and responsibilities, in terms of budget and employees, the greater its capacity to implement and utilize performance measurement and management systems. In addition, the more expansive the agency’s size and responsibility, the greater the need for performance measurement and management to ensure accountability. For example, Thomson (2010) found that staff and budget size are related to the extent of performance measurement in nonprofit organizations.

We also expect that agencies with greater management capacity use performance measurement and management more concretely, such as having quantifiable performance targets. Consistent with the Government Performance Project (GPP) approach, we define management capacity as “government’s intrinsic ability to marshal, develop, direct, and control its human, physical, and information capital to support the discharge of its policy directions” (Ingraham & Donahue, 2000)( p. 294).

This emphasis on management capacity is important because strong capacity is the platform for high performance (Ingraham, Joyce, & Donahue, 2003; Ingraham & Moynihan, 2001).

The second category of factors includes a wide range of governance and legislative factors, which mirror the external interest groups element of de Lancer Julnes and Holzer’s (2001) political/cultural factors. These external interest group factors were found to be associated with implementation of performance measurement systems. From a governance perspective, state DOTs may be implementing performance measurement and management systems in response to pressures from external stakeholders. These external stakeholders include the legislature and also external governing organizations, such as boards or commissions. O’Connell, Yusuf, and Hackbart (2009) identified that, as of 2004, 36 states had some type of external body connected to their state DOTs with the goal of improving the DOT’s accountability to the public. The characteristics of these external boards or commissions, particularly their composition and size, may influence the extent to which there are external pressures for accountability that may drive state DOTs to pursue performance measurement and management strategies. Larger boards or commissions and those with greater representation by citizen members, may provide greater push for agencies to introduce structural policies, such as performance measurement and management, to enhance accountability.

We also expect that the characteristics of the legislature will influence how it engages with and provides oversight of the state DOT. For example, greater legislative involvement in transportation planning, such as having extensive legislative-DOT communication and legislative involvement in project selection and prioritization, may be associated with greater legislative oversight via requirements for state DOTs to use performance measurement and management in transportation planning. Finally, more broadly, the type of legislature can play a role in how a legislature may push state DOTs to implement performance measurement and management. The NCSL organizes state legislatures into three categories based on level of professionalism, ranging from those having year-round legislative sessions, full-time legislators and large legislative staffs to those with limited or biennial sessions, part-time legislators and smaller staffs, and a middle group that is a hybrid of both (Rall et al., 2011). The most professional legislatures, having greater capacity to hold agencies more accountable, may be more likely to pressure state DOTs to utilize greater accountability mechanisms such as performance measurement and management.

Data and Method

To answer our research questions, we need to first categorize state DOTs in terms of the extent of their performance measurement and management systems. We begin with data from an assessment by Smart Growth America and Taxpayers for Common Sense (2011) of performance measures and targets for road pavement conditions. In 2010 and 2011, the Smart Growth America/Taxpayers for Common Sense project team collected data on state DOT practices regarding their use of pavement condition targets. Data collection sources included state DOT websites, DOT documents (such as departmental strategic plans, performance reports, improvement plans, or highway maintenance manuals), and phone and email correspondence with DOT staff (for more information about the methodology, see Smart Growth America and Taxpayers for Common Sense, 2011). For validation purposes, we compared the Smart Growth/Taxpayers for Common Sense data with those from a 2009 study by the Texas Transportation Institute (Papagiannakis, Gharaibeh, Weissmann, & Wimsatt, 2009; see Note 3).

For our analysis, we focus on whether the state DOT has quantifiable performance targets for pavement condition, which is consistent with the most common performance measure in transportation agencies (Baird & Stammer, 2000). From the qualitative assessment in the Smart Growth America and Taxpayers for Common Sense report, we created a dichotomous variable representing whether the state had quantifiable performance goals or targets for pavement conditions as of 2011. Thirty-nine states had quantifiable targets for pavement condition performance metrics. In these states, the DOTs rely on metrics based on standard ratings, such as the International Roughness Index (IRI) and the Pavement Serviceability Rating (PSR), and other state-specific ratings such as the Sufficiency Surface Condition (SSC) rating used in Michigan DOT and the Ride Index used by the Montana DOT. Examples of targets include those for a minimum percentage of pavement in “fair” condition or a maximum percentage of condition rated as “poor” (Smart Growth America and Taxpayers for Common Sense, 2011). One possible limitation is that we identify states as having quality performance targets simply if performance targets are quantifiable without any further assessment of type or effectiveness.

To answer the first research question linking performance measurement and management to transportation outcomes in the context of road repair and preservation, we use data on road pavement condition and annual spending on maintenance. Pavement condition outcomes are measured as the percent of urban interstates or principal rural arterials that are in poor condition averaged across the period from 2012 to 2014. We also focus on improvement in pavement conditions, measured as the decrease in the percent of such roads in poor condition compared with the period from 2006 to 2008. The maintenance spending outcome variable is measured as the average annual disbursement for maintenance, in dollars per lane-mile, averaged across 2012, 2013, and 2014.

To answer the second research question, we rely on state DOT governance data from a joint study by the American Association of State Highway and Transportation Officials (AASHTO) and NCSL (Rall et al., 2011) and from research by Yusuf and O’Connell (O’Connell et al., 2009; Yusuf, O’Connell, Hackbart, & Wallace, 2008) on transportation boards and commissions. The AASHTO/NCSL study provides information on legislature characterization, DOT size, legislative-DOT relationships, and legislative involvement in transportation planning and project selection. These 2010 data were collected from multiple sources, including original survey research, expert interviews, and legal legislative research (for more information about the methodology, see Rall et al., 2011). From the AASHTO/NCSL qualitative data, we created dichotomous variables measuring whether there is extensive legislature-DOT communication and whether the legislature has direct involvement in project selection and prioritization. We also utilize the NCSL (2009) classification of state legislatures to create a dichotomous variable for whether the state has a professional full-time legislature. From O’Connell et al. (2009), we obtained data on the size of the transportation board/commission and the number of citizen members on the board/commission. These data on board/commission were based on a 2004 review of state statutes.⁴

We use the infrastructure management results of the 2008 GPP as our measure of capital management capacity (Barrett & Greene, 2008; GPP, 2003). The Maxwell School of Citizenship and Public Affairs at Syracuse University in collaboration with Governing Magazine conducted a study, known as the GPP, which estimated the performance of state governments (Barrett & Greene, 2008; Ebdon, 2001, 2003; GPP, 2003). Each state was assigned a letter grade for infrastructure management capacity, which represents the state’s capacity to harness capital for policy implementation. The infrastructure management capacity grades were determined based on five major criteria: capital planning, project monitoring, maintenance, internal cooperation, and intergovernmental coordination (Jimenez & Pagano, 2012).⁵ Data underpinning the assessment came from a survey of the states, interviews with state officials, and analysis of government documents and reports (GPP, 2003; Ingraham et al., 2003; Jimenez & Pagano, 2012). We follow Yusuf et al. (2008) in using the infrastructure management grade as a proxy for DOT capital management capacity. We convert the letter grades for the GPP infrastructure management into numerical scores that correspond with academic letter grades used to calculate grade point averages (e.g., A = 4.000, A− = 3.667, B+ = 3.333, B = 3.000, etc.). The descriptive statistics for all variables used in our analysis are presented in Table 2.

Table 2.

Descriptive Statistics.

	Median	Mean	SD	Minimum/maximum	p value
Performance measurement
Quantifiable Pavement Condition Targets (2011)	1.00	0.78	0.42	0/1.00	.003
Agency factors
No. of FTE employees (2010)	3,952	4,888.57	3,545.48	780/18,406	<.001
Lane-miles in state highway system (2008)	25,975	37,083.44	38,613.10	2,477/193,188	<.001
Annual highway spending (average for 2004-2008)	851,678	1,197,510	1,221,350	158,073/6,553,244	<.001
Annual highway spending per lane-mile (average for 2004-2008)	38.61	47.12	34.88	13.88/208.91	<.001
Infrastructure management capacity (2008)	2.67	2.63	0.61	1.33/3.67	.932
Governance/legislative factors
Size of commission (2004)	7	6.52	5.59	0/20.00	.019
Percent of citizen members on commission (2004)	91.51	58.85	46.55	0/100.00	.010
Extensive legislature-DOT communication (2010)	0	0.14	0.35	0/1.00	<.001
Legislative involvement in project selection and prioritization (2010)	0	0.38	0.49	0/1.00	.715
Professional legislature (2009)	0	0.20	0.40	0/1.00	<.001
Repair and preservation outcomes
Percent of urban interstates in poor condition, average of 2012-2014	3.68	4.77	4.98	0.14/28.58	<.001
Percent of rural principal arterials in poor condition, average of 2012-2014	1.62	3.03	4.72	0.32/27.96	<.001
Improvement in urban interstates pavement condition, i.e., decrease in % of roads in poor condition (between 2006-2008 and 2012-2014)	−0.33	0.27	3.58	−5.78/11.05	.008
Improvement in rural principal arterials pavement condition, i.e., decrease in % of roads in poor condition (between 2006-2008 and 2012-2014)	−1.26	−2.12	2.88	−13.44/0.16	<.001
Annual spending for maintenance (average for 2012-2014), US$ per lane-mile	24,429.85	40,633.78	46,589.67	3,421/296,162	<.001

Note. p value is for the Shapiro–Wilk test for normality. Significant p value indicates nonnormal distribution. DOT = Department of Transportations.

We use data from various years to capture agency factors, legislative factors, use of performance measurement, and transportation outcomes. We also use variables that measure averages across multiple years to account for the possible lumpiness of annual and capital spending for transportation. The specific years for the different variables are included in Table 2. We acknowledge that using different years for data collection may be a limitation of the study, but we also recognize the challenges of obtaining consistent data across the needed years. Our data are consistent in that the performance measurement variable is measured in 2011, all transportation outcomes are measured for years after 2011, and all agency and governance/legislative variables are measured in years before 2010.

To conduct our analysis, we use the Kolmogorov–Smirnov equality-of-distribution test to compare across the two groups of states with different levels of performance measurement and management. The Kolmogorov–Smirnov test detects differences not only in the means or medians of the two groups but also differences in variances or standard deviations (Conover, 1999). To answer the first research question, we compare repair and preservation outcomes across the state DOTs according to whether they have quantifiable pavement condition performance targets. To answer the second research question, we compare agency and governance/legislative factors across the state DOTs using the same performance measurement and management categorization.

The two-sample Kolmogorov–Smirnov test is a nonparametric test for determining whether two independent samples have equal distribution functions. The two-sample Kolmogorov–Smirnov test is one of the most general nonparametric tests for comparing two samples as it compares the overall distribution and is sensitive to differences in both location and shape of the empirical cumulative distribution functions. The Kolmogorov–Smirnov test produces a d-test statistic which quantifies the absolute distance between the two empirical distribution functions. We rely on nonparametric statistical testing because (a) we have a small number of observations; (b) our data do not meet the normality and homogeneity of variance assumptions needed for traditional, parametric tests such as the two-sample t test or one-way ANOVA; (c) we have unequal groups (of 39 states with quantifiable performance targets and 11 without quantifiable targets); and (d) some of our variables are ordinal. For all comparisons, we conduct one-tailed tests of the hypotheses.

The Kolmogorov–Smirnov test lacks power against the null hypothesis when comparing differences (Dransfield & Brightwell, 2012). This is particularly true for small samples such as those in our study. The empirical distributions will need to be substantially different for the Kolmogorov–Smirnov test to show statistically significant differences. As such, the Kolmogorov–Smirnov test provides more conservative results of statistical significance. Any error in our results will be toward failing to detect or understating differences between the two groups of states. Given this conservative bias, we also focus on effect size in addition to statistical significance.

Results and Findings

Our first research question is as follows: Does use of quantifiable performance targets make a difference in transportation outcomes as measured by pavement condition and repair and maintenance spending? We expect that state DOTs with quantifiable performance targets will have higher quality roads, spend more and invest more on meeting repair and preservation needs. These comparisons are summarized in Table 3. This table compares the median, mean, and standard deviation of the transportation outcomes across the two groups of states—those with and without quantifiable performance condition targets. The table shows the p value for the one-tailed Kolmogorov–Smirnov d-test statistic, which measures the deviations or differences between the empirical distributions of the two groups.

Table 3.

Comparison of Transportation Outcomes by Use of Quantifiable Pavement Condition Performance Targets—Median, Mean, and Standard Deviation.

	Quantifiable pavement condition targets: Yes (n = 39)	Quantifiable pavement condition targets: No (n = 11)	p value
Percent of urban interstates in poor condition, average of 2012-2014	Median = 4.09 Mean = 4.95	Median = 2.95 Mean = 4.14	.414
Percent of rural principal arterials in poor condition, average of 2012-2014	Median = 1.58 Mean = 2.93	Median = 2.07 Mean = 3.37	.134
Improvement in urban interstates pavement condition, i.e., decrease in % of roads in poor condition (between 2006-2008 and 2012-2014)	Median = −0.55 Mean = 0.04	Median = 0.17 Mean = 1.05	.474
Improvement in rural principal arterials pavement condition, i.e., decrease in % of roads in poor condition (between 2006-2008 and 2012-2014)	Median = −1.27 Mean = −2.13	Median = −1.25 Mean = −2.10	.300
Annual spending for maintenance (average for 2012-2014), US$ per lane-mile	Median = 27,611 Mean = 43,410	Median = 16,635 Mean = 30,790	.106

Note. p value is for one-tailed test of the d-test statistic for the two-sample Kolmogorov–Smirnov test of equality of distributions.

The results of the two-sample Kolmogorov–Smirnov test of equality-of-distribution show that the differences in infrastructure quality (i.e., road conditions) were not statistically significantly between state DOTs with quantifiable performance targets and those without quantifiable performance targets. While not statistically significant, differences across the two categories of state DOTs were greater when considering pavement condition of urban interstates compared with rural principal arterials. Surprisingly, state DOTs with quantifiable performance targets had more urban interstates in poor condition (based on comparison of median and mean values) than those state DOTs without such performance targets. In addition, the former group also saw increases, over time, in the percent of urban interstates in poor condition, while the latter group saw a small decrease in the percent of urban interstates in poor condition.

We also expected that spending decisions that prioritize repair and preservation would vary across state DOTs in terms of their use of performance measures. Our analysis showed no statistically significant differences in terms of annual spending for maintenance at the p < .10 level. However, states with quantitative pavement condition targets had higher mean and median values for annual maintenance spending per lane-mile, compared with those states without such pavement condition targets. Combined, the results suggest that having quantifiable performance targets related to pavement condition may direct DOT efforts toward spending more annually on maintenance, but that such spending decisions have not resulted in improved road conditions. This finding, then, raises interesting additional questions.

One such question is the focus of our second research question: How are state DOTs that use quantifiable performance targets different from those DOTs that do not? As a preliminary analysis, we examined whether there were obvious geographic or size differences across the states. The 11 states that do not have quantifiable pavement condition targets are as follows: Alabama, Arkansas, Connecticut, Maine, Mississippi, New Hampshire, New York, Oklahoma, Rhode Island, South Carolina, and West Virginia. There does not appear to be any specific patterns in terms of geographic size. However, states without quantifiable pavement condition targets do appear clustered in the northeastern and mid-southern regions of the country. This may suggest that, in addition to being driven by state-specific factors, use of performance measurement may also be driven by policy diffusion from neighboring states.

We also examined whether differences in agency and governance/legislative factors were associated with state DOT use of performance measurement and management. The results of our comparison of these factors across state DOTs depending on whether they had quantifiable pavement condition targets are shown in Table 4. Similar to Table 3, this table compares the median, mean, and standard deviation of the agency and legislative/governance variables for states with and without quantifiable pavement condition targets. The table shows the p value for the one-tailed Kolmogorov–Smirnov d-test statistic. The results show that state DOTs with and without quantifiable pavement condition performance targets differ significantly in terms of total highway spending, infrastructure management capacity, and size and composition of the external commission or boards.

Table 4.

Comparison of Agency and Governance/Legislative Factors by Use of Quantifiable Pavement Condition Performance Targets—Median, Mean, and Standard Deviation.

	Quantifiable pavement condition targets: Yes (n = 39)	Quantifiable pavement condition targets: No (n = 11)	p value
No. of FTE employees	Median = 4,498 Mean = 5,171 SD = 3,746	Median = 3,430 Mean = 3,758 SD = 2,418	.592
Lane-miles in state highway system	Median = 23,321 Mean = 29,726 SD = 26,724	Median = 24,530 Mean = 22,368 SD = 14,268	.608
Total highway spending	Median = 1,045,327 Mean = 1,348,927 SD = 1,317,969	Median = 668,040 Mean = 591,844 SD = 291,169	.034
Spending per lane-mile	Median = 39.80 Mean = 50.35 SD = 37.28	Median = 30.32 Mean = 34.20 SD = 19.14	.267
Infrastructure management capacity	Median = 2.67 Mean = 2.78 SD = 0.57	Median = 2.33 Mean = 2.02 SD = 0.38	.001
Size of commission	Median = 7 Mean = 7.58 SD = 5.57	Median = 0 Mean = 2.30 SD = 3.23	.005
Percent of citizen members on commission	Median = 100 Mean = 68.70 SD = 42.59	Median = 0 Mean = 31.43 SD = 47.52	.011
Extensive legislature-DOT communication	Median = 0 Mean = 0.15 SD = 0.36	Median = 0 Mean = 0.10 SD = 0.32	.961
Legislative involvement in project selection and prioritization	Median = 0 Mean = 0.38 SD = 0.49	Median = 0 Mean = 0.40 SD = 0.52	.998
Professional legislature	Median = 0 Mean = 0.25 SD = 0.44	Median = 0 Mean = 0.00 SD = 0.00	.715

Note. p value is for one-tailed test of the d-test statistic for the two-sample Kolmogorov–Smirnov test of equality of distributions.FTE= Full-time equivalent.

In terms of agency factors, we expected that state DOT of larger size and greater responsibility would have greater capacity and need for performance measurement. Similarly, greater management capacity would provide support for performance measurement. State DOTs with quantifiable pavement condition targets have larger budgets for highway spending; the differences in median and mean values compared with those states DOTs without performance targets are US$377,288 and US$757,083, respectively. The state DOTs with quantifiable pavement condition targets also have statistically significantly higher infrastructure management capacity (p = .001). Their GPP infrastructure management capacity grade was 2.67 (out of 4.00), compared with 2.33 for state DOTs without quantifiable performance measures. These support the argument that resources and capacity play an important role in performance measurement.

In terms of governance/legislative factors, we found empirical results are as expected. Board or commission size increases as the state DOT uses performance measures more concretely (i.e., using quantifiable pavement condition targets), with average sizes of eight members, compared with two members for those state DOTs without quantitative performance targets. The median size is also higher (7 compared with 0 members). State DOTs with quantifiable pavement condition targets had significantly greater representation of citizens on external governing boards or commissions. These results suggest that the size and composition of the board or commission appear to be relevant as a mechanism to exert external accountability pressures for state DOTs to put in place performance measurement and management.

We expected the characteristics of the legislature and its relationship with the state DOT would influence the extent of legislative oversight of the state DOT via performance measurement and management. While these variables were not statistically significantly different across the two groups of state DOTs, the magnitude of the differences are interesting to highlight. For example, 25% of states with quantifiable performance targets had a professional legislature. In contrast, all 11 states without targets had either part-time or hybrid legislature. In addition, 15% of the state DOTs with quantifiable targets had additional accountability mechanisms through extensive legislature-DOT communications, compared with 10% of state DOTs without quantifiable pavement condition targets.

Our finding regarding the lack of a connection between use of quantifiable performance measures and transportation outcomes also raised the interesting question of whether focus on performance measurement and management was also driven by poor transportation outcomes. Specifically, were states with poor road quality driven to implement quantifiable performance targets due to having under-performing transportation infrastructure? While we are unable to test this causal connection, we conducted preliminary analysis that compared infrastructure quality for states with and without the quantitative performance standards, for the period from 2006 to 2008. We found that the percent of urban interstates in poor condition, averaged across 2006 through 2008, was not statistically significantly different across the two categories of state DOTs. The mean and median values were quite comparable. For example, the mean percent of urban interstates in poor condition was 5.00% and 5.20% for states with and without performance targets, respectively. However, the differences in the condition of rural principal arterials were statistically significant. From 2006 to 2008, states with quantitative performance targets had fewer percentages of rural principal arterials in poor condition compared with those without performance targets. This suggests that states with higher performing rural roads infrastructure were more likely to put in place quantitative performance targets. That these same states subsequently, in the period from 2012 to 2014, see deterioration in their urban road infrastructure is an interesting finding that may need further study.

Conclusion and Discussion

We set out to examine the use of performance measurement and management systems by state DOTs. We focused on the use of quantifiable performance measures. For our first research question, we found that there were no statistically significant differences in terms of transportation outcomes when state DOTs with quantitative performance targets are compared with those without such targets. Perhaps this reflects an industry standard in maintenance and repair of roads based more on procedure and process rather than the quality of performance targets. A variation in outcomes may exist across types of quantifiable performance metrics that was not measured or captured in this study.

There is no statistical difference in maintenance spending between states with quantifiable performance targets and states that do not have quantifiable targets. However, analysis of the magnitudes of the differences across the two categories of state DOTs do suggest that state DOTs with quantitative performance targets are spending more on maintenance, but such spending has not resulted in much improvement in road conditions. Our study’s findings raise questions about the role performance measurement and management may play in ensuring more efficient and targeted distribution of limited resources to meet quantifiable organizational goals and objectives.

However, pairing quantifiable performance measures with specific and related performance outcomes is useful for creating structural policies that can be visible accountability tools. Citizens, political actors, and other elements of the multiparty accountability environment can use these target outcomes to make decisions about priorities, support, and funding. Our findings show that the use of quantifiable measures is more associated with states with commissions that are larger and populated with more citizens. This suggests that external accountability bodies, such as the commission, can use the quality performance targets to make the corrective funding decisions as called for in MAP-21. This ultimately is the intent of MAP-21; however, our findings do not support the connection between performance measurement and specific infrastructure quality outcomes.

We also found that DOT performance measurement and management differ according to agency factors such as total highway spending and infrastructure management capacity. The results suggest that agency capacity matters in terms of management and resources in establishing quantifiable targets, which is consistent with previous literature. In conclusion, when it comes to state DOT use of quantifiable performance measures and management, external accountability factors and capacity in terms of management and resources matter

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Juita-Elena (Wie) Yusuf is an Associate Professor in the School of Public Service at Old Dominion University. Her areas of research include transportation policy, environmental policy and climate change, and public budgeting and finance. Her work has been published in Public Works Management and Policy, Public Budgeting and Finance, American Review of Public Administration, Public Performance and Management Review, and other journals.

Meagan M. Jordan is an Associate Professor of Public Administration in the School of Public Service at Old Dominion University. Her research interests and publications include state and local budgeting and revenue policy, performance budgeting, and strategic planning.

References

American Association of State Highway and Transportation Officials Center for Excellence in Project Finance. (2010). A CFO’s Handbook on performance management. Washington, DC: American Association of State Highway and Transportation Officials.

American Society of Civil Engineers. (2013). 2013 report card for America’s infrastructure. Reston, VA: Author.

Ammons

D. N.

Rivenbark

W. C.

(2008). Factors influencing the use of performance data to improve municipal services: Evidence from the North Carolina benchmarking project. Public Administration Review, 68, 304-318.

Baird

M. E.

Stammer

R. E.

Jr. (2000). Measuring the performance of state transportation agencies: Three perspectives. Transportation Research Record: Journal of the Transportation Research Board, 1729, 26-34.

Barrett

Greene

(2008, March 3). A management report: Grading the states. In Governing (pp. 24-95). Retrieved from http://www.pewtrusts.org/en/research-and-analysis/reports/2008/03/03/grading-the-states-2008-report

Behn

R. D.

(2003). Why measure performance? Different purposes require different measures. Public Administration Review, 63, 586-606.

Bremmer

Cotton

Hamilton

(2005). Emerging performance measurement responses to changing political pressures at state departments of transportation: Practitioners’ perspective. Transportation Research Record: Journal of the Transportation Research Board, 1924, 175-183.

Boyne

G. A.

Chen

A. A.

(2007). Performance targets and public service improvement. Journal of Public Administration Research and Theory, 17, 455-477.

Conover

W. J.

(1999). Practical nonparametric statistics (3rd ed.). New York, NY: Wiley.

10.

de Lancer Julnes

Holzer

. (2001). Promoting the utilization of performance measures in public organizations: An empirical study of factors affecting adoption and implementation. Public Administration Review, 61, 693-708.

11.

Dransfield

R. D.

Brightwell

(2012). How to get on top of statistics: Design and analysis for biologists, with R. London, England: InfluentialPoints LLP.

12.

Ebdon

(2001). Capital planning in the states: Internal decision-making. Learning Paper Series. Retrieved from www.maxwell.syr.edu/gpp/pdfs/CM_State_Learning_Paper.pdf

13.

Ebdon

(2003). Capital management. In ( Alan

. (Ed.), Paths to performance in state and local government: Final report from the Maxwell School (pp. 71-100). Syracuse, NY: Syracuse University, Campbell Public Affairs Institute.

14.

Falcocchio

(2004). Performance measures for evaluating transportation systems: Stakeholder.

15.

Gifford

Carlisle

(2004). Virginia Department of Transportation’s dashboard performance measurement and reporting system: Going the full monty. Transportation Research Record: Journal of the Transportation Research Board, 1895, 207-219.

16.

Government Performance Project. (2003). Paths to performance in state and local government. Syracuse, NY: Alan K. Campbell Public Affairs Institute, Syracuse University.

17.

Hatry

H. P.

(2006). Performance measurement: Getting results (2nd ed.). Washington, DC: Urban Institute.

18.

Hartgen

D. T.

Fields

M. G.

Feigenbaum

(2014). 21st Annual Report on the Performance of State Highway Systems (1984–2012). Los Angeles, CA: The Reason Foundation. Retrieved from: http://reason.org/files/21st_annual_highway_report.pdf

19.

Heinrich

C. J.

(2002). Outcomes-based performance management in the public sector: Implications for government accountability and effectiveness. Public Administration Review, 62, 712-725.

20.

Ingraham

P. W.

Donahue

A. K.

(2000). Dissecting the black box revisited: Characterizing government management capacity. In Heinrich

C. J.

Lynn

L. E.

Jr . (Eds.), Governance and performance: New perspectives (pp. 292-318). Washington, DC: Georgetown University Press.

21.

Ingraham

P. W.

Joyce

P. G.

Donahue

A. D.

(2003). Government performance: Why management matters. Baltimore, MD: Johns Hopkins University Press.

22.

Ingraham

P. W.

Moynihan

D. P.

(2001). Beyond measurement: Managing for results in state government. In Forsythe

(Ed.), Quicker, better, cheaper? Managing performance in American government (pp. 311-335). Albany: State University of New York Press.

23.

Jimenez

B. S.

Pagano

M. A.

(2012). What factors affect management quality? State infrastructure management and the government performance project. Public Works Management & Policy, 17, 124-151.

24.

Kelly

J. M.

(2003). Citizen satisfaction and administrative performance measures: Is there really a link? Urban Affairs Review, 38, 855-866.

25.

Latham

G. P.

(2004). The motivational benefits of goal-setting. The Academy of Management Executive, 18(4), 126-129.

26.

Locke

E. A.

Latham

G. P.

(2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57, 705-717.

27.

Marr

(2009). Managing and delivering performance. New York, NY: Routledge.

28.

National Conference of State Legislatures. (2008). Legislative performance budgeting. Retrieved from http://www.ncsl.org/research/fiscal-policy/legislative-performance-budgeting.aspx

29.

National Conference of State Legislatures. (2009). Full- and part-time legislatures. Retrieved from http://www.ncsl.org/research/about-state-legislatures/full-and-part-time-legislatures.aspx

30.

National Cooperative Highway Research Program. (2006). Performance measures and targets for transportation asset management (Report 551). Washington, DC: Transportation Research Board.

31.

O’Connell

Yusuf

J.-E.

Hackbart

(2009). Transportation commissions as accountability structures: A review of their statutory roles and other attributes. American Review of Public Administration, 39, 409-424.

32.

Papagiannakis

Gharaibeh

Weissmann

Wimsatt

(2009). Pavement score synthesis (Report No. FHWA/TX-09/0-6386-1). College Station: Texas Transportation Institute. Retrieved from http://tti.tamu.edu/documents/0-6386-1.pdf

33.

Pew Center on the States and The Rockefeller Foundation. (2011). Measuring transportation investments: The road to results. Washington, DC: The Pew Charitable Trusts and The Rockefeller Foundation.

34.

Pickrell

Neumann

(2000). Use of performance measures in transportation decision making. Paper presented at the Conference on Performance Measures to Improve Transportation Systems and Agency Operations, Irvine, CA.

35.

Plant

J. F.

(2007). The big questions of transportation policy and administration in the twenty-first century. In Plant

J. F.

(Ed.), Handbook of transportation policy and administration (pp. 3-9). New York, NY: CRC Press.

36.

Poister

T. H.

(1997). A survey of performance measurement systems in state transportation departments. Public Works Management & Policy, 1(4), 323-341.

37.

Poister

T. H.

(2007). Performance measurement in transportation agencies: State of the practice. In Plant

J. F.

(Ed.), Handbook of transportation policy and administration (pp. 485-504). New York, NY: CRC Press.

38.

Poister

T. H.

(2008). Measuring performance in public and nonprofit organizations. San Francisco, CA: Wiley.

39.

Poister

T. H.

Pasha

O. Q.

Edwards

L. H.

(2013). Does performance management lead to better outcomes? Evidence from the US Public Transit Industry. Public Administration Review, 73, 625-636.

40.

Rall

Wheet

Farber

N. J.

Reed

J. B.

(2011). Transportation governance and finance: A 50-state review of state legislatures and departments of transportation. Washington, DC: National Conference of State Legislatures and the AASHTO Center for Excellence in Project Finance.

41.

Rodgers

Hunter

J. E.

(1992). A foundation of good management practice in government: Management by objectives. Public Administration Review, 52, 27-39.

42.

Salisbury

Heinz

(1970). A theory of policy analysis and some preliminary applications. In Sharkansky

(Ed.), Policy analysis in political science (pp. 39-60). Chicago, IL: Markham.

43.

Smart Growth America and Taxpayers for Common Sense. (2011). Repair priorities: Transportation spending strategies to save taxpayer dollars and improve roads. Washington, DC: Author.

44.

Smart Growth America and Taxpayers for Common Sense. (2014). Repair priorities 2014: Transportation spending strategies to save taxpayer dollars and improve roads. Washington, DC: Author.

45.

Thomson

D. E.

(2010). Exploring the role of funders’ performance reporting mandates in nonprofit performance measurement. Nonprofit and Voluntary Sector Quarterly, 39, 611-629.

46.

U.S. General Accounting Office. (1996). Executive guide: Effectively implementing the Government Performance and Results Act. Washington, DC: Author.

47.

U.S. Government Accountability Office. (2010). Statewide transportation planning: Opportunities exist to transition to performance-based planning and federal oversight (Report to the Chairman, Committee on Transportation and Infrastructure, House of Representatives). Washington, DC: Author.

48.

U.S. Government Accountability Office. (2015). Surface transportation: DOT is progressing toward a performance-based approach, but states and grantees report potential implementation challenges (Report to Congressional Addressees). Washington, DC: Author.

49.

Van Dooren

Bouckaert

Halligan

. (2015). Performance management in the public sector. New York, NY: Routledge.

50.

Verbeeten

F. H.

(2008). Performance management practices in public sector organizations: Impact on performance. Accounting, Auditing & Accountability Journal, 21, 427-454.

51.

Walker

R. M.

Damanpour

Devece

C. A.

(2011). Management innovation and organizational performance: The mediating effect of performance management. Journal of Public Administration Research and Theory, 21, 367-386.

52.

Wholey

J. S.

(1999). Performance-based management: Responding to the challenges. Public Productivity and Management Review, 22, 288-307.

53.

Wholey

J. S.

Hatry

H. P.

(1992). The case for performance monitoring. Public Administration Review, 52, 604-610.

54.

Yusuf

J.-E.

Leavitt

W. M.

(2014). The case for performance management in public works and infrastructure. Public Works Management and Policy, 9, 204-218.

55.

Yusuf

J.-E.

O’Connell

Hackbart

Wallace

(2008). An empirical examination of the statutory characteristics and effects of highway and transportation commissions on DOT capital management capacity. Public Works Management and Policy, 12, 533-543.