Abstract
Recent economic problems have led to a potential increase in the adoption of an availability payments approach by U.S. transportation agencies when creating public-private partnerships (P3s). In doing so, the public sector assumes 100% of the demand risk, relying much more heavily on performance management systems that contain key performance indicators (KPIs) to ensure optimal private partner performance during operations and maintenance. Many existing performance management systems, however, suffer from a lack of alignment among KPIs chosen to ensure that all P3s meet societal or agency goals as well as project goals. Furthermore, the choice of KPIs and accompanying standards has resulted in standards that are too easily met or inappropriately weighed. Examples are provided from P3 contracts for projects in Australia and Canada.
Keywords
Recent economic problems have highlighted negative results associated with U.S. transportation-related public-private partnership (P3) contracts that assign all demand risk to the private partners. Many private partners, operating as part of a Design–Build–Finance–Operate–Maintain (DBFOM) P3 agreement, have experienced lower than anticipated revenue and, in some cases, bankruptcy. 1 These instances support a more recent interest 2 in adopting different approaches to assuming demand risk: (a) sharing of demand risk among both public and private partners, or (b) the public sector assumes all demand risk. The former is implemented using approaches such as shadow tolling, various revenue sharing formulas, and sharing refinancing gains, 3 and is outside the scope of this discussion. The latter approach adopts the use of availability payments (APs).
APs are periodic payments, made by public partners over the life of a P3 contract, that respond to specific efforts made by the private partner or concessionaire. If these efforts meet predetermined standards, payments are made. Determination of relevant performance standards can be identified as part of a contractually identified performance management system. Such standards are based on key performance indicators (KPIs) defined as,
more specific milestones in or components of performance measures that serve as precursors to indicate progress toward the eventual achievement of the desired performance measures. (Garvin, Molenaar, Navarro, & Proctor, 2011, p. 5)
The adoption of KPIs can ensure continued high quality performance from the private partner, especially during the operations and maintenance phases of the P3 contract.
By using an AP approach, the public partner assumes all demand risk because payments to the private partners are identified by contract and do not change because of any variation in the number of travelers using the roadway. All revenue from collected tolls is deposited with the public partner and is removed as the only source of funding for the private partners.
Without an effective performance management system, however, one that contains KPI-related performance standards that reflect agency, regional, and larger societal goals, as well as project-related goals, the risk is that insufficient incentives will exist for private partners to achieve optimal performance. A review of existing transportation P3 contracts suggests that presently adopted performance management systems contain project level KPIs that are not sufficiently aligned—if at all—with agency and societal goals. Performance standards reflecting optimal asset management, (e.g., pavement smoothness) are insufficient to ensure optimal efforts to lower accident and fatality rates and meet system or statewide safety goals.
Additional problems include identifying performance standards that are much too easy to meet, using a “pass/fail” approach, as well as assigning weights to standards that do not accurately reflect goal priorities. Reviews of examples from two P3 contracts, the East Link Tunnel Project (Australia) and the Sea to Sky Highway Improvement Project (Canada), illustrate these problems.
Both cases were chosen as part of research funded by the Florida Department of Transportation. The Melbourne (Australia) East Link P3 features 45 km of freeway, including 39 km of tolled roadway and 6 km of non-tolled bypasses, “17 interchanges, 88 bridges, and twin 1.6 km three-lane tunnels” (Melbourne’s Motorway Masterpiece, n.d., p. 2). Constructed at a cost of Aus$2.5 billion, it opened in June 2008, many months ahead of schedule. It is now widely regarded as a state of the art example of a successful P3.
The Sea to Sky highway is an 80-plus mile section of Highway 99 that begins at the U.S./Canadian border. In 2005, the British Columbia Ministry of Transportation entered a 25 year Can$600 million DBFOM P3 contract with S25 Transportation Group. This case study is an example of the use of APs and additional incentives supporting a DBFOM contract.
APs: Definition, Benefits, and Challenges
APs are made, usually monthly or annually, over the length of the 30- to 50-year P3 contract, for example, when a greenway roadway is “available”—open for travel, beginning when construction is completed.By implication, the travel experience is also meant to be optimal, that is, traveling at comfortable speeds without congestion, without encountering potholes or damaged roadways, or not facing delays due to accidents or roadway maintenance.
In its simplest form, during the operations period, if a roadway remains open for a specified time period, then a corresponding AP will be made. If lanes on a roadway are closed for any reason, such as for maintenance purposes, incident management, or snow removal, then the AP will be reduced according to a specified formula. This situation may reflect conditions of “pure availability”: unobstructed travel by motorists on the roadway (Dochia & Parker, 2009).
During the operations and maintenance periods, APs are tied to performance standards reflected by KPIs. Data must be collected and monitored relevant to the KPIs, with corresponding levels of APs related to the KPI’s achieved. If the data are not collected, or if monitoring of KPIs does not occur, then the use of APs is simply another way for the public sector to borrow funds to build roadways or other infrastructure (Farquharson, Torres de Mastle, & Yescombe, 2011).
Another category of APs refers to “constructive availability”: the use of
performance, safety and quality criteria—specified in the contract, often providing the public owner with stronger metrics and management tools to assure a high quality service than it may be able to apply to services it self-performs. (Dochia & Parker, 2009)
These could include, for example, level-of-service performance measures, used as the primary payment measure involving the implementation of managed lanes.
Benefits of APs
If demand is highly uncertain, and revenue from user fees is difficult to predict, then compared to other P3s using an AP approach can include the following benefits:
The use of KPIs increases the likelihood that private partner performance will remain at high, specified levels. (Sharma & Cui, 2012)
Private partners are more attracted to P3 projects, as there is less risk that would be transferred to them.
The potential for private partner bankruptcy is greatly reduced.
Public contributions to the overall cost of the P3 are capped, as no more than the contract specified amount will be paid. 4 (U.S. Department of Transportation [USDOT], Federal Highway Administration, Office of Innovative Delivery, n.d.)
Private profit is also capped, eliminating the likelihood of “windfall” profits. 5 (KPMG, 2009)
Financing costs are less, as public bonds and other debt mechanisms can be accessed at a lower cost. 6
The cost of debt is lower, as debt service coverage ratios would be lower, and private partner debt reviewed more favorably by rating agencies. (Goldman Sachs, 2008; KPMG, 2011)
Greater transparency and public support for the P3 project can be more easily achieved because payment amounts are known.
Maintenance and future capital renewal costs are fully funded. (Seliga, Paasman, & Jansen, 2011)
If demand is deemed to be more highly certain, and revenues projected to be very positive, then the AP approach provides a new revenue stream for the public partners. 7
Challenges/Potential Problems With APs
Availability risk does not disappear, and may increase. Availability risk, occurring especially during the operational and maintenance phases, can be defined as
an underperformance linked to the state of the PPP assets [that] results in services being partially or wholly unavailable, or where these services fail to meet the quality standards specified in the PPP contract. (European PPP Expertise Centre, 2011)
By removing the demand risk from the P3 contract, there is a potential disincentive for the developer during construction to use resources in ways that optimize asset usage. During operations, the private partner may meet minimum standards or incur penalties to limit availability risk but will have no incentive to go beyond these standards as the payment received remains constant. Furthermore, if more travelers use the road, operations and maintenance costs may increase, further reducing profit. (Ugarte, Guiterrez, & Phillips, 2012)
As will be discussed below, additional related challenges refer to the KPIs chosen and the resulting performance management system. Optimal KPI choice results in measures that reflect achievement of project and societal goals. The choice of penalties and enforcement of them by public partners must be an ongoing, accepted part of the relationship among partners after contract award.
Overall, the risk is that with an AP P3 approach a true partnership will not occur, and that the relationship among the public agencies and private contractors reflect behavior more likely found in privatization contracts and not in P3s. 8 Only with a fully integrated agency performance management system can the potential disadvantages associated with APs be overcome.
Performance Management Systems: Trends and Challenges
To improve performance of agencies and programs, over the past three decades, governments have increasingly adopted Performance Management Systems (Moynihan & Pandey, 2010; Van Dooren, 2011). In the United States, many state governments have benefitted from statewide efforts to adopt specific performance measures (Aristegueta & Zarook, 2011). Similarly, as stated by Cempel (2010), over the past decade U.S. transportation agencies have progressed from identifying measures and tracking performance to making decisions based on performance data. Key aspects of all such performance management efforts include measurable performance goals, specific performance standards, and valid quantifiable measures (Newcomer & Caudle, 2011).
For transportation agencies, specifically those at the state level, identified best practices include,
the integration of performance measures throughout the agency both vertically and horizontally, leading to the identification of more valid performance standards and related efforts;
the application of performance measurement in a systematic, documented way, leading to greater understanding and consistency across measures, goals, and objectives; and
transparency of performance results and their implications for transportation customers and stakeholders, leading to better results, greater accountability and improved communication. (Cempel, 2010, p. 4)
The adoption of these and other best practices by transportation agencies, however, is inadequate in many cases. Although agencies, as well as larger governments have adopted measures and are collecting data, much less effort is placed in integrating these measures and adopting valid performance standards (Moynihan & Pandey, 2010). A lack of measurement integration is reflected in a disconnect between larger societal measures (e.g., roadway safety) and those associated with asset management programs that focus on measures that relate to pavement smoothness (USDOT, 2010; Van Dooren, 2011)
Characteristics of an Optimal Transportation P3 Performance Management System
Ideally, the best practices found in state transportation agency performance management system should be reflected in a specific P3 contract managed by that agency. As discussed in more detail below, the P3 contract should specify,
the indicators by which the performance of the private partner will be measured;
the mechanisms or processes by which performance data will be collected and evaluated;
the standards by which the measures will be assessed and penalties and/or incentives applied; and
the payment amount and mechanism resulting from the performance, including penalties for not meeting identified standards.
The standards chosen and the effectiveness of these standards are reflected by the identification of milestones or components that are an essential aspect of KPIs. To further illustrate, KPIs can be categorized by the following:
Project Faults: Roadway may be judged constructively unavailable even if travel on the roadway still occurs; for example, failure to meet pavement smoothness standards; reduction in payments as a result can be adjusted by segment and time of day.
Non-Compliance Points: failure to meet minimum performance requirements; failure to meet maintenance requirements (Harder, 2009).
To further assess relevant issues, it is useful to identify different levels or degrees of KPIs or performance measures. Each level can be distinguished by the degree of control that the private and public partners of the P3 project have over the potential outcomes or outputs. At the broadest or highest level, societal goals such as safety or congestion mitigation may be likely. More relevant to a P3 contractual arrangement are agency goals and P3 project goals. To the extent that all three goals are the same or can be aligned, the P3 project has greater potential for being judged effective or successful.
A related issue is the extent to which the agency or the contractor specifies or agrees to the measure prior to contract award; plus, the degree to which the private partner is given control over choosing appropriate materials and processes without agency specification. With P3 contracts, the intent is to allow contractor discretion as much as possible, supporting potential innovation and permitting the contractor to align construction efforts in light of saving future operations and maintenance costs. The challenge, then, is to align the outcomes of decisions related to lower level KPIs with higher level agency and societal goals and corresponding measures, as well as choosing measures that most validly reflect goal achievement.
The Dutch Transportation Performance Model is useful because it provides five levels—listed here from low to high—of performance measures relevant to this analysis:
Five. Basic materials and processes: meeting minimally acceptable standard specification levels and processes.
Four. Materials properties: aspects that the contractor may have some control over, and are specified by the agency in traditional project delivery, for example, fatigue, elasticity, compactability.
Three. Construction behavior and practices: including, for example, identifying appropriate labor needs, use of specified equipment, and adherence to optimal schedules.
Two. Functional performance: Measures in this level relate to the experience that the road user encounters when traveling on the highway—obviously after the construction has been completed and the roadway is open. Relevant measures here include skid resistance, pavement smoothness, noise reduction, load-level capacity, and snow and ice removal.
One. User and societal demands: Measures related to safety, accessibility, and congestion levels are appropriate here (Garvin et al., 2011).
With a performance management system, problems may occur for a variety of reasons. First, measures, especially for higher Levels 2 and 1, may not be specified or insufficiently specified in the P3 contract. Second, the public partner monitoring or contract management processes for any level of measurement may be inadequate or non-existent. Third, there may be a lack of understanding by private partners on the degree to which specific decisions or choices, for lower Levels 3, 4, and 5, affect the degree of alignment with measures for higher Levels 2 and 1. Fourth, penalties or incentives for non-performance may not be sufficiently specified by the P3 contract.
There is the commonly accepted assumption that with traditional design bid build construction contracts and separately awarded contracts for operations and maintenance, optimal efficiencies are not realized; for example, those in charge of construction do not have sufficient incentives to implement construction practices and materials that will limit operations and maintenance costs. 9 DBFOM P3 contracts, in theory, provide incentives for private partners to link efforts made relevant to performance measures for Levels 3, 4, and 5 to Levels 1 and 2. The risk, though, is that the linkage between specific lower level measures and the higher level ones may not be fully understood. For example, pavement smoothness (Level 2) may be optimal because of choices in materials (Level 3), but desired safety goals (Level 1: number of accidents per time period) may not be met.
To illustrate these different levels of measures, the KPIs for the I595 P3 contract (Florida) includes those for pavement smoothness (e.g., “rutting maintained at a depth of less than 0.25 inches”). This is one KPI—a level two measure—identified as part of efforts to achieve the goal of “maintain flexible pavement at acceptable level of safety for the traveling public” (Florida Department of Transportation [FDOT], 2009; Garvin et al., 2011, p. 83). Found elsewhere, on a separate website, is the Florida 2012 Safety and Security Performance Report 10 that states a safety-related goal for the state agency is “to reduce highway fatalities and injuries by 5% annually”—a Level 1 measure. No reference to reducing highway fatalities is found in the P3 contract, nor does the latter report contain any mention of pavement smoothness.
Optimal KPI Choice and P3 Contract Components
Optimal KPI choice should result from an ongoing process in which appropriate analysis should be given to achieving P3 Project goals as well as larger regional or societal goals. As such, attention should be paid to creating an integrated performance management system, one in which measures representing a range of levels are found. Initial KPIs should be chosen early in the procurement process, as part of a benefit cost analysis, for example, and appear in initial procurement documents such as a Request for Proposal (RFP).
Once negotiations with potential private partners occur, modifications to the initially chosen KPIs may occur. Initial management plans and policies that affect KPI achievement should be identified as part of discussions establishing a performance management system. Data collection hardware/software and appropriate review processes should be identified and chosen. Initial standards and penalties can be negotiated, with the understanding that KPIs may evolve over time and require additional modification.
Although a P3 contract is designed to give the private partners flexibility in choosing the means by which goals are achieved, it is anticipated that creating such a contract that contains the above characteristics may be unfamiliar or daunting for prospective private partners in some contexts. As much as possible, the RFP should identify relevant aspects if necessary. For example, in collecting data about average travel times, relevant to achieving congestion relief goals, the use of appropriate sensing technology could be suggested or encouraged. Furthermore, choosing safety measures should require the private partners to collaborate closely with law enforcement personnel as well as already existing incident management programs managed by public partners.
All penalties and incentives should be linked to both performance standards and data collection and reporting policies. As discussed below, a sliding scale reflecting the degree to which standards have been met is preferable to a “pass/fail” approach in which if the standard is not met, no incentive payment is made or penalty is paid. In addition, given that data are collected and self-reported by the private partners, if the data are not reported by the appropriate deadline or are found to be invalid, additional penalties should occur.
The following are examples from two P3 contracts: East Link Project (Australia) and Sea to Sky Highway Improvement Project (British Columbia). In neither case is there any indication that an integrated performance management system has been created.
East Link Project
Five categories of performance measures were identified, with an overall total of 51 measures:
customer service and satisfaction—11 measures
road condition: incident response, lane availability, and tunnel closures—14 measures
landscape and architectural maintenance—1 measure
tolling accuracy—22 measures 11
environment—2 responses
The private partner, ConnectEast, is also directed to report all data on a quarterly basis. If standards are not met, penalties will occur in the form of credits issued to travelers who pay tolls to use the roadway. If performance is deemed to be unsatisfactory, ConnectEast must provide a plan to improve performance (Batchelor & ConnectEast, 2004).
Good performance is rewarded by a lack of accumulation of KPI points or credits, totaled from across all performance categories. Payments made to East Link customers are based on a specified scale according to the amount of points or credits accumulated. For example, beginning with the 4th year of operations, 500 KPI points results in a Aus$2.5-million payment, with a maximum of 2,000 KPI points resulting in a Aus$15-million payment. (Batchelor & ConnectEast, 2004). In other words, if less than 500 KPI points are accumulated in any given year, no payment is made to any travelers.
These KPI examples reflect measure and performance standard choices that may not lead to creating optimal performance management systems (see Table 1). If the system combines measures and standards from a variety of operational areas, careful consideration must be given to the weight or importance of each measure/standard. In addition, “pass/fail” standards are likely not to provide sufficient incentives to ensure appropriate performance effort.
Societal and User Demands: KPI Examples—East Link Project.
Note. KPI = key performance indicator. (Batcherlor and ConnectEast, 2004: 364,365,367,371)
In part, determination of appropriate measures and standards should reflect the project and the agency or societal goals, as well as stakeholder input/feedback. In addition, the impact of not meeting the goal or standard should be assessed. For example, safety goals that are met in part by speedier response to accidents may be given more weight than goals to respond to customer phone calls. The impact of delays in getting a victim to the hospital may be judged as far greater than a customer who receives a busy signal when calling.
The differences in weight can be reflected by differences in the standards chosen. Given the examples above, a 10-point penalty for arriving late at the scene of an accident seems insufficient, compared with the 10-point penalty for not responding to a customer complaint within 2 days. In addition, the lack of a sliding scale for standards is apparent, as the time periods required for response times are reflected in pass/fail standards. The penalty for responding to an accident in 11 min—1 min over the standard of 10 min—is the same as responding to it in 11 hr, or, not responding at all!
Not reflected in these measures are several considerations in estimating the amount of resources necessary to meet specified standards. If there is an already existing Incident Management Program in the region or nearby, efforts at collaboration and sharing of resources may reduce overall cost. At a minimum, efforts must be made to ensure understanding of how key terms are understood in terms of programs, defining, for example, what response means when measuring response times to an incident. 12
Level 2 measures, those titled “pavement performance measures,” refer to those that measure characteristics such as pavement smoothness. The KPI points for these are “10 points outside of the KPI Benchmark.” This penalty further supports the lack of any link between Level 1 and 2 measures, as it is the same as that for the Level 1 examples cited above. There is no other indication in the P3 contract that any linkage exists.
These considerations should also reflect the acceptance of a need to change measures, standards, and efforts needed to perform at appropriate levels. To the extent that data concerning workload cannot be accurately forecast, there may be need to add measures and change standards to more appropriately reflect measures such as numbers of incidents and customer complaints. Estimated resources may be insufficient, or not as many personnel may be required. It may be recognized that originally agreed standards may be too low to provide sufficient incentives leading to appropriate performance efforts. There must be recognition that incorporating design and construction practices that increase roadway safety may not be sufficient to reach identified safety goals during operations.
Sea to Sky Highway Improvement Project
Many fewer performance measures are identified in the Sea to Sky Highway Improvement Project (STS) than found in the East Link example, as tolls are not collected. To illustrate relevant issues and challenges, a performance measure and standard related to safety are discussed.
The relevant measures are the number of fatalities, injuries, and property damage crashes along the 81.5 kilometer highway. The standard for these measures is calculated by a formula that compares the record of the STS highway with the accident record of similar Provincial highways, more specifically, a “portfolio” of similar highways, including Kicking Horse Canyon and Coquihalla Highways. The formula (Equation 1) states that if the accident rate is no more than 1.5 times the accident rate on these similar highways, the agreed annual payment for safety will be made (Ministry of Transportation, British Columbia, 2005: Schedule 10 part 6: 29).
Sea to Sky Highway Improvement Project: Safety KPI and Standard.
Several issues are raised with the standard calculated at 1.5 times the provincial average. First, to maximize accountability to citizens residing in surrounding communities, plus to those traveling on the STS, stakeholder input should have occurred in determining this standard. It is doubtful that such input did occur, as public agreement to a standard higher than for other similar highways in the Province seems unlikely.
Second, similar to the above discussion, it is a pass/fail standard, as 1 or 100 incidents above the 1.5 times standard results in no payment made. Providing a sliding payment scale, so that greater rewards are forthcoming for increased safety as measured, would lead to a more fully thought-out performance management plan that identifies efforts to improve safety. The challenge for the optimal performance management system, then, is to ensure that appropriate private partner efforts are being made to increase or improve safety. With a pass/fail measure, higher than the average, the risk is that no plans will be created, as private partners may rely only on highway construction efforts to improve safety. Without any requirement for Incident Management Programs, for example, efforts to respond to accidents may not be sufficiently resourced.
Conclusion
The choice of APs, in which the government or public partner assumes all demand risk, has several advantages, including a more transparent, lower project cost, leading to potentially greater public support and acceptance of a specific P3 project. Potential negative outcomes such as private partner receiving windfall profits—if demand is higher than expected—or experiencing bankruptcy—if demand is lower than expected—are eliminated or greatly reduced.
The use of APs can result in the effectiveness or success of a specific P3 project, however, only if an optimal performance management system is required. The choice of KPIs that reflect both specific measures and appropriate performance standards determines the extent to which a highly effective and integrated performance management system is in place. Measure choice should reflect different levels, ranging from those dealing with safety to pavement smoothness, with an overall understanding of how efforts to support one measurement level can complement efforts to support another level. As illustrated by the two case examples, the risk is that standards do not adequately reflect the impact or importance that each measure contributes to overall P3 project goals (East Link), or set standards on a pass/fail basis that are too easily attained (STS).
Once demand risk is removed from private partners, incentives and sanctions that accompany the stated performance standards during operations and maintenance are the only means by which optimal performance can be ensured. An optimal performance management system for private partners in a P3 contract does not dictate the means by which private partners achieve performance standards but should require identification of existing efforts and improvements if results do not meet existing standards.
Although the discussion here focuses on transportation infrastructure–related P3s, implications for other types of infrastructure P3s are relevant, whether they be county courthouses, wastewater treatment plants, or schools. 13 If private partners will oversee operations and maintenance after construction is completed, the implementation of an optimal performance management system is a key means to ensure continued high quality performance that results in confidence that public support will be maintained throughout the life of the contract.
If the goal of a performance management system is to improve performance, there will be a need to adjust measures and standards over time. To the extent that initial standards were set too low, raising standards may be favored by public partners and resisted by private partners if additional resources may be required to meet higher standards. There is also the risk that over time complacency may occur, as the focus is on the process of reporting and reviewing accident data, for example, without consideration of changes that may be needed to improve results. A cultural change may be needed, creating a greater understanding of the value of collecting data, comparing results with agreed-upon standards, and making adjustments in processes designed to improve performance.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this research was received from a grant funded by the Florida Department of Transportation.