Campbell’s Law: implications for health care

Introduction

Holding people accountable to a set of highly incentivized performance targets is plainly problematic. Such pressure can result in actors in a system omitting important processes that do not directly influence targets, engaging in selective external reporting and resorting to even more significant deception and corruption. This issue has been highlighted in the health care domain. ¹ Campbell’s Law states, ‘the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor’. ² This axiom was developed in the 1970s by the celebrated American social scientist, Donald T Campbell. He argued that if a particular metric is used to measure success and to drive system changes, that metric’s ability to accurately measure success tends to be compromised. He noted that such metrics become unreliable as actors in the system are pressured to reach targets at the expense of any other aspect of the endeavour at hand, lie in the reporting of such metrics and otherwise adopt corrupt practices for purposes of achieving targets.

Historical examples

A few specific examples Campbell cites illustrate the law well. For instance, he points to examples of police departments held accountable for crime clearance rates (i.e. the percentage of crimes solved) failing to record every complaint or postponing recording complaints until and unless they are solved. Campbell also points to an early period in the Vietnam War when the use of “body count” (the number of the enemy dead left on the battlefield) was used to evaluate the effectiveness of US military units. He notes that how optimizing this quantitative metric began to compete with and even supplant more traditional goals of warfare to the detriment of the US war effort. He also highlights the infamous Texarkana educational performance experiment. Contractors were hired to provide supplementary education and tutoring to students. The contractors were paid based on student’s improvement on standardized tests. As it turns out, the hired tutors were teaching items taken directly from the final test to be administered, rather than tutoring robustly on math, science and reading.

Health care analogues

These specific examples ought to serve as a cautionary tale for the pay-for-performance/pay-for-quality faithful. For years, various health care delivery interest groups, experts and other stakeholders have decried the perverse incentives wrought by the fee-for-service system of provider payments. Reimbursement systems with the flavour of pay-for-performance and pay-for-quality are gaining popularity.

As we drive deeper into these novel payment arrangements, we ought to take stock of the examples of Campbell’s Law that have already been exhibited in health care in the United States (US). For instance, since 1992, the Pennsylvania Cost Containment Council has published a public report, “A Consumer Guide to Coronary Artery Bypass Graft Surgery”, highlighting in-hospital mortality and readmission rates for certain cardiac procedures, broken down by facility and by individual surgeon. In a study published in 1996, a majority of surveyed cardiac surgeons in the state reported that this annual report made them less willing to operate on critically ill patients needing the procedures, though it is unclear if their practice patterns were actually altered as a result. ³ To highlight another example, we can look at the impact of treatment guidelines for ST-elevation myocardial infarction (i.e. STEMI). The recommended door-to-balloon time (i.e. the time from when the patient arrives at the hospital to when that patient undergoes percutaneous coronary intervention) established by the American Heart Association and the American College of Cardiology is 90 minutes or less. ‘Time is muscle’, as the old adage goes; and the faster the artery blockage is dealt with, the better the patient will fare. However, researchers found that while national door-to-balloon times declined significantly in the years following the establishment of the treatment guidelines, in-hospital mortality remained unchanged. ⁴ In light of these findings (which have been repeated in other studies), there is notable concern that while door-to-balloon time is indeed an important aspect of improving care for STEMI patients, an overemphasis of this metric may also be diverting health care providers’ attention and resources away from other drivers of effective treatment in these cases. ⁵

A final (and more controversial) example of Campbell’s Law at work deals with the interplay of hospital readmissions and hospital observation stays. In 2012, per the Affordable Care Act, Medicare began issuing financial penalties to hospitals exhibiting “greater than expected” 30-day readmission rates. Since that time, the average 30-day hospital readmission rate for Medicare fee-for-service beneficiaries has declined. The Center for Medicare and Medicaid Services and other stakeholders attribute this improvement to improved transitions of care efforts and enhanced care coordination initiatives led by hospitals. Others believe that the continued rise in hospital observation stays (i.e. stays where patients are treated for one or more days in the hospital, technically as outpatients), which do not count towards hospitals’ readmission tallies, are also partly responsible for this trend.

Mitigation strategies

Fortunately, there are ways to mitigate the effects of Campbell’s Law. In general, there are two strategies available: (1) divorce the overarching goals of the health care delivery system (i.e. keeping populations healthy, delivering cost-effective and high-quality care, etc.) from the measures used to track progress towards that ideal or (2) implement necessary safeguards to minimize corruption and distortion of the metrics used for accountability. The first strategy entails measuring the relevant outcome, process, structural and quality metrics but not directly tying these to any form or system of reward or punishment. Health care providers would utilize these measures internally to make continuous improvements in the name of quality and patient safety. Campbell himself supported such a strategy for education—he asserted that standardized testing may be valuable “under conditions of normal teaching aimed at general competence”, that is, essentially under appropriate and typical professional behaviour. ² Of course, the principal issue with this strategy is how to enforce accountability in an environment where objectives are completely separated from the measures used to track success towards those goals. Many may doubt that health care providers will do the right thing without a proper incentive system.

In light of this shortcoming, the alternative strategy to mitigate Campbell’s Law may be more palatable. The idea behind the second strategy is that metrics can be valuable indicators and drivers of quality and performance if care is taken to minimize corruption pressures. One specific tactic includes utilizing multiple measures to evaluate care in order to decrease the pressure to manipulate any single metric. Medicare’s Hospital Value-Based Purchasing initiative in the US is a good example of this, as it incorporates multiple measures along different dimensions (e.g. efficiency, patient experience, clinical process, and outcomes). The diversification of measures used should paint a clearer picture of overall system performance. Such diversification becomes increasingly important as more consequences are attached to the metrics. Another useful safeguarding tactic is independent verification of reported data. In the health care context, this would entail regular auditing of relevant documentation and periodic observation visits from external inspectors. The Joint Commission, a United States-based nonprofit organization that accredits health care organizations in the United States, performs some aspects of this with its periodic surveys as part of its accreditation process, but a much more robust system would have to be implemented to ensure the veracity of reported data from health care providers, and this could be quite costly.

Conclusion

Campbell’s Law states that anytime a measure of success becomes a target for players in a system that measure loses its value as an indicator of success and tends to distort processes within the system. The ramifications of Campbell’s Law have been described across industries including health care. As health care systems continue to move away from fee-for-service reimbursement towards so-called value-based payment models, health policy makers and other stakeholders ought to be aware of Campbell’s Law and its implications. The detrimental consequences of Campbell’s Law can be mitigated with a thoughtful approach to utilizing metrics. Systems will have to decide how best to balance the need for undistorted data and uncorrupted behaviour with the need not to burden the health care system with extra bureaucracy and additional administrative costs.