EBM: Rationalist Fever Dreams

Rule Bound or Highly Adaptive?

In many domains, in many contexts, there has been an overwhelming desire to rationalize practice through plans, procedures, algorithms, automation, and standards, including best practice guidelines. These tactics can be useful aids to improve practice in a world of finite resources, surprises, and change. The “rationalist fever dream” casts these support mechanisms in a different light, though: It asserts that effective practice consists of complying with rules; however, these rules are embodied. Even more, the rationalist fever dream imagines that the only standard for rational behavior is compliance with plans, procedures, algorithms, and standards. As the fever worsens, the pressures to conform to rules grow, which further narrows the range of performance possible—work to the rule/work to the plan/delegate judgments to algorithms.

We assert that the rationalist fever dream is a kind of oversimplification that affects managers, researchers, and practitioners in many domains (Feltovich, Spiro, & Coulson, 1989). The fever is manifested by the strength of exhortations to rely just on procedures, principles, rules, algorithms, and standards. When the rationalist fever dream takes hold, various problems ensue, and these are evident in the case of evidence-based medicine (EBM).

We appreciate that EBM is not supposed to be about compliance with rules. It is foremost about the effort needed to build a stronger evidence base regarding the effectiveness of different types of interventions that can be used to support situation decisions and actions. Nevertheless, as EBM efforts typically seek to synthesize the evidence into best practices guidelines, it becomes a natural next step to ramp up the pressure to comply with those guidelines out of context. We object to the tendency for EBM to lead to a rule-bound system as this pressure for compliance builds.

Research into human-machine systems and naturalistic decision making (NDM) over the last 35 years has shown a gap between work as imagined from distant parties and work as done, because practitioners have to cope with inevitable complexities (Braithewaite, Wears, & Hollnagel, 2017; Perry & Wears, 2012). As Suchman (1987) noted, plans are resources for action and not specification of action. For example, plans and rules exist out of time and therefore are unable to help keep pace with events and activities as a patient’s difficulties grow. The fundamental limits of rules and plans require adaptation in the face of inevitable surprises. In fact, the ability to recognize the limits and brittleness of plans/rules can be valuable as one aid to see the unexpected and drive adaptation (Cook, Render, & Woods, 2000; G. Klein, Pliske, Crandall, & Woods, 2005).

The rationalist stance misses the fundamental limits of work to rule and work to plan. To make matters worse, the rationalist fever dream imagines that rational behavior is compliance with rules and procedures. Under this mind-set, management’s duty is to increase pressures for compliance (Dekker, 2018).

However, research studies continue to reveal how increasing pressures for compliance with rules undermine the ability to adapt. Roth, Bennett, and Woods (1987) described the performance constrictions that arise when a system is rule bound through the latest technology for automation, and they showed how people in responsible roles had to adapt to make the system work. Finkel (2011) documented how rule-bound military organizations were unable to effectively adapt to surprising events and changes. When organizations increase pressure to work to plan/rule, workers engage in covert work practices to get the work done despite inadequate systems—these underground adaptations expand the gap between work as imagined and work as practiced (Perry & Wears, 2012).

Falzer (2018) uses the example of treatment for rheumatoid arthritis. The format and content of this set of guidelines illustrate the wide variation in presentation of a single disease. Even more striking, however, these guidelines illustrate how the experts contributing to them acknowledge and accept the necessity for adaptive decision making by clinicians to accommodate patients’ needs. As opposed to firm EBM controls, this set of rheumatoid arthritis guidelines, with its mixture of approaches, is actually a more rational and reasonable stance because it is guided by evidence of the reality of what is required to work in a dynamic, variable, and uncertain environment: adaptive, rather than constrained, decision making.

The contrast between rule-bound and highly adaptive systems becomes even more important as complexity grows—which is the situation with modern health care. Rule-bound organizations exerting more and more pressures for compliance are unable to keep up with the pace of change in the increasingly interdependent systems of today (Woods, in press). For the rationalist stance, there is a best model of work; so, of course, all should work to that model. The findings from studies of adaptive systems in complex worlds show that these models are guaranteed to become stale and inadequate. Effective performance depends on being able to revise or update the models/plans/rules that guide work, rather than rigidly adhering to the current models/plans/rules in play (Finkel, 2011; Woods, 2017).

Of course, the rules keep changing—that is called progress. A recent example of this in health care just occurred with regard to acute stroke management algorithms to reduce tissue damage, which increased the window from 3 to 6 hours (the rigid, hard rule for the past decade) to up to 16 hours because of new tools and techniques. Another example is the inclusion of emergency medical services personnel in treatment procedures for heart attacks such that the work of problem detection and identification now begins at the scene. Emergency medical services team can, at times, facilitate getting the patient from home straight to the catheterization laboratory within minutes on the basis of their assessment. These examples illustrate the ability of health care to adapt and expand the boundaries of the sociotechnical system within which clinical work occurs.

Reducing the Fever

In health care, counter to the results of NDM and related research, EBM and best practices have become entangled in the rationalist fever dream. Our original paper (D. E. Klein, Woods, Klein, & Perry, 2016) attempted to reduce the fever by pointing to six challenges:

Falzer’s paper leads us to suggest some additional cognitive challenges. One is to determine the currency of the evidence—judging whether the findings have become stale or obsolete. Falzer notes that by the time carefully controlled, double-blind studies have been designed, funded, implemented, submitted for publication, and finally disseminated, the treatment being evaluated may be superseded by different or at least modified methods. So practitioners have to consider this dissemination lag and scrutinize the details of the treatment being assessed.

This is related to Challenge 5—revising treatment plans that do not seem to be working—but it addresses perturbations in the system rather than inadequacies of the initial plan. The initial plan may have been fine: a skilled decision maker will be alert to the need for changes necessary to support workflow and will anticipate potential bottlenecks ahead to be prepared to compensate. An example is a technician placing a second intravenous (IV) site for an elderly patient with chest pain, despite the absence of an order. On the basis of a cold and clammy appearance, the patient may require more than one IV medication and need a second site. The technician’s adaptation to the situation supports several goals—fewer needle sticks to the patient, reduced chance of workload bottlenecks should she or he be occupied with another patient when the first patient requires another IV site, and reduced risk of being unable to get an additional IV site placed should the patient suddenly lose blood pressure or go into arrest.

Work as performed depends on the expertise of the practitioners, and evidence-based approaches set up a very unfortunate conflict between taking advantage of expertise in context versus relying on study results in general when both are valuable. Wears and Klein (2017) described the ways that the medical community has come to discount and ignore the strengths of clinical judgments. G. Klein, Shneiderman, Hoffman, and Wears (in press) described how the evidence-based movement tends to dismiss the existence of expertise. We agree with EBM about the value of experimentation to identify ineffective treatments, but we see little value in polemics against practitioner expertise; these polemics ignore the considerable evidence demonstrating practitioner skills. The NDM approach would seek to accelerate the growth of expertise in health care practice rather than constraining clinician judgments and creating a rule-bound system.

Evidence, plans, automation, and procedures do not have to become a rule-bound system. All can, should, and have been part of highly adaptive systems (e.g., the organizations that were able to adapt to surprise in Finkel’s analyses [2011]). Adaptive systems recognize when unexpected events and changes challenge the boundaries of work to plan and invoke the ability to adapt/update plans to fit situations (Woods, in press). Organizations can be designed for adaptability that builds on plans rather than be constrained to just comply with plans.