On Messes,Systems Thinking,and Evaluation

Continuous development of a profession and professionals requires at least as much concern with those they try to serve as with the means by which they try to serve them.

Russell Ackoff (1991, p. 154) Recent research on evaluation illustrates that, despite many noteworthy efforts to integrate evaluation’s theory into its practice and its practice into its theory, more often than not, the gap between theory and practice remains a chasm (Christie, 2003; Christie, Quinones, & Fiero, 2014; Coryn, Noakes, Westine, & Schröter, 2011; Miller & Campbell, 2006). According to Patton (this issue), his Developmental Evaluation (DE) model is among the latest approaches to suffer infidelity in its application. Patton observes DE has been applied in situations for which it was not designed and in ways he and other expert developmental evaluators do not recognize as in keeping with the aims and spirit of the approach.

To help evaluation professionals better understand what an authentic attempt at DE ought to look like and when it is appropriate to use, Patton offers a list of sensitizing concepts that, taken together, succinctly define the approach. Patton argues his approach is best conceived of in terms of sensitizing concepts, noting these orienting ideas are especially well suited to the case for which DE was originally created: situations in which people are attempting to develop what they believe are innovative solutions to intractable problems. Renowned systems thinker Russell Ackoff characterized these kinds of situations as “messes” (Ackoff, 1999), akin to Rittel and Webber’s (1973) concept of a wicked problem. Messes are complex, interacting, and synergistic problems. Messes are characterized by tremendous uncertainty about how they ought to be conceptualized and understood (Ackoff, 1999; Flood, 1999; Midgley, 2000). Messes have no obvious, single, or best solution and defy our intuition (Forrester, 1993). Messes require innovation.

A key test of any evaluation theory is whether it is replicable by others, feasible to conduct, and has a unique and recognizable signature each time it is applied (Miller, 2010). Patton wants DE to do well on this test. His essay attempts to distill the approach to its essence. Can these eight concepts help evaluators assist their clients to navigate the messes that call for innovative problem solving? Are they sufficient to clarify when and how to conduct a DE? Sensitizing concepts are orienting in the sense they suggest the evaluator focus on some things and not on others. To be sensitized to a concept requires more than a passing grasp of loosely defined terms. And herein lies the challenge Patton has given to those practitioners who might pursue the approach. To orient to all eight concepts and to use each in combination, to do work that truly qualifies as DE, an evaluator must have depth of command in at least three distinct areas of scholarship and in the diverse methods that accompany each area. Patton’s concepts derive from his utilization-focused evaluation approach, with which many are well familiar and consider a mainstay of their practice. But they also derive from multiple schools of systems scholarship, methods, and practice with which many practitioners are unlikely to be well versed, and from collaborative inquiry models from within and outside systems scholarship. In addition, the sensitizing concepts Patton offers require the evaluator to have highly evolved skills in ferreting out when stakeholders have genuine commitment to innovative processes and in forecasting that innovative solutions have a reasonable likelihood of evolving from those processes.

I share Patton’s commitment to systems thinking, as do many other evaluators, and agree wholeheartedly that systems thinking offers the potential to enrich evaluation practice in important ways. Systems thinking is especially well suited to the task of grappling with the messes that prompt innovation because systems thinking requires that problem solvers resist conceptualizing problems and solutions in overly simplistic, acontextual, and ahistorical ways. Systems thinking demands problem solvers avoid working through problems on the basis of a single perspective (Flood, 1999; Midgley, 2000). As Reynolds and Holwell note, systems thinking is an antidote to reductionism and dogmatism in social problem solving (Reynolds & Holwell, 2010). Applying systems thinking to problem solving can also lead to the identification of very powerful interventions (Meadows, 2008).

Mastery of any of the major systems thinking traditions that inform DE—complexity theory, soft systems, and critical and complex adaptive systems—or of other schools Patton referenced in early treatments on DE (e.g., cybernetics, system dynamics) is no small feat. Each comes with particular methodology as well as a unique perspective on how to understand a problem situation and determine its multiple boundaries. Each also brings an ideological stance on which users are important and each suggests how local users ought to be engaged by system practitioners. For instance, critical systems heuristics has as its primary sensitizing idea boundary critique. It also prescribes a multiple-step participatory team process for planning (Ulrich, 1996; Ulrich & Reynolds, 2010). System dynamics, by contrast, orients around feedback and delay within systems and the development of computer simulation models to gain insight into the dynamics of a problem (Forrester, 1993; Hirsch, Levine, & Miller, 2007; Sterman, 2000). System dynamics has developed its own tradition of collaborative modeling to capture multiple perspectives and identify promising solutions (e.g., Homer, Hirsch, Minniti, & Pierson, 2004; Hovmand, 2014; van den Belt, 2004). Yes, there are broad, shared similarities among various schools of thought, as Patton notes, but at the razor’s edge of practice, the differences in concepts, methods, and practice can be stark. And, the translation of system scholarship, perspectives, methodologies, and practices to evaluation is not necessarily straightforward, particularly when overlaid with other imperatives such as foreseeing innovation, providing rapid feedback, and engaging in utilization-focused and collaborative inquiry.

DE lays out a set of appealing ideals, which is no doubt why so many evaluators have gravitated toward the approach. But I’m willing to lay bets it is the rare evaluator among them who can boast sufficient depth and capability in all of the areas Patton identifies to conduct a successful DE as a solo practitioner. No other evaluation approach of which I am aware involves an evaluand that is potentially in a state of perpetual emergence. For an evaluation to proceed under the circumstances for which DE was developed, the integration of systems expertise seems helpful if not essential. Patton’s point—all eight principles must be strongly in evidence to make the claim a DE occurred—bears noting. Systems thinking offers ways to learn in the context of unknowable futures (Flood, 1999) but not through mere use of its concepts alone. If DE is to be done well, then the practitioner must learn what systems concepts mean and how to carry out the methods that go with them, so these can be mindfully adapted to the evaluation context. As Jay Wright Forrester, the founder of system dynamics wisely observed in a 1992 interview, it is not enough to engage in an undisciplined use of select system concepts (e.g., boundaries, perspectives, and relationships) if the goal is to understand complex problems and engage in innovative social problem solving. Systems thinking alone, says Forrester, “brings you less than 5% of the way towards a genuine understanding of systems” (as cited in Richmond, 1994). Patton’s provocative essay will not substitute for investing in the training in systems thinking and methods that realizing his vision for fidelity in DE practice requires.

Patton is among our most appealing theorists because his theoretical work reflects the careful integration of his experiences in practice gained over years of trial. He attends closely to what his clients find helpful. He uses his practice as the place to discover and test emerging ideas. He pays close attention to practitioners’ experiences applying his ideas. He aspires to get it right and say it clearly. He writes with the larger goal of helping practitioners and their clients get the best possible evaluation for their particular needs. Indeed, over the course of the few months following my first read of his essay, he dramatically revised it, making it more focused, clarifying definitions, and sharpening arguments. The sensitizing concepts Patton articulates in this essay (which are fully elaborated in his latest book, Patton, McKegg, & Wehipeihana, 2015) make plain DE requires evaluators: accept the logical contradiction that one can know innovation will emerge in advance rather than, as is ordinarily the case, with the benefit of hindsight; carry an unwavering belief in the power of systems thinking and its methods; and know how to fulfill the unique role evaluation might play in supporting innovation. Patton gives aspirants of the approach a tall order, asking they make a serious commitment to and investment in careful study of systems thinking and methods. Eight seemingly simple sensitizing concepts are manifestly appealing on their face. But I foresee many years of studying and practice-informed tinkering before the average evaluator will find adequate and manageable guidance in this elegant set of orienting concepts for evaluating in the context of emergent innovation.