Reconstructing professional ethics and responsibility: Implications of critical systems thinking

Historians and sociologists remind us that it is disingenuous to ignore the fact that despite differences in research practices as well as the degree of scientization and compartmentalization of the social sciences between the United States and Europe there have been continuous communications between Europeans and Americans over their meaning, purpose, and methods (Mancias, 1987; Wagner et al., 1991). A similar observation about the field of evaluation is warranted, not least since the advent of Evaluation founded in 1995 as ‘an international journal with strong European roots, but with a commitment to encourage dialogue between European, Scandinavian, Northern American, Asian, Australasian and other existing and emergent evaluation communities’ (Stern, 1995: 6). One issue of growing concern in the transnational evaluation conversation that Evaluation encourages, and one that has received a fair amount of attention from European contributors ¹ to Evaluation in the past 20 years, is the relevance of the scientific study of complex systems (i.e. complexity science) and systems approaches to managing complex issues (i.e. systems thinking) to the theory and practice of evaluation. Looking forward, I offer a few comments on one key idea in this conversation.

Evaluation scholars and others have argued that value is not inherent in an intervention (object or activity) being evaluated, but rather is something ascribed to it by those involved in observing it (Stake and Schwandt, 2006; Wadsworth, 1997). Scholar-practitioners working in the tradition of critical systems thinking and specifically in critical systems heuristics (CSH) (e.g. Midgley, 2007; Reynolds, 2014; Ulrich and Reynolds, 2010; Williams, 2015) have expanded this insight by developing a particular approach to making value judgments called boundary critique. Following a brief sketch of this approach, I turn specifically to its implications for the current transnational conversation on the competencies, capabilities, and professional obligations demanded of evaluators.

While judging value is presumably the very purpose of evaluation, evaluators have long struggled with multiple issues surrounding making judgments or appraisals of the value (merit, worth, significance) of an intervention (e.g. activity, policy, program, and so on). Debates have centered on how explicit an evaluator must be about the criteria employed for judgment – compare for example the fairly precise approach to identifying criteria and performance on those criteria as advocated by Scriven (1994) and Davidson (2005) to Stake’s (Stake et al., 1997) wholesale opposition to ‘criterial thinking.’ Controversy is associated with the extent to which and the manner in which stakeholders (and which ones – managers, commissioners, beneficiaries?) are to be involved in determining criteria (Cullen et al., 2011; Mark and Shotland, 1985), and whether the evaluator per se is the person ultimately responsible for the judgment (Alkin et al., 2012). Disagreement surrounds whether there ought to be a universal set of criteria (e.g. the OECD/DAC criteria of relevance, effectiveness, efficiency, impact, and sustainability for evaluating development assistance), whether a particular criterion or set of criteria are always paramount (e.g. cultural responsiveness, gender-responsiveness, equity-focus), or whether criteria are always matters to be negotiated in particular settings. Equally troublesome is the fact of considerable disagreement on how best to deal with multiple criteria, the so-called aggregation or synthesis problem. And, finally, disputes surround both the applicability and the justification of various methods for valuing (e.g. determining impact via experiments; judging efficiency by means of cost-benefit techniques) in specific contexts (Julnes, 2012; Stern et al., 2012).

To this contested context of valuing in evaluation characterized by lack of agreement on methodologies as well as responsibilities, the tradition of critical systems thinking adds a new approach known as boundary critique. Every evaluative inquiry is ‘bounded’ in the sense that particular facts and values bearing on determining the value of the intervention under consideration are either included or excluded from analysis. Certain criteria of performance, for example, are considered more or less relevant, and certain kinds of evidence of performance are considered more or less important. These boundaries or choices are not naturally given (e.g. as features of the context) but social (and personal) constructions that define what is to be taken as germane to the analysis of value. For example, in a recent project at the International Development Research Centre (IDRC) in Canada in which I was involved, the task was to develop an approach to judging the quality of research for development. That inquiry was ‘bounded’ in different ways by different stakeholders who had different stakes in the activity. Researchers in the global South, IDRC’s partners in development, tended to place high value not simply on the scientific merit of the research but on its relevance to local needs and circumstances. Some members of IDRC management, reasonably concerned about IDRC’s position in the global conversation about research quality taking place with other funding partners (e.g. Bill and Melinda Gates Foundation, Canadian Institutes of Health Research, the William and Flora Hewlett Foundation), tended to argue for the primacy of traditional criteria of rigor or scientific merit. In critical systems thinking, the primary concern is how these different ways in which an inquiry such as this is bounded are negotiated. This involves an activity known as boundary critique that is a matter of making and appraising judgments that determine which facts and evidence and which value considerations (criteria) count as relevant and what others are considered marginalized or irrelevant and left out. Moreover, because the boundary judgments we make are closely tied to the values we hold, boundary critique supports moral reflection on the purpose of an intervention itself (i.e. whether it is the right thing to do). Hence, critical systems thinking emphasizes the importance of dialogue across different value framings of an intervention or activity, that is, what purposes it promotes (Williams, 2015). The objective of boundary critique is to make value judgments transparent: ‘to identify the exact nature and scope of the claims to which boundary judgments give rise – for example, what is claimed to be achieved and who is supposed to benefit, and how can this choice be justified rationally – and to submit these claims to the critique of the different parties concerned’ (Ulrich and Reynolds, 2010: 263).

As an aid in making such judgments CSH offers a set of 12 boundary questions as a heuristic device for exploring sources of motivation, control, knowledge, and legitimacy across the roles, specific concerns, and issues of stakeholder groups that include beneficiaries, decision makers, experts, and those individuals or groups negatively affected by but not involved with the activity, project or policy in question (Ulrich and Reynolds, 2010). This heuristic can be used in a self-critical way to question one’s own judgments as well as dialogically to explore whether judgments across concerned parties conflict or agree and why. Boundary critique involves rationally justifying the drawing of those boundaries, and that in turn involves moral argument about what and who may be marginalized by boundary judgments (Midgley, 2000). Critical systems thinking holds that this kind of argument should proceed dialogically; in other words, boundaries are defined by means of a dialogue among those affected by and involved in an intervention, and, in some circumstances, also via the involvement of ‘people who might not be defined as directly affected or involved, but who may nevertheless have an important perspective to bring to bear on the boundaries of the intervention’ (Midgley et al., 1998: 470).

Much greater detail about critical systems thinking and CSH is provided in the references cited here. Suffice it to say for present purposes that there are three central claims underlying the idea of boundary critique as an approach to evaluative judgment. First, is the idea that all claims about the merits of interventions are partial in the sense that they are selective with respect to what factual knowledge and value perspectives are taken as relevant and in that they inevitably benefit some parties more than others. Given such a condition, ‘the only practical approach is to examine the different selectivity of alternative proposals, as a basis for well-informed and transparent processes of opinion forming and decision making within democratically legitimate institutional settings’ (Ulrich and Reynolds, 2010: 263). The second claim is that practical decision making – involved in deciding what we ought to do now in this situation given this set of circumstances (see Schwandt, 2002) – cannot be merely a matter of instrumental reasoning. That form of reason is adequate to the task of appraising different means to a given end, but has nothing to say about the ends it serves. Our decisions about what is correct and what is right to do have a normative foundation, that is, they involve value judgments. Thus, what it means to act rationally in situations demanding practical decisions and action means attending to both empirical and normative considerations. The third claim holds that knowledge is not solely the province of experts, but is the product of a dialogue-driven approach involving both experts (professionals) and citizens.

Making boundary judgments – grounded in a rationale for critical systems thinking – is a new methodology for valuing in evaluation that invites significant reconsideration of the professional competencies and ethical obligations of evaluators. Ulrich and Reynolds (2010: 287) argue that CSH encourages a ‘new ethos of professional responsibility’ that involves a keen intellectual awareness that statements about the value (or lack thereof) of an intervention are conditioned by assumptions and that, consequently, evaluators ought to be more modest with respect to their own claims while simultaneously exhibiting mutual tolerance toward the claims made by others. Acknowledging the limits of one’s knowledge is of course an important virtue, but this new ethos involves more profound considerations. The idea that valuing involves boundary critique draws attention specifically to matters of moral argumentation and democratic professionalism in the practice of evaluation.

Appraising claims about the criteria (values), and subsequently the evidence, employed by experts and non-experts in judging the rightness, wrongness, or neutrality of interventions activities, projects, programs, policies, and so on entails knowing about the justification of moral claims (claims about what it is right or wrong to do, good or bad to be, and what should or should not happen). Understanding how to evaluate moral arguments is a necessity because reasoning about boundaries involves normative and not simply empirical concerns. Evaluators must be capable not only of assessing the properties of evidence offered in support of a boundary judgment but also of recognizing and appraising their own moral arguments and the arguments of others. This involves cultivating capacities in moral reasoning including moral imagination and moral sensitivity (i.e. recognizing the moral issue at stake) as well as an ability to recognize flawed moral arguments (e.g. those that employ faulty analogies, beg the question, use a straw man, or equivocate). Moreover, public argument about values is not easy, not least, as Zarefsky (2014) has pointed out, because democratic discussion and debate assumes some measure of human fallibility while individuals normally hold moral principles with near absolute certainty. Critical systems thinking presents a major challenge to the field of evaluation to explore the circumstances and conditions of responsible public argument about differing value perspectives. Attention to the skills, abilities, and dispositions demanded in moral argumentation requires much more than awareness of a broad set of ethical principles involved in designing, conducting, and reporting an evaluation, as currently promoted in efforts to define professional competencies needed by evaluators.

In the view of a leading proponent of CSH, critically systemic thinking – specifically, engaging in making and appraising boundary judgments – ought to be viewed as a common reflective competence of professionals and citizens alike (Ulrich, 2000). Yet more is at stake for evaluators than developing such competence. CSH invites us to consider the model of professionalism operative in the field of evaluation. I have been critical of an emerging model of technical and commercial (market-oriented) professionalism in evaluation (Schwandt, 2008, 2015). Technical professionalism is a kind of ethos and attitude as well as an institutional pattern of work less dedicated to popularly informed democratic decision making and more concerned with expert-informed policymaking (Fischer, 1990). Technical professionals ‘depoliticize issues by translating social problems into questions that can be solved by using the methods in which they are trained and over which they have greater command than the public’ (Dzur, 2008). In stark contrast to this model of professionalism stands the ideal of democratic professionalism (Fischer, 2000; Sullivan, 1995) defined by Dzur (2008) in the following way: ‘Sharing previously professionalized tasks and encouraging lay participation in ways that enhance and enable broader public engagement and deliberation about major social issues inside and outside professional domains’ (p. 130). Democratic professionals aim to bring affected publics (stakeholders) into problem-solving work. In so doing they restrain their own expertise and engage in a dialogue with local knowledge. This is just a sketch of this idea. There is much to explore here including the responsibility of evaluators for establishing democratic processes for ethical deliberation that allow for moral disagreements and the capacity of evaluators to engage in systemic mediation of differing moral frameworks (Midgley and Pinzón, 2013).

Critical system thinking draws our attention to the idea of ‘professionalism’ as a particular kind of ethic. An ethic grounded in reciprocity of accountability and responsibility from professionals and the active participation and public concern on the part of citizens served by professions (Sullivan, 2004). This matter of defining (and enacting) what professionalism actually means in evaluation is, in my judgment, a far more significant pursuit than is the current transnational conversation directed at professionalizing the practice by means of credentialing and certification.